You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by QShu <sh...@gmail.com> on 2014/02/20 08:09:06 UTC

A question about Balancer in HDFS

Hi, Guys:

	The HDFS Balancer will stuck after running a while in my cluster now, the following is the log, after 14:38:52, the Balancer stuccoed and never continue balancing.
	Can anybody give me some help, thank you!

==============================================log==============================================
14/02/20 14:38:25 INFO balancer.Balancer: namenodes = [hdfs://hstore]
14/02/20 14:38:25 INFO balancer.Balancer: p         = Balancer.Parameters[BalancingPolicy.Node, threshold=10.0]
Time Stamp               Iteration#  Bytes Already Moved  Bytes Left To Move  Bytes Being Moved
14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/172.19.105.111:50010
14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/172.19.105.103:50010
14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/172.19.105.108:50010
14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/172.19.105.102:50010
14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack1/172.19.105.23:50010
14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/172.19.105.112:50010
14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/172.19.105.106:50010
14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/172.19.105.101:50010
14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack1/172.19.105.28:50010
14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/172.19.105.113:50010
14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/172.19.105.110:50010
14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack1/172.19.105.27:50010
14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/172.19.105.116:50010
14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/172.19.105.105:50010
14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack1/172.19.105.29:50010
14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack1/172.19.105.25:50010
14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack1/172.19.105.26:50010
14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack1/172.19.105.24:50010
14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/172.19.105.114:50010
14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/172.19.105.107:50010
14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/172.19.105.109:50010
14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack1/172.19.105.30:50010
14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/172.19.105.115:50010
14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/172.19.105.104:50010
14/02/20 14:38:26 INFO balancer.Balancer: 9 over-utilized: [Source[172.19.105.102:50010, utilization=42.58400812492816], Source[172.19.105.23:50010, utilization=41.94449751804609], Source[172.19.105.101:50010, utilization=49.25646255747681], Source[172.19.105.27:50010, utilization=42.13637544454659], Source[172.19.105.29:50010, utilization=42.253685824453484], Source[172.19.105.25:50010, utilization=41.29288653593026], Source[172.19.105.26:50010, utilization=42.58689306059978], Source[172.19.105.24:50010, utilization=42.681174402528306], Source[172.19.105.30:50010, utilization=42.111934753109566]]
14/02/20 14:38:26 INFO balancer.Balancer: 1 underutilized: [BalancerDatanode[172.19.105.28:50010, utilization=5.073501546034823]]
14/02/20 14:38:26 INFO balancer.Balancer: Need to move 1.23 TB to make the cluster balanced.
14/02/20 14:38:26 INFO balancer.Balancer: Decided to move 10 GB bytes from 172.19.105.102:50010 to 172.19.105.28:50010
14/02/20 14:38:26 INFO balancer.Balancer: Decided to move 10 GB bytes from 172.19.105.23:50010 to 172.19.105.111:50010
14/02/20 14:38:26 INFO balancer.Balancer: Decided to move 10 GB bytes from 172.19.105.101:50010 to 172.19.105.103:50010
14/02/20 14:38:26 INFO balancer.Balancer: Decided to move 10 GB bytes from 172.19.105.27:50010 to 172.19.105.108:50010
14/02/20 14:38:26 INFO balancer.Balancer: Decided to move 10 GB bytes from 172.19.105.29:50010 to 172.19.105.112:50010
14/02/20 14:38:26 INFO balancer.Balancer: Decided to move 10 GB bytes from 172.19.105.25:50010 to 172.19.105.106:50010
14/02/20 14:38:26 INFO balancer.Balancer: Decided to move 10 GB bytes from 172.19.105.26:50010 to 172.19.105.113:50010
14/02/20 14:38:26 INFO balancer.Balancer: Decided to move 10 GB bytes from 172.19.105.24:50010 to 172.19.105.110:50010
14/02/20 14:38:26 INFO balancer.Balancer: Decided to move 10 GB bytes from 172.19.105.30:50010 to 172.19.105.116:50010
14/02/20 14:38:26 INFO balancer.Balancer: Will move 90 GB in this iteration
2014-2-20 14:38:26                0                  0 B             1.23 TB              90 GB
14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1074013803 with a length of 128 MB bytes from 172.19.105.26:50010 to 172.19.105.113:50010 using proxy source 172.19.105.102:50010
14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1073936074 with a length of 128 MB bytes from 172.19.105.101:50010 to 172.19.105.103:50010 using proxy source 172.19.105.101:50010
14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1073961733 with a length of 128 MB bytes from 172.19.105.30:50010 to 172.19.105.116:50010 using proxy source 172.19.105.101:50010
14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1073922224 with a length of 128 MB bytes from 172.19.105.23:50010 to 172.19.105.111:50010 using proxy source 172.19.105.102:50010
14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1074018599 with a length of 26.12 MB bytes from 172.19.105.27:50010 to 172.19.105.108:50010 using proxy source 172.19.105.102:50010
14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1074013794 with a length of 128 MB bytes from 172.19.105.26:50010 to 172.19.105.113:50010 using proxy source 172.19.105.101:50010
14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074013803 from 172.19.105.102:50010 to 172.19.105.113:50010
14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1074013798 with a length of 128 MB bytes from 172.19.105.26:50010 to 172.19.105.113:50010 using proxy source 172.19.105.101:50010
14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074013794 from 172.19.105.101:50010 to 172.19.105.113:50010
14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1074018603 with a length of 128 MB bytes from 172.19.105.27:50010 to 172.19.105.108:50010 using proxy source 172.19.105.102:50010
14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074018599 from 172.19.105.102:50010 to 172.19.105.108:50010
14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1074008476 with a length of 128 MB bytes from 172.19.105.25:50010 to 172.19.105.106:50010 using proxy source 172.19.105.107:50010
14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1073922231 with a length of 128 MB bytes from 172.19.105.23:50010 to 172.19.105.111:50010 using proxy source 172.19.105.114:50010
14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073922224 from 172.19.105.102:50010 to 172.19.105.111:50010
14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1073812199 with a length of 64.75 MB bytes from 172.19.105.102:50010 to 172.19.105.28:50010 using proxy source 172.19.105.24:50010
14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1073991855 with a length of 73.88 MB bytes from 172.19.105.24:50010 to 172.19.105.110:50010 using proxy source 172.19.105.102:50010
14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1074035591 with a length of 128 MB bytes from 172.19.105.29:50010 to 172.19.105.112:50010 using proxy source 172.19.105.101:50010
14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1074008479 with a length of 128 MB bytes from 172.19.105.25:50010 to 172.19.105.106:50010 using proxy source 172.19.105.107:50010
14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074008476 from 172.19.105.107:50010 to 172.19.105.106:50010
14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1074035589 with a length of 6.51 KB bytes from 172.19.105.29:50010 to 172.19.105.112:50010 using proxy source 172.19.105.103:50010
14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074035591 from 172.19.105.101:50010 to 172.19.105.112:50010
14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1073997260 with a length of 85.24 KB bytes from 172.19.105.101:50010 to 172.19.105.103:50010 using proxy source 172.19.105.106:50010
14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1073992320 with a length of 128 MB bytes from 172.19.105.30:50010 to 172.19.105.116:50010 using proxy source 172.19.105.105:50010
14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073961733 from 172.19.105.101:50010 to 172.19.105.116:50010
14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073936074 from 172.19.105.101:50010 to 172.19.105.103:50010
14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1074009436 with a length of 128 MB bytes from 172.19.105.101:50010 to 172.19.105.103:50010 using proxy source 172.19.105.28:50010
14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073997260 from 172.19.105.106:50010 to 172.19.105.103:50010
14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073991855 from 172.19.105.102:50010 to 172.19.105.110:50010
14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1074009439 with a length of 128 MB bytes from 172.19.105.101:50010 to 172.19.105.103:50010 using proxy source 172.19.105.27:50010
14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074009436 from 172.19.105.28:50010 to 172.19.105.103:50010
14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073812199 from 172.19.105.24:50010 to 172.19.105.28:50010
14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1073883114 with a length of 128 MB bytes from 172.19.105.102:50010 to 172.19.105.28:50010 using proxy source 172.19.105.27:50010
14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1074018604 with a length of 128 MB bytes from 172.19.105.27:50010 to 172.19.105.108:50010 using proxy source 172.19.105.24:50010
14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074018603 from 172.19.105.102:50010 to 172.19.105.108:50010
14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1074013709 with a length of 128 MB bytes from 172.19.105.26:50010 to 172.19.105.113:50010 using proxy source 172.19.105.27:50010
14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074013798 from 172.19.105.101:50010 to 172.19.105.113:50010
14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074009439 from 172.19.105.27:50010 to 172.19.105.103:50010
14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1073987402 with a length of 75.99 MB bytes from 172.19.105.24:50010 to 172.19.105.110:50010 using proxy source 172.19.105.116:50010
14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1073987410 with a length of 128 MB bytes from 172.19.105.24:50010 to 172.19.105.110:50010 using proxy source 172.19.105.24:50010
14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073987402 from 172.19.105.116:50010 to 172.19.105.110:50010
14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1073987422 with a length of 128 MB bytes from 172.19.105.24:50010 to 172.19.105.110:50010 using proxy source 172.19.105.30:50010
14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1073922228 with a length of 128 MB bytes from 172.19.105.23:50010 to 172.19.105.111:50010 using proxy source 172.19.105.23:50010
14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073987410 from 172.19.105.24:50010 to 172.19.105.110:50010
14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073922231 from 172.19.105.114:50010 to 172.19.105.111:50010
14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1073922442 with a length of 128 MB bytes from 172.19.105.30:50010 to 172.19.105.116:50010 using proxy source 172.19.105.26:50010
14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073992320 from 172.19.105.105:50010 to 172.19.105.116:50010
14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1073992340 with a length of 128 MB bytes from 172.19.105.30:50010 to 172.19.105.116:50010 using proxy source 172.19.105.111:50010
14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073922442 from 172.19.105.26:50010 to 172.19.105.116:50010
14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073992340 from 172.19.105.111:50010 to 172.19.105.116:50010
14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1074008469 with a length of 128 MB bytes from 172.19.105.25:50010 to 172.19.105.106:50010 using proxy source 172.19.105.107:50010
14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074008479 from 172.19.105.107:50010 to 172.19.105.106:50010
14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1074035711 with a length of 128 MB bytes from 172.19.105.29:50010 to 172.19.105.112:50010 using proxy source 172.19.105.29:50010
14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074035589 from 172.19.105.103:50010 to 172.19.105.112:50010
14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1073992638 with a length of 128 MB bytes from 172.19.105.23:50010 to 172.19.105.111:50010 using proxy source 172.19.105.23:50010
14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073922228 from 172.19.105.23:50010 to 172.19.105.111:50010
14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1073917025 with a length of 128 MB bytes from 172.19.105.24:50010 to 172.19.105.110:50010 using proxy source 172.19.105.26:50010
14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1073892376 with a length of 128 MB bytes from 172.19.105.26:50010 to 172.19.105.113:50010 using proxy source 172.19.105.106:50010
14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073883114 from 172.19.105.27:50010 to 172.19.105.28:50010
14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1073917544 with a length of 128 MB bytes from 172.19.105.102:50010 to 172.19.105.28:50010 using proxy source 172.19.105.108:50010
14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1073990086 with a length of 128 MB bytes from 172.19.105.27:50010 to 172.19.105.108:50010 using proxy source 172.19.105.27:50010
14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073987422 from 172.19.105.30:50010 to 172.19.105.110:50010
14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074018604 from 172.19.105.24:50010 to 172.19.105.108:50010
14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074013709 from 172.19.105.27:50010 to 172.19.105.113:50010
14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073992638 from 172.19.105.23:50010 to 172.19.105.111:50010
14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1074035705 with a length of 128 MB bytes from 172.19.105.29:50010 to 172.19.105.112:50010 using proxy source 172.19.105.23:50010
14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074035711 from 172.19.105.29:50010 to 172.19.105.112:50010
14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074008469 from 172.19.105.107:50010 to 172.19.105.106:50010
14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073917544 from 172.19.105.108:50010 to 172.19.105.28:50010
14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073990086 from 172.19.105.27:50010 to 172.19.105.108:50010
14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1073911911 with a length of 128 MB bytes from 172.19.105.29:50010 to 172.19.105.112:50010 using proxy source 172.19.105.29:50010
14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073917025 from 172.19.105.26:50010 to 172.19.105.110:50010
14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073892376 from 172.19.105.106:50010 to 172.19.105.113:50010
14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074035705 from 172.19.105.23:50010 to 172.19.105.112:50010
14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073911911 from 172.19.105.29:50010 to 172.19.105.112:50010
14/02/20 14:38:26 INFO balancer.Balancer: Moving block 1074035589 from 172.19.105.29:50010 to 172.19.105.112:50010 through 172.19.105.103:50010 is succeeded.
14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1074035693 with a length of 128 MB bytes from 172.19.105.29:50010 to 172.19.105.112:50010 using proxy source 172.19.105.29:50010
14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1074009432 with a length of 128 MB bytes from 172.19.105.101:50010 to 172.19.105.103:50010 using proxy source 172.19.105.29:50010
14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074035693 from 172.19.105.29:50010 to 172.19.105.112:50010
14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074009432 from 172.19.105.29:50010 to 172.19.105.103:50010
14/02/20 14:38:26 INFO balancer.Balancer: Moving block 1073997260 from 172.19.105.101:50010 to 172.19.105.103:50010 through 172.19.105.106:50010 is succeeded.
14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1074009435 with a length of 128 MB bytes from 172.19.105.101:50010 to 172.19.105.103:50010 using proxy source 172.19.105.27:50010
14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1074008471 with a length of 128 MB bytes from 172.19.105.25:50010 to 172.19.105.106:50010 using proxy source 172.19.105.107:50010
14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074009435 from 172.19.105.27:50010 to 172.19.105.103:50010
14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074008471 from 172.19.105.107:50010 to 172.19.105.106:50010
14/02/20 14:38:32 INFO balancer.Balancer: Moving block 1074018599 from 172.19.105.27:50010 to 172.19.105.108:50010 through 172.19.105.102:50010 is succeeded.
14/02/20 14:38:32 INFO balancer.Balancer: Moving block 1073812199 from 172.19.105.102:50010 to 172.19.105.28:50010 through 172.19.105.24:50010 is succeeded.
14/02/20 14:38:33 INFO balancer.Balancer: Moving block 1073987402 from 172.19.105.24:50010 to 172.19.105.110:50010 through 172.19.105.116:50010 is succeeded.
14/02/20 14:38:37 INFO balancer.Balancer: Moving block 1073922231 from 172.19.105.23:50010 to 172.19.105.111:50010 through 172.19.105.114:50010 is succeeded.
14/02/20 14:38:38 INFO balancer.Balancer: Moving block 1073917544 from 172.19.105.102:50010 to 172.19.105.28:50010 through 172.19.105.108:50010 is succeeded.
14/02/20 14:38:39 INFO balancer.Balancer: Moving block 1073922224 from 172.19.105.23:50010 to 172.19.105.111:50010 through 172.19.105.102:50010 is succeeded.
14/02/20 14:38:39 INFO balancer.Balancer: Moving block 1074013803 from 172.19.105.26:50010 to 172.19.105.113:50010 through 172.19.105.102:50010 is succeeded.
14/02/20 14:38:40 INFO balancer.Balancer: Moving block 1074035591 from 172.19.105.29:50010 to 172.19.105.112:50010 through 172.19.105.101:50010 is succeeded.
14/02/20 14:38:40 INFO balancer.Balancer: Moving block 1073992320 from 172.19.105.30:50010 to 172.19.105.116:50010 through 172.19.105.105:50010 is succeeded.
14/02/20 14:38:41 INFO balancer.Balancer: Moving block 1073961733 from 172.19.105.30:50010 to 172.19.105.116:50010 through 172.19.105.101:50010 is succeeded.
14/02/20 14:38:42 INFO balancer.Balancer: Moving block 1073991855 from 172.19.105.24:50010 to 172.19.105.110:50010 through 172.19.105.102:50010 is succeeded.
14/02/20 14:38:42 INFO balancer.Balancer: Moving block 1074008479 from 172.19.105.25:50010 to 172.19.105.106:50010 through 172.19.105.107:50010 is succeeded.
14/02/20 14:38:43 INFO balancer.Balancer: Moving block 1074009436 from 172.19.105.101:50010 to 172.19.105.103:50010 through 172.19.105.28:50010 is succeeded.
14/02/20 14:38:44 INFO balancer.Balancer: Moving block 1074008469 from 172.19.105.25:50010 to 172.19.105.106:50010 through 172.19.105.107:50010 is succeeded.
14/02/20 14:38:44 INFO balancer.Balancer: Moving block 1073883114 from 172.19.105.102:50010 to 172.19.105.28:50010 through 172.19.105.27:50010 is succeeded.
14/02/20 14:38:44 INFO balancer.Balancer: Moving block 1073911911 from 172.19.105.29:50010 to 172.19.105.112:50010 through 172.19.105.29:50010 is succeeded.
14/02/20 14:38:44 INFO balancer.Balancer: Moving block 1073990086 from 172.19.105.27:50010 to 172.19.105.108:50010 through 172.19.105.27:50010 is succeeded.
14/02/20 14:38:44 INFO balancer.Balancer: Moving block 1074008476 from 172.19.105.25:50010 to 172.19.105.106:50010 through 172.19.105.107:50010 is succeeded.
14/02/20 14:38:45 INFO balancer.Balancer: Moving block 1073987422 from 172.19.105.24:50010 to 172.19.105.110:50010 through 172.19.105.30:50010 is succeeded.
14/02/20 14:38:46 INFO balancer.Balancer: Moving block 1074013794 from 172.19.105.26:50010 to 172.19.105.113:50010 through 172.19.105.101:50010 is succeeded.
14/02/20 14:38:46 INFO balancer.Balancer: Moving block 1074009432 from 172.19.105.101:50010 to 172.19.105.103:50010 through 172.19.105.29:50010 is succeeded.
14/02/20 14:38:46 INFO balancer.Balancer: Moving block 1073936074 from 172.19.105.101:50010 to 172.19.105.103:50010 through 172.19.105.101:50010 is succeeded.
14/02/20 14:38:47 INFO balancer.Balancer: Moving block 1074035711 from 172.19.105.29:50010 to 172.19.105.112:50010 through 172.19.105.29:50010 is succeeded.
14/02/20 14:38:47 INFO balancer.Balancer: Moving block 1074013709 from 172.19.105.26:50010 to 172.19.105.113:50010 through 172.19.105.27:50010 is succeeded.
14/02/20 14:38:47 INFO balancer.Balancer: Moving block 1074008471 from 172.19.105.25:50010 to 172.19.105.106:50010 through 172.19.105.107:50010 is succeeded.
14/02/20 14:38:47 INFO balancer.Balancer: Moving block 1073992340 from 172.19.105.30:50010 to 172.19.105.116:50010 through 172.19.105.111:50010 is succeeded.
14/02/20 14:38:47 INFO balancer.Balancer: Moving block 1074018604 from 172.19.105.27:50010 to 172.19.105.108:50010 through 172.19.105.24:50010 is succeeded.
14/02/20 14:38:47 INFO balancer.Balancer: Moving block 1073917025 from 172.19.105.24:50010 to 172.19.105.110:50010 through 172.19.105.26:50010 is succeeded.
14/02/20 14:38:48 INFO balancer.Balancer: Moving block 1074009439 from 172.19.105.101:50010 to 172.19.105.103:50010 through 172.19.105.27:50010 is succeeded.
14/02/20 14:38:48 INFO balancer.Balancer: Moving block 1074013798 from 172.19.105.26:50010 to 172.19.105.113:50010 through 172.19.105.101:50010 is succeeded.
14/02/20 14:38:48 INFO balancer.Balancer: Moving block 1074018603 from 172.19.105.27:50010 to 172.19.105.108:50010 through 172.19.105.102:50010 is succeeded.
14/02/20 14:38:48 INFO balancer.Balancer: Moving block 1074035693 from 172.19.105.29:50010 to 172.19.105.112:50010 through 172.19.105.29:50010 is succeeded.
14/02/20 14:38:48 INFO balancer.Balancer: Moving block 1073922442 from 172.19.105.30:50010 to 172.19.105.116:50010 through 172.19.105.26:50010 is succeeded.
14/02/20 14:38:49 INFO balancer.Balancer: Moving block 1073892376 from 172.19.105.26:50010 to 172.19.105.113:50010 through 172.19.105.106:50010 is succeeded.
14/02/20 14:38:50 INFO balancer.Balancer: Moving block 1073987410 from 172.19.105.24:50010 to 172.19.105.110:50010 through 172.19.105.24:50010 is succeeded.
14/02/20 14:38:50 INFO balancer.Balancer: Moving block 1073922228 from 172.19.105.23:50010 to 172.19.105.111:50010 through 172.19.105.23:50010 is succeeded.
14/02/20 14:38:50 INFO balancer.Balancer: Moving block 1073992638 from 172.19.105.23:50010 to 172.19.105.111:50010 through 172.19.105.23:50010 is succeeded.
14/02/20 14:38:50 INFO balancer.Balancer: Moving block 1074009435 from 172.19.105.101:50010 to 172.19.105.103:50010 through 172.19.105.27:50010 is succeeded.
14/02/20 14:38:52 INFO balancer.Balancer: Moving block 1074035705 from 172.19.105.29:50010 to 172.19.105.112:50010 through 172.19.105.23:50010 is succeeded.

Re: A question about Balancer in HDFS

Posted by praveenesh kumar <pr...@gmail.com>.
Cool, thanks for sharing this info. Cheers

Regards
Prav


On Fri, Feb 21, 2014 at 6:53 AM, QShu <sh...@gmail.com> wrote:

> Hi, Prav:
>
> Thanks for your help!
> Now I found out why the balancer stopped working. My Cluster has 2 racks,
> and the balancer follows the rule that can’t change the number of blocks of
> a rack.
> At the same time, the 2 racks is not in the same disk usage. After I
> change all the node into 1 rack, balancer works fine.
>
> Regards
> QiShu
>
>
> 在 2014年2月20日，19:16，praveenesh kumar <pr...@gmail.com> 写道：
>
> Have you noticed any balancing in your datanodes by looking at the
> namenode url (where it shows data-node level usage). You should have.
> I think its not stuck, it just waiting for looking for more data blocks
> which it can move around to balance the cluster.
> There is threshold limit (if I remember it correctly), which you can also
> set while running the balancer (otherwise it uses some default value) and
> keeps on running till it reaches that threshold. I think it would be safe
> to just stop the process, as it has done balancing.
>
> Sorry I don't have the exact answer, just sharing my experiences. Someone
> else might throw better light in here.
>
> Regards
> Prav
>
>
> On Thu, Feb 20, 2014 at 7:09 AM, QShu <sh...@gmail.com> wrote:
>
>> Hi, Guys:
>>
>>         The HDFS Balancer will stuck after running a while in my cluster
>> now, the following is the log, after 14:38:52, the Balancer stuccoed and
>> never continue balancing.
>>         Can anybody give me some help, thank you!
>>
>>
>> ==============================================log==============================================
>> 14/02/20 14:38:25 INFO balancer.Balancer: namenodes = [hdfs://hstore]
>> 14/02/20 14:38:25 INFO balancer.Balancer: p         =
>> Balancer.Parameters[BalancingPolicy.Node, threshold=10.0]
>> Time Stamp               Iteration#  Bytes Already Moved  Bytes Left To
>> Move  Bytes Being Moved
>> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/
>> 172.19.105.111:50010
>> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/
>> 172.19.105.103:50010
>> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/
>> 172.19.105.108:50010
>> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/
>> 172.19.105.102:50010
>> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack1/
>> 172.19.105.23:50010
>> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/
>> 172.19.105.112:50010
>> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/
>> 172.19.105.106:50010
>> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/
>> 172.19.105.101:50010
>> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack1/
>> 172.19.105.28:50010
>> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/
>> 172.19.105.113:50010
>> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/
>> 172.19.105.110:50010
>> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack1/
>> 172.19.105.27:50010
>> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/
>> 172.19.105.116:50010
>> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/
>> 172.19.105.105:50010
>> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack1/
>> 172.19.105.29:50010
>> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack1/
>> 172.19.105.25:50010
>> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack1/
>> 172.19.105.26:50010
>> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack1/
>> 172.19.105.24:50010
>> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/
>> 172.19.105.114:50010
>> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/
>> 172.19.105.107:50010
>> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/
>> 172.19.105.109:50010
>> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack1/
>> 172.19.105.30:50010
>> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/
>> 172.19.105.115:50010
>> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/
>> 172.19.105.104:50010
>> 14/02/20 14:38:26 INFO balancer.Balancer: 9 over-utilized: [Source[
>> 172.19.105.102:50010, utilization=42.58400812492816], Source[
>> 172.19.105.23:50010, utilization=41.94449751804609], Source[
>> 172.19.105.101:50010, utilization=49.25646255747681], Source[
>> 172.19.105.27:50010, utilization=42.13637544454659], Source[
>> 172.19.105.29:50010, utilization=42.253685824453484], Source[
>> 172.19.105.25:50010, utilization=41.29288653593026], Source[
>> 172.19.105.26:50010, utilization=42.58689306059978], Source[
>> 172.19.105.24:50010, utilization=42.681174402528306], Source[
>> 172.19.105.30:50010, utilization=42.111934753109566]]
>> 14/02/20 14:38:26 INFO balancer.Balancer: 1 underutilized:
>> [BalancerDatanode[172.19.105.28:50010, utilization=5.073501546034823]]
>> 14/02/20 14:38:26 INFO balancer.Balancer: Need to move 1.23 TB to make
>> the cluster balanced.
>> 14/02/20 14:38:26 INFO balancer.Balancer: Decided to move 10 GB bytes
>> from 172.19.105.102:50010 to 172.19.105.28:50010
>> 14/02/20 14:38:26 INFO balancer.Balancer: Decided to move 10 GB bytes
>> from 172.19.105.23:50010 to 172.19.105.111:50010
>> 14/02/20 14:38:26 INFO balancer.Balancer: Decided to move 10 GB bytes
>> from 172.19.105.101:50010 to 172.19.105.103:50010
>> 14/02/20 14:38:26 INFO balancer.Balancer: Decided to move 10 GB bytes
>> from 172.19.105.27:50010 to 172.19.105.108:50010
>> 14/02/20 14:38:26 INFO balancer.Balancer: Decided to move 10 GB bytes
>> from 172.19.105.29:50010 to 172.19.105.112:50010
>> 14/02/20 14:38:26 INFO balancer.Balancer: Decided to move 10 GB bytes
>> from 172.19.105.25:50010 to 172.19.105.106:50010
>> 14/02/20 14:38:26 INFO balancer.Balancer: Decided to move 10 GB bytes
>> from 172.19.105.26:50010 to 172.19.105.113:50010
>> 14/02/20 14:38:26 INFO balancer.Balancer: Decided to move 10 GB bytes
>> from 172.19.105.24:50010 to 172.19.105.110:50010
>> 14/02/20 14:38:26 INFO balancer.Balancer: Decided to move 10 GB bytes
>> from 172.19.105.30:50010 to 172.19.105.116:50010
>> 14/02/20 14:38:26 INFO balancer.Balancer: Will move 90 GB in this
>> iteration
>> 2014-2-20 14:38:26                0                  0 B             1.23
>> TB              90 GB
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
>> 1074013803 with a length of 128 MB bytes from 172.19.105.26:50010 to
>> 172.19.105.113:50010 using proxy source 172.19.105.102:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
>> 1073936074 with a length of 128 MB bytes from 172.19.105.101:50010 to
>> 172.19.105.103:50010 using proxy source 172.19.105.101:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
>> 1073961733 with a length of 128 MB bytes from 172.19.105.30:50010 to
>> 172.19.105.116:50010 using proxy source 172.19.105.101:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
>> 1073922224 with a length of 128 MB bytes from 172.19.105.23:50010 to
>> 172.19.105.111:50010 using proxy source 172.19.105.102:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
>> 1074018599 with a length of 26.12 MB bytes from 172.19.105.27:50010 to
>> 172.19.105.108:50010 using proxy source 172.19.105.102:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
>> 1074013794 with a length of 128 MB bytes from 172.19.105.26:50010 to
>> 172.19.105.113:50010 using proxy source 172.19.105.101:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074013803
>> from 172.19.105.102:50010 to 172.19.105.113:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
>> 1074013798 with a length of 128 MB bytes from 172.19.105.26:50010 to
>> 172.19.105.113:50010 using proxy source 172.19.105.101:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074013794
>> from 172.19.105.101:50010 to 172.19.105.113:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
>> 1074018603 with a length of 128 MB bytes from 172.19.105.27:50010 to
>> 172.19.105.108:50010 using proxy source 172.19.105.102:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074018599
>> from 172.19.105.102:50010 to 172.19.105.108:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
>> 1074008476 with a length of 128 MB bytes from 172.19.105.25:50010 to
>> 172.19.105.106:50010 using proxy source 172.19.105.107:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
>> 1073922231 with a length of 128 MB bytes from 172.19.105.23:50010 to
>> 172.19.105.111:50010 using proxy source 172.19.105.114:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073922224
>> from 172.19.105.102:50010 to 172.19.105.111:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
>> 1073812199 with a length of 64.75 MB bytes from 172.19.105.102:50010 to
>> 172.19.105.28:50010 using proxy source 172.19.105.24:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
>> 1073991855 with a length of 73.88 MB bytes from 172.19.105.24:50010 to
>> 172.19.105.110:50010 using proxy source 172.19.105.102:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
>> 1074035591 with a length of 128 MB bytes from 172.19.105.29:50010 to
>> 172.19.105.112:50010 using proxy source 172.19.105.101:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
>> 1074008479 with a length of 128 MB bytes from 172.19.105.25:50010 to
>> 172.19.105.106:50010 using proxy source 172.19.105.107:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074008476
>> from 172.19.105.107:50010 to 172.19.105.106:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
>> 1074035589 with a length of 6.51 KB bytes from 172.19.105.29:50010 to
>> 172.19.105.112:50010 using proxy source 172.19.105.103:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074035591
>> from 172.19.105.101:50010 to 172.19.105.112:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
>> 1073997260 with a length of 85.24 KB bytes from 172.19.105.101:50010 to
>> 172.19.105.103:50010 using proxy source 172.19.105.106:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
>> 1073992320 with a length of 128 MB bytes from 172.19.105.30:50010 to
>> 172.19.105.116:50010 using proxy source 172.19.105.105:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073961733
>> from 172.19.105.101:50010 to 172.19.105.116:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073936074
>> from 172.19.105.101:50010 to 172.19.105.103:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
>> 1074009436 with a length of 128 MB bytes from 172.19.105.101:50010 to
>> 172.19.105.103:50010 using proxy source 172.19.105.28:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073997260
>> from 172.19.105.106:50010 to 172.19.105.103:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073991855
>> from 172.19.105.102:50010 to 172.19.105.110:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
>> 1074009439 with a length of 128 MB bytes from 172.19.105.101:50010 to
>> 172.19.105.103:50010 using proxy source 172.19.105.27:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074009436
>> from 172.19.105.28:50010 to 172.19.105.103:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073812199
>> from 172.19.105.24:50010 to 172.19.105.28:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
>> 1073883114 with a length of 128 MB bytes from 172.19.105.102:50010 to
>> 172.19.105.28:50010 using proxy source 172.19.105.27:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
>> 1074018604 with a length of 128 MB bytes from 172.19.105.27:50010 to
>> 172.19.105.108:50010 using proxy source 172.19.105.24:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074018603
>> from 172.19.105.102:50010 to 172.19.105.108:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
>> 1074013709 with a length of 128 MB bytes from 172.19.105.26:50010 to
>> 172.19.105.113:50010 using proxy source 172.19.105.27:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074013798
>> from 172.19.105.101:50010 to 172.19.105.113:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074009439
>> from 172.19.105.27:50010 to 172.19.105.103:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
>> 1073987402 with a length of 75.99 MB bytes from 172.19.105.24:50010 to
>> 172.19.105.110:50010 using proxy source 172.19.105.116:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
>> 1073987410 with a length of 128 MB bytes from 172.19.105.24:50010 to
>> 172.19.105.110:50010 using proxy source 172.19.105.24:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073987402
>> from 172.19.105.116:50010 to 172.19.105.110:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
>> 1073987422 with a length of 128 MB bytes from 172.19.105.24:50010 to
>> 172.19.105.110:50010 using proxy source 172.19.105.30:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
>> 1073922228 with a length of 128 MB bytes from 172.19.105.23:50010 to
>> 172.19.105.111:50010 using proxy source 172.19.105.23:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073987410
>> from 172.19.105.24:50010 to 172.19.105.110:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073922231
>> from 172.19.105.114:50010 to 172.19.105.111:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
>> 1073922442 with a length of 128 MB bytes from 172.19.105.30:50010 to
>> 172.19.105.116:50010 using proxy source 172.19.105.26:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073992320
>> from 172.19.105.105:50010 to 172.19.105.116:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
>> 1073992340 with a length of 128 MB bytes from 172.19.105.30:50010 to
>> 172.19.105.116:50010 using proxy source 172.19.105.111:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073922442
>> from 172.19.105.26:50010 to 172.19.105.116:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073992340
>> from 172.19.105.111:50010 to 172.19.105.116:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
>> 1074008469 with a length of 128 MB bytes from 172.19.105.25:50010 to
>> 172.19.105.106:50010 using proxy source 172.19.105.107:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074008479
>> from 172.19.105.107:50010 to 172.19.105.106:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
>> 1074035711 with a length of 128 MB bytes from 172.19.105.29:50010 to
>> 172.19.105.112:50010 using proxy source 172.19.105.29:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074035589
>> from 172.19.105.103:50010 to 172.19.105.112:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
>> 1073992638 with a length of 128 MB bytes from 172.19.105.23:50010 to
>> 172.19.105.111:50010 using proxy source 172.19.105.23:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073922228
>> from 172.19.105.23:50010 to 172.19.105.111:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
>> 1073917025 with a length of 128 MB bytes from 172.19.105.24:50010 to
>> 172.19.105.110:50010 using proxy source 172.19.105.26:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
>> 1073892376 with a length of 128 MB bytes from 172.19.105.26:50010 to
>> 172.19.105.113:50010 using proxy source 172.19.105.106:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073883114
>> from 172.19.105.27:50010 to 172.19.105.28:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
>> 1073917544 with a length of 128 MB bytes from 172.19.105.102:50010 to
>> 172.19.105.28:50010 using proxy source 172.19.105.108:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
>> 1073990086 with a length of 128 MB bytes from 172.19.105.27:50010 to
>> 172.19.105.108:50010 using proxy source 172.19.105.27:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073987422
>> from 172.19.105.30:50010 to 172.19.105.110:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074018604
>> from 172.19.105.24:50010 to 172.19.105.108:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074013709
>> from 172.19.105.27:50010 to 172.19.105.113:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073992638
>> from 172.19.105.23:50010 to 172.19.105.111:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
>> 1074035705 with a length of 128 MB bytes from 172.19.105.29:50010 to
>> 172.19.105.112:50010 using proxy source 172.19.105.23:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074035711
>> from 172.19.105.29:50010 to 172.19.105.112:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074008469
>> from 172.19.105.107:50010 to 172.19.105.106:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073917544
>> from 172.19.105.108:50010 to 172.19.105.28:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073990086
>> from 172.19.105.27:50010 to 172.19.105.108:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
>> 1073911911 with a length of 128 MB bytes from 172.19.105.29:50010 to
>> 172.19.105.112:50010 using proxy source 172.19.105.29:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073917025
>> from 172.19.105.26:50010 to 172.19.105.110:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073892376
>> from 172.19.105.106:50010 to 172.19.105.113:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074035705
>> from 172.19.105.23:50010 to 172.19.105.112:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073911911
>> from 172.19.105.29:50010 to 172.19.105.112:50010
>> 14/02/20 14:38:26 INFO balancer.Balancer: Moving block 1074035589 from
>> 172.19.105.29:50010 to 172.19.105.112:50010 through 172.19.105.103:50010is succeeded.
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
>> 1074035693 with a length of 128 MB bytes from 172.19.105.29:50010 to
>> 172.19.105.112:50010 using proxy source 172.19.105.29:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
>> 1074009432 with a length of 128 MB bytes from 172.19.105.101:50010 to
>> 172.19.105.103:50010 using proxy source 172.19.105.29:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074035693
>> from 172.19.105.29:50010 to 172.19.105.112:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074009432
>> from 172.19.105.29:50010 to 172.19.105.103:50010
>> 14/02/20 14:38:26 INFO balancer.Balancer: Moving block 1073997260 from
>> 172.19.105.101:50010 to 172.19.105.103:50010 through 172.19.105.106:50010is succeeded.
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
>> 1074009435 with a length of 128 MB bytes from 172.19.105.101:50010 to
>> 172.19.105.103:50010 using proxy source 172.19.105.27:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
>> 1074008471 with a length of 128 MB bytes from 172.19.105.25:50010 to
>> 172.19.105.106:50010 using proxy source 172.19.105.107:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074009435
>> from 172.19.105.27:50010 to 172.19.105.103:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074008471
>> from 172.19.105.107:50010 to 172.19.105.106:50010
>> 14/02/20 14:38:32 INFO balancer.Balancer: Moving block 1074018599 from
>> 172.19.105.27:50010 to 172.19.105.108:50010 through 172.19.105.102:50010is succeeded.
>> 14/02/20 14:38:32 INFO balancer.Balancer: Moving block 1073812199 from
>> 172.19.105.102:50010 to 172.19.105.28:50010 through 172.19.105.24:50010is succeeded.
>> 14/02/20 14:38:33 INFO balancer.Balancer: Moving block 1073987402 from
>> 172.19.105.24:50010 to 172.19.105.110:50010 through 172.19.105.116:50010is succeeded.
>> 14/02/20 14:38:37 INFO balancer.Balancer: Moving block 1073922231 from
>> 172.19.105.23:50010 to 172.19.105.111:50010 through 172.19.105.114:50010is succeeded.
>> 14/02/20 14:38:38 INFO balancer.Balancer: Moving block 1073917544 from
>> 172.19.105.102:50010 to 172.19.105.28:50010 through 172.19.105.108:50010is succeeded.
>> 14/02/20 14:38:39 INFO balancer.Balancer: Moving block 1073922224 from
>> 172.19.105.23:50010 to 172.19.105.111:50010 through 172.19.105.102:50010is succeeded.
>> 14/02/20 14:38:39 INFO balancer.Balancer: Moving block 1074013803 from
>> 172.19.105.26:50010 to 172.19.105.113:50010 through 172.19.105.102:50010is succeeded.
>> 14/02/20 14:38:40 INFO balancer.Balancer: Moving block 1074035591 from
>> 172.19.105.29:50010 to 172.19.105.112:50010 through 172.19.105.101:50010is succeeded.
>> 14/02/20 14:38:40 INFO balancer.Balancer: Moving block 1073992320 from
>> 172.19.105.30:50010 to 172.19.105.116:50010 through 172.19.105.105:50010is succeeded.
>> 14/02/20 14:38:41 INFO balancer.Balancer: Moving block 1073961733 from
>> 172.19.105.30:50010 to 172.19.105.116:50010 through 172.19.105.101:50010is succeeded.
>> 14/02/20 14:38:42 INFO balancer.Balancer: Moving block 1073991855 from
>> 172.19.105.24:50010 to 172.19.105.110:50010 through 172.19.105.102:50010is succeeded.
>> 14/02/20 14:38:42 INFO balancer.Balancer: Moving block 1074008479 from
>> 172.19.105.25:50010 to 172.19.105.106:50010 through 172.19.105.107:50010is succeeded.
>> 14/02/20 14:38:43 INFO balancer.Balancer: Moving block 1074009436 from
>> 172.19.105.101:50010 to 172.19.105.103:50010 through 172.19.105.28:50010is succeeded.
>> 14/02/20 14:38:44 INFO balancer.Balancer: Moving block 1074008469 from
>> 172.19.105.25:50010 to 172.19.105.106:50010 through 172.19.105.107:50010is succeeded.
>> 14/02/20 14:38:44 INFO balancer.Balancer: Moving block 1073883114 from
>> 172.19.105.102:50010 to 172.19.105.28:50010 through 172.19.105.27:50010is succeeded.
>> 14/02/20 14:38:44 INFO balancer.Balancer: Moving block 1073911911 from
>> 172.19.105.29:50010 to 172.19.105.112:50010 through 172.19.105.29:50010is succeeded.
>> 14/02/20 14:38:44 INFO balancer.Balancer: Moving block 1073990086 from
>> 172.19.105.27:50010 to 172.19.105.108:50010 through 172.19.105.27:50010is succeeded.
>> 14/02/20 14:38:44 INFO balancer.Balancer: Moving block 1074008476 from
>> 172.19.105.25:50010 to 172.19.105.106:50010 through 172.19.105.107:50010is succeeded.
>> 14/02/20 14:38:45 INFO balancer.Balancer: Moving block 1073987422 from
>> 172.19.105.24:50010 to 172.19.105.110:50010 through 172.19.105.30:50010is succeeded.
>> 14/02/20 14:38:46 INFO balancer.Balancer: Moving block 1074013794 from
>> 172.19.105.26:50010 to 172.19.105.113:50010 through 172.19.105.101:50010is succeeded.
>> 14/02/20 14:38:46 INFO balancer.Balancer: Moving block 1074009432 from
>> 172.19.105.101:50010 to 172.19.105.103:50010 through 172.19.105.29:50010is succeeded.
>> 14/02/20 14:38:46 INFO balancer.Balancer: Moving block 1073936074 from
>> 172.19.105.101:50010 to 172.19.105.103:50010 through 172.19.105.101:50010is succeeded.
>> 14/02/20 14:38:47 INFO balancer.Balancer: Moving block 1074035711 from
>> 172.19.105.29:50010 to 172.19.105.112:50010 through 172.19.105.29:50010is succeeded.
>> 14/02/20 14:38:47 INFO balancer.Balancer: Moving block 1074013709 from
>> 172.19.105.26:50010 to 172.19.105.113:50010 through 172.19.105.27:50010is succeeded.
>> 14/02/20 14:38:47 INFO balancer.Balancer: Moving block 1074008471 from
>> 172.19.105.25:50010 to 172.19.105.106:50010 through 172.19.105.107:50010is succeeded.
>> 14/02/20 14:38:47 INFO balancer.Balancer: Moving block 1073992340 from
>> 172.19.105.30:50010 to 172.19.105.116:50010 through 172.19.105.111:50010is succeeded.
>> 14/02/20 14:38:47 INFO balancer.Balancer: Moving block 1074018604 from
>> 172.19.105.27:50010 to 172.19.105.108:50010 through 172.19.105.24:50010is succeeded.
>> 14/02/20 14:38:47 INFO balancer.Balancer: Moving block 1073917025 from
>> 172.19.105.24:50010 to 172.19.105.110:50010 through 172.19.105.26:50010is succeeded.
>> 14/02/20 14:38:48 INFO balancer.Balancer: Moving block 1074009439 from
>> 172.19.105.101:50010 to 172.19.105.103:50010 through 172.19.105.27:50010is succeeded.
>> 14/02/20 14:38:48 INFO balancer.Balancer: Moving block 1074013798 from
>> 172.19.105.26:50010 to 172.19.105.113:50010 through 172.19.105.101:50010is succeeded.
>> 14/02/20 14:38:48 INFO balancer.Balancer: Moving block 1074018603 from
>> 172.19.105.27:50010 to 172.19.105.108:50010 through 172.19.105.102:50010is succeeded.
>> 14/02/20 14:38:48 INFO balancer.Balancer: Moving block 1074035693 from
>> 172.19.105.29:50010 to 172.19.105.112:50010 through 172.19.105.29:50010is succeeded.
>> 14/02/20 14:38:48 INFO balancer.Balancer: Moving block 1073922442 from
>> 172.19.105.30:50010 to 172.19.105.116:50010 through 172.19.105.26:50010is succeeded.
>> 14/02/20 14:38:49 INFO balancer.Balancer: Moving block 1073892376 from
>> 172.19.105.26:50010 to 172.19.105.113:50010 through 172.19.105.106:50010is succeeded.
>> 14/02/20 14:38:50 INFO balancer.Balancer: Moving block 1073987410 from
>> 172.19.105.24:50010 to 172.19.105.110:50010 through 172.19.105.24:50010is succeeded.
>> 14/02/20 14:38:50 INFO balancer.Balancer: Moving block 1073922228 from
>> 172.19.105.23:50010 to 172.19.105.111:50010 through 172.19.105.23:50010is succeeded.
>> 14/02/20 14:38:50 INFO balancer.Balancer: Moving block 1073992638 from
>> 172.19.105.23:50010 to 172.19.105.111:50010 through 172.19.105.23:50010is succeeded.
>> 14/02/20 14:38:50 INFO balancer.Balancer: Moving block 1074009435 from
>> 172.19.105.101:50010 to 172.19.105.103:50010 through 172.19.105.27:50010is succeeded.
>> 14/02/20 14:38:52 INFO balancer.Balancer: Moving block 1074035705 from
>> 172.19.105.29:50010 to 172.19.105.112:50010 through 172.19.105.23:50010is succeeded.
>
>
>
>

Re: A question about Balancer in HDFS

Posted by praveenesh kumar <pr...@gmail.com>.
Cool, thanks for sharing this info. Cheers

Regards
Prav


On Fri, Feb 21, 2014 at 6:53 AM, QShu <sh...@gmail.com> wrote:

> Hi, Prav:
>
> Thanks for your help!
> Now I found out why the balancer stopped working. My Cluster has 2 racks,
> and the balancer follows the rule that can’t change the number of blocks of
> a rack.
> At the same time, the 2 racks is not in the same disk usage. After I
> change all the node into 1 rack, balancer works fine.
>
> Regards
> QiShu
>
>
> 在 2014年2月20日，19:16，praveenesh kumar <pr...@gmail.com> 写道：
>
> Have you noticed any balancing in your datanodes by looking at the
> namenode url (where it shows data-node level usage). You should have.
> I think its not stuck, it just waiting for looking for more data blocks
> which it can move around to balance the cluster.
> There is threshold limit (if I remember it correctly), which you can also
> set while running the balancer (otherwise it uses some default value) and
> keeps on running till it reaches that threshold. I think it would be safe
> to just stop the process, as it has done balancing.
>
> Sorry I don't have the exact answer, just sharing my experiences. Someone
> else might throw better light in here.
>
> Regards
> Prav
>
>
> On Thu, Feb 20, 2014 at 7:09 AM, QShu <sh...@gmail.com> wrote:
>
>> Hi, Guys:
>>
>>         The HDFS Balancer will stuck after running a while in my cluster
>> now, the following is the log, after 14:38:52, the Balancer stuccoed and
>> never continue balancing.
>>         Can anybody give me some help, thank you!
>>
>>
>> ==============================================log==============================================
>> 14/02/20 14:38:25 INFO balancer.Balancer: namenodes = [hdfs://hstore]
>> 14/02/20 14:38:25 INFO balancer.Balancer: p         =
>> Balancer.Parameters[BalancingPolicy.Node, threshold=10.0]
>> Time Stamp               Iteration#  Bytes Already Moved  Bytes Left To
>> Move  Bytes Being Moved
>> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/
>> 172.19.105.111:50010
>> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/
>> 172.19.105.103:50010
>> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/
>> 172.19.105.108:50010
>> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/
>> 172.19.105.102:50010
>> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack1/
>> 172.19.105.23:50010
>> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/
>> 172.19.105.112:50010
>> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/
>> 172.19.105.106:50010
>> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/
>> 172.19.105.101:50010
>> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack1/
>> 172.19.105.28:50010
>> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/
>> 172.19.105.113:50010
>> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/
>> 172.19.105.110:50010
>> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack1/
>> 172.19.105.27:50010
>> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/
>> 172.19.105.116:50010
>> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/
>> 172.19.105.105:50010
>> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack1/
>> 172.19.105.29:50010
>> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack1/
>> 172.19.105.25:50010
>> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack1/
>> 172.19.105.26:50010
>> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack1/
>> 172.19.105.24:50010
>> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/
>> 172.19.105.114:50010
>> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/
>> 172.19.105.107:50010
>> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/
>> 172.19.105.109:50010
>> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack1/
>> 172.19.105.30:50010
>> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/
>> 172.19.105.115:50010
>> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/
>> 172.19.105.104:50010
>> 14/02/20 14:38:26 INFO balancer.Balancer: 9 over-utilized: [Source[
>> 172.19.105.102:50010, utilization=42.58400812492816], Source[
>> 172.19.105.23:50010, utilization=41.94449751804609], Source[
>> 172.19.105.101:50010, utilization=49.25646255747681], Source[
>> 172.19.105.27:50010, utilization=42.13637544454659], Source[
>> 172.19.105.29:50010, utilization=42.253685824453484], Source[
>> 172.19.105.25:50010, utilization=41.29288653593026], Source[
>> 172.19.105.26:50010, utilization=42.58689306059978], Source[
>> 172.19.105.24:50010, utilization=42.681174402528306], Source[
>> 172.19.105.30:50010, utilization=42.111934753109566]]
>> 14/02/20 14:38:26 INFO balancer.Balancer: 1 underutilized:
>> [BalancerDatanode[172.19.105.28:50010, utilization=5.073501546034823]]
>> 14/02/20 14:38:26 INFO balancer.Balancer: Need to move 1.23 TB to make
>> the cluster balanced.
>> 14/02/20 14:38:26 INFO balancer.Balancer: Decided to move 10 GB bytes
>> from 172.19.105.102:50010 to 172.19.105.28:50010
>> 14/02/20 14:38:26 INFO balancer.Balancer: Decided to move 10 GB bytes
>> from 172.19.105.23:50010 to 172.19.105.111:50010
>> 14/02/20 14:38:26 INFO balancer.Balancer: Decided to move 10 GB bytes
>> from 172.19.105.101:50010 to 172.19.105.103:50010
>> 14/02/20 14:38:26 INFO balancer.Balancer: Decided to move 10 GB bytes
>> from 172.19.105.27:50010 to 172.19.105.108:50010
>> 14/02/20 14:38:26 INFO balancer.Balancer: Decided to move 10 GB bytes
>> from 172.19.105.29:50010 to 172.19.105.112:50010
>> 14/02/20 14:38:26 INFO balancer.Balancer: Decided to move 10 GB bytes
>> from 172.19.105.25:50010 to 172.19.105.106:50010
>> 14/02/20 14:38:26 INFO balancer.Balancer: Decided to move 10 GB bytes
>> from 172.19.105.26:50010 to 172.19.105.113:50010
>> 14/02/20 14:38:26 INFO balancer.Balancer: Decided to move 10 GB bytes
>> from 172.19.105.24:50010 to 172.19.105.110:50010
>> 14/02/20 14:38:26 INFO balancer.Balancer: Decided to move 10 GB bytes
>> from 172.19.105.30:50010 to 172.19.105.116:50010
>> 14/02/20 14:38:26 INFO balancer.Balancer: Will move 90 GB in this
>> iteration
>> 2014-2-20 14:38:26                0                  0 B             1.23
>> TB              90 GB
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
>> 1074013803 with a length of 128 MB bytes from 172.19.105.26:50010 to
>> 172.19.105.113:50010 using proxy source 172.19.105.102:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
>> 1073936074 with a length of 128 MB bytes from 172.19.105.101:50010 to
>> 172.19.105.103:50010 using proxy source 172.19.105.101:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
>> 1073961733 with a length of 128 MB bytes from 172.19.105.30:50010 to
>> 172.19.105.116:50010 using proxy source 172.19.105.101:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
>> 1073922224 with a length of 128 MB bytes from 172.19.105.23:50010 to
>> 172.19.105.111:50010 using proxy source 172.19.105.102:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
>> 1074018599 with a length of 26.12 MB bytes from 172.19.105.27:50010 to
>> 172.19.105.108:50010 using proxy source 172.19.105.102:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
>> 1074013794 with a length of 128 MB bytes from 172.19.105.26:50010 to
>> 172.19.105.113:50010 using proxy source 172.19.105.101:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074013803
>> from 172.19.105.102:50010 to 172.19.105.113:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
>> 1074013798 with a length of 128 MB bytes from 172.19.105.26:50010 to
>> 172.19.105.113:50010 using proxy source 172.19.105.101:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074013794
>> from 172.19.105.101:50010 to 172.19.105.113:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
>> 1074018603 with a length of 128 MB bytes from 172.19.105.27:50010 to
>> 172.19.105.108:50010 using proxy source 172.19.105.102:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074018599
>> from 172.19.105.102:50010 to 172.19.105.108:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
>> 1074008476 with a length of 128 MB bytes from 172.19.105.25:50010 to
>> 172.19.105.106:50010 using proxy source 172.19.105.107:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
>> 1073922231 with a length of 128 MB bytes from 172.19.105.23:50010 to
>> 172.19.105.111:50010 using proxy source 172.19.105.114:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073922224
>> from 172.19.105.102:50010 to 172.19.105.111:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
>> 1073812199 with a length of 64.75 MB bytes from 172.19.105.102:50010 to
>> 172.19.105.28:50010 using proxy source 172.19.105.24:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
>> 1073991855 with a length of 73.88 MB bytes from 172.19.105.24:50010 to
>> 172.19.105.110:50010 using proxy source 172.19.105.102:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
>> 1074035591 with a length of 128 MB bytes from 172.19.105.29:50010 to
>> 172.19.105.112:50010 using proxy source 172.19.105.101:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
>> 1074008479 with a length of 128 MB bytes from 172.19.105.25:50010 to
>> 172.19.105.106:50010 using proxy source 172.19.105.107:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074008476
>> from 172.19.105.107:50010 to 172.19.105.106:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
>> 1074035589 with a length of 6.51 KB bytes from 172.19.105.29:50010 to
>> 172.19.105.112:50010 using proxy source 172.19.105.103:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074035591
>> from 172.19.105.101:50010 to 172.19.105.112:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
>> 1073997260 with a length of 85.24 KB bytes from 172.19.105.101:50010 to
>> 172.19.105.103:50010 using proxy source 172.19.105.106:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
>> 1073992320 with a length of 128 MB bytes from 172.19.105.30:50010 to
>> 172.19.105.116:50010 using proxy source 172.19.105.105:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073961733
>> from 172.19.105.101:50010 to 172.19.105.116:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073936074
>> from 172.19.105.101:50010 to 172.19.105.103:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
>> 1074009436 with a length of 128 MB bytes from 172.19.105.101:50010 to
>> 172.19.105.103:50010 using proxy source 172.19.105.28:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073997260
>> from 172.19.105.106:50010 to 172.19.105.103:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073991855
>> from 172.19.105.102:50010 to 172.19.105.110:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
>> 1074009439 with a length of 128 MB bytes from 172.19.105.101:50010 to
>> 172.19.105.103:50010 using proxy source 172.19.105.27:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074009436
>> from 172.19.105.28:50010 to 172.19.105.103:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073812199
>> from 172.19.105.24:50010 to 172.19.105.28:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
>> 1073883114 with a length of 128 MB bytes from 172.19.105.102:50010 to
>> 172.19.105.28:50010 using proxy source 172.19.105.27:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
>> 1074018604 with a length of 128 MB bytes from 172.19.105.27:50010 to
>> 172.19.105.108:50010 using proxy source 172.19.105.24:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074018603
>> from 172.19.105.102:50010 to 172.19.105.108:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
>> 1074013709 with a length of 128 MB bytes from 172.19.105.26:50010 to
>> 172.19.105.113:50010 using proxy source 172.19.105.27:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074013798
>> from 172.19.105.101:50010 to 172.19.105.113:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074009439
>> from 172.19.105.27:50010 to 172.19.105.103:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
>> 1073987402 with a length of 75.99 MB bytes from 172.19.105.24:50010 to
>> 172.19.105.110:50010 using proxy source 172.19.105.116:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
>> 1073987410 with a length of 128 MB bytes from 172.19.105.24:50010 to
>> 172.19.105.110:50010 using proxy source 172.19.105.24:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073987402
>> from 172.19.105.116:50010 to 172.19.105.110:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
>> 1073987422 with a length of 128 MB bytes from 172.19.105.24:50010 to
>> 172.19.105.110:50010 using proxy source 172.19.105.30:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
>> 1073922228 with a length of 128 MB bytes from 172.19.105.23:50010 to
>> 172.19.105.111:50010 using proxy source 172.19.105.23:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073987410
>> from 172.19.105.24:50010 to 172.19.105.110:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073922231
>> from 172.19.105.114:50010 to 172.19.105.111:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
>> 1073922442 with a length of 128 MB bytes from 172.19.105.30:50010 to
>> 172.19.105.116:50010 using proxy source 172.19.105.26:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073992320
>> from 172.19.105.105:50010 to 172.19.105.116:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
>> 1073992340 with a length of 128 MB bytes from 172.19.105.30:50010 to
>> 172.19.105.116:50010 using proxy source 172.19.105.111:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073922442
>> from 172.19.105.26:50010 to 172.19.105.116:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073992340
>> from 172.19.105.111:50010 to 172.19.105.116:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
>> 1074008469 with a length of 128 MB bytes from 172.19.105.25:50010 to
>> 172.19.105.106:50010 using proxy source 172.19.105.107:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074008479
>> from 172.19.105.107:50010 to 172.19.105.106:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
>> 1074035711 with a length of 128 MB bytes from 172.19.105.29:50010 to
>> 172.19.105.112:50010 using proxy source 172.19.105.29:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074035589
>> from 172.19.105.103:50010 to 172.19.105.112:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
>> 1073992638 with a length of 128 MB bytes from 172.19.105.23:50010 to
>> 172.19.105.111:50010 using proxy source 172.19.105.23:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073922228
>> from 172.19.105.23:50010 to 172.19.105.111:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
>> 1073917025 with a length of 128 MB bytes from 172.19.105.24:50010 to
>> 172.19.105.110:50010 using proxy source 172.19.105.26:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
>> 1073892376 with a length of 128 MB bytes from 172.19.105.26:50010 to
>> 172.19.105.113:50010 using proxy source 172.19.105.106:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073883114
>> from 172.19.105.27:50010 to 172.19.105.28:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
>> 1073917544 with a length of 128 MB bytes from 172.19.105.102:50010 to
>> 172.19.105.28:50010 using proxy source 172.19.105.108:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
>> 1073990086 with a length of 128 MB bytes from 172.19.105.27:50010 to
>> 172.19.105.108:50010 using proxy source 172.19.105.27:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073987422
>> from 172.19.105.30:50010 to 172.19.105.110:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074018604
>> from 172.19.105.24:50010 to 172.19.105.108:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074013709
>> from 172.19.105.27:50010 to 172.19.105.113:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073992638
>> from 172.19.105.23:50010 to 172.19.105.111:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
>> 1074035705 with a length of 128 MB bytes from 172.19.105.29:50010 to
>> 172.19.105.112:50010 using proxy source 172.19.105.23:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074035711
>> from 172.19.105.29:50010 to 172.19.105.112:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074008469
>> from 172.19.105.107:50010 to 172.19.105.106:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073917544
>> from 172.19.105.108:50010 to 172.19.105.28:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073990086
>> from 172.19.105.27:50010 to 172.19.105.108:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
>> 1073911911 with a length of 128 MB bytes from 172.19.105.29:50010 to
>> 172.19.105.112:50010 using proxy source 172.19.105.29:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073917025
>> from 172.19.105.26:50010 to 172.19.105.110:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073892376
>> from 172.19.105.106:50010 to 172.19.105.113:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074035705
>> from 172.19.105.23:50010 to 172.19.105.112:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073911911
>> from 172.19.105.29:50010 to 172.19.105.112:50010
>> 14/02/20 14:38:26 INFO balancer.Balancer: Moving block 1074035589 from
>> 172.19.105.29:50010 to 172.19.105.112:50010 through 172.19.105.103:50010is succeeded.
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
>> 1074035693 with a length of 128 MB bytes from 172.19.105.29:50010 to
>> 172.19.105.112:50010 using proxy source 172.19.105.29:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
>> 1074009432 with a length of 128 MB bytes from 172.19.105.101:50010 to
>> 172.19.105.103:50010 using proxy source 172.19.105.29:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074035693
>> from 172.19.105.29:50010 to 172.19.105.112:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074009432
>> from 172.19.105.29:50010 to 172.19.105.103:50010
>> 14/02/20 14:38:26 INFO balancer.Balancer: Moving block 1073997260 from
>> 172.19.105.101:50010 to 172.19.105.103:50010 through 172.19.105.106:50010is succeeded.
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
>> 1074009435 with a length of 128 MB bytes from 172.19.105.101:50010 to
>> 172.19.105.103:50010 using proxy source 172.19.105.27:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
>> 1074008471 with a length of 128 MB bytes from 172.19.105.25:50010 to
>> 172.19.105.106:50010 using proxy source 172.19.105.107:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074009435
>> from 172.19.105.27:50010 to 172.19.105.103:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074008471
>> from 172.19.105.107:50010 to 172.19.105.106:50010
>> 14/02/20 14:38:32 INFO balancer.Balancer: Moving block 1074018599 from
>> 172.19.105.27:50010 to 172.19.105.108:50010 through 172.19.105.102:50010is succeeded.
>> 14/02/20 14:38:32 INFO balancer.Balancer: Moving block 1073812199 from
>> 172.19.105.102:50010 to 172.19.105.28:50010 through 172.19.105.24:50010is succeeded.
>> 14/02/20 14:38:33 INFO balancer.Balancer: Moving block 1073987402 from
>> 172.19.105.24:50010 to 172.19.105.110:50010 through 172.19.105.116:50010is succeeded.
>> 14/02/20 14:38:37 INFO balancer.Balancer: Moving block 1073922231 from
>> 172.19.105.23:50010 to 172.19.105.111:50010 through 172.19.105.114:50010is succeeded.
>> 14/02/20 14:38:38 INFO balancer.Balancer: Moving block 1073917544 from
>> 172.19.105.102:50010 to 172.19.105.28:50010 through 172.19.105.108:50010is succeeded.
>> 14/02/20 14:38:39 INFO balancer.Balancer: Moving block 1073922224 from
>> 172.19.105.23:50010 to 172.19.105.111:50010 through 172.19.105.102:50010is succeeded.
>> 14/02/20 14:38:39 INFO balancer.Balancer: Moving block 1074013803 from
>> 172.19.105.26:50010 to 172.19.105.113:50010 through 172.19.105.102:50010is succeeded.
>> 14/02/20 14:38:40 INFO balancer.Balancer: Moving block 1074035591 from
>> 172.19.105.29:50010 to 172.19.105.112:50010 through 172.19.105.101:50010is succeeded.
>> 14/02/20 14:38:40 INFO balancer.Balancer: Moving block 1073992320 from
>> 172.19.105.30:50010 to 172.19.105.116:50010 through 172.19.105.105:50010is succeeded.
>> 14/02/20 14:38:41 INFO balancer.Balancer: Moving block 1073961733 from
>> 172.19.105.30:50010 to 172.19.105.116:50010 through 172.19.105.101:50010is succeeded.
>> 14/02/20 14:38:42 INFO balancer.Balancer: Moving block 1073991855 from
>> 172.19.105.24:50010 to 172.19.105.110:50010 through 172.19.105.102:50010is succeeded.
>> 14/02/20 14:38:42 INFO balancer.Balancer: Moving block 1074008479 from
>> 172.19.105.25:50010 to 172.19.105.106:50010 through 172.19.105.107:50010is succeeded.
>> 14/02/20 14:38:43 INFO balancer.Balancer: Moving block 1074009436 from
>> 172.19.105.101:50010 to 172.19.105.103:50010 through 172.19.105.28:50010is succeeded.
>> 14/02/20 14:38:44 INFO balancer.Balancer: Moving block 1074008469 from
>> 172.19.105.25:50010 to 172.19.105.106:50010 through 172.19.105.107:50010is succeeded.
>> 14/02/20 14:38:44 INFO balancer.Balancer: Moving block 1073883114 from
>> 172.19.105.102:50010 to 172.19.105.28:50010 through 172.19.105.27:50010is succeeded.
>> 14/02/20 14:38:44 INFO balancer.Balancer: Moving block 1073911911 from
>> 172.19.105.29:50010 to 172.19.105.112:50010 through 172.19.105.29:50010is succeeded.
>> 14/02/20 14:38:44 INFO balancer.Balancer: Moving block 1073990086 from
>> 172.19.105.27:50010 to 172.19.105.108:50010 through 172.19.105.27:50010is succeeded.
>> 14/02/20 14:38:44 INFO balancer.Balancer: Moving block 1074008476 from
>> 172.19.105.25:50010 to 172.19.105.106:50010 through 172.19.105.107:50010is succeeded.
>> 14/02/20 14:38:45 INFO balancer.Balancer: Moving block 1073987422 from
>> 172.19.105.24:50010 to 172.19.105.110:50010 through 172.19.105.30:50010is succeeded.
>> 14/02/20 14:38:46 INFO balancer.Balancer: Moving block 1074013794 from
>> 172.19.105.26:50010 to 172.19.105.113:50010 through 172.19.105.101:50010is succeeded.
>> 14/02/20 14:38:46 INFO balancer.Balancer: Moving block 1074009432 from
>> 172.19.105.101:50010 to 172.19.105.103:50010 through 172.19.105.29:50010is succeeded.
>> 14/02/20 14:38:46 INFO balancer.Balancer: Moving block 1073936074 from
>> 172.19.105.101:50010 to 172.19.105.103:50010 through 172.19.105.101:50010is succeeded.
>> 14/02/20 14:38:47 INFO balancer.Balancer: Moving block 1074035711 from
>> 172.19.105.29:50010 to 172.19.105.112:50010 through 172.19.105.29:50010is succeeded.
>> 14/02/20 14:38:47 INFO balancer.Balancer: Moving block 1074013709 from
>> 172.19.105.26:50010 to 172.19.105.113:50010 through 172.19.105.27:50010is succeeded.
>> 14/02/20 14:38:47 INFO balancer.Balancer: Moving block 1074008471 from
>> 172.19.105.25:50010 to 172.19.105.106:50010 through 172.19.105.107:50010is succeeded.
>> 14/02/20 14:38:47 INFO balancer.Balancer: Moving block 1073992340 from
>> 172.19.105.30:50010 to 172.19.105.116:50010 through 172.19.105.111:50010is succeeded.
>> 14/02/20 14:38:47 INFO balancer.Balancer: Moving block 1074018604 from
>> 172.19.105.27:50010 to 172.19.105.108:50010 through 172.19.105.24:50010is succeeded.
>> 14/02/20 14:38:47 INFO balancer.Balancer: Moving block 1073917025 from
>> 172.19.105.24:50010 to 172.19.105.110:50010 through 172.19.105.26:50010is succeeded.
>> 14/02/20 14:38:48 INFO balancer.Balancer: Moving block 1074009439 from
>> 172.19.105.101:50010 to 172.19.105.103:50010 through 172.19.105.27:50010is succeeded.
>> 14/02/20 14:38:48 INFO balancer.Balancer: Moving block 1074013798 from
>> 172.19.105.26:50010 to 172.19.105.113:50010 through 172.19.105.101:50010is succeeded.
>> 14/02/20 14:38:48 INFO balancer.Balancer: Moving block 1074018603 from
>> 172.19.105.27:50010 to 172.19.105.108:50010 through 172.19.105.102:50010is succeeded.
>> 14/02/20 14:38:48 INFO balancer.Balancer: Moving block 1074035693 from
>> 172.19.105.29:50010 to 172.19.105.112:50010 through 172.19.105.29:50010is succeeded.
>> 14/02/20 14:38:48 INFO balancer.Balancer: Moving block 1073922442 from
>> 172.19.105.30:50010 to 172.19.105.116:50010 through 172.19.105.26:50010is succeeded.
>> 14/02/20 14:38:49 INFO balancer.Balancer: Moving block 1073892376 from
>> 172.19.105.26:50010 to 172.19.105.113:50010 through 172.19.105.106:50010is succeeded.
>> 14/02/20 14:38:50 INFO balancer.Balancer: Moving block 1073987410 from
>> 172.19.105.24:50010 to 172.19.105.110:50010 through 172.19.105.24:50010is succeeded.
>> 14/02/20 14:38:50 INFO balancer.Balancer: Moving block 1073922228 from
>> 172.19.105.23:50010 to 172.19.105.111:50010 through 172.19.105.23:50010is succeeded.
>> 14/02/20 14:38:50 INFO balancer.Balancer: Moving block 1073992638 from
>> 172.19.105.23:50010 to 172.19.105.111:50010 through 172.19.105.23:50010is succeeded.
>> 14/02/20 14:38:50 INFO balancer.Balancer: Moving block 1074009435 from
>> 172.19.105.101:50010 to 172.19.105.103:50010 through 172.19.105.27:50010is succeeded.
>> 14/02/20 14:38:52 INFO balancer.Balancer: Moving block 1074035705 from
>> 172.19.105.29:50010 to 172.19.105.112:50010 through 172.19.105.23:50010is succeeded.
>
>
>
>

Re: A question about Balancer in HDFS

Posted by praveenesh kumar <pr...@gmail.com>.
Cool, thanks for sharing this info. Cheers

Regards
Prav


On Fri, Feb 21, 2014 at 6:53 AM, QShu <sh...@gmail.com> wrote:

> Hi, Prav:
>
> Thanks for your help!
> Now I found out why the balancer stopped working. My Cluster has 2 racks,
> and the balancer follows the rule that can’t change the number of blocks of
> a rack.
> At the same time, the 2 racks is not in the same disk usage. After I
> change all the node into 1 rack, balancer works fine.
>
> Regards
> QiShu
>
>
> 在 2014年2月20日，19:16，praveenesh kumar <pr...@gmail.com> 写道：
>
> Have you noticed any balancing in your datanodes by looking at the
> namenode url (where it shows data-node level usage). You should have.
> I think its not stuck, it just waiting for looking for more data blocks
> which it can move around to balance the cluster.
> There is threshold limit (if I remember it correctly), which you can also
> set while running the balancer (otherwise it uses some default value) and
> keeps on running till it reaches that threshold. I think it would be safe
> to just stop the process, as it has done balancing.
>
> Sorry I don't have the exact answer, just sharing my experiences. Someone
> else might throw better light in here.
>
> Regards
> Prav
>
>
> On Thu, Feb 20, 2014 at 7:09 AM, QShu <sh...@gmail.com> wrote:
>
>> Hi, Guys:
>>
>>         The HDFS Balancer will stuck after running a while in my cluster
>> now, the following is the log, after 14:38:52, the Balancer stuccoed and
>> never continue balancing.
>>         Can anybody give me some help, thank you!
>>
>>
>> ==============================================log==============================================
>> 14/02/20 14:38:25 INFO balancer.Balancer: namenodes = [hdfs://hstore]
>> 14/02/20 14:38:25 INFO balancer.Balancer: p         =
>> Balancer.Parameters[BalancingPolicy.Node, threshold=10.0]
>> Time Stamp               Iteration#  Bytes Already Moved  Bytes Left To
>> Move  Bytes Being Moved
>> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/
>> 172.19.105.111:50010
>> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/
>> 172.19.105.103:50010
>> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/
>> 172.19.105.108:50010
>> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/
>> 172.19.105.102:50010
>> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack1/
>> 172.19.105.23:50010
>> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/
>> 172.19.105.112:50010
>> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/
>> 172.19.105.106:50010
>> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/
>> 172.19.105.101:50010
>> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack1/
>> 172.19.105.28:50010
>> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/
>> 172.19.105.113:50010
>> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/
>> 172.19.105.110:50010
>> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack1/
>> 172.19.105.27:50010
>> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/
>> 172.19.105.116:50010
>> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/
>> 172.19.105.105:50010
>> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack1/
>> 172.19.105.29:50010
>> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack1/
>> 172.19.105.25:50010
>> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack1/
>> 172.19.105.26:50010
>> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack1/
>> 172.19.105.24:50010
>> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/
>> 172.19.105.114:50010
>> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/
>> 172.19.105.107:50010
>> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/
>> 172.19.105.109:50010
>> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack1/
>> 172.19.105.30:50010
>> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/
>> 172.19.105.115:50010
>> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/
>> 172.19.105.104:50010
>> 14/02/20 14:38:26 INFO balancer.Balancer: 9 over-utilized: [Source[
>> 172.19.105.102:50010, utilization=42.58400812492816], Source[
>> 172.19.105.23:50010, utilization=41.94449751804609], Source[
>> 172.19.105.101:50010, utilization=49.25646255747681], Source[
>> 172.19.105.27:50010, utilization=42.13637544454659], Source[
>> 172.19.105.29:50010, utilization=42.253685824453484], Source[
>> 172.19.105.25:50010, utilization=41.29288653593026], Source[
>> 172.19.105.26:50010, utilization=42.58689306059978], Source[
>> 172.19.105.24:50010, utilization=42.681174402528306], Source[
>> 172.19.105.30:50010, utilization=42.111934753109566]]
>> 14/02/20 14:38:26 INFO balancer.Balancer: 1 underutilized:
>> [BalancerDatanode[172.19.105.28:50010, utilization=5.073501546034823]]
>> 14/02/20 14:38:26 INFO balancer.Balancer: Need to move 1.23 TB to make
>> the cluster balanced.
>> 14/02/20 14:38:26 INFO balancer.Balancer: Decided to move 10 GB bytes
>> from 172.19.105.102:50010 to 172.19.105.28:50010
>> 14/02/20 14:38:26 INFO balancer.Balancer: Decided to move 10 GB bytes
>> from 172.19.105.23:50010 to 172.19.105.111:50010
>> 14/02/20 14:38:26 INFO balancer.Balancer: Decided to move 10 GB bytes
>> from 172.19.105.101:50010 to 172.19.105.103:50010
>> 14/02/20 14:38:26 INFO balancer.Balancer: Decided to move 10 GB bytes
>> from 172.19.105.27:50010 to 172.19.105.108:50010
>> 14/02/20 14:38:26 INFO balancer.Balancer: Decided to move 10 GB bytes
>> from 172.19.105.29:50010 to 172.19.105.112:50010
>> 14/02/20 14:38:26 INFO balancer.Balancer: Decided to move 10 GB bytes
>> from 172.19.105.25:50010 to 172.19.105.106:50010
>> 14/02/20 14:38:26 INFO balancer.Balancer: Decided to move 10 GB bytes
>> from 172.19.105.26:50010 to 172.19.105.113:50010
>> 14/02/20 14:38:26 INFO balancer.Balancer: Decided to move 10 GB bytes
>> from 172.19.105.24:50010 to 172.19.105.110:50010
>> 14/02/20 14:38:26 INFO balancer.Balancer: Decided to move 10 GB bytes
>> from 172.19.105.30:50010 to 172.19.105.116:50010
>> 14/02/20 14:38:26 INFO balancer.Balancer: Will move 90 GB in this
>> iteration
>> 2014-2-20 14:38:26                0                  0 B             1.23
>> TB              90 GB
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
>> 1074013803 with a length of 128 MB bytes from 172.19.105.26:50010 to
>> 172.19.105.113:50010 using proxy source 172.19.105.102:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
>> 1073936074 with a length of 128 MB bytes from 172.19.105.101:50010 to
>> 172.19.105.103:50010 using proxy source 172.19.105.101:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
>> 1073961733 with a length of 128 MB bytes from 172.19.105.30:50010 to
>> 172.19.105.116:50010 using proxy source 172.19.105.101:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
>> 1073922224 with a length of 128 MB bytes from 172.19.105.23:50010 to
>> 172.19.105.111:50010 using proxy source 172.19.105.102:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
>> 1074018599 with a length of 26.12 MB bytes from 172.19.105.27:50010 to
>> 172.19.105.108:50010 using proxy source 172.19.105.102:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
>> 1074013794 with a length of 128 MB bytes from 172.19.105.26:50010 to
>> 172.19.105.113:50010 using proxy source 172.19.105.101:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074013803
>> from 172.19.105.102:50010 to 172.19.105.113:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
>> 1074013798 with a length of 128 MB bytes from 172.19.105.26:50010 to
>> 172.19.105.113:50010 using proxy source 172.19.105.101:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074013794
>> from 172.19.105.101:50010 to 172.19.105.113:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
>> 1074018603 with a length of 128 MB bytes from 172.19.105.27:50010 to
>> 172.19.105.108:50010 using proxy source 172.19.105.102:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074018599
>> from 172.19.105.102:50010 to 172.19.105.108:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
>> 1074008476 with a length of 128 MB bytes from 172.19.105.25:50010 to
>> 172.19.105.106:50010 using proxy source 172.19.105.107:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
>> 1073922231 with a length of 128 MB bytes from 172.19.105.23:50010 to
>> 172.19.105.111:50010 using proxy source 172.19.105.114:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073922224
>> from 172.19.105.102:50010 to 172.19.105.111:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
>> 1073812199 with a length of 64.75 MB bytes from 172.19.105.102:50010 to
>> 172.19.105.28:50010 using proxy source 172.19.105.24:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
>> 1073991855 with a length of 73.88 MB bytes from 172.19.105.24:50010 to
>> 172.19.105.110:50010 using proxy source 172.19.105.102:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
>> 1074035591 with a length of 128 MB bytes from 172.19.105.29:50010 to
>> 172.19.105.112:50010 using proxy source 172.19.105.101:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
>> 1074008479 with a length of 128 MB bytes from 172.19.105.25:50010 to
>> 172.19.105.106:50010 using proxy source 172.19.105.107:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074008476
>> from 172.19.105.107:50010 to 172.19.105.106:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
>> 1074035589 with a length of 6.51 KB bytes from 172.19.105.29:50010 to
>> 172.19.105.112:50010 using proxy source 172.19.105.103:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074035591
>> from 172.19.105.101:50010 to 172.19.105.112:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
>> 1073997260 with a length of 85.24 KB bytes from 172.19.105.101:50010 to
>> 172.19.105.103:50010 using proxy source 172.19.105.106:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
>> 1073992320 with a length of 128 MB bytes from 172.19.105.30:50010 to
>> 172.19.105.116:50010 using proxy source 172.19.105.105:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073961733
>> from 172.19.105.101:50010 to 172.19.105.116:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073936074
>> from 172.19.105.101:50010 to 172.19.105.103:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
>> 1074009436 with a length of 128 MB bytes from 172.19.105.101:50010 to
>> 172.19.105.103:50010 using proxy source 172.19.105.28:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073997260
>> from 172.19.105.106:50010 to 172.19.105.103:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073991855
>> from 172.19.105.102:50010 to 172.19.105.110:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
>> 1074009439 with a length of 128 MB bytes from 172.19.105.101:50010 to
>> 172.19.105.103:50010 using proxy source 172.19.105.27:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074009436
>> from 172.19.105.28:50010 to 172.19.105.103:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073812199
>> from 172.19.105.24:50010 to 172.19.105.28:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
>> 1073883114 with a length of 128 MB bytes from 172.19.105.102:50010 to
>> 172.19.105.28:50010 using proxy source 172.19.105.27:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
>> 1074018604 with a length of 128 MB bytes from 172.19.105.27:50010 to
>> 172.19.105.108:50010 using proxy source 172.19.105.24:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074018603
>> from 172.19.105.102:50010 to 172.19.105.108:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
>> 1074013709 with a length of 128 MB bytes from 172.19.105.26:50010 to
>> 172.19.105.113:50010 using proxy source 172.19.105.27:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074013798
>> from 172.19.105.101:50010 to 172.19.105.113:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074009439
>> from 172.19.105.27:50010 to 172.19.105.103:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
>> 1073987402 with a length of 75.99 MB bytes from 172.19.105.24:50010 to
>> 172.19.105.110:50010 using proxy source 172.19.105.116:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
>> 1073987410 with a length of 128 MB bytes from 172.19.105.24:50010 to
>> 172.19.105.110:50010 using proxy source 172.19.105.24:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073987402
>> from 172.19.105.116:50010 to 172.19.105.110:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
>> 1073987422 with a length of 128 MB bytes from 172.19.105.24:50010 to
>> 172.19.105.110:50010 using proxy source 172.19.105.30:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
>> 1073922228 with a length of 128 MB bytes from 172.19.105.23:50010 to
>> 172.19.105.111:50010 using proxy source 172.19.105.23:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073987410
>> from 172.19.105.24:50010 to 172.19.105.110:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073922231
>> from 172.19.105.114:50010 to 172.19.105.111:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
>> 1073922442 with a length of 128 MB bytes from 172.19.105.30:50010 to
>> 172.19.105.116:50010 using proxy source 172.19.105.26:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073992320
>> from 172.19.105.105:50010 to 172.19.105.116:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
>> 1073992340 with a length of 128 MB bytes from 172.19.105.30:50010 to
>> 172.19.105.116:50010 using proxy source 172.19.105.111:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073922442
>> from 172.19.105.26:50010 to 172.19.105.116:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073992340
>> from 172.19.105.111:50010 to 172.19.105.116:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
>> 1074008469 with a length of 128 MB bytes from 172.19.105.25:50010 to
>> 172.19.105.106:50010 using proxy source 172.19.105.107:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074008479
>> from 172.19.105.107:50010 to 172.19.105.106:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
>> 1074035711 with a length of 128 MB bytes from 172.19.105.29:50010 to
>> 172.19.105.112:50010 using proxy source 172.19.105.29:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074035589
>> from 172.19.105.103:50010 to 172.19.105.112:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
>> 1073992638 with a length of 128 MB bytes from 172.19.105.23:50010 to
>> 172.19.105.111:50010 using proxy source 172.19.105.23:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073922228
>> from 172.19.105.23:50010 to 172.19.105.111:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
>> 1073917025 with a length of 128 MB bytes from 172.19.105.24:50010 to
>> 172.19.105.110:50010 using proxy source 172.19.105.26:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
>> 1073892376 with a length of 128 MB bytes from 172.19.105.26:50010 to
>> 172.19.105.113:50010 using proxy source 172.19.105.106:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073883114
>> from 172.19.105.27:50010 to 172.19.105.28:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
>> 1073917544 with a length of 128 MB bytes from 172.19.105.102:50010 to
>> 172.19.105.28:50010 using proxy source 172.19.105.108:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
>> 1073990086 with a length of 128 MB bytes from 172.19.105.27:50010 to
>> 172.19.105.108:50010 using proxy source 172.19.105.27:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073987422
>> from 172.19.105.30:50010 to 172.19.105.110:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074018604
>> from 172.19.105.24:50010 to 172.19.105.108:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074013709
>> from 172.19.105.27:50010 to 172.19.105.113:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073992638
>> from 172.19.105.23:50010 to 172.19.105.111:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
>> 1074035705 with a length of 128 MB bytes from 172.19.105.29:50010 to
>> 172.19.105.112:50010 using proxy source 172.19.105.23:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074035711
>> from 172.19.105.29:50010 to 172.19.105.112:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074008469
>> from 172.19.105.107:50010 to 172.19.105.106:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073917544
>> from 172.19.105.108:50010 to 172.19.105.28:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073990086
>> from 172.19.105.27:50010 to 172.19.105.108:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
>> 1073911911 with a length of 128 MB bytes from 172.19.105.29:50010 to
>> 172.19.105.112:50010 using proxy source 172.19.105.29:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073917025
>> from 172.19.105.26:50010 to 172.19.105.110:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073892376
>> from 172.19.105.106:50010 to 172.19.105.113:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074035705
>> from 172.19.105.23:50010 to 172.19.105.112:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073911911
>> from 172.19.105.29:50010 to 172.19.105.112:50010
>> 14/02/20 14:38:26 INFO balancer.Balancer: Moving block 1074035589 from
>> 172.19.105.29:50010 to 172.19.105.112:50010 through 172.19.105.103:50010is succeeded.
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
>> 1074035693 with a length of 128 MB bytes from 172.19.105.29:50010 to
>> 172.19.105.112:50010 using proxy source 172.19.105.29:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
>> 1074009432 with a length of 128 MB bytes from 172.19.105.101:50010 to
>> 172.19.105.103:50010 using proxy source 172.19.105.29:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074035693
>> from 172.19.105.29:50010 to 172.19.105.112:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074009432
>> from 172.19.105.29:50010 to 172.19.105.103:50010
>> 14/02/20 14:38:26 INFO balancer.Balancer: Moving block 1073997260 from
>> 172.19.105.101:50010 to 172.19.105.103:50010 through 172.19.105.106:50010is succeeded.
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
>> 1074009435 with a length of 128 MB bytes from 172.19.105.101:50010 to
>> 172.19.105.103:50010 using proxy source 172.19.105.27:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
>> 1074008471 with a length of 128 MB bytes from 172.19.105.25:50010 to
>> 172.19.105.106:50010 using proxy source 172.19.105.107:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074009435
>> from 172.19.105.27:50010 to 172.19.105.103:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074008471
>> from 172.19.105.107:50010 to 172.19.105.106:50010
>> 14/02/20 14:38:32 INFO balancer.Balancer: Moving block 1074018599 from
>> 172.19.105.27:50010 to 172.19.105.108:50010 through 172.19.105.102:50010is succeeded.
>> 14/02/20 14:38:32 INFO balancer.Balancer: Moving block 1073812199 from
>> 172.19.105.102:50010 to 172.19.105.28:50010 through 172.19.105.24:50010is succeeded.
>> 14/02/20 14:38:33 INFO balancer.Balancer: Moving block 1073987402 from
>> 172.19.105.24:50010 to 172.19.105.110:50010 through 172.19.105.116:50010is succeeded.
>> 14/02/20 14:38:37 INFO balancer.Balancer: Moving block 1073922231 from
>> 172.19.105.23:50010 to 172.19.105.111:50010 through 172.19.105.114:50010is succeeded.
>> 14/02/20 14:38:38 INFO balancer.Balancer: Moving block 1073917544 from
>> 172.19.105.102:50010 to 172.19.105.28:50010 through 172.19.105.108:50010is succeeded.
>> 14/02/20 14:38:39 INFO balancer.Balancer: Moving block 1073922224 from
>> 172.19.105.23:50010 to 172.19.105.111:50010 through 172.19.105.102:50010is succeeded.
>> 14/02/20 14:38:39 INFO balancer.Balancer: Moving block 1074013803 from
>> 172.19.105.26:50010 to 172.19.105.113:50010 through 172.19.105.102:50010is succeeded.
>> 14/02/20 14:38:40 INFO balancer.Balancer: Moving block 1074035591 from
>> 172.19.105.29:50010 to 172.19.105.112:50010 through 172.19.105.101:50010is succeeded.
>> 14/02/20 14:38:40 INFO balancer.Balancer: Moving block 1073992320 from
>> 172.19.105.30:50010 to 172.19.105.116:50010 through 172.19.105.105:50010is succeeded.
>> 14/02/20 14:38:41 INFO balancer.Balancer: Moving block 1073961733 from
>> 172.19.105.30:50010 to 172.19.105.116:50010 through 172.19.105.101:50010is succeeded.
>> 14/02/20 14:38:42 INFO balancer.Balancer: Moving block 1073991855 from
>> 172.19.105.24:50010 to 172.19.105.110:50010 through 172.19.105.102:50010is succeeded.
>> 14/02/20 14:38:42 INFO balancer.Balancer: Moving block 1074008479 from
>> 172.19.105.25:50010 to 172.19.105.106:50010 through 172.19.105.107:50010is succeeded.
>> 14/02/20 14:38:43 INFO balancer.Balancer: Moving block 1074009436 from
>> 172.19.105.101:50010 to 172.19.105.103:50010 through 172.19.105.28:50010is succeeded.
>> 14/02/20 14:38:44 INFO balancer.Balancer: Moving block 1074008469 from
>> 172.19.105.25:50010 to 172.19.105.106:50010 through 172.19.105.107:50010is succeeded.
>> 14/02/20 14:38:44 INFO balancer.Balancer: Moving block 1073883114 from
>> 172.19.105.102:50010 to 172.19.105.28:50010 through 172.19.105.27:50010is succeeded.
>> 14/02/20 14:38:44 INFO balancer.Balancer: Moving block 1073911911 from
>> 172.19.105.29:50010 to 172.19.105.112:50010 through 172.19.105.29:50010is succeeded.
>> 14/02/20 14:38:44 INFO balancer.Balancer: Moving block 1073990086 from
>> 172.19.105.27:50010 to 172.19.105.108:50010 through 172.19.105.27:50010is succeeded.
>> 14/02/20 14:38:44 INFO balancer.Balancer: Moving block 1074008476 from
>> 172.19.105.25:50010 to 172.19.105.106:50010 through 172.19.105.107:50010is succeeded.
>> 14/02/20 14:38:45 INFO balancer.Balancer: Moving block 1073987422 from
>> 172.19.105.24:50010 to 172.19.105.110:50010 through 172.19.105.30:50010is succeeded.
>> 14/02/20 14:38:46 INFO balancer.Balancer: Moving block 1074013794 from
>> 172.19.105.26:50010 to 172.19.105.113:50010 through 172.19.105.101:50010is succeeded.
>> 14/02/20 14:38:46 INFO balancer.Balancer: Moving block 1074009432 from
>> 172.19.105.101:50010 to 172.19.105.103:50010 through 172.19.105.29:50010is succeeded.
>> 14/02/20 14:38:46 INFO balancer.Balancer: Moving block 1073936074 from
>> 172.19.105.101:50010 to 172.19.105.103:50010 through 172.19.105.101:50010is succeeded.
>> 14/02/20 14:38:47 INFO balancer.Balancer: Moving block 1074035711 from
>> 172.19.105.29:50010 to 172.19.105.112:50010 through 172.19.105.29:50010is succeeded.
>> 14/02/20 14:38:47 INFO balancer.Balancer: Moving block 1074013709 from
>> 172.19.105.26:50010 to 172.19.105.113:50010 through 172.19.105.27:50010is succeeded.
>> 14/02/20 14:38:47 INFO balancer.Balancer: Moving block 1074008471 from
>> 172.19.105.25:50010 to 172.19.105.106:50010 through 172.19.105.107:50010is succeeded.
>> 14/02/20 14:38:47 INFO balancer.Balancer: Moving block 1073992340 from
>> 172.19.105.30:50010 to 172.19.105.116:50010 through 172.19.105.111:50010is succeeded.
>> 14/02/20 14:38:47 INFO balancer.Balancer: Moving block 1074018604 from
>> 172.19.105.27:50010 to 172.19.105.108:50010 through 172.19.105.24:50010is succeeded.
>> 14/02/20 14:38:47 INFO balancer.Balancer: Moving block 1073917025 from
>> 172.19.105.24:50010 to 172.19.105.110:50010 through 172.19.105.26:50010is succeeded.
>> 14/02/20 14:38:48 INFO balancer.Balancer: Moving block 1074009439 from
>> 172.19.105.101:50010 to 172.19.105.103:50010 through 172.19.105.27:50010is succeeded.
>> 14/02/20 14:38:48 INFO balancer.Balancer: Moving block 1074013798 from
>> 172.19.105.26:50010 to 172.19.105.113:50010 through 172.19.105.101:50010is succeeded.
>> 14/02/20 14:38:48 INFO balancer.Balancer: Moving block 1074018603 from
>> 172.19.105.27:50010 to 172.19.105.108:50010 through 172.19.105.102:50010is succeeded.
>> 14/02/20 14:38:48 INFO balancer.Balancer: Moving block 1074035693 from
>> 172.19.105.29:50010 to 172.19.105.112:50010 through 172.19.105.29:50010is succeeded.
>> 14/02/20 14:38:48 INFO balancer.Balancer: Moving block 1073922442 from
>> 172.19.105.30:50010 to 172.19.105.116:50010 through 172.19.105.26:50010is succeeded.
>> 14/02/20 14:38:49 INFO balancer.Balancer: Moving block 1073892376 from
>> 172.19.105.26:50010 to 172.19.105.113:50010 through 172.19.105.106:50010is succeeded.
>> 14/02/20 14:38:50 INFO balancer.Balancer: Moving block 1073987410 from
>> 172.19.105.24:50010 to 172.19.105.110:50010 through 172.19.105.24:50010is succeeded.
>> 14/02/20 14:38:50 INFO balancer.Balancer: Moving block 1073922228 from
>> 172.19.105.23:50010 to 172.19.105.111:50010 through 172.19.105.23:50010is succeeded.
>> 14/02/20 14:38:50 INFO balancer.Balancer: Moving block 1073992638 from
>> 172.19.105.23:50010 to 172.19.105.111:50010 through 172.19.105.23:50010is succeeded.
>> 14/02/20 14:38:50 INFO balancer.Balancer: Moving block 1074009435 from
>> 172.19.105.101:50010 to 172.19.105.103:50010 through 172.19.105.27:50010is succeeded.
>> 14/02/20 14:38:52 INFO balancer.Balancer: Moving block 1074035705 from
>> 172.19.105.29:50010 to 172.19.105.112:50010 through 172.19.105.23:50010is succeeded.
>
>
>
>

Re: A question about Balancer in HDFS

Posted by praveenesh kumar <pr...@gmail.com>.
Cool, thanks for sharing this info. Cheers

Regards
Prav


On Fri, Feb 21, 2014 at 6:53 AM, QShu <sh...@gmail.com> wrote:

> Hi, Prav:
>
> Thanks for your help!
> Now I found out why the balancer stopped working. My Cluster has 2 racks,
> and the balancer follows the rule that can’t change the number of blocks of
> a rack.
> At the same time, the 2 racks is not in the same disk usage. After I
> change all the node into 1 rack, balancer works fine.
>
> Regards
> QiShu
>
>
> 在 2014年2月20日，19:16，praveenesh kumar <pr...@gmail.com> 写道：
>
> Have you noticed any balancing in your datanodes by looking at the
> namenode url (where it shows data-node level usage). You should have.
> I think its not stuck, it just waiting for looking for more data blocks
> which it can move around to balance the cluster.
> There is threshold limit (if I remember it correctly), which you can also
> set while running the balancer (otherwise it uses some default value) and
> keeps on running till it reaches that threshold. I think it would be safe
> to just stop the process, as it has done balancing.
>
> Sorry I don't have the exact answer, just sharing my experiences. Someone
> else might throw better light in here.
>
> Regards
> Prav
>
>
> On Thu, Feb 20, 2014 at 7:09 AM, QShu <sh...@gmail.com> wrote:
>
>> Hi, Guys:
>>
>>         The HDFS Balancer will stuck after running a while in my cluster
>> now, the following is the log, after 14:38:52, the Balancer stuccoed and
>> never continue balancing.
>>         Can anybody give me some help, thank you!
>>
>>
>> ==============================================log==============================================
>> 14/02/20 14:38:25 INFO balancer.Balancer: namenodes = [hdfs://hstore]
>> 14/02/20 14:38:25 INFO balancer.Balancer: p         =
>> Balancer.Parameters[BalancingPolicy.Node, threshold=10.0]
>> Time Stamp               Iteration#  Bytes Already Moved  Bytes Left To
>> Move  Bytes Being Moved
>> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/
>> 172.19.105.111:50010
>> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/
>> 172.19.105.103:50010
>> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/
>> 172.19.105.108:50010
>> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/
>> 172.19.105.102:50010
>> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack1/
>> 172.19.105.23:50010
>> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/
>> 172.19.105.112:50010
>> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/
>> 172.19.105.106:50010
>> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/
>> 172.19.105.101:50010
>> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack1/
>> 172.19.105.28:50010
>> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/
>> 172.19.105.113:50010
>> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/
>> 172.19.105.110:50010
>> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack1/
>> 172.19.105.27:50010
>> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/
>> 172.19.105.116:50010
>> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/
>> 172.19.105.105:50010
>> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack1/
>> 172.19.105.29:50010
>> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack1/
>> 172.19.105.25:50010
>> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack1/
>> 172.19.105.26:50010
>> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack1/
>> 172.19.105.24:50010
>> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/
>> 172.19.105.114:50010
>> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/
>> 172.19.105.107:50010
>> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/
>> 172.19.105.109:50010
>> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack1/
>> 172.19.105.30:50010
>> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/
>> 172.19.105.115:50010
>> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/
>> 172.19.105.104:50010
>> 14/02/20 14:38:26 INFO balancer.Balancer: 9 over-utilized: [Source[
>> 172.19.105.102:50010, utilization=42.58400812492816], Source[
>> 172.19.105.23:50010, utilization=41.94449751804609], Source[
>> 172.19.105.101:50010, utilization=49.25646255747681], Source[
>> 172.19.105.27:50010, utilization=42.13637544454659], Source[
>> 172.19.105.29:50010, utilization=42.253685824453484], Source[
>> 172.19.105.25:50010, utilization=41.29288653593026], Source[
>> 172.19.105.26:50010, utilization=42.58689306059978], Source[
>> 172.19.105.24:50010, utilization=42.681174402528306], Source[
>> 172.19.105.30:50010, utilization=42.111934753109566]]
>> 14/02/20 14:38:26 INFO balancer.Balancer: 1 underutilized:
>> [BalancerDatanode[172.19.105.28:50010, utilization=5.073501546034823]]
>> 14/02/20 14:38:26 INFO balancer.Balancer: Need to move 1.23 TB to make
>> the cluster balanced.
>> 14/02/20 14:38:26 INFO balancer.Balancer: Decided to move 10 GB bytes
>> from 172.19.105.102:50010 to 172.19.105.28:50010
>> 14/02/20 14:38:26 INFO balancer.Balancer: Decided to move 10 GB bytes
>> from 172.19.105.23:50010 to 172.19.105.111:50010
>> 14/02/20 14:38:26 INFO balancer.Balancer: Decided to move 10 GB bytes
>> from 172.19.105.101:50010 to 172.19.105.103:50010
>> 14/02/20 14:38:26 INFO balancer.Balancer: Decided to move 10 GB bytes
>> from 172.19.105.27:50010 to 172.19.105.108:50010
>> 14/02/20 14:38:26 INFO balancer.Balancer: Decided to move 10 GB bytes
>> from 172.19.105.29:50010 to 172.19.105.112:50010
>> 14/02/20 14:38:26 INFO balancer.Balancer: Decided to move 10 GB bytes
>> from 172.19.105.25:50010 to 172.19.105.106:50010
>> 14/02/20 14:38:26 INFO balancer.Balancer: Decided to move 10 GB bytes
>> from 172.19.105.26:50010 to 172.19.105.113:50010
>> 14/02/20 14:38:26 INFO balancer.Balancer: Decided to move 10 GB bytes
>> from 172.19.105.24:50010 to 172.19.105.110:50010
>> 14/02/20 14:38:26 INFO balancer.Balancer: Decided to move 10 GB bytes
>> from 172.19.105.30:50010 to 172.19.105.116:50010
>> 14/02/20 14:38:26 INFO balancer.Balancer: Will move 90 GB in this
>> iteration
>> 2014-2-20 14:38:26                0                  0 B             1.23
>> TB              90 GB
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
>> 1074013803 with a length of 128 MB bytes from 172.19.105.26:50010 to
>> 172.19.105.113:50010 using proxy source 172.19.105.102:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
>> 1073936074 with a length of 128 MB bytes from 172.19.105.101:50010 to
>> 172.19.105.103:50010 using proxy source 172.19.105.101:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
>> 1073961733 with a length of 128 MB bytes from 172.19.105.30:50010 to
>> 172.19.105.116:50010 using proxy source 172.19.105.101:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
>> 1073922224 with a length of 128 MB bytes from 172.19.105.23:50010 to
>> 172.19.105.111:50010 using proxy source 172.19.105.102:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
>> 1074018599 with a length of 26.12 MB bytes from 172.19.105.27:50010 to
>> 172.19.105.108:50010 using proxy source 172.19.105.102:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
>> 1074013794 with a length of 128 MB bytes from 172.19.105.26:50010 to
>> 172.19.105.113:50010 using proxy source 172.19.105.101:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074013803
>> from 172.19.105.102:50010 to 172.19.105.113:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
>> 1074013798 with a length of 128 MB bytes from 172.19.105.26:50010 to
>> 172.19.105.113:50010 using proxy source 172.19.105.101:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074013794
>> from 172.19.105.101:50010 to 172.19.105.113:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
>> 1074018603 with a length of 128 MB bytes from 172.19.105.27:50010 to
>> 172.19.105.108:50010 using proxy source 172.19.105.102:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074018599
>> from 172.19.105.102:50010 to 172.19.105.108:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
>> 1074008476 with a length of 128 MB bytes from 172.19.105.25:50010 to
>> 172.19.105.106:50010 using proxy source 172.19.105.107:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
>> 1073922231 with a length of 128 MB bytes from 172.19.105.23:50010 to
>> 172.19.105.111:50010 using proxy source 172.19.105.114:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073922224
>> from 172.19.105.102:50010 to 172.19.105.111:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
>> 1073812199 with a length of 64.75 MB bytes from 172.19.105.102:50010 to
>> 172.19.105.28:50010 using proxy source 172.19.105.24:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
>> 1073991855 with a length of 73.88 MB bytes from 172.19.105.24:50010 to
>> 172.19.105.110:50010 using proxy source 172.19.105.102:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
>> 1074035591 with a length of 128 MB bytes from 172.19.105.29:50010 to
>> 172.19.105.112:50010 using proxy source 172.19.105.101:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
>> 1074008479 with a length of 128 MB bytes from 172.19.105.25:50010 to
>> 172.19.105.106:50010 using proxy source 172.19.105.107:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074008476
>> from 172.19.105.107:50010 to 172.19.105.106:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
>> 1074035589 with a length of 6.51 KB bytes from 172.19.105.29:50010 to
>> 172.19.105.112:50010 using proxy source 172.19.105.103:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074035591
>> from 172.19.105.101:50010 to 172.19.105.112:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
>> 1073997260 with a length of 85.24 KB bytes from 172.19.105.101:50010 to
>> 172.19.105.103:50010 using proxy source 172.19.105.106:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
>> 1073992320 with a length of 128 MB bytes from 172.19.105.30:50010 to
>> 172.19.105.116:50010 using proxy source 172.19.105.105:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073961733
>> from 172.19.105.101:50010 to 172.19.105.116:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073936074
>> from 172.19.105.101:50010 to 172.19.105.103:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
>> 1074009436 with a length of 128 MB bytes from 172.19.105.101:50010 to
>> 172.19.105.103:50010 using proxy source 172.19.105.28:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073997260
>> from 172.19.105.106:50010 to 172.19.105.103:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073991855
>> from 172.19.105.102:50010 to 172.19.105.110:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
>> 1074009439 with a length of 128 MB bytes from 172.19.105.101:50010 to
>> 172.19.105.103:50010 using proxy source 172.19.105.27:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074009436
>> from 172.19.105.28:50010 to 172.19.105.103:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073812199
>> from 172.19.105.24:50010 to 172.19.105.28:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
>> 1073883114 with a length of 128 MB bytes from 172.19.105.102:50010 to
>> 172.19.105.28:50010 using proxy source 172.19.105.27:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
>> 1074018604 with a length of 128 MB bytes from 172.19.105.27:50010 to
>> 172.19.105.108:50010 using proxy source 172.19.105.24:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074018603
>> from 172.19.105.102:50010 to 172.19.105.108:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
>> 1074013709 with a length of 128 MB bytes from 172.19.105.26:50010 to
>> 172.19.105.113:50010 using proxy source 172.19.105.27:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074013798
>> from 172.19.105.101:50010 to 172.19.105.113:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074009439
>> from 172.19.105.27:50010 to 172.19.105.103:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
>> 1073987402 with a length of 75.99 MB bytes from 172.19.105.24:50010 to
>> 172.19.105.110:50010 using proxy source 172.19.105.116:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
>> 1073987410 with a length of 128 MB bytes from 172.19.105.24:50010 to
>> 172.19.105.110:50010 using proxy source 172.19.105.24:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073987402
>> from 172.19.105.116:50010 to 172.19.105.110:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
>> 1073987422 with a length of 128 MB bytes from 172.19.105.24:50010 to
>> 172.19.105.110:50010 using proxy source 172.19.105.30:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
>> 1073922228 with a length of 128 MB bytes from 172.19.105.23:50010 to
>> 172.19.105.111:50010 using proxy source 172.19.105.23:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073987410
>> from 172.19.105.24:50010 to 172.19.105.110:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073922231
>> from 172.19.105.114:50010 to 172.19.105.111:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
>> 1073922442 with a length of 128 MB bytes from 172.19.105.30:50010 to
>> 172.19.105.116:50010 using proxy source 172.19.105.26:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073992320
>> from 172.19.105.105:50010 to 172.19.105.116:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
>> 1073992340 with a length of 128 MB bytes from 172.19.105.30:50010 to
>> 172.19.105.116:50010 using proxy source 172.19.105.111:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073922442
>> from 172.19.105.26:50010 to 172.19.105.116:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073992340
>> from 172.19.105.111:50010 to 172.19.105.116:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
>> 1074008469 with a length of 128 MB bytes from 172.19.105.25:50010 to
>> 172.19.105.106:50010 using proxy source 172.19.105.107:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074008479
>> from 172.19.105.107:50010 to 172.19.105.106:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
>> 1074035711 with a length of 128 MB bytes from 172.19.105.29:50010 to
>> 172.19.105.112:50010 using proxy source 172.19.105.29:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074035589
>> from 172.19.105.103:50010 to 172.19.105.112:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
>> 1073992638 with a length of 128 MB bytes from 172.19.105.23:50010 to
>> 172.19.105.111:50010 using proxy source 172.19.105.23:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073922228
>> from 172.19.105.23:50010 to 172.19.105.111:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
>> 1073917025 with a length of 128 MB bytes from 172.19.105.24:50010 to
>> 172.19.105.110:50010 using proxy source 172.19.105.26:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
>> 1073892376 with a length of 128 MB bytes from 172.19.105.26:50010 to
>> 172.19.105.113:50010 using proxy source 172.19.105.106:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073883114
>> from 172.19.105.27:50010 to 172.19.105.28:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
>> 1073917544 with a length of 128 MB bytes from 172.19.105.102:50010 to
>> 172.19.105.28:50010 using proxy source 172.19.105.108:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
>> 1073990086 with a length of 128 MB bytes from 172.19.105.27:50010 to
>> 172.19.105.108:50010 using proxy source 172.19.105.27:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073987422
>> from 172.19.105.30:50010 to 172.19.105.110:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074018604
>> from 172.19.105.24:50010 to 172.19.105.108:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074013709
>> from 172.19.105.27:50010 to 172.19.105.113:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073992638
>> from 172.19.105.23:50010 to 172.19.105.111:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
>> 1074035705 with a length of 128 MB bytes from 172.19.105.29:50010 to
>> 172.19.105.112:50010 using proxy source 172.19.105.23:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074035711
>> from 172.19.105.29:50010 to 172.19.105.112:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074008469
>> from 172.19.105.107:50010 to 172.19.105.106:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073917544
>> from 172.19.105.108:50010 to 172.19.105.28:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073990086
>> from 172.19.105.27:50010 to 172.19.105.108:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
>> 1073911911 with a length of 128 MB bytes from 172.19.105.29:50010 to
>> 172.19.105.112:50010 using proxy source 172.19.105.29:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073917025
>> from 172.19.105.26:50010 to 172.19.105.110:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073892376
>> from 172.19.105.106:50010 to 172.19.105.113:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074035705
>> from 172.19.105.23:50010 to 172.19.105.112:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073911911
>> from 172.19.105.29:50010 to 172.19.105.112:50010
>> 14/02/20 14:38:26 INFO balancer.Balancer: Moving block 1074035589 from
>> 172.19.105.29:50010 to 172.19.105.112:50010 through 172.19.105.103:50010is succeeded.
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
>> 1074035693 with a length of 128 MB bytes from 172.19.105.29:50010 to
>> 172.19.105.112:50010 using proxy source 172.19.105.29:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
>> 1074009432 with a length of 128 MB bytes from 172.19.105.101:50010 to
>> 172.19.105.103:50010 using proxy source 172.19.105.29:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074035693
>> from 172.19.105.29:50010 to 172.19.105.112:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074009432
>> from 172.19.105.29:50010 to 172.19.105.103:50010
>> 14/02/20 14:38:26 INFO balancer.Balancer: Moving block 1073997260 from
>> 172.19.105.101:50010 to 172.19.105.103:50010 through 172.19.105.106:50010is succeeded.
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
>> 1074009435 with a length of 128 MB bytes from 172.19.105.101:50010 to
>> 172.19.105.103:50010 using proxy source 172.19.105.27:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
>> 1074008471 with a length of 128 MB bytes from 172.19.105.25:50010 to
>> 172.19.105.106:50010 using proxy source 172.19.105.107:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074009435
>> from 172.19.105.27:50010 to 172.19.105.103:50010
>> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074008471
>> from 172.19.105.107:50010 to 172.19.105.106:50010
>> 14/02/20 14:38:32 INFO balancer.Balancer: Moving block 1074018599 from
>> 172.19.105.27:50010 to 172.19.105.108:50010 through 172.19.105.102:50010is succeeded.
>> 14/02/20 14:38:32 INFO balancer.Balancer: Moving block 1073812199 from
>> 172.19.105.102:50010 to 172.19.105.28:50010 through 172.19.105.24:50010is succeeded.
>> 14/02/20 14:38:33 INFO balancer.Balancer: Moving block 1073987402 from
>> 172.19.105.24:50010 to 172.19.105.110:50010 through 172.19.105.116:50010is succeeded.
>> 14/02/20 14:38:37 INFO balancer.Balancer: Moving block 1073922231 from
>> 172.19.105.23:50010 to 172.19.105.111:50010 through 172.19.105.114:50010is succeeded.
>> 14/02/20 14:38:38 INFO balancer.Balancer: Moving block 1073917544 from
>> 172.19.105.102:50010 to 172.19.105.28:50010 through 172.19.105.108:50010is succeeded.
>> 14/02/20 14:38:39 INFO balancer.Balancer: Moving block 1073922224 from
>> 172.19.105.23:50010 to 172.19.105.111:50010 through 172.19.105.102:50010is succeeded.
>> 14/02/20 14:38:39 INFO balancer.Balancer: Moving block 1074013803 from
>> 172.19.105.26:50010 to 172.19.105.113:50010 through 172.19.105.102:50010is succeeded.
>> 14/02/20 14:38:40 INFO balancer.Balancer: Moving block 1074035591 from
>> 172.19.105.29:50010 to 172.19.105.112:50010 through 172.19.105.101:50010is succeeded.
>> 14/02/20 14:38:40 INFO balancer.Balancer: Moving block 1073992320 from
>> 172.19.105.30:50010 to 172.19.105.116:50010 through 172.19.105.105:50010is succeeded.
>> 14/02/20 14:38:41 INFO balancer.Balancer: Moving block 1073961733 from
>> 172.19.105.30:50010 to 172.19.105.116:50010 through 172.19.105.101:50010is succeeded.
>> 14/02/20 14:38:42 INFO balancer.Balancer: Moving block 1073991855 from
>> 172.19.105.24:50010 to 172.19.105.110:50010 through 172.19.105.102:50010is succeeded.
>> 14/02/20 14:38:42 INFO balancer.Balancer: Moving block 1074008479 from
>> 172.19.105.25:50010 to 172.19.105.106:50010 through 172.19.105.107:50010is succeeded.
>> 14/02/20 14:38:43 INFO balancer.Balancer: Moving block 1074009436 from
>> 172.19.105.101:50010 to 172.19.105.103:50010 through 172.19.105.28:50010is succeeded.
>> 14/02/20 14:38:44 INFO balancer.Balancer: Moving block 1074008469 from
>> 172.19.105.25:50010 to 172.19.105.106:50010 through 172.19.105.107:50010is succeeded.
>> 14/02/20 14:38:44 INFO balancer.Balancer: Moving block 1073883114 from
>> 172.19.105.102:50010 to 172.19.105.28:50010 through 172.19.105.27:50010is succeeded.
>> 14/02/20 14:38:44 INFO balancer.Balancer: Moving block 1073911911 from
>> 172.19.105.29:50010 to 172.19.105.112:50010 through 172.19.105.29:50010is succeeded.
>> 14/02/20 14:38:44 INFO balancer.Balancer: Moving block 1073990086 from
>> 172.19.105.27:50010 to 172.19.105.108:50010 through 172.19.105.27:50010is succeeded.
>> 14/02/20 14:38:44 INFO balancer.Balancer: Moving block 1074008476 from
>> 172.19.105.25:50010 to 172.19.105.106:50010 through 172.19.105.107:50010is succeeded.
>> 14/02/20 14:38:45 INFO balancer.Balancer: Moving block 1073987422 from
>> 172.19.105.24:50010 to 172.19.105.110:50010 through 172.19.105.30:50010is succeeded.
>> 14/02/20 14:38:46 INFO balancer.Balancer: Moving block 1074013794 from
>> 172.19.105.26:50010 to 172.19.105.113:50010 through 172.19.105.101:50010is succeeded.
>> 14/02/20 14:38:46 INFO balancer.Balancer: Moving block 1074009432 from
>> 172.19.105.101:50010 to 172.19.105.103:50010 through 172.19.105.29:50010is succeeded.
>> 14/02/20 14:38:46 INFO balancer.Balancer: Moving block 1073936074 from
>> 172.19.105.101:50010 to 172.19.105.103:50010 through 172.19.105.101:50010is succeeded.
>> 14/02/20 14:38:47 INFO balancer.Balancer: Moving block 1074035711 from
>> 172.19.105.29:50010 to 172.19.105.112:50010 through 172.19.105.29:50010is succeeded.
>> 14/02/20 14:38:47 INFO balancer.Balancer: Moving block 1074013709 from
>> 172.19.105.26:50010 to 172.19.105.113:50010 through 172.19.105.27:50010is succeeded.
>> 14/02/20 14:38:47 INFO balancer.Balancer: Moving block 1074008471 from
>> 172.19.105.25:50010 to 172.19.105.106:50010 through 172.19.105.107:50010is succeeded.
>> 14/02/20 14:38:47 INFO balancer.Balancer: Moving block 1073992340 from
>> 172.19.105.30:50010 to 172.19.105.116:50010 through 172.19.105.111:50010is succeeded.
>> 14/02/20 14:38:47 INFO balancer.Balancer: Moving block 1074018604 from
>> 172.19.105.27:50010 to 172.19.105.108:50010 through 172.19.105.24:50010is succeeded.
>> 14/02/20 14:38:47 INFO balancer.Balancer: Moving block 1073917025 from
>> 172.19.105.24:50010 to 172.19.105.110:50010 through 172.19.105.26:50010is succeeded.
>> 14/02/20 14:38:48 INFO balancer.Balancer: Moving block 1074009439 from
>> 172.19.105.101:50010 to 172.19.105.103:50010 through 172.19.105.27:50010is succeeded.
>> 14/02/20 14:38:48 INFO balancer.Balancer: Moving block 1074013798 from
>> 172.19.105.26:50010 to 172.19.105.113:50010 through 172.19.105.101:50010is succeeded.
>> 14/02/20 14:38:48 INFO balancer.Balancer: Moving block 1074018603 from
>> 172.19.105.27:50010 to 172.19.105.108:50010 through 172.19.105.102:50010is succeeded.
>> 14/02/20 14:38:48 INFO balancer.Balancer: Moving block 1074035693 from
>> 172.19.105.29:50010 to 172.19.105.112:50010 through 172.19.105.29:50010is succeeded.
>> 14/02/20 14:38:48 INFO balancer.Balancer: Moving block 1073922442 from
>> 172.19.105.30:50010 to 172.19.105.116:50010 through 172.19.105.26:50010is succeeded.
>> 14/02/20 14:38:49 INFO balancer.Balancer: Moving block 1073892376 from
>> 172.19.105.26:50010 to 172.19.105.113:50010 through 172.19.105.106:50010is succeeded.
>> 14/02/20 14:38:50 INFO balancer.Balancer: Moving block 1073987410 from
>> 172.19.105.24:50010 to 172.19.105.110:50010 through 172.19.105.24:50010is succeeded.
>> 14/02/20 14:38:50 INFO balancer.Balancer: Moving block 1073922228 from
>> 172.19.105.23:50010 to 172.19.105.111:50010 through 172.19.105.23:50010is succeeded.
>> 14/02/20 14:38:50 INFO balancer.Balancer: Moving block 1073992638 from
>> 172.19.105.23:50010 to 172.19.105.111:50010 through 172.19.105.23:50010is succeeded.
>> 14/02/20 14:38:50 INFO balancer.Balancer: Moving block 1074009435 from
>> 172.19.105.101:50010 to 172.19.105.103:50010 through 172.19.105.27:50010is succeeded.
>> 14/02/20 14:38:52 INFO balancer.Balancer: Moving block 1074035705 from
>> 172.19.105.29:50010 to 172.19.105.112:50010 through 172.19.105.23:50010is succeeded.
>
>
>
>

Re: A question about Balancer in HDFS

Posted by QShu <sh...@gmail.com>.
Hi, Prav:

	Thanks for your help!
	Now I found out why the balancer stopped working. My Cluster has 2 racks, and the balancer follows the rule that can’t change the number of blocks of a rack. 
	At the same time, the 2 racks is not in the same disk usage. After I change all the node into 1 rack, balancer works fine.

Regards
QiShu


在 2014年2月20日，19:16，praveenesh kumar <pr...@gmail.com> 写道：

> Have you noticed any balancing in your datanodes by looking at the namenode url (where it shows data-node level usage). You should have.
> I think its not stuck, it just waiting for looking for more data blocks which it can move around to balance the cluster.
> There is threshold limit (if I remember it correctly), which you can also set while running the balancer (otherwise it uses some default value) and keeps on running till it reaches that threshold. I think it would be safe to just stop the process, as it has done balancing. 
> 
> Sorry I don't have the exact answer, just sharing my experiences. Someone else might throw better light in here.
> 
> Regards
> Prav
> 
> 
> On Thu, Feb 20, 2014 at 7:09 AM, QShu <sh...@gmail.com> wrote:
> Hi, Guys:
> 
>         The HDFS Balancer will stuck after running a while in my cluster now, the following is the log, after 14:38:52, the Balancer stuccoed and never continue balancing.
>         Can anybody give me some help, thank you!
> 
> ==============================================log==============================================
> 14/02/20 14:38:25 INFO balancer.Balancer: namenodes = [hdfs://hstore]
> 14/02/20 14:38:25 INFO balancer.Balancer: p         = Balancer.Parameters[BalancingPolicy.Node, threshold=10.0]
> Time Stamp               Iteration#  Bytes Already Moved  Bytes Left To Move  Bytes Being Moved
> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/172.19.105.111:50010
> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/172.19.105.103:50010
> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/172.19.105.108:50010
> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/172.19.105.102:50010
> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack1/172.19.105.23:50010
> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/172.19.105.112:50010
> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/172.19.105.106:50010
> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/172.19.105.101:50010
> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack1/172.19.105.28:50010
> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/172.19.105.113:50010
> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/172.19.105.110:50010
> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack1/172.19.105.27:50010
> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/172.19.105.116:50010
> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/172.19.105.105:50010
> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack1/172.19.105.29:50010
> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack1/172.19.105.25:50010
> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack1/172.19.105.26:50010
> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack1/172.19.105.24:50010
> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/172.19.105.114:50010
> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/172.19.105.107:50010
> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/172.19.105.109:50010
> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack1/172.19.105.30:50010
> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/172.19.105.115:50010
> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/172.19.105.104:50010
> 14/02/20 14:38:26 INFO balancer.Balancer: 9 over-utilized: [Source[172.19.105.102:50010, utilization=42.58400812492816], Source[172.19.105.23:50010, utilization=41.94449751804609], Source[172.19.105.101:50010, utilization=49.25646255747681], Source[172.19.105.27:50010, utilization=42.13637544454659], Source[172.19.105.29:50010, utilization=42.253685824453484], Source[172.19.105.25:50010, utilization=41.29288653593026], Source[172.19.105.26:50010, utilization=42.58689306059978], Source[172.19.105.24:50010, utilization=42.681174402528306], Source[172.19.105.30:50010, utilization=42.111934753109566]]
> 14/02/20 14:38:26 INFO balancer.Balancer: 1 underutilized: [BalancerDatanode[172.19.105.28:50010, utilization=5.073501546034823]]
> 14/02/20 14:38:26 INFO balancer.Balancer: Need to move 1.23 TB to make the cluster balanced.
> 14/02/20 14:38:26 INFO balancer.Balancer: Decided to move 10 GB bytes from 172.19.105.102:50010 to 172.19.105.28:50010
> 14/02/20 14:38:26 INFO balancer.Balancer: Decided to move 10 GB bytes from 172.19.105.23:50010 to 172.19.105.111:50010
> 14/02/20 14:38:26 INFO balancer.Balancer: Decided to move 10 GB bytes from 172.19.105.101:50010 to 172.19.105.103:50010
> 14/02/20 14:38:26 INFO balancer.Balancer: Decided to move 10 GB bytes from 172.19.105.27:50010 to 172.19.105.108:50010
> 14/02/20 14:38:26 INFO balancer.Balancer: Decided to move 10 GB bytes from 172.19.105.29:50010 to 172.19.105.112:50010
> 14/02/20 14:38:26 INFO balancer.Balancer: Decided to move 10 GB bytes from 172.19.105.25:50010 to 172.19.105.106:50010
> 14/02/20 14:38:26 INFO balancer.Balancer: Decided to move 10 GB bytes from 172.19.105.26:50010 to 172.19.105.113:50010
> 14/02/20 14:38:26 INFO balancer.Balancer: Decided to move 10 GB bytes from 172.19.105.24:50010 to 172.19.105.110:50010
> 14/02/20 14:38:26 INFO balancer.Balancer: Decided to move 10 GB bytes from 172.19.105.30:50010 to 172.19.105.116:50010
> 14/02/20 14:38:26 INFO balancer.Balancer: Will move 90 GB in this iteration
> 2014-2-20 14:38:26                0                  0 B             1.23 TB              90 GB
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1074013803 with a length of 128 MB bytes from 172.19.105.26:50010 to 172.19.105.113:50010 using proxy source 172.19.105.102:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1073936074 with a length of 128 MB bytes from 172.19.105.101:50010 to 172.19.105.103:50010 using proxy source 172.19.105.101:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1073961733 with a length of 128 MB bytes from 172.19.105.30:50010 to 172.19.105.116:50010 using proxy source 172.19.105.101:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1073922224 with a length of 128 MB bytes from 172.19.105.23:50010 to 172.19.105.111:50010 using proxy source 172.19.105.102:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1074018599 with a length of 26.12 MB bytes from 172.19.105.27:50010 to 172.19.105.108:50010 using proxy source 172.19.105.102:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1074013794 with a length of 128 MB bytes from 172.19.105.26:50010 to 172.19.105.113:50010 using proxy source 172.19.105.101:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074013803 from 172.19.105.102:50010 to 172.19.105.113:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1074013798 with a length of 128 MB bytes from 172.19.105.26:50010 to 172.19.105.113:50010 using proxy source 172.19.105.101:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074013794 from 172.19.105.101:50010 to 172.19.105.113:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1074018603 with a length of 128 MB bytes from 172.19.105.27:50010 to 172.19.105.108:50010 using proxy source 172.19.105.102:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074018599 from 172.19.105.102:50010 to 172.19.105.108:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1074008476 with a length of 128 MB bytes from 172.19.105.25:50010 to 172.19.105.106:50010 using proxy source 172.19.105.107:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1073922231 with a length of 128 MB bytes from 172.19.105.23:50010 to 172.19.105.111:50010 using proxy source 172.19.105.114:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073922224 from 172.19.105.102:50010 to 172.19.105.111:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1073812199 with a length of 64.75 MB bytes from 172.19.105.102:50010 to 172.19.105.28:50010 using proxy source 172.19.105.24:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1073991855 with a length of 73.88 MB bytes from 172.19.105.24:50010 to 172.19.105.110:50010 using proxy source 172.19.105.102:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1074035591 with a length of 128 MB bytes from 172.19.105.29:50010 to 172.19.105.112:50010 using proxy source 172.19.105.101:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1074008479 with a length of 128 MB bytes from 172.19.105.25:50010 to 172.19.105.106:50010 using proxy source 172.19.105.107:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074008476 from 172.19.105.107:50010 to 172.19.105.106:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1074035589 with a length of 6.51 KB bytes from 172.19.105.29:50010 to 172.19.105.112:50010 using proxy source 172.19.105.103:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074035591 from 172.19.105.101:50010 to 172.19.105.112:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1073997260 with a length of 85.24 KB bytes from 172.19.105.101:50010 to 172.19.105.103:50010 using proxy source 172.19.105.106:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1073992320 with a length of 128 MB bytes from 172.19.105.30:50010 to 172.19.105.116:50010 using proxy source 172.19.105.105:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073961733 from 172.19.105.101:50010 to 172.19.105.116:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073936074 from 172.19.105.101:50010 to 172.19.105.103:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1074009436 with a length of 128 MB bytes from 172.19.105.101:50010 to 172.19.105.103:50010 using proxy source 172.19.105.28:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073997260 from 172.19.105.106:50010 to 172.19.105.103:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073991855 from 172.19.105.102:50010 to 172.19.105.110:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1074009439 with a length of 128 MB bytes from 172.19.105.101:50010 to 172.19.105.103:50010 using proxy source 172.19.105.27:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074009436 from 172.19.105.28:50010 to 172.19.105.103:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073812199 from 172.19.105.24:50010 to 172.19.105.28:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1073883114 with a length of 128 MB bytes from 172.19.105.102:50010 to 172.19.105.28:50010 using proxy source 172.19.105.27:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1074018604 with a length of 128 MB bytes from 172.19.105.27:50010 to 172.19.105.108:50010 using proxy source 172.19.105.24:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074018603 from 172.19.105.102:50010 to 172.19.105.108:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1074013709 with a length of 128 MB bytes from 172.19.105.26:50010 to 172.19.105.113:50010 using proxy source 172.19.105.27:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074013798 from 172.19.105.101:50010 to 172.19.105.113:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074009439 from 172.19.105.27:50010 to 172.19.105.103:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1073987402 with a length of 75.99 MB bytes from 172.19.105.24:50010 to 172.19.105.110:50010 using proxy source 172.19.105.116:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1073987410 with a length of 128 MB bytes from 172.19.105.24:50010 to 172.19.105.110:50010 using proxy source 172.19.105.24:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073987402 from 172.19.105.116:50010 to 172.19.105.110:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1073987422 with a length of 128 MB bytes from 172.19.105.24:50010 to 172.19.105.110:50010 using proxy source 172.19.105.30:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1073922228 with a length of 128 MB bytes from 172.19.105.23:50010 to 172.19.105.111:50010 using proxy source 172.19.105.23:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073987410 from 172.19.105.24:50010 to 172.19.105.110:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073922231 from 172.19.105.114:50010 to 172.19.105.111:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1073922442 with a length of 128 MB bytes from 172.19.105.30:50010 to 172.19.105.116:50010 using proxy source 172.19.105.26:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073992320 from 172.19.105.105:50010 to 172.19.105.116:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1073992340 with a length of 128 MB bytes from 172.19.105.30:50010 to 172.19.105.116:50010 using proxy source 172.19.105.111:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073922442 from 172.19.105.26:50010 to 172.19.105.116:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073992340 from 172.19.105.111:50010 to 172.19.105.116:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1074008469 with a length of 128 MB bytes from 172.19.105.25:50010 to 172.19.105.106:50010 using proxy source 172.19.105.107:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074008479 from 172.19.105.107:50010 to 172.19.105.106:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1074035711 with a length of 128 MB bytes from 172.19.105.29:50010 to 172.19.105.112:50010 using proxy source 172.19.105.29:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074035589 from 172.19.105.103:50010 to 172.19.105.112:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1073992638 with a length of 128 MB bytes from 172.19.105.23:50010 to 172.19.105.111:50010 using proxy source 172.19.105.23:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073922228 from 172.19.105.23:50010 to 172.19.105.111:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1073917025 with a length of 128 MB bytes from 172.19.105.24:50010 to 172.19.105.110:50010 using proxy source 172.19.105.26:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1073892376 with a length of 128 MB bytes from 172.19.105.26:50010 to 172.19.105.113:50010 using proxy source 172.19.105.106:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073883114 from 172.19.105.27:50010 to 172.19.105.28:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1073917544 with a length of 128 MB bytes from 172.19.105.102:50010 to 172.19.105.28:50010 using proxy source 172.19.105.108:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1073990086 with a length of 128 MB bytes from 172.19.105.27:50010 to 172.19.105.108:50010 using proxy source 172.19.105.27:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073987422 from 172.19.105.30:50010 to 172.19.105.110:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074018604 from 172.19.105.24:50010 to 172.19.105.108:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074013709 from 172.19.105.27:50010 to 172.19.105.113:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073992638 from 172.19.105.23:50010 to 172.19.105.111:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1074035705 with a length of 128 MB bytes from 172.19.105.29:50010 to 172.19.105.112:50010 using proxy source 172.19.105.23:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074035711 from 172.19.105.29:50010 to 172.19.105.112:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074008469 from 172.19.105.107:50010 to 172.19.105.106:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073917544 from 172.19.105.108:50010 to 172.19.105.28:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073990086 from 172.19.105.27:50010 to 172.19.105.108:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1073911911 with a length of 128 MB bytes from 172.19.105.29:50010 to 172.19.105.112:50010 using proxy source 172.19.105.29:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073917025 from 172.19.105.26:50010 to 172.19.105.110:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073892376 from 172.19.105.106:50010 to 172.19.105.113:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074035705 from 172.19.105.23:50010 to 172.19.105.112:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073911911 from 172.19.105.29:50010 to 172.19.105.112:50010
> 14/02/20 14:38:26 INFO balancer.Balancer: Moving block 1074035589 from 172.19.105.29:50010 to 172.19.105.112:50010 through 172.19.105.103:50010 is succeeded.
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1074035693 with a length of 128 MB bytes from 172.19.105.29:50010 to 172.19.105.112:50010 using proxy source 172.19.105.29:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1074009432 with a length of 128 MB bytes from 172.19.105.101:50010 to 172.19.105.103:50010 using proxy source 172.19.105.29:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074035693 from 172.19.105.29:50010 to 172.19.105.112:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074009432 from 172.19.105.29:50010 to 172.19.105.103:50010
> 14/02/20 14:38:26 INFO balancer.Balancer: Moving block 1073997260 from 172.19.105.101:50010 to 172.19.105.103:50010 through 172.19.105.106:50010 is succeeded.
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1074009435 with a length of 128 MB bytes from 172.19.105.101:50010 to 172.19.105.103:50010 using proxy source 172.19.105.27:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1074008471 with a length of 128 MB bytes from 172.19.105.25:50010 to 172.19.105.106:50010 using proxy source 172.19.105.107:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074009435 from 172.19.105.27:50010 to 172.19.105.103:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074008471 from 172.19.105.107:50010 to 172.19.105.106:50010
> 14/02/20 14:38:32 INFO balancer.Balancer: Moving block 1074018599 from 172.19.105.27:50010 to 172.19.105.108:50010 through 172.19.105.102:50010 is succeeded.
> 14/02/20 14:38:32 INFO balancer.Balancer: Moving block 1073812199 from 172.19.105.102:50010 to 172.19.105.28:50010 through 172.19.105.24:50010 is succeeded.
> 14/02/20 14:38:33 INFO balancer.Balancer: Moving block 1073987402 from 172.19.105.24:50010 to 172.19.105.110:50010 through 172.19.105.116:50010 is succeeded.
> 14/02/20 14:38:37 INFO balancer.Balancer: Moving block 1073922231 from 172.19.105.23:50010 to 172.19.105.111:50010 through 172.19.105.114:50010 is succeeded.
> 14/02/20 14:38:38 INFO balancer.Balancer: Moving block 1073917544 from 172.19.105.102:50010 to 172.19.105.28:50010 through 172.19.105.108:50010 is succeeded.
> 14/02/20 14:38:39 INFO balancer.Balancer: Moving block 1073922224 from 172.19.105.23:50010 to 172.19.105.111:50010 through 172.19.105.102:50010 is succeeded.
> 14/02/20 14:38:39 INFO balancer.Balancer: Moving block 1074013803 from 172.19.105.26:50010 to 172.19.105.113:50010 through 172.19.105.102:50010 is succeeded.
> 14/02/20 14:38:40 INFO balancer.Balancer: Moving block 1074035591 from 172.19.105.29:50010 to 172.19.105.112:50010 through 172.19.105.101:50010 is succeeded.
> 14/02/20 14:38:40 INFO balancer.Balancer: Moving block 1073992320 from 172.19.105.30:50010 to 172.19.105.116:50010 through 172.19.105.105:50010 is succeeded.
> 14/02/20 14:38:41 INFO balancer.Balancer: Moving block 1073961733 from 172.19.105.30:50010 to 172.19.105.116:50010 through 172.19.105.101:50010 is succeeded.
> 14/02/20 14:38:42 INFO balancer.Balancer: Moving block 1073991855 from 172.19.105.24:50010 to 172.19.105.110:50010 through 172.19.105.102:50010 is succeeded.
> 14/02/20 14:38:42 INFO balancer.Balancer: Moving block 1074008479 from 172.19.105.25:50010 to 172.19.105.106:50010 through 172.19.105.107:50010 is succeeded.
> 14/02/20 14:38:43 INFO balancer.Balancer: Moving block 1074009436 from 172.19.105.101:50010 to 172.19.105.103:50010 through 172.19.105.28:50010 is succeeded.
> 14/02/20 14:38:44 INFO balancer.Balancer: Moving block 1074008469 from 172.19.105.25:50010 to 172.19.105.106:50010 through 172.19.105.107:50010 is succeeded.
> 14/02/20 14:38:44 INFO balancer.Balancer: Moving block 1073883114 from 172.19.105.102:50010 to 172.19.105.28:50010 through 172.19.105.27:50010 is succeeded.
> 14/02/20 14:38:44 INFO balancer.Balancer: Moving block 1073911911 from 172.19.105.29:50010 to 172.19.105.112:50010 through 172.19.105.29:50010 is succeeded.
> 14/02/20 14:38:44 INFO balancer.Balancer: Moving block 1073990086 from 172.19.105.27:50010 to 172.19.105.108:50010 through 172.19.105.27:50010 is succeeded.
> 14/02/20 14:38:44 INFO balancer.Balancer: Moving block 1074008476 from 172.19.105.25:50010 to 172.19.105.106:50010 through 172.19.105.107:50010 is succeeded.
> 14/02/20 14:38:45 INFO balancer.Balancer: Moving block 1073987422 from 172.19.105.24:50010 to 172.19.105.110:50010 through 172.19.105.30:50010 is succeeded.
> 14/02/20 14:38:46 INFO balancer.Balancer: Moving block 1074013794 from 172.19.105.26:50010 to 172.19.105.113:50010 through 172.19.105.101:50010 is succeeded.
> 14/02/20 14:38:46 INFO balancer.Balancer: Moving block 1074009432 from 172.19.105.101:50010 to 172.19.105.103:50010 through 172.19.105.29:50010 is succeeded.
> 14/02/20 14:38:46 INFO balancer.Balancer: Moving block 1073936074 from 172.19.105.101:50010 to 172.19.105.103:50010 through 172.19.105.101:50010 is succeeded.
> 14/02/20 14:38:47 INFO balancer.Balancer: Moving block 1074035711 from 172.19.105.29:50010 to 172.19.105.112:50010 through 172.19.105.29:50010 is succeeded.
> 14/02/20 14:38:47 INFO balancer.Balancer: Moving block 1074013709 from 172.19.105.26:50010 to 172.19.105.113:50010 through 172.19.105.27:50010 is succeeded.
> 14/02/20 14:38:47 INFO balancer.Balancer: Moving block 1074008471 from 172.19.105.25:50010 to 172.19.105.106:50010 through 172.19.105.107:50010 is succeeded.
> 14/02/20 14:38:47 INFO balancer.Balancer: Moving block 1073992340 from 172.19.105.30:50010 to 172.19.105.116:50010 through 172.19.105.111:50010 is succeeded.
> 14/02/20 14:38:47 INFO balancer.Balancer: Moving block 1074018604 from 172.19.105.27:50010 to 172.19.105.108:50010 through 172.19.105.24:50010 is succeeded.
> 14/02/20 14:38:47 INFO balancer.Balancer: Moving block 1073917025 from 172.19.105.24:50010 to 172.19.105.110:50010 through 172.19.105.26:50010 is succeeded.
> 14/02/20 14:38:48 INFO balancer.Balancer: Moving block 1074009439 from 172.19.105.101:50010 to 172.19.105.103:50010 through 172.19.105.27:50010 is succeeded.
> 14/02/20 14:38:48 INFO balancer.Balancer: Moving block 1074013798 from 172.19.105.26:50010 to 172.19.105.113:50010 through 172.19.105.101:50010 is succeeded.
> 14/02/20 14:38:48 INFO balancer.Balancer: Moving block 1074018603 from 172.19.105.27:50010 to 172.19.105.108:50010 through 172.19.105.102:50010 is succeeded.
> 14/02/20 14:38:48 INFO balancer.Balancer: Moving block 1074035693 from 172.19.105.29:50010 to 172.19.105.112:50010 through 172.19.105.29:50010 is succeeded.
> 14/02/20 14:38:48 INFO balancer.Balancer: Moving block 1073922442 from 172.19.105.30:50010 to 172.19.105.116:50010 through 172.19.105.26:50010 is succeeded.
> 14/02/20 14:38:49 INFO balancer.Balancer: Moving block 1073892376 from 172.19.105.26:50010 to 172.19.105.113:50010 through 172.19.105.106:50010 is succeeded.
> 14/02/20 14:38:50 INFO balancer.Balancer: Moving block 1073987410 from 172.19.105.24:50010 to 172.19.105.110:50010 through 172.19.105.24:50010 is succeeded.
> 14/02/20 14:38:50 INFO balancer.Balancer: Moving block 1073922228 from 172.19.105.23:50010 to 172.19.105.111:50010 through 172.19.105.23:50010 is succeeded.
> 14/02/20 14:38:50 INFO balancer.Balancer: Moving block 1073992638 from 172.19.105.23:50010 to 172.19.105.111:50010 through 172.19.105.23:50010 is succeeded.
> 14/02/20 14:38:50 INFO balancer.Balancer: Moving block 1074009435 from 172.19.105.101:50010 to 172.19.105.103:50010 through 172.19.105.27:50010 is succeeded.
> 14/02/20 14:38:52 INFO balancer.Balancer: Moving block 1074035705 from 172.19.105.29:50010 to 172.19.105.112:50010 through 172.19.105.23:50010 is succeeded.
>

Re: A question about Balancer in HDFS

Posted by QShu <sh...@gmail.com>.
Hi, Prav:

	Thanks for your help!
	Now I found out why the balancer stopped working. My Cluster has 2 racks, and the balancer follows the rule that can’t change the number of blocks of a rack. 
	At the same time, the 2 racks is not in the same disk usage. After I change all the node into 1 rack, balancer works fine.

Regards
QiShu


在 2014年2月20日，19:16，praveenesh kumar <pr...@gmail.com> 写道：

> Have you noticed any balancing in your datanodes by looking at the namenode url (where it shows data-node level usage). You should have.
> I think its not stuck, it just waiting for looking for more data blocks which it can move around to balance the cluster.
> There is threshold limit (if I remember it correctly), which you can also set while running the balancer (otherwise it uses some default value) and keeps on running till it reaches that threshold. I think it would be safe to just stop the process, as it has done balancing. 
> 
> Sorry I don't have the exact answer, just sharing my experiences. Someone else might throw better light in here.
> 
> Regards
> Prav
> 
> 
> On Thu, Feb 20, 2014 at 7:09 AM, QShu <sh...@gmail.com> wrote:
> Hi, Guys:
> 
>         The HDFS Balancer will stuck after running a while in my cluster now, the following is the log, after 14:38:52, the Balancer stuccoed and never continue balancing.
>         Can anybody give me some help, thank you!
> 
> ==============================================log==============================================
> 14/02/20 14:38:25 INFO balancer.Balancer: namenodes = [hdfs://hstore]
> 14/02/20 14:38:25 INFO balancer.Balancer: p         = Balancer.Parameters[BalancingPolicy.Node, threshold=10.0]
> Time Stamp               Iteration#  Bytes Already Moved  Bytes Left To Move  Bytes Being Moved
> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/172.19.105.111:50010
> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/172.19.105.103:50010
> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/172.19.105.108:50010
> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/172.19.105.102:50010
> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack1/172.19.105.23:50010
> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/172.19.105.112:50010
> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/172.19.105.106:50010
> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/172.19.105.101:50010
> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack1/172.19.105.28:50010
> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/172.19.105.113:50010
> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/172.19.105.110:50010
> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack1/172.19.105.27:50010
> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/172.19.105.116:50010
> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/172.19.105.105:50010
> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack1/172.19.105.29:50010
> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack1/172.19.105.25:50010
> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack1/172.19.105.26:50010
> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack1/172.19.105.24:50010
> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/172.19.105.114:50010
> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/172.19.105.107:50010
> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/172.19.105.109:50010
> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack1/172.19.105.30:50010
> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/172.19.105.115:50010
> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/172.19.105.104:50010
> 14/02/20 14:38:26 INFO balancer.Balancer: 9 over-utilized: [Source[172.19.105.102:50010, utilization=42.58400812492816], Source[172.19.105.23:50010, utilization=41.94449751804609], Source[172.19.105.101:50010, utilization=49.25646255747681], Source[172.19.105.27:50010, utilization=42.13637544454659], Source[172.19.105.29:50010, utilization=42.253685824453484], Source[172.19.105.25:50010, utilization=41.29288653593026], Source[172.19.105.26:50010, utilization=42.58689306059978], Source[172.19.105.24:50010, utilization=42.681174402528306], Source[172.19.105.30:50010, utilization=42.111934753109566]]
> 14/02/20 14:38:26 INFO balancer.Balancer: 1 underutilized: [BalancerDatanode[172.19.105.28:50010, utilization=5.073501546034823]]
> 14/02/20 14:38:26 INFO balancer.Balancer: Need to move 1.23 TB to make the cluster balanced.
> 14/02/20 14:38:26 INFO balancer.Balancer: Decided to move 10 GB bytes from 172.19.105.102:50010 to 172.19.105.28:50010
> 14/02/20 14:38:26 INFO balancer.Balancer: Decided to move 10 GB bytes from 172.19.105.23:50010 to 172.19.105.111:50010
> 14/02/20 14:38:26 INFO balancer.Balancer: Decided to move 10 GB bytes from 172.19.105.101:50010 to 172.19.105.103:50010
> 14/02/20 14:38:26 INFO balancer.Balancer: Decided to move 10 GB bytes from 172.19.105.27:50010 to 172.19.105.108:50010
> 14/02/20 14:38:26 INFO balancer.Balancer: Decided to move 10 GB bytes from 172.19.105.29:50010 to 172.19.105.112:50010
> 14/02/20 14:38:26 INFO balancer.Balancer: Decided to move 10 GB bytes from 172.19.105.25:50010 to 172.19.105.106:50010
> 14/02/20 14:38:26 INFO balancer.Balancer: Decided to move 10 GB bytes from 172.19.105.26:50010 to 172.19.105.113:50010
> 14/02/20 14:38:26 INFO balancer.Balancer: Decided to move 10 GB bytes from 172.19.105.24:50010 to 172.19.105.110:50010
> 14/02/20 14:38:26 INFO balancer.Balancer: Decided to move 10 GB bytes from 172.19.105.30:50010 to 172.19.105.116:50010
> 14/02/20 14:38:26 INFO balancer.Balancer: Will move 90 GB in this iteration
> 2014-2-20 14:38:26                0                  0 B             1.23 TB              90 GB
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1074013803 with a length of 128 MB bytes from 172.19.105.26:50010 to 172.19.105.113:50010 using proxy source 172.19.105.102:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1073936074 with a length of 128 MB bytes from 172.19.105.101:50010 to 172.19.105.103:50010 using proxy source 172.19.105.101:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1073961733 with a length of 128 MB bytes from 172.19.105.30:50010 to 172.19.105.116:50010 using proxy source 172.19.105.101:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1073922224 with a length of 128 MB bytes from 172.19.105.23:50010 to 172.19.105.111:50010 using proxy source 172.19.105.102:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1074018599 with a length of 26.12 MB bytes from 172.19.105.27:50010 to 172.19.105.108:50010 using proxy source 172.19.105.102:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1074013794 with a length of 128 MB bytes from 172.19.105.26:50010 to 172.19.105.113:50010 using proxy source 172.19.105.101:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074013803 from 172.19.105.102:50010 to 172.19.105.113:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1074013798 with a length of 128 MB bytes from 172.19.105.26:50010 to 172.19.105.113:50010 using proxy source 172.19.105.101:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074013794 from 172.19.105.101:50010 to 172.19.105.113:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1074018603 with a length of 128 MB bytes from 172.19.105.27:50010 to 172.19.105.108:50010 using proxy source 172.19.105.102:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074018599 from 172.19.105.102:50010 to 172.19.105.108:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1074008476 with a length of 128 MB bytes from 172.19.105.25:50010 to 172.19.105.106:50010 using proxy source 172.19.105.107:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1073922231 with a length of 128 MB bytes from 172.19.105.23:50010 to 172.19.105.111:50010 using proxy source 172.19.105.114:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073922224 from 172.19.105.102:50010 to 172.19.105.111:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1073812199 with a length of 64.75 MB bytes from 172.19.105.102:50010 to 172.19.105.28:50010 using proxy source 172.19.105.24:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1073991855 with a length of 73.88 MB bytes from 172.19.105.24:50010 to 172.19.105.110:50010 using proxy source 172.19.105.102:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1074035591 with a length of 128 MB bytes from 172.19.105.29:50010 to 172.19.105.112:50010 using proxy source 172.19.105.101:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1074008479 with a length of 128 MB bytes from 172.19.105.25:50010 to 172.19.105.106:50010 using proxy source 172.19.105.107:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074008476 from 172.19.105.107:50010 to 172.19.105.106:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1074035589 with a length of 6.51 KB bytes from 172.19.105.29:50010 to 172.19.105.112:50010 using proxy source 172.19.105.103:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074035591 from 172.19.105.101:50010 to 172.19.105.112:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1073997260 with a length of 85.24 KB bytes from 172.19.105.101:50010 to 172.19.105.103:50010 using proxy source 172.19.105.106:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1073992320 with a length of 128 MB bytes from 172.19.105.30:50010 to 172.19.105.116:50010 using proxy source 172.19.105.105:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073961733 from 172.19.105.101:50010 to 172.19.105.116:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073936074 from 172.19.105.101:50010 to 172.19.105.103:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1074009436 with a length of 128 MB bytes from 172.19.105.101:50010 to 172.19.105.103:50010 using proxy source 172.19.105.28:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073997260 from 172.19.105.106:50010 to 172.19.105.103:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073991855 from 172.19.105.102:50010 to 172.19.105.110:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1074009439 with a length of 128 MB bytes from 172.19.105.101:50010 to 172.19.105.103:50010 using proxy source 172.19.105.27:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074009436 from 172.19.105.28:50010 to 172.19.105.103:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073812199 from 172.19.105.24:50010 to 172.19.105.28:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1073883114 with a length of 128 MB bytes from 172.19.105.102:50010 to 172.19.105.28:50010 using proxy source 172.19.105.27:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1074018604 with a length of 128 MB bytes from 172.19.105.27:50010 to 172.19.105.108:50010 using proxy source 172.19.105.24:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074018603 from 172.19.105.102:50010 to 172.19.105.108:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1074013709 with a length of 128 MB bytes from 172.19.105.26:50010 to 172.19.105.113:50010 using proxy source 172.19.105.27:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074013798 from 172.19.105.101:50010 to 172.19.105.113:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074009439 from 172.19.105.27:50010 to 172.19.105.103:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1073987402 with a length of 75.99 MB bytes from 172.19.105.24:50010 to 172.19.105.110:50010 using proxy source 172.19.105.116:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1073987410 with a length of 128 MB bytes from 172.19.105.24:50010 to 172.19.105.110:50010 using proxy source 172.19.105.24:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073987402 from 172.19.105.116:50010 to 172.19.105.110:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1073987422 with a length of 128 MB bytes from 172.19.105.24:50010 to 172.19.105.110:50010 using proxy source 172.19.105.30:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1073922228 with a length of 128 MB bytes from 172.19.105.23:50010 to 172.19.105.111:50010 using proxy source 172.19.105.23:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073987410 from 172.19.105.24:50010 to 172.19.105.110:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073922231 from 172.19.105.114:50010 to 172.19.105.111:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1073922442 with a length of 128 MB bytes from 172.19.105.30:50010 to 172.19.105.116:50010 using proxy source 172.19.105.26:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073992320 from 172.19.105.105:50010 to 172.19.105.116:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1073992340 with a length of 128 MB bytes from 172.19.105.30:50010 to 172.19.105.116:50010 using proxy source 172.19.105.111:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073922442 from 172.19.105.26:50010 to 172.19.105.116:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073992340 from 172.19.105.111:50010 to 172.19.105.116:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1074008469 with a length of 128 MB bytes from 172.19.105.25:50010 to 172.19.105.106:50010 using proxy source 172.19.105.107:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074008479 from 172.19.105.107:50010 to 172.19.105.106:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1074035711 with a length of 128 MB bytes from 172.19.105.29:50010 to 172.19.105.112:50010 using proxy source 172.19.105.29:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074035589 from 172.19.105.103:50010 to 172.19.105.112:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1073992638 with a length of 128 MB bytes from 172.19.105.23:50010 to 172.19.105.111:50010 using proxy source 172.19.105.23:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073922228 from 172.19.105.23:50010 to 172.19.105.111:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1073917025 with a length of 128 MB bytes from 172.19.105.24:50010 to 172.19.105.110:50010 using proxy source 172.19.105.26:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1073892376 with a length of 128 MB bytes from 172.19.105.26:50010 to 172.19.105.113:50010 using proxy source 172.19.105.106:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073883114 from 172.19.105.27:50010 to 172.19.105.28:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1073917544 with a length of 128 MB bytes from 172.19.105.102:50010 to 172.19.105.28:50010 using proxy source 172.19.105.108:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1073990086 with a length of 128 MB bytes from 172.19.105.27:50010 to 172.19.105.108:50010 using proxy source 172.19.105.27:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073987422 from 172.19.105.30:50010 to 172.19.105.110:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074018604 from 172.19.105.24:50010 to 172.19.105.108:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074013709 from 172.19.105.27:50010 to 172.19.105.113:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073992638 from 172.19.105.23:50010 to 172.19.105.111:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1074035705 with a length of 128 MB bytes from 172.19.105.29:50010 to 172.19.105.112:50010 using proxy source 172.19.105.23:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074035711 from 172.19.105.29:50010 to 172.19.105.112:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074008469 from 172.19.105.107:50010 to 172.19.105.106:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073917544 from 172.19.105.108:50010 to 172.19.105.28:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073990086 from 172.19.105.27:50010 to 172.19.105.108:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1073911911 with a length of 128 MB bytes from 172.19.105.29:50010 to 172.19.105.112:50010 using proxy source 172.19.105.29:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073917025 from 172.19.105.26:50010 to 172.19.105.110:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073892376 from 172.19.105.106:50010 to 172.19.105.113:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074035705 from 172.19.105.23:50010 to 172.19.105.112:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073911911 from 172.19.105.29:50010 to 172.19.105.112:50010
> 14/02/20 14:38:26 INFO balancer.Balancer: Moving block 1074035589 from 172.19.105.29:50010 to 172.19.105.112:50010 through 172.19.105.103:50010 is succeeded.
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1074035693 with a length of 128 MB bytes from 172.19.105.29:50010 to 172.19.105.112:50010 using proxy source 172.19.105.29:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1074009432 with a length of 128 MB bytes from 172.19.105.101:50010 to 172.19.105.103:50010 using proxy source 172.19.105.29:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074035693 from 172.19.105.29:50010 to 172.19.105.112:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074009432 from 172.19.105.29:50010 to 172.19.105.103:50010
> 14/02/20 14:38:26 INFO balancer.Balancer: Moving block 1073997260 from 172.19.105.101:50010 to 172.19.105.103:50010 through 172.19.105.106:50010 is succeeded.
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1074009435 with a length of 128 MB bytes from 172.19.105.101:50010 to 172.19.105.103:50010 using proxy source 172.19.105.27:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1074008471 with a length of 128 MB bytes from 172.19.105.25:50010 to 172.19.105.106:50010 using proxy source 172.19.105.107:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074009435 from 172.19.105.27:50010 to 172.19.105.103:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074008471 from 172.19.105.107:50010 to 172.19.105.106:50010
> 14/02/20 14:38:32 INFO balancer.Balancer: Moving block 1074018599 from 172.19.105.27:50010 to 172.19.105.108:50010 through 172.19.105.102:50010 is succeeded.
> 14/02/20 14:38:32 INFO balancer.Balancer: Moving block 1073812199 from 172.19.105.102:50010 to 172.19.105.28:50010 through 172.19.105.24:50010 is succeeded.
> 14/02/20 14:38:33 INFO balancer.Balancer: Moving block 1073987402 from 172.19.105.24:50010 to 172.19.105.110:50010 through 172.19.105.116:50010 is succeeded.
> 14/02/20 14:38:37 INFO balancer.Balancer: Moving block 1073922231 from 172.19.105.23:50010 to 172.19.105.111:50010 through 172.19.105.114:50010 is succeeded.
> 14/02/20 14:38:38 INFO balancer.Balancer: Moving block 1073917544 from 172.19.105.102:50010 to 172.19.105.28:50010 through 172.19.105.108:50010 is succeeded.
> 14/02/20 14:38:39 INFO balancer.Balancer: Moving block 1073922224 from 172.19.105.23:50010 to 172.19.105.111:50010 through 172.19.105.102:50010 is succeeded.
> 14/02/20 14:38:39 INFO balancer.Balancer: Moving block 1074013803 from 172.19.105.26:50010 to 172.19.105.113:50010 through 172.19.105.102:50010 is succeeded.
> 14/02/20 14:38:40 INFO balancer.Balancer: Moving block 1074035591 from 172.19.105.29:50010 to 172.19.105.112:50010 through 172.19.105.101:50010 is succeeded.
> 14/02/20 14:38:40 INFO balancer.Balancer: Moving block 1073992320 from 172.19.105.30:50010 to 172.19.105.116:50010 through 172.19.105.105:50010 is succeeded.
> 14/02/20 14:38:41 INFO balancer.Balancer: Moving block 1073961733 from 172.19.105.30:50010 to 172.19.105.116:50010 through 172.19.105.101:50010 is succeeded.
> 14/02/20 14:38:42 INFO balancer.Balancer: Moving block 1073991855 from 172.19.105.24:50010 to 172.19.105.110:50010 through 172.19.105.102:50010 is succeeded.
> 14/02/20 14:38:42 INFO balancer.Balancer: Moving block 1074008479 from 172.19.105.25:50010 to 172.19.105.106:50010 through 172.19.105.107:50010 is succeeded.
> 14/02/20 14:38:43 INFO balancer.Balancer: Moving block 1074009436 from 172.19.105.101:50010 to 172.19.105.103:50010 through 172.19.105.28:50010 is succeeded.
> 14/02/20 14:38:44 INFO balancer.Balancer: Moving block 1074008469 from 172.19.105.25:50010 to 172.19.105.106:50010 through 172.19.105.107:50010 is succeeded.
> 14/02/20 14:38:44 INFO balancer.Balancer: Moving block 1073883114 from 172.19.105.102:50010 to 172.19.105.28:50010 through 172.19.105.27:50010 is succeeded.
> 14/02/20 14:38:44 INFO balancer.Balancer: Moving block 1073911911 from 172.19.105.29:50010 to 172.19.105.112:50010 through 172.19.105.29:50010 is succeeded.
> 14/02/20 14:38:44 INFO balancer.Balancer: Moving block 1073990086 from 172.19.105.27:50010 to 172.19.105.108:50010 through 172.19.105.27:50010 is succeeded.
> 14/02/20 14:38:44 INFO balancer.Balancer: Moving block 1074008476 from 172.19.105.25:50010 to 172.19.105.106:50010 through 172.19.105.107:50010 is succeeded.
> 14/02/20 14:38:45 INFO balancer.Balancer: Moving block 1073987422 from 172.19.105.24:50010 to 172.19.105.110:50010 through 172.19.105.30:50010 is succeeded.
> 14/02/20 14:38:46 INFO balancer.Balancer: Moving block 1074013794 from 172.19.105.26:50010 to 172.19.105.113:50010 through 172.19.105.101:50010 is succeeded.
> 14/02/20 14:38:46 INFO balancer.Balancer: Moving block 1074009432 from 172.19.105.101:50010 to 172.19.105.103:50010 through 172.19.105.29:50010 is succeeded.
> 14/02/20 14:38:46 INFO balancer.Balancer: Moving block 1073936074 from 172.19.105.101:50010 to 172.19.105.103:50010 through 172.19.105.101:50010 is succeeded.
> 14/02/20 14:38:47 INFO balancer.Balancer: Moving block 1074035711 from 172.19.105.29:50010 to 172.19.105.112:50010 through 172.19.105.29:50010 is succeeded.
> 14/02/20 14:38:47 INFO balancer.Balancer: Moving block 1074013709 from 172.19.105.26:50010 to 172.19.105.113:50010 through 172.19.105.27:50010 is succeeded.
> 14/02/20 14:38:47 INFO balancer.Balancer: Moving block 1074008471 from 172.19.105.25:50010 to 172.19.105.106:50010 through 172.19.105.107:50010 is succeeded.
> 14/02/20 14:38:47 INFO balancer.Balancer: Moving block 1073992340 from 172.19.105.30:50010 to 172.19.105.116:50010 through 172.19.105.111:50010 is succeeded.
> 14/02/20 14:38:47 INFO balancer.Balancer: Moving block 1074018604 from 172.19.105.27:50010 to 172.19.105.108:50010 through 172.19.105.24:50010 is succeeded.
> 14/02/20 14:38:47 INFO balancer.Balancer: Moving block 1073917025 from 172.19.105.24:50010 to 172.19.105.110:50010 through 172.19.105.26:50010 is succeeded.
> 14/02/20 14:38:48 INFO balancer.Balancer: Moving block 1074009439 from 172.19.105.101:50010 to 172.19.105.103:50010 through 172.19.105.27:50010 is succeeded.
> 14/02/20 14:38:48 INFO balancer.Balancer: Moving block 1074013798 from 172.19.105.26:50010 to 172.19.105.113:50010 through 172.19.105.101:50010 is succeeded.
> 14/02/20 14:38:48 INFO balancer.Balancer: Moving block 1074018603 from 172.19.105.27:50010 to 172.19.105.108:50010 through 172.19.105.102:50010 is succeeded.
> 14/02/20 14:38:48 INFO balancer.Balancer: Moving block 1074035693 from 172.19.105.29:50010 to 172.19.105.112:50010 through 172.19.105.29:50010 is succeeded.
> 14/02/20 14:38:48 INFO balancer.Balancer: Moving block 1073922442 from 172.19.105.30:50010 to 172.19.105.116:50010 through 172.19.105.26:50010 is succeeded.
> 14/02/20 14:38:49 INFO balancer.Balancer: Moving block 1073892376 from 172.19.105.26:50010 to 172.19.105.113:50010 through 172.19.105.106:50010 is succeeded.
> 14/02/20 14:38:50 INFO balancer.Balancer: Moving block 1073987410 from 172.19.105.24:50010 to 172.19.105.110:50010 through 172.19.105.24:50010 is succeeded.
> 14/02/20 14:38:50 INFO balancer.Balancer: Moving block 1073922228 from 172.19.105.23:50010 to 172.19.105.111:50010 through 172.19.105.23:50010 is succeeded.
> 14/02/20 14:38:50 INFO balancer.Balancer: Moving block 1073992638 from 172.19.105.23:50010 to 172.19.105.111:50010 through 172.19.105.23:50010 is succeeded.
> 14/02/20 14:38:50 INFO balancer.Balancer: Moving block 1074009435 from 172.19.105.101:50010 to 172.19.105.103:50010 through 172.19.105.27:50010 is succeeded.
> 14/02/20 14:38:52 INFO balancer.Balancer: Moving block 1074035705 from 172.19.105.29:50010 to 172.19.105.112:50010 through 172.19.105.23:50010 is succeeded.
>

Re: A question about Balancer in HDFS

Posted by QShu <sh...@gmail.com>.
Hi, Prav:

	Thanks for your help!
	Now I found out why the balancer stopped working. My Cluster has 2 racks, and the balancer follows the rule that can’t change the number of blocks of a rack. 
	At the same time, the 2 racks is not in the same disk usage. After I change all the node into 1 rack, balancer works fine.

Regards
QiShu


在 2014年2月20日，19:16，praveenesh kumar <pr...@gmail.com> 写道：

> Have you noticed any balancing in your datanodes by looking at the namenode url (where it shows data-node level usage). You should have.
> I think its not stuck, it just waiting for looking for more data blocks which it can move around to balance the cluster.
> There is threshold limit (if I remember it correctly), which you can also set while running the balancer (otherwise it uses some default value) and keeps on running till it reaches that threshold. I think it would be safe to just stop the process, as it has done balancing. 
> 
> Sorry I don't have the exact answer, just sharing my experiences. Someone else might throw better light in here.
> 
> Regards
> Prav
> 
> 
> On Thu, Feb 20, 2014 at 7:09 AM, QShu <sh...@gmail.com> wrote:
> Hi, Guys:
> 
>         The HDFS Balancer will stuck after running a while in my cluster now, the following is the log, after 14:38:52, the Balancer stuccoed and never continue balancing.
>         Can anybody give me some help, thank you!
> 
> ==============================================log==============================================
> 14/02/20 14:38:25 INFO balancer.Balancer: namenodes = [hdfs://hstore]
> 14/02/20 14:38:25 INFO balancer.Balancer: p         = Balancer.Parameters[BalancingPolicy.Node, threshold=10.0]
> Time Stamp               Iteration#  Bytes Already Moved  Bytes Left To Move  Bytes Being Moved
> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/172.19.105.111:50010
> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/172.19.105.103:50010
> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/172.19.105.108:50010
> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/172.19.105.102:50010
> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack1/172.19.105.23:50010
> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/172.19.105.112:50010
> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/172.19.105.106:50010
> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/172.19.105.101:50010
> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack1/172.19.105.28:50010
> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/172.19.105.113:50010
> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/172.19.105.110:50010
> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack1/172.19.105.27:50010
> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/172.19.105.116:50010
> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/172.19.105.105:50010
> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack1/172.19.105.29:50010
> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack1/172.19.105.25:50010
> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack1/172.19.105.26:50010
> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack1/172.19.105.24:50010
> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/172.19.105.114:50010
> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/172.19.105.107:50010
> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/172.19.105.109:50010
> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack1/172.19.105.30:50010
> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/172.19.105.115:50010
> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/172.19.105.104:50010
> 14/02/20 14:38:26 INFO balancer.Balancer: 9 over-utilized: [Source[172.19.105.102:50010, utilization=42.58400812492816], Source[172.19.105.23:50010, utilization=41.94449751804609], Source[172.19.105.101:50010, utilization=49.25646255747681], Source[172.19.105.27:50010, utilization=42.13637544454659], Source[172.19.105.29:50010, utilization=42.253685824453484], Source[172.19.105.25:50010, utilization=41.29288653593026], Source[172.19.105.26:50010, utilization=42.58689306059978], Source[172.19.105.24:50010, utilization=42.681174402528306], Source[172.19.105.30:50010, utilization=42.111934753109566]]
> 14/02/20 14:38:26 INFO balancer.Balancer: 1 underutilized: [BalancerDatanode[172.19.105.28:50010, utilization=5.073501546034823]]
> 14/02/20 14:38:26 INFO balancer.Balancer: Need to move 1.23 TB to make the cluster balanced.
> 14/02/20 14:38:26 INFO balancer.Balancer: Decided to move 10 GB bytes from 172.19.105.102:50010 to 172.19.105.28:50010
> 14/02/20 14:38:26 INFO balancer.Balancer: Decided to move 10 GB bytes from 172.19.105.23:50010 to 172.19.105.111:50010
> 14/02/20 14:38:26 INFO balancer.Balancer: Decided to move 10 GB bytes from 172.19.105.101:50010 to 172.19.105.103:50010
> 14/02/20 14:38:26 INFO balancer.Balancer: Decided to move 10 GB bytes from 172.19.105.27:50010 to 172.19.105.108:50010
> 14/02/20 14:38:26 INFO balancer.Balancer: Decided to move 10 GB bytes from 172.19.105.29:50010 to 172.19.105.112:50010
> 14/02/20 14:38:26 INFO balancer.Balancer: Decided to move 10 GB bytes from 172.19.105.25:50010 to 172.19.105.106:50010
> 14/02/20 14:38:26 INFO balancer.Balancer: Decided to move 10 GB bytes from 172.19.105.26:50010 to 172.19.105.113:50010
> 14/02/20 14:38:26 INFO balancer.Balancer: Decided to move 10 GB bytes from 172.19.105.24:50010 to 172.19.105.110:50010
> 14/02/20 14:38:26 INFO balancer.Balancer: Decided to move 10 GB bytes from 172.19.105.30:50010 to 172.19.105.116:50010
> 14/02/20 14:38:26 INFO balancer.Balancer: Will move 90 GB in this iteration
> 2014-2-20 14:38:26                0                  0 B             1.23 TB              90 GB
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1074013803 with a length of 128 MB bytes from 172.19.105.26:50010 to 172.19.105.113:50010 using proxy source 172.19.105.102:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1073936074 with a length of 128 MB bytes from 172.19.105.101:50010 to 172.19.105.103:50010 using proxy source 172.19.105.101:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1073961733 with a length of 128 MB bytes from 172.19.105.30:50010 to 172.19.105.116:50010 using proxy source 172.19.105.101:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1073922224 with a length of 128 MB bytes from 172.19.105.23:50010 to 172.19.105.111:50010 using proxy source 172.19.105.102:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1074018599 with a length of 26.12 MB bytes from 172.19.105.27:50010 to 172.19.105.108:50010 using proxy source 172.19.105.102:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1074013794 with a length of 128 MB bytes from 172.19.105.26:50010 to 172.19.105.113:50010 using proxy source 172.19.105.101:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074013803 from 172.19.105.102:50010 to 172.19.105.113:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1074013798 with a length of 128 MB bytes from 172.19.105.26:50010 to 172.19.105.113:50010 using proxy source 172.19.105.101:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074013794 from 172.19.105.101:50010 to 172.19.105.113:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1074018603 with a length of 128 MB bytes from 172.19.105.27:50010 to 172.19.105.108:50010 using proxy source 172.19.105.102:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074018599 from 172.19.105.102:50010 to 172.19.105.108:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1074008476 with a length of 128 MB bytes from 172.19.105.25:50010 to 172.19.105.106:50010 using proxy source 172.19.105.107:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1073922231 with a length of 128 MB bytes from 172.19.105.23:50010 to 172.19.105.111:50010 using proxy source 172.19.105.114:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073922224 from 172.19.105.102:50010 to 172.19.105.111:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1073812199 with a length of 64.75 MB bytes from 172.19.105.102:50010 to 172.19.105.28:50010 using proxy source 172.19.105.24:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1073991855 with a length of 73.88 MB bytes from 172.19.105.24:50010 to 172.19.105.110:50010 using proxy source 172.19.105.102:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1074035591 with a length of 128 MB bytes from 172.19.105.29:50010 to 172.19.105.112:50010 using proxy source 172.19.105.101:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1074008479 with a length of 128 MB bytes from 172.19.105.25:50010 to 172.19.105.106:50010 using proxy source 172.19.105.107:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074008476 from 172.19.105.107:50010 to 172.19.105.106:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1074035589 with a length of 6.51 KB bytes from 172.19.105.29:50010 to 172.19.105.112:50010 using proxy source 172.19.105.103:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074035591 from 172.19.105.101:50010 to 172.19.105.112:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1073997260 with a length of 85.24 KB bytes from 172.19.105.101:50010 to 172.19.105.103:50010 using proxy source 172.19.105.106:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1073992320 with a length of 128 MB bytes from 172.19.105.30:50010 to 172.19.105.116:50010 using proxy source 172.19.105.105:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073961733 from 172.19.105.101:50010 to 172.19.105.116:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073936074 from 172.19.105.101:50010 to 172.19.105.103:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1074009436 with a length of 128 MB bytes from 172.19.105.101:50010 to 172.19.105.103:50010 using proxy source 172.19.105.28:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073997260 from 172.19.105.106:50010 to 172.19.105.103:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073991855 from 172.19.105.102:50010 to 172.19.105.110:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1074009439 with a length of 128 MB bytes from 172.19.105.101:50010 to 172.19.105.103:50010 using proxy source 172.19.105.27:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074009436 from 172.19.105.28:50010 to 172.19.105.103:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073812199 from 172.19.105.24:50010 to 172.19.105.28:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1073883114 with a length of 128 MB bytes from 172.19.105.102:50010 to 172.19.105.28:50010 using proxy source 172.19.105.27:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1074018604 with a length of 128 MB bytes from 172.19.105.27:50010 to 172.19.105.108:50010 using proxy source 172.19.105.24:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074018603 from 172.19.105.102:50010 to 172.19.105.108:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1074013709 with a length of 128 MB bytes from 172.19.105.26:50010 to 172.19.105.113:50010 using proxy source 172.19.105.27:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074013798 from 172.19.105.101:50010 to 172.19.105.113:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074009439 from 172.19.105.27:50010 to 172.19.105.103:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1073987402 with a length of 75.99 MB bytes from 172.19.105.24:50010 to 172.19.105.110:50010 using proxy source 172.19.105.116:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1073987410 with a length of 128 MB bytes from 172.19.105.24:50010 to 172.19.105.110:50010 using proxy source 172.19.105.24:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073987402 from 172.19.105.116:50010 to 172.19.105.110:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1073987422 with a length of 128 MB bytes from 172.19.105.24:50010 to 172.19.105.110:50010 using proxy source 172.19.105.30:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1073922228 with a length of 128 MB bytes from 172.19.105.23:50010 to 172.19.105.111:50010 using proxy source 172.19.105.23:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073987410 from 172.19.105.24:50010 to 172.19.105.110:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073922231 from 172.19.105.114:50010 to 172.19.105.111:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1073922442 with a length of 128 MB bytes from 172.19.105.30:50010 to 172.19.105.116:50010 using proxy source 172.19.105.26:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073992320 from 172.19.105.105:50010 to 172.19.105.116:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1073992340 with a length of 128 MB bytes from 172.19.105.30:50010 to 172.19.105.116:50010 using proxy source 172.19.105.111:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073922442 from 172.19.105.26:50010 to 172.19.105.116:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073992340 from 172.19.105.111:50010 to 172.19.105.116:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1074008469 with a length of 128 MB bytes from 172.19.105.25:50010 to 172.19.105.106:50010 using proxy source 172.19.105.107:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074008479 from 172.19.105.107:50010 to 172.19.105.106:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1074035711 with a length of 128 MB bytes from 172.19.105.29:50010 to 172.19.105.112:50010 using proxy source 172.19.105.29:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074035589 from 172.19.105.103:50010 to 172.19.105.112:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1073992638 with a length of 128 MB bytes from 172.19.105.23:50010 to 172.19.105.111:50010 using proxy source 172.19.105.23:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073922228 from 172.19.105.23:50010 to 172.19.105.111:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1073917025 with a length of 128 MB bytes from 172.19.105.24:50010 to 172.19.105.110:50010 using proxy source 172.19.105.26:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1073892376 with a length of 128 MB bytes from 172.19.105.26:50010 to 172.19.105.113:50010 using proxy source 172.19.105.106:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073883114 from 172.19.105.27:50010 to 172.19.105.28:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1073917544 with a length of 128 MB bytes from 172.19.105.102:50010 to 172.19.105.28:50010 using proxy source 172.19.105.108:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1073990086 with a length of 128 MB bytes from 172.19.105.27:50010 to 172.19.105.108:50010 using proxy source 172.19.105.27:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073987422 from 172.19.105.30:50010 to 172.19.105.110:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074018604 from 172.19.105.24:50010 to 172.19.105.108:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074013709 from 172.19.105.27:50010 to 172.19.105.113:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073992638 from 172.19.105.23:50010 to 172.19.105.111:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1074035705 with a length of 128 MB bytes from 172.19.105.29:50010 to 172.19.105.112:50010 using proxy source 172.19.105.23:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074035711 from 172.19.105.29:50010 to 172.19.105.112:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074008469 from 172.19.105.107:50010 to 172.19.105.106:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073917544 from 172.19.105.108:50010 to 172.19.105.28:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073990086 from 172.19.105.27:50010 to 172.19.105.108:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1073911911 with a length of 128 MB bytes from 172.19.105.29:50010 to 172.19.105.112:50010 using proxy source 172.19.105.29:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073917025 from 172.19.105.26:50010 to 172.19.105.110:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073892376 from 172.19.105.106:50010 to 172.19.105.113:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074035705 from 172.19.105.23:50010 to 172.19.105.112:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073911911 from 172.19.105.29:50010 to 172.19.105.112:50010
> 14/02/20 14:38:26 INFO balancer.Balancer: Moving block 1074035589 from 172.19.105.29:50010 to 172.19.105.112:50010 through 172.19.105.103:50010 is succeeded.
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1074035693 with a length of 128 MB bytes from 172.19.105.29:50010 to 172.19.105.112:50010 using proxy source 172.19.105.29:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1074009432 with a length of 128 MB bytes from 172.19.105.101:50010 to 172.19.105.103:50010 using proxy source 172.19.105.29:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074035693 from 172.19.105.29:50010 to 172.19.105.112:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074009432 from 172.19.105.29:50010 to 172.19.105.103:50010
> 14/02/20 14:38:26 INFO balancer.Balancer: Moving block 1073997260 from 172.19.105.101:50010 to 172.19.105.103:50010 through 172.19.105.106:50010 is succeeded.
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1074009435 with a length of 128 MB bytes from 172.19.105.101:50010 to 172.19.105.103:50010 using proxy source 172.19.105.27:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1074008471 with a length of 128 MB bytes from 172.19.105.25:50010 to 172.19.105.106:50010 using proxy source 172.19.105.107:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074009435 from 172.19.105.27:50010 to 172.19.105.103:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074008471 from 172.19.105.107:50010 to 172.19.105.106:50010
> 14/02/20 14:38:32 INFO balancer.Balancer: Moving block 1074018599 from 172.19.105.27:50010 to 172.19.105.108:50010 through 172.19.105.102:50010 is succeeded.
> 14/02/20 14:38:32 INFO balancer.Balancer: Moving block 1073812199 from 172.19.105.102:50010 to 172.19.105.28:50010 through 172.19.105.24:50010 is succeeded.
> 14/02/20 14:38:33 INFO balancer.Balancer: Moving block 1073987402 from 172.19.105.24:50010 to 172.19.105.110:50010 through 172.19.105.116:50010 is succeeded.
> 14/02/20 14:38:37 INFO balancer.Balancer: Moving block 1073922231 from 172.19.105.23:50010 to 172.19.105.111:50010 through 172.19.105.114:50010 is succeeded.
> 14/02/20 14:38:38 INFO balancer.Balancer: Moving block 1073917544 from 172.19.105.102:50010 to 172.19.105.28:50010 through 172.19.105.108:50010 is succeeded.
> 14/02/20 14:38:39 INFO balancer.Balancer: Moving block 1073922224 from 172.19.105.23:50010 to 172.19.105.111:50010 through 172.19.105.102:50010 is succeeded.
> 14/02/20 14:38:39 INFO balancer.Balancer: Moving block 1074013803 from 172.19.105.26:50010 to 172.19.105.113:50010 through 172.19.105.102:50010 is succeeded.
> 14/02/20 14:38:40 INFO balancer.Balancer: Moving block 1074035591 from 172.19.105.29:50010 to 172.19.105.112:50010 through 172.19.105.101:50010 is succeeded.
> 14/02/20 14:38:40 INFO balancer.Balancer: Moving block 1073992320 from 172.19.105.30:50010 to 172.19.105.116:50010 through 172.19.105.105:50010 is succeeded.
> 14/02/20 14:38:41 INFO balancer.Balancer: Moving block 1073961733 from 172.19.105.30:50010 to 172.19.105.116:50010 through 172.19.105.101:50010 is succeeded.
> 14/02/20 14:38:42 INFO balancer.Balancer: Moving block 1073991855 from 172.19.105.24:50010 to 172.19.105.110:50010 through 172.19.105.102:50010 is succeeded.
> 14/02/20 14:38:42 INFO balancer.Balancer: Moving block 1074008479 from 172.19.105.25:50010 to 172.19.105.106:50010 through 172.19.105.107:50010 is succeeded.
> 14/02/20 14:38:43 INFO balancer.Balancer: Moving block 1074009436 from 172.19.105.101:50010 to 172.19.105.103:50010 through 172.19.105.28:50010 is succeeded.
> 14/02/20 14:38:44 INFO balancer.Balancer: Moving block 1074008469 from 172.19.105.25:50010 to 172.19.105.106:50010 through 172.19.105.107:50010 is succeeded.
> 14/02/20 14:38:44 INFO balancer.Balancer: Moving block 1073883114 from 172.19.105.102:50010 to 172.19.105.28:50010 through 172.19.105.27:50010 is succeeded.
> 14/02/20 14:38:44 INFO balancer.Balancer: Moving block 1073911911 from 172.19.105.29:50010 to 172.19.105.112:50010 through 172.19.105.29:50010 is succeeded.
> 14/02/20 14:38:44 INFO balancer.Balancer: Moving block 1073990086 from 172.19.105.27:50010 to 172.19.105.108:50010 through 172.19.105.27:50010 is succeeded.
> 14/02/20 14:38:44 INFO balancer.Balancer: Moving block 1074008476 from 172.19.105.25:50010 to 172.19.105.106:50010 through 172.19.105.107:50010 is succeeded.
> 14/02/20 14:38:45 INFO balancer.Balancer: Moving block 1073987422 from 172.19.105.24:50010 to 172.19.105.110:50010 through 172.19.105.30:50010 is succeeded.
> 14/02/20 14:38:46 INFO balancer.Balancer: Moving block 1074013794 from 172.19.105.26:50010 to 172.19.105.113:50010 through 172.19.105.101:50010 is succeeded.
> 14/02/20 14:38:46 INFO balancer.Balancer: Moving block 1074009432 from 172.19.105.101:50010 to 172.19.105.103:50010 through 172.19.105.29:50010 is succeeded.
> 14/02/20 14:38:46 INFO balancer.Balancer: Moving block 1073936074 from 172.19.105.101:50010 to 172.19.105.103:50010 through 172.19.105.101:50010 is succeeded.
> 14/02/20 14:38:47 INFO balancer.Balancer: Moving block 1074035711 from 172.19.105.29:50010 to 172.19.105.112:50010 through 172.19.105.29:50010 is succeeded.
> 14/02/20 14:38:47 INFO balancer.Balancer: Moving block 1074013709 from 172.19.105.26:50010 to 172.19.105.113:50010 through 172.19.105.27:50010 is succeeded.
> 14/02/20 14:38:47 INFO balancer.Balancer: Moving block 1074008471 from 172.19.105.25:50010 to 172.19.105.106:50010 through 172.19.105.107:50010 is succeeded.
> 14/02/20 14:38:47 INFO balancer.Balancer: Moving block 1073992340 from 172.19.105.30:50010 to 172.19.105.116:50010 through 172.19.105.111:50010 is succeeded.
> 14/02/20 14:38:47 INFO balancer.Balancer: Moving block 1074018604 from 172.19.105.27:50010 to 172.19.105.108:50010 through 172.19.105.24:50010 is succeeded.
> 14/02/20 14:38:47 INFO balancer.Balancer: Moving block 1073917025 from 172.19.105.24:50010 to 172.19.105.110:50010 through 172.19.105.26:50010 is succeeded.
> 14/02/20 14:38:48 INFO balancer.Balancer: Moving block 1074009439 from 172.19.105.101:50010 to 172.19.105.103:50010 through 172.19.105.27:50010 is succeeded.
> 14/02/20 14:38:48 INFO balancer.Balancer: Moving block 1074013798 from 172.19.105.26:50010 to 172.19.105.113:50010 through 172.19.105.101:50010 is succeeded.
> 14/02/20 14:38:48 INFO balancer.Balancer: Moving block 1074018603 from 172.19.105.27:50010 to 172.19.105.108:50010 through 172.19.105.102:50010 is succeeded.
> 14/02/20 14:38:48 INFO balancer.Balancer: Moving block 1074035693 from 172.19.105.29:50010 to 172.19.105.112:50010 through 172.19.105.29:50010 is succeeded.
> 14/02/20 14:38:48 INFO balancer.Balancer: Moving block 1073922442 from 172.19.105.30:50010 to 172.19.105.116:50010 through 172.19.105.26:50010 is succeeded.
> 14/02/20 14:38:49 INFO balancer.Balancer: Moving block 1073892376 from 172.19.105.26:50010 to 172.19.105.113:50010 through 172.19.105.106:50010 is succeeded.
> 14/02/20 14:38:50 INFO balancer.Balancer: Moving block 1073987410 from 172.19.105.24:50010 to 172.19.105.110:50010 through 172.19.105.24:50010 is succeeded.
> 14/02/20 14:38:50 INFO balancer.Balancer: Moving block 1073922228 from 172.19.105.23:50010 to 172.19.105.111:50010 through 172.19.105.23:50010 is succeeded.
> 14/02/20 14:38:50 INFO balancer.Balancer: Moving block 1073992638 from 172.19.105.23:50010 to 172.19.105.111:50010 through 172.19.105.23:50010 is succeeded.
> 14/02/20 14:38:50 INFO balancer.Balancer: Moving block 1074009435 from 172.19.105.101:50010 to 172.19.105.103:50010 through 172.19.105.27:50010 is succeeded.
> 14/02/20 14:38:52 INFO balancer.Balancer: Moving block 1074035705 from 172.19.105.29:50010 to 172.19.105.112:50010 through 172.19.105.23:50010 is succeeded.
>

Re: A question about Balancer in HDFS

Posted by QShu <sh...@gmail.com>.
Hi, Prav:

	Thanks for your help!
	Now I found out why the balancer stopped working. My Cluster has 2 racks, and the balancer follows the rule that can’t change the number of blocks of a rack. 
	At the same time, the 2 racks is not in the same disk usage. After I change all the node into 1 rack, balancer works fine.

Regards
QiShu


在 2014年2月20日，19:16，praveenesh kumar <pr...@gmail.com> 写道：

> Have you noticed any balancing in your datanodes by looking at the namenode url (where it shows data-node level usage). You should have.
> I think its not stuck, it just waiting for looking for more data blocks which it can move around to balance the cluster.
> There is threshold limit (if I remember it correctly), which you can also set while running the balancer (otherwise it uses some default value) and keeps on running till it reaches that threshold. I think it would be safe to just stop the process, as it has done balancing. 
> 
> Sorry I don't have the exact answer, just sharing my experiences. Someone else might throw better light in here.
> 
> Regards
> Prav
> 
> 
> On Thu, Feb 20, 2014 at 7:09 AM, QShu <sh...@gmail.com> wrote:
> Hi, Guys:
> 
>         The HDFS Balancer will stuck after running a while in my cluster now, the following is the log, after 14:38:52, the Balancer stuccoed and never continue balancing.
>         Can anybody give me some help, thank you!
> 
> ==============================================log==============================================
> 14/02/20 14:38:25 INFO balancer.Balancer: namenodes = [hdfs://hstore]
> 14/02/20 14:38:25 INFO balancer.Balancer: p         = Balancer.Parameters[BalancingPolicy.Node, threshold=10.0]
> Time Stamp               Iteration#  Bytes Already Moved  Bytes Left To Move  Bytes Being Moved
> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/172.19.105.111:50010
> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/172.19.105.103:50010
> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/172.19.105.108:50010
> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/172.19.105.102:50010
> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack1/172.19.105.23:50010
> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/172.19.105.112:50010
> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/172.19.105.106:50010
> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/172.19.105.101:50010
> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack1/172.19.105.28:50010
> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/172.19.105.113:50010
> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/172.19.105.110:50010
> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack1/172.19.105.27:50010
> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/172.19.105.116:50010
> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/172.19.105.105:50010
> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack1/172.19.105.29:50010
> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack1/172.19.105.25:50010
> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack1/172.19.105.26:50010
> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack1/172.19.105.24:50010
> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/172.19.105.114:50010
> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/172.19.105.107:50010
> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/172.19.105.109:50010
> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack1/172.19.105.30:50010
> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/172.19.105.115:50010
> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/172.19.105.104:50010
> 14/02/20 14:38:26 INFO balancer.Balancer: 9 over-utilized: [Source[172.19.105.102:50010, utilization=42.58400812492816], Source[172.19.105.23:50010, utilization=41.94449751804609], Source[172.19.105.101:50010, utilization=49.25646255747681], Source[172.19.105.27:50010, utilization=42.13637544454659], Source[172.19.105.29:50010, utilization=42.253685824453484], Source[172.19.105.25:50010, utilization=41.29288653593026], Source[172.19.105.26:50010, utilization=42.58689306059978], Source[172.19.105.24:50010, utilization=42.681174402528306], Source[172.19.105.30:50010, utilization=42.111934753109566]]
> 14/02/20 14:38:26 INFO balancer.Balancer: 1 underutilized: [BalancerDatanode[172.19.105.28:50010, utilization=5.073501546034823]]
> 14/02/20 14:38:26 INFO balancer.Balancer: Need to move 1.23 TB to make the cluster balanced.
> 14/02/20 14:38:26 INFO balancer.Balancer: Decided to move 10 GB bytes from 172.19.105.102:50010 to 172.19.105.28:50010
> 14/02/20 14:38:26 INFO balancer.Balancer: Decided to move 10 GB bytes from 172.19.105.23:50010 to 172.19.105.111:50010
> 14/02/20 14:38:26 INFO balancer.Balancer: Decided to move 10 GB bytes from 172.19.105.101:50010 to 172.19.105.103:50010
> 14/02/20 14:38:26 INFO balancer.Balancer: Decided to move 10 GB bytes from 172.19.105.27:50010 to 172.19.105.108:50010
> 14/02/20 14:38:26 INFO balancer.Balancer: Decided to move 10 GB bytes from 172.19.105.29:50010 to 172.19.105.112:50010
> 14/02/20 14:38:26 INFO balancer.Balancer: Decided to move 10 GB bytes from 172.19.105.25:50010 to 172.19.105.106:50010
> 14/02/20 14:38:26 INFO balancer.Balancer: Decided to move 10 GB bytes from 172.19.105.26:50010 to 172.19.105.113:50010
> 14/02/20 14:38:26 INFO balancer.Balancer: Decided to move 10 GB bytes from 172.19.105.24:50010 to 172.19.105.110:50010
> 14/02/20 14:38:26 INFO balancer.Balancer: Decided to move 10 GB bytes from 172.19.105.30:50010 to 172.19.105.116:50010
> 14/02/20 14:38:26 INFO balancer.Balancer: Will move 90 GB in this iteration
> 2014-2-20 14:38:26                0                  0 B             1.23 TB              90 GB
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1074013803 with a length of 128 MB bytes from 172.19.105.26:50010 to 172.19.105.113:50010 using proxy source 172.19.105.102:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1073936074 with a length of 128 MB bytes from 172.19.105.101:50010 to 172.19.105.103:50010 using proxy source 172.19.105.101:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1073961733 with a length of 128 MB bytes from 172.19.105.30:50010 to 172.19.105.116:50010 using proxy source 172.19.105.101:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1073922224 with a length of 128 MB bytes from 172.19.105.23:50010 to 172.19.105.111:50010 using proxy source 172.19.105.102:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1074018599 with a length of 26.12 MB bytes from 172.19.105.27:50010 to 172.19.105.108:50010 using proxy source 172.19.105.102:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1074013794 with a length of 128 MB bytes from 172.19.105.26:50010 to 172.19.105.113:50010 using proxy source 172.19.105.101:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074013803 from 172.19.105.102:50010 to 172.19.105.113:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1074013798 with a length of 128 MB bytes from 172.19.105.26:50010 to 172.19.105.113:50010 using proxy source 172.19.105.101:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074013794 from 172.19.105.101:50010 to 172.19.105.113:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1074018603 with a length of 128 MB bytes from 172.19.105.27:50010 to 172.19.105.108:50010 using proxy source 172.19.105.102:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074018599 from 172.19.105.102:50010 to 172.19.105.108:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1074008476 with a length of 128 MB bytes from 172.19.105.25:50010 to 172.19.105.106:50010 using proxy source 172.19.105.107:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1073922231 with a length of 128 MB bytes from 172.19.105.23:50010 to 172.19.105.111:50010 using proxy source 172.19.105.114:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073922224 from 172.19.105.102:50010 to 172.19.105.111:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1073812199 with a length of 64.75 MB bytes from 172.19.105.102:50010 to 172.19.105.28:50010 using proxy source 172.19.105.24:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1073991855 with a length of 73.88 MB bytes from 172.19.105.24:50010 to 172.19.105.110:50010 using proxy source 172.19.105.102:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1074035591 with a length of 128 MB bytes from 172.19.105.29:50010 to 172.19.105.112:50010 using proxy source 172.19.105.101:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1074008479 with a length of 128 MB bytes from 172.19.105.25:50010 to 172.19.105.106:50010 using proxy source 172.19.105.107:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074008476 from 172.19.105.107:50010 to 172.19.105.106:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1074035589 with a length of 6.51 KB bytes from 172.19.105.29:50010 to 172.19.105.112:50010 using proxy source 172.19.105.103:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074035591 from 172.19.105.101:50010 to 172.19.105.112:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1073997260 with a length of 85.24 KB bytes from 172.19.105.101:50010 to 172.19.105.103:50010 using proxy source 172.19.105.106:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1073992320 with a length of 128 MB bytes from 172.19.105.30:50010 to 172.19.105.116:50010 using proxy source 172.19.105.105:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073961733 from 172.19.105.101:50010 to 172.19.105.116:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073936074 from 172.19.105.101:50010 to 172.19.105.103:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1074009436 with a length of 128 MB bytes from 172.19.105.101:50010 to 172.19.105.103:50010 using proxy source 172.19.105.28:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073997260 from 172.19.105.106:50010 to 172.19.105.103:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073991855 from 172.19.105.102:50010 to 172.19.105.110:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1074009439 with a length of 128 MB bytes from 172.19.105.101:50010 to 172.19.105.103:50010 using proxy source 172.19.105.27:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074009436 from 172.19.105.28:50010 to 172.19.105.103:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073812199 from 172.19.105.24:50010 to 172.19.105.28:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1073883114 with a length of 128 MB bytes from 172.19.105.102:50010 to 172.19.105.28:50010 using proxy source 172.19.105.27:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1074018604 with a length of 128 MB bytes from 172.19.105.27:50010 to 172.19.105.108:50010 using proxy source 172.19.105.24:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074018603 from 172.19.105.102:50010 to 172.19.105.108:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1074013709 with a length of 128 MB bytes from 172.19.105.26:50010 to 172.19.105.113:50010 using proxy source 172.19.105.27:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074013798 from 172.19.105.101:50010 to 172.19.105.113:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074009439 from 172.19.105.27:50010 to 172.19.105.103:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1073987402 with a length of 75.99 MB bytes from 172.19.105.24:50010 to 172.19.105.110:50010 using proxy source 172.19.105.116:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1073987410 with a length of 128 MB bytes from 172.19.105.24:50010 to 172.19.105.110:50010 using proxy source 172.19.105.24:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073987402 from 172.19.105.116:50010 to 172.19.105.110:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1073987422 with a length of 128 MB bytes from 172.19.105.24:50010 to 172.19.105.110:50010 using proxy source 172.19.105.30:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1073922228 with a length of 128 MB bytes from 172.19.105.23:50010 to 172.19.105.111:50010 using proxy source 172.19.105.23:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073987410 from 172.19.105.24:50010 to 172.19.105.110:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073922231 from 172.19.105.114:50010 to 172.19.105.111:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1073922442 with a length of 128 MB bytes from 172.19.105.30:50010 to 172.19.105.116:50010 using proxy source 172.19.105.26:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073992320 from 172.19.105.105:50010 to 172.19.105.116:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1073992340 with a length of 128 MB bytes from 172.19.105.30:50010 to 172.19.105.116:50010 using proxy source 172.19.105.111:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073922442 from 172.19.105.26:50010 to 172.19.105.116:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073992340 from 172.19.105.111:50010 to 172.19.105.116:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1074008469 with a length of 128 MB bytes from 172.19.105.25:50010 to 172.19.105.106:50010 using proxy source 172.19.105.107:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074008479 from 172.19.105.107:50010 to 172.19.105.106:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1074035711 with a length of 128 MB bytes from 172.19.105.29:50010 to 172.19.105.112:50010 using proxy source 172.19.105.29:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074035589 from 172.19.105.103:50010 to 172.19.105.112:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1073992638 with a length of 128 MB bytes from 172.19.105.23:50010 to 172.19.105.111:50010 using proxy source 172.19.105.23:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073922228 from 172.19.105.23:50010 to 172.19.105.111:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1073917025 with a length of 128 MB bytes from 172.19.105.24:50010 to 172.19.105.110:50010 using proxy source 172.19.105.26:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1073892376 with a length of 128 MB bytes from 172.19.105.26:50010 to 172.19.105.113:50010 using proxy source 172.19.105.106:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073883114 from 172.19.105.27:50010 to 172.19.105.28:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1073917544 with a length of 128 MB bytes from 172.19.105.102:50010 to 172.19.105.28:50010 using proxy source 172.19.105.108:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1073990086 with a length of 128 MB bytes from 172.19.105.27:50010 to 172.19.105.108:50010 using proxy source 172.19.105.27:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073987422 from 172.19.105.30:50010 to 172.19.105.110:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074018604 from 172.19.105.24:50010 to 172.19.105.108:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074013709 from 172.19.105.27:50010 to 172.19.105.113:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073992638 from 172.19.105.23:50010 to 172.19.105.111:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1074035705 with a length of 128 MB bytes from 172.19.105.29:50010 to 172.19.105.112:50010 using proxy source 172.19.105.23:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074035711 from 172.19.105.29:50010 to 172.19.105.112:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074008469 from 172.19.105.107:50010 to 172.19.105.106:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073917544 from 172.19.105.108:50010 to 172.19.105.28:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073990086 from 172.19.105.27:50010 to 172.19.105.108:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1073911911 with a length of 128 MB bytes from 172.19.105.29:50010 to 172.19.105.112:50010 using proxy source 172.19.105.29:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073917025 from 172.19.105.26:50010 to 172.19.105.110:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073892376 from 172.19.105.106:50010 to 172.19.105.113:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074035705 from 172.19.105.23:50010 to 172.19.105.112:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073911911 from 172.19.105.29:50010 to 172.19.105.112:50010
> 14/02/20 14:38:26 INFO balancer.Balancer: Moving block 1074035589 from 172.19.105.29:50010 to 172.19.105.112:50010 through 172.19.105.103:50010 is succeeded.
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1074035693 with a length of 128 MB bytes from 172.19.105.29:50010 to 172.19.105.112:50010 using proxy source 172.19.105.29:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1074009432 with a length of 128 MB bytes from 172.19.105.101:50010 to 172.19.105.103:50010 using proxy source 172.19.105.29:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074035693 from 172.19.105.29:50010 to 172.19.105.112:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074009432 from 172.19.105.29:50010 to 172.19.105.103:50010
> 14/02/20 14:38:26 INFO balancer.Balancer: Moving block 1073997260 from 172.19.105.101:50010 to 172.19.105.103:50010 through 172.19.105.106:50010 is succeeded.
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1074009435 with a length of 128 MB bytes from 172.19.105.101:50010 to 172.19.105.103:50010 using proxy source 172.19.105.27:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1074008471 with a length of 128 MB bytes from 172.19.105.25:50010 to 172.19.105.106:50010 using proxy source 172.19.105.107:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074009435 from 172.19.105.27:50010 to 172.19.105.103:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074008471 from 172.19.105.107:50010 to 172.19.105.106:50010
> 14/02/20 14:38:32 INFO balancer.Balancer: Moving block 1074018599 from 172.19.105.27:50010 to 172.19.105.108:50010 through 172.19.105.102:50010 is succeeded.
> 14/02/20 14:38:32 INFO balancer.Balancer: Moving block 1073812199 from 172.19.105.102:50010 to 172.19.105.28:50010 through 172.19.105.24:50010 is succeeded.
> 14/02/20 14:38:33 INFO balancer.Balancer: Moving block 1073987402 from 172.19.105.24:50010 to 172.19.105.110:50010 through 172.19.105.116:50010 is succeeded.
> 14/02/20 14:38:37 INFO balancer.Balancer: Moving block 1073922231 from 172.19.105.23:50010 to 172.19.105.111:50010 through 172.19.105.114:50010 is succeeded.
> 14/02/20 14:38:38 INFO balancer.Balancer: Moving block 1073917544 from 172.19.105.102:50010 to 172.19.105.28:50010 through 172.19.105.108:50010 is succeeded.
> 14/02/20 14:38:39 INFO balancer.Balancer: Moving block 1073922224 from 172.19.105.23:50010 to 172.19.105.111:50010 through 172.19.105.102:50010 is succeeded.
> 14/02/20 14:38:39 INFO balancer.Balancer: Moving block 1074013803 from 172.19.105.26:50010 to 172.19.105.113:50010 through 172.19.105.102:50010 is succeeded.
> 14/02/20 14:38:40 INFO balancer.Balancer: Moving block 1074035591 from 172.19.105.29:50010 to 172.19.105.112:50010 through 172.19.105.101:50010 is succeeded.
> 14/02/20 14:38:40 INFO balancer.Balancer: Moving block 1073992320 from 172.19.105.30:50010 to 172.19.105.116:50010 through 172.19.105.105:50010 is succeeded.
> 14/02/20 14:38:41 INFO balancer.Balancer: Moving block 1073961733 from 172.19.105.30:50010 to 172.19.105.116:50010 through 172.19.105.101:50010 is succeeded.
> 14/02/20 14:38:42 INFO balancer.Balancer: Moving block 1073991855 from 172.19.105.24:50010 to 172.19.105.110:50010 through 172.19.105.102:50010 is succeeded.
> 14/02/20 14:38:42 INFO balancer.Balancer: Moving block 1074008479 from 172.19.105.25:50010 to 172.19.105.106:50010 through 172.19.105.107:50010 is succeeded.
> 14/02/20 14:38:43 INFO balancer.Balancer: Moving block 1074009436 from 172.19.105.101:50010 to 172.19.105.103:50010 through 172.19.105.28:50010 is succeeded.
> 14/02/20 14:38:44 INFO balancer.Balancer: Moving block 1074008469 from 172.19.105.25:50010 to 172.19.105.106:50010 through 172.19.105.107:50010 is succeeded.
> 14/02/20 14:38:44 INFO balancer.Balancer: Moving block 1073883114 from 172.19.105.102:50010 to 172.19.105.28:50010 through 172.19.105.27:50010 is succeeded.
> 14/02/20 14:38:44 INFO balancer.Balancer: Moving block 1073911911 from 172.19.105.29:50010 to 172.19.105.112:50010 through 172.19.105.29:50010 is succeeded.
> 14/02/20 14:38:44 INFO balancer.Balancer: Moving block 1073990086 from 172.19.105.27:50010 to 172.19.105.108:50010 through 172.19.105.27:50010 is succeeded.
> 14/02/20 14:38:44 INFO balancer.Balancer: Moving block 1074008476 from 172.19.105.25:50010 to 172.19.105.106:50010 through 172.19.105.107:50010 is succeeded.
> 14/02/20 14:38:45 INFO balancer.Balancer: Moving block 1073987422 from 172.19.105.24:50010 to 172.19.105.110:50010 through 172.19.105.30:50010 is succeeded.
> 14/02/20 14:38:46 INFO balancer.Balancer: Moving block 1074013794 from 172.19.105.26:50010 to 172.19.105.113:50010 through 172.19.105.101:50010 is succeeded.
> 14/02/20 14:38:46 INFO balancer.Balancer: Moving block 1074009432 from 172.19.105.101:50010 to 172.19.105.103:50010 through 172.19.105.29:50010 is succeeded.
> 14/02/20 14:38:46 INFO balancer.Balancer: Moving block 1073936074 from 172.19.105.101:50010 to 172.19.105.103:50010 through 172.19.105.101:50010 is succeeded.
> 14/02/20 14:38:47 INFO balancer.Balancer: Moving block 1074035711 from 172.19.105.29:50010 to 172.19.105.112:50010 through 172.19.105.29:50010 is succeeded.
> 14/02/20 14:38:47 INFO balancer.Balancer: Moving block 1074013709 from 172.19.105.26:50010 to 172.19.105.113:50010 through 172.19.105.27:50010 is succeeded.
> 14/02/20 14:38:47 INFO balancer.Balancer: Moving block 1074008471 from 172.19.105.25:50010 to 172.19.105.106:50010 through 172.19.105.107:50010 is succeeded.
> 14/02/20 14:38:47 INFO balancer.Balancer: Moving block 1073992340 from 172.19.105.30:50010 to 172.19.105.116:50010 through 172.19.105.111:50010 is succeeded.
> 14/02/20 14:38:47 INFO balancer.Balancer: Moving block 1074018604 from 172.19.105.27:50010 to 172.19.105.108:50010 through 172.19.105.24:50010 is succeeded.
> 14/02/20 14:38:47 INFO balancer.Balancer: Moving block 1073917025 from 172.19.105.24:50010 to 172.19.105.110:50010 through 172.19.105.26:50010 is succeeded.
> 14/02/20 14:38:48 INFO balancer.Balancer: Moving block 1074009439 from 172.19.105.101:50010 to 172.19.105.103:50010 through 172.19.105.27:50010 is succeeded.
> 14/02/20 14:38:48 INFO balancer.Balancer: Moving block 1074013798 from 172.19.105.26:50010 to 172.19.105.113:50010 through 172.19.105.101:50010 is succeeded.
> 14/02/20 14:38:48 INFO balancer.Balancer: Moving block 1074018603 from 172.19.105.27:50010 to 172.19.105.108:50010 through 172.19.105.102:50010 is succeeded.
> 14/02/20 14:38:48 INFO balancer.Balancer: Moving block 1074035693 from 172.19.105.29:50010 to 172.19.105.112:50010 through 172.19.105.29:50010 is succeeded.
> 14/02/20 14:38:48 INFO balancer.Balancer: Moving block 1073922442 from 172.19.105.30:50010 to 172.19.105.116:50010 through 172.19.105.26:50010 is succeeded.
> 14/02/20 14:38:49 INFO balancer.Balancer: Moving block 1073892376 from 172.19.105.26:50010 to 172.19.105.113:50010 through 172.19.105.106:50010 is succeeded.
> 14/02/20 14:38:50 INFO balancer.Balancer: Moving block 1073987410 from 172.19.105.24:50010 to 172.19.105.110:50010 through 172.19.105.24:50010 is succeeded.
> 14/02/20 14:38:50 INFO balancer.Balancer: Moving block 1073922228 from 172.19.105.23:50010 to 172.19.105.111:50010 through 172.19.105.23:50010 is succeeded.
> 14/02/20 14:38:50 INFO balancer.Balancer: Moving block 1073992638 from 172.19.105.23:50010 to 172.19.105.111:50010 through 172.19.105.23:50010 is succeeded.
> 14/02/20 14:38:50 INFO balancer.Balancer: Moving block 1074009435 from 172.19.105.101:50010 to 172.19.105.103:50010 through 172.19.105.27:50010 is succeeded.
> 14/02/20 14:38:52 INFO balancer.Balancer: Moving block 1074035705 from 172.19.105.29:50010 to 172.19.105.112:50010 through 172.19.105.23:50010 is succeeded.
>

Re: A question about Balancer in HDFS

Posted by praveenesh kumar <pr...@gmail.com>.
Have you noticed any balancing in your datanodes by looking at the namenode
url (where it shows data-node level usage). You should have.
I think its not stuck, it just waiting for looking for more data blocks
which it can move around to balance the cluster.
There is threshold limit (if I remember it correctly), which you can also
set while running the balancer (otherwise it uses some default value) and
keeps on running till it reaches that threshold. I think it would be safe
to just stop the process, as it has done balancing.

Sorry I don't have the exact answer, just sharing my experiences. Someone
else might throw better light in here.

Regards
Prav


On Thu, Feb 20, 2014 at 7:09 AM, QShu <sh...@gmail.com> wrote:

> Hi, Guys:
>
>         The HDFS Balancer will stuck after running a while in my cluster
> now, the following is the log, after 14:38:52, the Balancer stuccoed and
> never continue balancing.
>         Can anybody give me some help, thank you!
>
>
> ==============================================log==============================================
> 14/02/20 14:38:25 INFO balancer.Balancer: namenodes = [hdfs://hstore]
> 14/02/20 14:38:25 INFO balancer.Balancer: p         =
> Balancer.Parameters[BalancingPolicy.Node, threshold=10.0]
> Time Stamp               Iteration#  Bytes Already Moved  Bytes Left To
> Move  Bytes Being Moved
> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/
> 172.19.105.111:50010
> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/
> 172.19.105.103:50010
> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/
> 172.19.105.108:50010
> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/
> 172.19.105.102:50010
> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack1/
> 172.19.105.23:50010
> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/
> 172.19.105.112:50010
> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/
> 172.19.105.106:50010
> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/
> 172.19.105.101:50010
> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack1/
> 172.19.105.28:50010
> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/
> 172.19.105.113:50010
> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/
> 172.19.105.110:50010
> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack1/
> 172.19.105.27:50010
> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/
> 172.19.105.116:50010
> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/
> 172.19.105.105:50010
> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack1/
> 172.19.105.29:50010
> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack1/
> 172.19.105.25:50010
> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack1/
> 172.19.105.26:50010
> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack1/
> 172.19.105.24:50010
> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/
> 172.19.105.114:50010
> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/
> 172.19.105.107:50010
> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/
> 172.19.105.109:50010
> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack1/
> 172.19.105.30:50010
> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/
> 172.19.105.115:50010
> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/
> 172.19.105.104:50010
> 14/02/20 14:38:26 INFO balancer.Balancer: 9 over-utilized: [Source[
> 172.19.105.102:50010, utilization=42.58400812492816], Source[
> 172.19.105.23:50010, utilization=41.94449751804609], Source[
> 172.19.105.101:50010, utilization=49.25646255747681], Source[
> 172.19.105.27:50010, utilization=42.13637544454659], Source[
> 172.19.105.29:50010, utilization=42.253685824453484], Source[
> 172.19.105.25:50010, utilization=41.29288653593026], Source[
> 172.19.105.26:50010, utilization=42.58689306059978], Source[
> 172.19.105.24:50010, utilization=42.681174402528306], Source[
> 172.19.105.30:50010, utilization=42.111934753109566]]
> 14/02/20 14:38:26 INFO balancer.Balancer: 1 underutilized:
> [BalancerDatanode[172.19.105.28:50010, utilization=5.073501546034823]]
> 14/02/20 14:38:26 INFO balancer.Balancer: Need to move 1.23 TB to make the
> cluster balanced.
> 14/02/20 14:38:26 INFO balancer.Balancer: Decided to move 10 GB bytes from
> 172.19.105.102:50010 to 172.19.105.28:50010
> 14/02/20 14:38:26 INFO balancer.Balancer: Decided to move 10 GB bytes from
> 172.19.105.23:50010 to 172.19.105.111:50010
> 14/02/20 14:38:26 INFO balancer.Balancer: Decided to move 10 GB bytes from
> 172.19.105.101:50010 to 172.19.105.103:50010
> 14/02/20 14:38:26 INFO balancer.Balancer: Decided to move 10 GB bytes from
> 172.19.105.27:50010 to 172.19.105.108:50010
> 14/02/20 14:38:26 INFO balancer.Balancer: Decided to move 10 GB bytes from
> 172.19.105.29:50010 to 172.19.105.112:50010
> 14/02/20 14:38:26 INFO balancer.Balancer: Decided to move 10 GB bytes from
> 172.19.105.25:50010 to 172.19.105.106:50010
> 14/02/20 14:38:26 INFO balancer.Balancer: Decided to move 10 GB bytes from
> 172.19.105.26:50010 to 172.19.105.113:50010
> 14/02/20 14:38:26 INFO balancer.Balancer: Decided to move 10 GB bytes from
> 172.19.105.24:50010 to 172.19.105.110:50010
> 14/02/20 14:38:26 INFO balancer.Balancer: Decided to move 10 GB bytes from
> 172.19.105.30:50010 to 172.19.105.116:50010
> 14/02/20 14:38:26 INFO balancer.Balancer: Will move 90 GB in this iteration
> 2014-2-20 14:38:26                0                  0 B             1.23
> TB              90 GB
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
> 1074013803 with a length of 128 MB bytes from 172.19.105.26:50010 to
> 172.19.105.113:50010 using proxy source 172.19.105.102:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
> 1073936074 with a length of 128 MB bytes from 172.19.105.101:50010 to
> 172.19.105.103:50010 using proxy source 172.19.105.101:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
> 1073961733 with a length of 128 MB bytes from 172.19.105.30:50010 to
> 172.19.105.116:50010 using proxy source 172.19.105.101:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
> 1073922224 with a length of 128 MB bytes from 172.19.105.23:50010 to
> 172.19.105.111:50010 using proxy source 172.19.105.102:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
> 1074018599 with a length of 26.12 MB bytes from 172.19.105.27:50010 to
> 172.19.105.108:50010 using proxy source 172.19.105.102:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
> 1074013794 with a length of 128 MB bytes from 172.19.105.26:50010 to
> 172.19.105.113:50010 using proxy source 172.19.105.101:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074013803 from
> 172.19.105.102:50010 to 172.19.105.113:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
> 1074013798 with a length of 128 MB bytes from 172.19.105.26:50010 to
> 172.19.105.113:50010 using proxy source 172.19.105.101:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074013794 from
> 172.19.105.101:50010 to 172.19.105.113:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
> 1074018603 with a length of 128 MB bytes from 172.19.105.27:50010 to
> 172.19.105.108:50010 using proxy source 172.19.105.102:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074018599 from
> 172.19.105.102:50010 to 172.19.105.108:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
> 1074008476 with a length of 128 MB bytes from 172.19.105.25:50010 to
> 172.19.105.106:50010 using proxy source 172.19.105.107:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
> 1073922231 with a length of 128 MB bytes from 172.19.105.23:50010 to
> 172.19.105.111:50010 using proxy source 172.19.105.114:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073922224 from
> 172.19.105.102:50010 to 172.19.105.111:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
> 1073812199 with a length of 64.75 MB bytes from 172.19.105.102:50010 to
> 172.19.105.28:50010 using proxy source 172.19.105.24:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
> 1073991855 with a length of 73.88 MB bytes from 172.19.105.24:50010 to
> 172.19.105.110:50010 using proxy source 172.19.105.102:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
> 1074035591 with a length of 128 MB bytes from 172.19.105.29:50010 to
> 172.19.105.112:50010 using proxy source 172.19.105.101:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
> 1074008479 with a length of 128 MB bytes from 172.19.105.25:50010 to
> 172.19.105.106:50010 using proxy source 172.19.105.107:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074008476 from
> 172.19.105.107:50010 to 172.19.105.106:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
> 1074035589 with a length of 6.51 KB bytes from 172.19.105.29:50010 to
> 172.19.105.112:50010 using proxy source 172.19.105.103:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074035591 from
> 172.19.105.101:50010 to 172.19.105.112:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
> 1073997260 with a length of 85.24 KB bytes from 172.19.105.101:50010 to
> 172.19.105.103:50010 using proxy source 172.19.105.106:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
> 1073992320 with a length of 128 MB bytes from 172.19.105.30:50010 to
> 172.19.105.116:50010 using proxy source 172.19.105.105:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073961733 from
> 172.19.105.101:50010 to 172.19.105.116:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073936074 from
> 172.19.105.101:50010 to 172.19.105.103:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
> 1074009436 with a length of 128 MB bytes from 172.19.105.101:50010 to
> 172.19.105.103:50010 using proxy source 172.19.105.28:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073997260 from
> 172.19.105.106:50010 to 172.19.105.103:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073991855 from
> 172.19.105.102:50010 to 172.19.105.110:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
> 1074009439 with a length of 128 MB bytes from 172.19.105.101:50010 to
> 172.19.105.103:50010 using proxy source 172.19.105.27:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074009436 from
> 172.19.105.28:50010 to 172.19.105.103:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073812199 from
> 172.19.105.24:50010 to 172.19.105.28:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
> 1073883114 with a length of 128 MB bytes from 172.19.105.102:50010 to
> 172.19.105.28:50010 using proxy source 172.19.105.27:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
> 1074018604 with a length of 128 MB bytes from 172.19.105.27:50010 to
> 172.19.105.108:50010 using proxy source 172.19.105.24:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074018603 from
> 172.19.105.102:50010 to 172.19.105.108:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
> 1074013709 with a length of 128 MB bytes from 172.19.105.26:50010 to
> 172.19.105.113:50010 using proxy source 172.19.105.27:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074013798 from
> 172.19.105.101:50010 to 172.19.105.113:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074009439 from
> 172.19.105.27:50010 to 172.19.105.103:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
> 1073987402 with a length of 75.99 MB bytes from 172.19.105.24:50010 to
> 172.19.105.110:50010 using proxy source 172.19.105.116:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
> 1073987410 with a length of 128 MB bytes from 172.19.105.24:50010 to
> 172.19.105.110:50010 using proxy source 172.19.105.24:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073987402 from
> 172.19.105.116:50010 to 172.19.105.110:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
> 1073987422 with a length of 128 MB bytes from 172.19.105.24:50010 to
> 172.19.105.110:50010 using proxy source 172.19.105.30:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
> 1073922228 with a length of 128 MB bytes from 172.19.105.23:50010 to
> 172.19.105.111:50010 using proxy source 172.19.105.23:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073987410 from
> 172.19.105.24:50010 to 172.19.105.110:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073922231 from
> 172.19.105.114:50010 to 172.19.105.111:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
> 1073922442 with a length of 128 MB bytes from 172.19.105.30:50010 to
> 172.19.105.116:50010 using proxy source 172.19.105.26:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073992320 from
> 172.19.105.105:50010 to 172.19.105.116:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
> 1073992340 with a length of 128 MB bytes from 172.19.105.30:50010 to
> 172.19.105.116:50010 using proxy source 172.19.105.111:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073922442 from
> 172.19.105.26:50010 to 172.19.105.116:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073992340 from
> 172.19.105.111:50010 to 172.19.105.116:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
> 1074008469 with a length of 128 MB bytes from 172.19.105.25:50010 to
> 172.19.105.106:50010 using proxy source 172.19.105.107:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074008479 from
> 172.19.105.107:50010 to 172.19.105.106:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
> 1074035711 with a length of 128 MB bytes from 172.19.105.29:50010 to
> 172.19.105.112:50010 using proxy source 172.19.105.29:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074035589 from
> 172.19.105.103:50010 to 172.19.105.112:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
> 1073992638 with a length of 128 MB bytes from 172.19.105.23:50010 to
> 172.19.105.111:50010 using proxy source 172.19.105.23:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073922228 from
> 172.19.105.23:50010 to 172.19.105.111:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
> 1073917025 with a length of 128 MB bytes from 172.19.105.24:50010 to
> 172.19.105.110:50010 using proxy source 172.19.105.26:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
> 1073892376 with a length of 128 MB bytes from 172.19.105.26:50010 to
> 172.19.105.113:50010 using proxy source 172.19.105.106:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073883114 from
> 172.19.105.27:50010 to 172.19.105.28:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
> 1073917544 with a length of 128 MB bytes from 172.19.105.102:50010 to
> 172.19.105.28:50010 using proxy source 172.19.105.108:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
> 1073990086 with a length of 128 MB bytes from 172.19.105.27:50010 to
> 172.19.105.108:50010 using proxy source 172.19.105.27:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073987422 from
> 172.19.105.30:50010 to 172.19.105.110:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074018604 from
> 172.19.105.24:50010 to 172.19.105.108:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074013709 from
> 172.19.105.27:50010 to 172.19.105.113:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073992638 from
> 172.19.105.23:50010 to 172.19.105.111:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
> 1074035705 with a length of 128 MB bytes from 172.19.105.29:50010 to
> 172.19.105.112:50010 using proxy source 172.19.105.23:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074035711 from
> 172.19.105.29:50010 to 172.19.105.112:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074008469 from
> 172.19.105.107:50010 to 172.19.105.106:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073917544 from
> 172.19.105.108:50010 to 172.19.105.28:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073990086 from
> 172.19.105.27:50010 to 172.19.105.108:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
> 1073911911 with a length of 128 MB bytes from 172.19.105.29:50010 to
> 172.19.105.112:50010 using proxy source 172.19.105.29:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073917025 from
> 172.19.105.26:50010 to 172.19.105.110:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073892376 from
> 172.19.105.106:50010 to 172.19.105.113:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074035705 from
> 172.19.105.23:50010 to 172.19.105.112:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073911911 from
> 172.19.105.29:50010 to 172.19.105.112:50010
> 14/02/20 14:38:26 INFO balancer.Balancer: Moving block 1074035589 from
> 172.19.105.29:50010 to 172.19.105.112:50010 through 172.19.105.103:50010is succeeded.
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
> 1074035693 with a length of 128 MB bytes from 172.19.105.29:50010 to
> 172.19.105.112:50010 using proxy source 172.19.105.29:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
> 1074009432 with a length of 128 MB bytes from 172.19.105.101:50010 to
> 172.19.105.103:50010 using proxy source 172.19.105.29:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074035693 from
> 172.19.105.29:50010 to 172.19.105.112:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074009432 from
> 172.19.105.29:50010 to 172.19.105.103:50010
> 14/02/20 14:38:26 INFO balancer.Balancer: Moving block 1073997260 from
> 172.19.105.101:50010 to 172.19.105.103:50010 through 172.19.105.106:50010is succeeded.
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
> 1074009435 with a length of 128 MB bytes from 172.19.105.101:50010 to
> 172.19.105.103:50010 using proxy source 172.19.105.27:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
> 1074008471 with a length of 128 MB bytes from 172.19.105.25:50010 to
> 172.19.105.106:50010 using proxy source 172.19.105.107:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074009435 from
> 172.19.105.27:50010 to 172.19.105.103:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074008471 from
> 172.19.105.107:50010 to 172.19.105.106:50010
> 14/02/20 14:38:32 INFO balancer.Balancer: Moving block 1074018599 from
> 172.19.105.27:50010 to 172.19.105.108:50010 through 172.19.105.102:50010is succeeded.
> 14/02/20 14:38:32 INFO balancer.Balancer: Moving block 1073812199 from
> 172.19.105.102:50010 to 172.19.105.28:50010 through 172.19.105.24:50010is succeeded.
> 14/02/20 14:38:33 INFO balancer.Balancer: Moving block 1073987402 from
> 172.19.105.24:50010 to 172.19.105.110:50010 through 172.19.105.116:50010is succeeded.
> 14/02/20 14:38:37 INFO balancer.Balancer: Moving block 1073922231 from
> 172.19.105.23:50010 to 172.19.105.111:50010 through 172.19.105.114:50010is succeeded.
> 14/02/20 14:38:38 INFO balancer.Balancer: Moving block 1073917544 from
> 172.19.105.102:50010 to 172.19.105.28:50010 through 172.19.105.108:50010is succeeded.
> 14/02/20 14:38:39 INFO balancer.Balancer: Moving block 1073922224 from
> 172.19.105.23:50010 to 172.19.105.111:50010 through 172.19.105.102:50010is succeeded.
> 14/02/20 14:38:39 INFO balancer.Balancer: Moving block 1074013803 from
> 172.19.105.26:50010 to 172.19.105.113:50010 through 172.19.105.102:50010is succeeded.
> 14/02/20 14:38:40 INFO balancer.Balancer: Moving block 1074035591 from
> 172.19.105.29:50010 to 172.19.105.112:50010 through 172.19.105.101:50010is succeeded.
> 14/02/20 14:38:40 INFO balancer.Balancer: Moving block 1073992320 from
> 172.19.105.30:50010 to 172.19.105.116:50010 through 172.19.105.105:50010is succeeded.
> 14/02/20 14:38:41 INFO balancer.Balancer: Moving block 1073961733 from
> 172.19.105.30:50010 to 172.19.105.116:50010 through 172.19.105.101:50010is succeeded.
> 14/02/20 14:38:42 INFO balancer.Balancer: Moving block 1073991855 from
> 172.19.105.24:50010 to 172.19.105.110:50010 through 172.19.105.102:50010is succeeded.
> 14/02/20 14:38:42 INFO balancer.Balancer: Moving block 1074008479 from
> 172.19.105.25:50010 to 172.19.105.106:50010 through 172.19.105.107:50010is succeeded.
> 14/02/20 14:38:43 INFO balancer.Balancer: Moving block 1074009436 from
> 172.19.105.101:50010 to 172.19.105.103:50010 through 172.19.105.28:50010is succeeded.
> 14/02/20 14:38:44 INFO balancer.Balancer: Moving block 1074008469 from
> 172.19.105.25:50010 to 172.19.105.106:50010 through 172.19.105.107:50010is succeeded.
> 14/02/20 14:38:44 INFO balancer.Balancer: Moving block 1073883114 from
> 172.19.105.102:50010 to 172.19.105.28:50010 through 172.19.105.27:50010is succeeded.
> 14/02/20 14:38:44 INFO balancer.Balancer: Moving block 1073911911 from
> 172.19.105.29:50010 to 172.19.105.112:50010 through 172.19.105.29:50010is succeeded.
> 14/02/20 14:38:44 INFO balancer.Balancer: Moving block 1073990086 from
> 172.19.105.27:50010 to 172.19.105.108:50010 through 172.19.105.27:50010is succeeded.
> 14/02/20 14:38:44 INFO balancer.Balancer: Moving block 1074008476 from
> 172.19.105.25:50010 to 172.19.105.106:50010 through 172.19.105.107:50010is succeeded.
> 14/02/20 14:38:45 INFO balancer.Balancer: Moving block 1073987422 from
> 172.19.105.24:50010 to 172.19.105.110:50010 through 172.19.105.30:50010is succeeded.
> 14/02/20 14:38:46 INFO balancer.Balancer: Moving block 1074013794 from
> 172.19.105.26:50010 to 172.19.105.113:50010 through 172.19.105.101:50010is succeeded.
> 14/02/20 14:38:46 INFO balancer.Balancer: Moving block 1074009432 from
> 172.19.105.101:50010 to 172.19.105.103:50010 through 172.19.105.29:50010is succeeded.
> 14/02/20 14:38:46 INFO balancer.Balancer: Moving block 1073936074 from
> 172.19.105.101:50010 to 172.19.105.103:50010 through 172.19.105.101:50010is succeeded.
> 14/02/20 14:38:47 INFO balancer.Balancer: Moving block 1074035711 from
> 172.19.105.29:50010 to 172.19.105.112:50010 through 172.19.105.29:50010is succeeded.
> 14/02/20 14:38:47 INFO balancer.Balancer: Moving block 1074013709 from
> 172.19.105.26:50010 to 172.19.105.113:50010 through 172.19.105.27:50010is succeeded.
> 14/02/20 14:38:47 INFO balancer.Balancer: Moving block 1074008471 from
> 172.19.105.25:50010 to 172.19.105.106:50010 through 172.19.105.107:50010is succeeded.
> 14/02/20 14:38:47 INFO balancer.Balancer: Moving block 1073992340 from
> 172.19.105.30:50010 to 172.19.105.116:50010 through 172.19.105.111:50010is succeeded.
> 14/02/20 14:38:47 INFO balancer.Balancer: Moving block 1074018604 from
> 172.19.105.27:50010 to 172.19.105.108:50010 through 172.19.105.24:50010is succeeded.
> 14/02/20 14:38:47 INFO balancer.Balancer: Moving block 1073917025 from
> 172.19.105.24:50010 to 172.19.105.110:50010 through 172.19.105.26:50010is succeeded.
> 14/02/20 14:38:48 INFO balancer.Balancer: Moving block 1074009439 from
> 172.19.105.101:50010 to 172.19.105.103:50010 through 172.19.105.27:50010is succeeded.
> 14/02/20 14:38:48 INFO balancer.Balancer: Moving block 1074013798 from
> 172.19.105.26:50010 to 172.19.105.113:50010 through 172.19.105.101:50010is succeeded.
> 14/02/20 14:38:48 INFO balancer.Balancer: Moving block 1074018603 from
> 172.19.105.27:50010 to 172.19.105.108:50010 through 172.19.105.102:50010is succeeded.
> 14/02/20 14:38:48 INFO balancer.Balancer: Moving block 1074035693 from
> 172.19.105.29:50010 to 172.19.105.112:50010 through 172.19.105.29:50010is succeeded.
> 14/02/20 14:38:48 INFO balancer.Balancer: Moving block 1073922442 from
> 172.19.105.30:50010 to 172.19.105.116:50010 through 172.19.105.26:50010is succeeded.
> 14/02/20 14:38:49 INFO balancer.Balancer: Moving block 1073892376 from
> 172.19.105.26:50010 to 172.19.105.113:50010 through 172.19.105.106:50010is succeeded.
> 14/02/20 14:38:50 INFO balancer.Balancer: Moving block 1073987410 from
> 172.19.105.24:50010 to 172.19.105.110:50010 through 172.19.105.24:50010is succeeded.
> 14/02/20 14:38:50 INFO balancer.Balancer: Moving block 1073922228 from
> 172.19.105.23:50010 to 172.19.105.111:50010 through 172.19.105.23:50010is succeeded.
> 14/02/20 14:38:50 INFO balancer.Balancer: Moving block 1073992638 from
> 172.19.105.23:50010 to 172.19.105.111:50010 through 172.19.105.23:50010is succeeded.
> 14/02/20 14:38:50 INFO balancer.Balancer: Moving block 1074009435 from
> 172.19.105.101:50010 to 172.19.105.103:50010 through 172.19.105.27:50010is succeeded.
> 14/02/20 14:38:52 INFO balancer.Balancer: Moving block 1074035705 from
> 172.19.105.29:50010 to 172.19.105.112:50010 through 172.19.105.23:50010is succeeded.

Re: A question about Balancer in HDFS

Posted by praveenesh kumar <pr...@gmail.com>.
Have you noticed any balancing in your datanodes by looking at the namenode
url (where it shows data-node level usage). You should have.
I think its not stuck, it just waiting for looking for more data blocks
which it can move around to balance the cluster.
There is threshold limit (if I remember it correctly), which you can also
set while running the balancer (otherwise it uses some default value) and
keeps on running till it reaches that threshold. I think it would be safe
to just stop the process, as it has done balancing.

Sorry I don't have the exact answer, just sharing my experiences. Someone
else might throw better light in here.

Regards
Prav


On Thu, Feb 20, 2014 at 7:09 AM, QShu <sh...@gmail.com> wrote:

> Hi, Guys:
>
>         The HDFS Balancer will stuck after running a while in my cluster
> now, the following is the log, after 14:38:52, the Balancer stuccoed and
> never continue balancing.
>         Can anybody give me some help, thank you!
>
>
> ==============================================log==============================================
> 14/02/20 14:38:25 INFO balancer.Balancer: namenodes = [hdfs://hstore]
> 14/02/20 14:38:25 INFO balancer.Balancer: p         =
> Balancer.Parameters[BalancingPolicy.Node, threshold=10.0]
> Time Stamp               Iteration#  Bytes Already Moved  Bytes Left To
> Move  Bytes Being Moved
> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/
> 172.19.105.111:50010
> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/
> 172.19.105.103:50010
> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/
> 172.19.105.108:50010
> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/
> 172.19.105.102:50010
> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack1/
> 172.19.105.23:50010
> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/
> 172.19.105.112:50010
> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/
> 172.19.105.106:50010
> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/
> 172.19.105.101:50010
> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack1/
> 172.19.105.28:50010
> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/
> 172.19.105.113:50010
> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/
> 172.19.105.110:50010
> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack1/
> 172.19.105.27:50010
> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/
> 172.19.105.116:50010
> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/
> 172.19.105.105:50010
> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack1/
> 172.19.105.29:50010
> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack1/
> 172.19.105.25:50010
> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack1/
> 172.19.105.26:50010
> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack1/
> 172.19.105.24:50010
> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/
> 172.19.105.114:50010
> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/
> 172.19.105.107:50010
> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/
> 172.19.105.109:50010
> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack1/
> 172.19.105.30:50010
> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/
> 172.19.105.115:50010
> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/
> 172.19.105.104:50010
> 14/02/20 14:38:26 INFO balancer.Balancer: 9 over-utilized: [Source[
> 172.19.105.102:50010, utilization=42.58400812492816], Source[
> 172.19.105.23:50010, utilization=41.94449751804609], Source[
> 172.19.105.101:50010, utilization=49.25646255747681], Source[
> 172.19.105.27:50010, utilization=42.13637544454659], Source[
> 172.19.105.29:50010, utilization=42.253685824453484], Source[
> 172.19.105.25:50010, utilization=41.29288653593026], Source[
> 172.19.105.26:50010, utilization=42.58689306059978], Source[
> 172.19.105.24:50010, utilization=42.681174402528306], Source[
> 172.19.105.30:50010, utilization=42.111934753109566]]
> 14/02/20 14:38:26 INFO balancer.Balancer: 1 underutilized:
> [BalancerDatanode[172.19.105.28:50010, utilization=5.073501546034823]]
> 14/02/20 14:38:26 INFO balancer.Balancer: Need to move 1.23 TB to make the
> cluster balanced.
> 14/02/20 14:38:26 INFO balancer.Balancer: Decided to move 10 GB bytes from
> 172.19.105.102:50010 to 172.19.105.28:50010
> 14/02/20 14:38:26 INFO balancer.Balancer: Decided to move 10 GB bytes from
> 172.19.105.23:50010 to 172.19.105.111:50010
> 14/02/20 14:38:26 INFO balancer.Balancer: Decided to move 10 GB bytes from
> 172.19.105.101:50010 to 172.19.105.103:50010
> 14/02/20 14:38:26 INFO balancer.Balancer: Decided to move 10 GB bytes from
> 172.19.105.27:50010 to 172.19.105.108:50010
> 14/02/20 14:38:26 INFO balancer.Balancer: Decided to move 10 GB bytes from
> 172.19.105.29:50010 to 172.19.105.112:50010
> 14/02/20 14:38:26 INFO balancer.Balancer: Decided to move 10 GB bytes from
> 172.19.105.25:50010 to 172.19.105.106:50010
> 14/02/20 14:38:26 INFO balancer.Balancer: Decided to move 10 GB bytes from
> 172.19.105.26:50010 to 172.19.105.113:50010
> 14/02/20 14:38:26 INFO balancer.Balancer: Decided to move 10 GB bytes from
> 172.19.105.24:50010 to 172.19.105.110:50010
> 14/02/20 14:38:26 INFO balancer.Balancer: Decided to move 10 GB bytes from
> 172.19.105.30:50010 to 172.19.105.116:50010
> 14/02/20 14:38:26 INFO balancer.Balancer: Will move 90 GB in this iteration
> 2014-2-20 14:38:26                0                  0 B             1.23
> TB              90 GB
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
> 1074013803 with a length of 128 MB bytes from 172.19.105.26:50010 to
> 172.19.105.113:50010 using proxy source 172.19.105.102:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
> 1073936074 with a length of 128 MB bytes from 172.19.105.101:50010 to
> 172.19.105.103:50010 using proxy source 172.19.105.101:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
> 1073961733 with a length of 128 MB bytes from 172.19.105.30:50010 to
> 172.19.105.116:50010 using proxy source 172.19.105.101:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
> 1073922224 with a length of 128 MB bytes from 172.19.105.23:50010 to
> 172.19.105.111:50010 using proxy source 172.19.105.102:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
> 1074018599 with a length of 26.12 MB bytes from 172.19.105.27:50010 to
> 172.19.105.108:50010 using proxy source 172.19.105.102:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
> 1074013794 with a length of 128 MB bytes from 172.19.105.26:50010 to
> 172.19.105.113:50010 using proxy source 172.19.105.101:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074013803 from
> 172.19.105.102:50010 to 172.19.105.113:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
> 1074013798 with a length of 128 MB bytes from 172.19.105.26:50010 to
> 172.19.105.113:50010 using proxy source 172.19.105.101:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074013794 from
> 172.19.105.101:50010 to 172.19.105.113:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
> 1074018603 with a length of 128 MB bytes from 172.19.105.27:50010 to
> 172.19.105.108:50010 using proxy source 172.19.105.102:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074018599 from
> 172.19.105.102:50010 to 172.19.105.108:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
> 1074008476 with a length of 128 MB bytes from 172.19.105.25:50010 to
> 172.19.105.106:50010 using proxy source 172.19.105.107:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
> 1073922231 with a length of 128 MB bytes from 172.19.105.23:50010 to
> 172.19.105.111:50010 using proxy source 172.19.105.114:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073922224 from
> 172.19.105.102:50010 to 172.19.105.111:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
> 1073812199 with a length of 64.75 MB bytes from 172.19.105.102:50010 to
> 172.19.105.28:50010 using proxy source 172.19.105.24:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
> 1073991855 with a length of 73.88 MB bytes from 172.19.105.24:50010 to
> 172.19.105.110:50010 using proxy source 172.19.105.102:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
> 1074035591 with a length of 128 MB bytes from 172.19.105.29:50010 to
> 172.19.105.112:50010 using proxy source 172.19.105.101:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
> 1074008479 with a length of 128 MB bytes from 172.19.105.25:50010 to
> 172.19.105.106:50010 using proxy source 172.19.105.107:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074008476 from
> 172.19.105.107:50010 to 172.19.105.106:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
> 1074035589 with a length of 6.51 KB bytes from 172.19.105.29:50010 to
> 172.19.105.112:50010 using proxy source 172.19.105.103:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074035591 from
> 172.19.105.101:50010 to 172.19.105.112:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
> 1073997260 with a length of 85.24 KB bytes from 172.19.105.101:50010 to
> 172.19.105.103:50010 using proxy source 172.19.105.106:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
> 1073992320 with a length of 128 MB bytes from 172.19.105.30:50010 to
> 172.19.105.116:50010 using proxy source 172.19.105.105:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073961733 from
> 172.19.105.101:50010 to 172.19.105.116:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073936074 from
> 172.19.105.101:50010 to 172.19.105.103:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
> 1074009436 with a length of 128 MB bytes from 172.19.105.101:50010 to
> 172.19.105.103:50010 using proxy source 172.19.105.28:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073997260 from
> 172.19.105.106:50010 to 172.19.105.103:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073991855 from
> 172.19.105.102:50010 to 172.19.105.110:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
> 1074009439 with a length of 128 MB bytes from 172.19.105.101:50010 to
> 172.19.105.103:50010 using proxy source 172.19.105.27:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074009436 from
> 172.19.105.28:50010 to 172.19.105.103:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073812199 from
> 172.19.105.24:50010 to 172.19.105.28:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
> 1073883114 with a length of 128 MB bytes from 172.19.105.102:50010 to
> 172.19.105.28:50010 using proxy source 172.19.105.27:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
> 1074018604 with a length of 128 MB bytes from 172.19.105.27:50010 to
> 172.19.105.108:50010 using proxy source 172.19.105.24:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074018603 from
> 172.19.105.102:50010 to 172.19.105.108:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
> 1074013709 with a length of 128 MB bytes from 172.19.105.26:50010 to
> 172.19.105.113:50010 using proxy source 172.19.105.27:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074013798 from
> 172.19.105.101:50010 to 172.19.105.113:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074009439 from
> 172.19.105.27:50010 to 172.19.105.103:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
> 1073987402 with a length of 75.99 MB bytes from 172.19.105.24:50010 to
> 172.19.105.110:50010 using proxy source 172.19.105.116:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
> 1073987410 with a length of 128 MB bytes from 172.19.105.24:50010 to
> 172.19.105.110:50010 using proxy source 172.19.105.24:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073987402 from
> 172.19.105.116:50010 to 172.19.105.110:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
> 1073987422 with a length of 128 MB bytes from 172.19.105.24:50010 to
> 172.19.105.110:50010 using proxy source 172.19.105.30:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
> 1073922228 with a length of 128 MB bytes from 172.19.105.23:50010 to
> 172.19.105.111:50010 using proxy source 172.19.105.23:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073987410 from
> 172.19.105.24:50010 to 172.19.105.110:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073922231 from
> 172.19.105.114:50010 to 172.19.105.111:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
> 1073922442 with a length of 128 MB bytes from 172.19.105.30:50010 to
> 172.19.105.116:50010 using proxy source 172.19.105.26:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073992320 from
> 172.19.105.105:50010 to 172.19.105.116:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
> 1073992340 with a length of 128 MB bytes from 172.19.105.30:50010 to
> 172.19.105.116:50010 using proxy source 172.19.105.111:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073922442 from
> 172.19.105.26:50010 to 172.19.105.116:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073992340 from
> 172.19.105.111:50010 to 172.19.105.116:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
> 1074008469 with a length of 128 MB bytes from 172.19.105.25:50010 to
> 172.19.105.106:50010 using proxy source 172.19.105.107:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074008479 from
> 172.19.105.107:50010 to 172.19.105.106:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
> 1074035711 with a length of 128 MB bytes from 172.19.105.29:50010 to
> 172.19.105.112:50010 using proxy source 172.19.105.29:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074035589 from
> 172.19.105.103:50010 to 172.19.105.112:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
> 1073992638 with a length of 128 MB bytes from 172.19.105.23:50010 to
> 172.19.105.111:50010 using proxy source 172.19.105.23:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073922228 from
> 172.19.105.23:50010 to 172.19.105.111:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
> 1073917025 with a length of 128 MB bytes from 172.19.105.24:50010 to
> 172.19.105.110:50010 using proxy source 172.19.105.26:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
> 1073892376 with a length of 128 MB bytes from 172.19.105.26:50010 to
> 172.19.105.113:50010 using proxy source 172.19.105.106:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073883114 from
> 172.19.105.27:50010 to 172.19.105.28:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
> 1073917544 with a length of 128 MB bytes from 172.19.105.102:50010 to
> 172.19.105.28:50010 using proxy source 172.19.105.108:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
> 1073990086 with a length of 128 MB bytes from 172.19.105.27:50010 to
> 172.19.105.108:50010 using proxy source 172.19.105.27:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073987422 from
> 172.19.105.30:50010 to 172.19.105.110:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074018604 from
> 172.19.105.24:50010 to 172.19.105.108:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074013709 from
> 172.19.105.27:50010 to 172.19.105.113:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073992638 from
> 172.19.105.23:50010 to 172.19.105.111:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
> 1074035705 with a length of 128 MB bytes from 172.19.105.29:50010 to
> 172.19.105.112:50010 using proxy source 172.19.105.23:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074035711 from
> 172.19.105.29:50010 to 172.19.105.112:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074008469 from
> 172.19.105.107:50010 to 172.19.105.106:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073917544 from
> 172.19.105.108:50010 to 172.19.105.28:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073990086 from
> 172.19.105.27:50010 to 172.19.105.108:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
> 1073911911 with a length of 128 MB bytes from 172.19.105.29:50010 to
> 172.19.105.112:50010 using proxy source 172.19.105.29:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073917025 from
> 172.19.105.26:50010 to 172.19.105.110:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073892376 from
> 172.19.105.106:50010 to 172.19.105.113:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074035705 from
> 172.19.105.23:50010 to 172.19.105.112:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073911911 from
> 172.19.105.29:50010 to 172.19.105.112:50010
> 14/02/20 14:38:26 INFO balancer.Balancer: Moving block 1074035589 from
> 172.19.105.29:50010 to 172.19.105.112:50010 through 172.19.105.103:50010is succeeded.
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
> 1074035693 with a length of 128 MB bytes from 172.19.105.29:50010 to
> 172.19.105.112:50010 using proxy source 172.19.105.29:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
> 1074009432 with a length of 128 MB bytes from 172.19.105.101:50010 to
> 172.19.105.103:50010 using proxy source 172.19.105.29:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074035693 from
> 172.19.105.29:50010 to 172.19.105.112:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074009432 from
> 172.19.105.29:50010 to 172.19.105.103:50010
> 14/02/20 14:38:26 INFO balancer.Balancer: Moving block 1073997260 from
> 172.19.105.101:50010 to 172.19.105.103:50010 through 172.19.105.106:50010is succeeded.
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
> 1074009435 with a length of 128 MB bytes from 172.19.105.101:50010 to
> 172.19.105.103:50010 using proxy source 172.19.105.27:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
> 1074008471 with a length of 128 MB bytes from 172.19.105.25:50010 to
> 172.19.105.106:50010 using proxy source 172.19.105.107:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074009435 from
> 172.19.105.27:50010 to 172.19.105.103:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074008471 from
> 172.19.105.107:50010 to 172.19.105.106:50010
> 14/02/20 14:38:32 INFO balancer.Balancer: Moving block 1074018599 from
> 172.19.105.27:50010 to 172.19.105.108:50010 through 172.19.105.102:50010is succeeded.
> 14/02/20 14:38:32 INFO balancer.Balancer: Moving block 1073812199 from
> 172.19.105.102:50010 to 172.19.105.28:50010 through 172.19.105.24:50010is succeeded.
> 14/02/20 14:38:33 INFO balancer.Balancer: Moving block 1073987402 from
> 172.19.105.24:50010 to 172.19.105.110:50010 through 172.19.105.116:50010is succeeded.
> 14/02/20 14:38:37 INFO balancer.Balancer: Moving block 1073922231 from
> 172.19.105.23:50010 to 172.19.105.111:50010 through 172.19.105.114:50010is succeeded.
> 14/02/20 14:38:38 INFO balancer.Balancer: Moving block 1073917544 from
> 172.19.105.102:50010 to 172.19.105.28:50010 through 172.19.105.108:50010is succeeded.
> 14/02/20 14:38:39 INFO balancer.Balancer: Moving block 1073922224 from
> 172.19.105.23:50010 to 172.19.105.111:50010 through 172.19.105.102:50010is succeeded.
> 14/02/20 14:38:39 INFO balancer.Balancer: Moving block 1074013803 from
> 172.19.105.26:50010 to 172.19.105.113:50010 through 172.19.105.102:50010is succeeded.
> 14/02/20 14:38:40 INFO balancer.Balancer: Moving block 1074035591 from
> 172.19.105.29:50010 to 172.19.105.112:50010 through 172.19.105.101:50010is succeeded.
> 14/02/20 14:38:40 INFO balancer.Balancer: Moving block 1073992320 from
> 172.19.105.30:50010 to 172.19.105.116:50010 through 172.19.105.105:50010is succeeded.
> 14/02/20 14:38:41 INFO balancer.Balancer: Moving block 1073961733 from
> 172.19.105.30:50010 to 172.19.105.116:50010 through 172.19.105.101:50010is succeeded.
> 14/02/20 14:38:42 INFO balancer.Balancer: Moving block 1073991855 from
> 172.19.105.24:50010 to 172.19.105.110:50010 through 172.19.105.102:50010is succeeded.
> 14/02/20 14:38:42 INFO balancer.Balancer: Moving block 1074008479 from
> 172.19.105.25:50010 to 172.19.105.106:50010 through 172.19.105.107:50010is succeeded.
> 14/02/20 14:38:43 INFO balancer.Balancer: Moving block 1074009436 from
> 172.19.105.101:50010 to 172.19.105.103:50010 through 172.19.105.28:50010is succeeded.
> 14/02/20 14:38:44 INFO balancer.Balancer: Moving block 1074008469 from
> 172.19.105.25:50010 to 172.19.105.106:50010 through 172.19.105.107:50010is succeeded.
> 14/02/20 14:38:44 INFO balancer.Balancer: Moving block 1073883114 from
> 172.19.105.102:50010 to 172.19.105.28:50010 through 172.19.105.27:50010is succeeded.
> 14/02/20 14:38:44 INFO balancer.Balancer: Moving block 1073911911 from
> 172.19.105.29:50010 to 172.19.105.112:50010 through 172.19.105.29:50010is succeeded.
> 14/02/20 14:38:44 INFO balancer.Balancer: Moving block 1073990086 from
> 172.19.105.27:50010 to 172.19.105.108:50010 through 172.19.105.27:50010is succeeded.
> 14/02/20 14:38:44 INFO balancer.Balancer: Moving block 1074008476 from
> 172.19.105.25:50010 to 172.19.105.106:50010 through 172.19.105.107:50010is succeeded.
> 14/02/20 14:38:45 INFO balancer.Balancer: Moving block 1073987422 from
> 172.19.105.24:50010 to 172.19.105.110:50010 through 172.19.105.30:50010is succeeded.
> 14/02/20 14:38:46 INFO balancer.Balancer: Moving block 1074013794 from
> 172.19.105.26:50010 to 172.19.105.113:50010 through 172.19.105.101:50010is succeeded.
> 14/02/20 14:38:46 INFO balancer.Balancer: Moving block 1074009432 from
> 172.19.105.101:50010 to 172.19.105.103:50010 through 172.19.105.29:50010is succeeded.
> 14/02/20 14:38:46 INFO balancer.Balancer: Moving block 1073936074 from
> 172.19.105.101:50010 to 172.19.105.103:50010 through 172.19.105.101:50010is succeeded.
> 14/02/20 14:38:47 INFO balancer.Balancer: Moving block 1074035711 from
> 172.19.105.29:50010 to 172.19.105.112:50010 through 172.19.105.29:50010is succeeded.
> 14/02/20 14:38:47 INFO balancer.Balancer: Moving block 1074013709 from
> 172.19.105.26:50010 to 172.19.105.113:50010 through 172.19.105.27:50010is succeeded.
> 14/02/20 14:38:47 INFO balancer.Balancer: Moving block 1074008471 from
> 172.19.105.25:50010 to 172.19.105.106:50010 through 172.19.105.107:50010is succeeded.
> 14/02/20 14:38:47 INFO balancer.Balancer: Moving block 1073992340 from
> 172.19.105.30:50010 to 172.19.105.116:50010 through 172.19.105.111:50010is succeeded.
> 14/02/20 14:38:47 INFO balancer.Balancer: Moving block 1074018604 from
> 172.19.105.27:50010 to 172.19.105.108:50010 through 172.19.105.24:50010is succeeded.
> 14/02/20 14:38:47 INFO balancer.Balancer: Moving block 1073917025 from
> 172.19.105.24:50010 to 172.19.105.110:50010 through 172.19.105.26:50010is succeeded.
> 14/02/20 14:38:48 INFO balancer.Balancer: Moving block 1074009439 from
> 172.19.105.101:50010 to 172.19.105.103:50010 through 172.19.105.27:50010is succeeded.
> 14/02/20 14:38:48 INFO balancer.Balancer: Moving block 1074013798 from
> 172.19.105.26:50010 to 172.19.105.113:50010 through 172.19.105.101:50010is succeeded.
> 14/02/20 14:38:48 INFO balancer.Balancer: Moving block 1074018603 from
> 172.19.105.27:50010 to 172.19.105.108:50010 through 172.19.105.102:50010is succeeded.
> 14/02/20 14:38:48 INFO balancer.Balancer: Moving block 1074035693 from
> 172.19.105.29:50010 to 172.19.105.112:50010 through 172.19.105.29:50010is succeeded.
> 14/02/20 14:38:48 INFO balancer.Balancer: Moving block 1073922442 from
> 172.19.105.30:50010 to 172.19.105.116:50010 through 172.19.105.26:50010is succeeded.
> 14/02/20 14:38:49 INFO balancer.Balancer: Moving block 1073892376 from
> 172.19.105.26:50010 to 172.19.105.113:50010 through 172.19.105.106:50010is succeeded.
> 14/02/20 14:38:50 INFO balancer.Balancer: Moving block 1073987410 from
> 172.19.105.24:50010 to 172.19.105.110:50010 through 172.19.105.24:50010is succeeded.
> 14/02/20 14:38:50 INFO balancer.Balancer: Moving block 1073922228 from
> 172.19.105.23:50010 to 172.19.105.111:50010 through 172.19.105.23:50010is succeeded.
> 14/02/20 14:38:50 INFO balancer.Balancer: Moving block 1073992638 from
> 172.19.105.23:50010 to 172.19.105.111:50010 through 172.19.105.23:50010is succeeded.
> 14/02/20 14:38:50 INFO balancer.Balancer: Moving block 1074009435 from
> 172.19.105.101:50010 to 172.19.105.103:50010 through 172.19.105.27:50010is succeeded.
> 14/02/20 14:38:52 INFO balancer.Balancer: Moving block 1074035705 from
> 172.19.105.29:50010 to 172.19.105.112:50010 through 172.19.105.23:50010is succeeded.

Re: A question about Balancer in HDFS

Posted by praveenesh kumar <pr...@gmail.com>.
Have you noticed any balancing in your datanodes by looking at the namenode
url (where it shows data-node level usage). You should have.
I think its not stuck, it just waiting for looking for more data blocks
which it can move around to balance the cluster.
There is threshold limit (if I remember it correctly), which you can also
set while running the balancer (otherwise it uses some default value) and
keeps on running till it reaches that threshold. I think it would be safe
to just stop the process, as it has done balancing.

Sorry I don't have the exact answer, just sharing my experiences. Someone
else might throw better light in here.

Regards
Prav


On Thu, Feb 20, 2014 at 7:09 AM, QShu <sh...@gmail.com> wrote:

> Hi, Guys:
>
>         The HDFS Balancer will stuck after running a while in my cluster
> now, the following is the log, after 14:38:52, the Balancer stuccoed and
> never continue balancing.
>         Can anybody give me some help, thank you!
>
>
> ==============================================log==============================================
> 14/02/20 14:38:25 INFO balancer.Balancer: namenodes = [hdfs://hstore]
> 14/02/20 14:38:25 INFO balancer.Balancer: p         =
> Balancer.Parameters[BalancingPolicy.Node, threshold=10.0]
> Time Stamp               Iteration#  Bytes Already Moved  Bytes Left To
> Move  Bytes Being Moved
> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/
> 172.19.105.111:50010
> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/
> 172.19.105.103:50010
> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/
> 172.19.105.108:50010
> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/
> 172.19.105.102:50010
> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack1/
> 172.19.105.23:50010
> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/
> 172.19.105.112:50010
> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/
> 172.19.105.106:50010
> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/
> 172.19.105.101:50010
> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack1/
> 172.19.105.28:50010
> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/
> 172.19.105.113:50010
> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/
> 172.19.105.110:50010
> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack1/
> 172.19.105.27:50010
> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/
> 172.19.105.116:50010
> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/
> 172.19.105.105:50010
> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack1/
> 172.19.105.29:50010
> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack1/
> 172.19.105.25:50010
> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack1/
> 172.19.105.26:50010
> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack1/
> 172.19.105.24:50010
> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/
> 172.19.105.114:50010
> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/
> 172.19.105.107:50010
> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/
> 172.19.105.109:50010
> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack1/
> 172.19.105.30:50010
> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/
> 172.19.105.115:50010
> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/
> 172.19.105.104:50010
> 14/02/20 14:38:26 INFO balancer.Balancer: 9 over-utilized: [Source[
> 172.19.105.102:50010, utilization=42.58400812492816], Source[
> 172.19.105.23:50010, utilization=41.94449751804609], Source[
> 172.19.105.101:50010, utilization=49.25646255747681], Source[
> 172.19.105.27:50010, utilization=42.13637544454659], Source[
> 172.19.105.29:50010, utilization=42.253685824453484], Source[
> 172.19.105.25:50010, utilization=41.29288653593026], Source[
> 172.19.105.26:50010, utilization=42.58689306059978], Source[
> 172.19.105.24:50010, utilization=42.681174402528306], Source[
> 172.19.105.30:50010, utilization=42.111934753109566]]
> 14/02/20 14:38:26 INFO balancer.Balancer: 1 underutilized:
> [BalancerDatanode[172.19.105.28:50010, utilization=5.073501546034823]]
> 14/02/20 14:38:26 INFO balancer.Balancer: Need to move 1.23 TB to make the
> cluster balanced.
> 14/02/20 14:38:26 INFO balancer.Balancer: Decided to move 10 GB bytes from
> 172.19.105.102:50010 to 172.19.105.28:50010
> 14/02/20 14:38:26 INFO balancer.Balancer: Decided to move 10 GB bytes from
> 172.19.105.23:50010 to 172.19.105.111:50010
> 14/02/20 14:38:26 INFO balancer.Balancer: Decided to move 10 GB bytes from
> 172.19.105.101:50010 to 172.19.105.103:50010
> 14/02/20 14:38:26 INFO balancer.Balancer: Decided to move 10 GB bytes from
> 172.19.105.27:50010 to 172.19.105.108:50010
> 14/02/20 14:38:26 INFO balancer.Balancer: Decided to move 10 GB bytes from
> 172.19.105.29:50010 to 172.19.105.112:50010
> 14/02/20 14:38:26 INFO balancer.Balancer: Decided to move 10 GB bytes from
> 172.19.105.25:50010 to 172.19.105.106:50010
> 14/02/20 14:38:26 INFO balancer.Balancer: Decided to move 10 GB bytes from
> 172.19.105.26:50010 to 172.19.105.113:50010
> 14/02/20 14:38:26 INFO balancer.Balancer: Decided to move 10 GB bytes from
> 172.19.105.24:50010 to 172.19.105.110:50010
> 14/02/20 14:38:26 INFO balancer.Balancer: Decided to move 10 GB bytes from
> 172.19.105.30:50010 to 172.19.105.116:50010
> 14/02/20 14:38:26 INFO balancer.Balancer: Will move 90 GB in this iteration
> 2014-2-20 14:38:26                0                  0 B             1.23
> TB              90 GB
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
> 1074013803 with a length of 128 MB bytes from 172.19.105.26:50010 to
> 172.19.105.113:50010 using proxy source 172.19.105.102:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
> 1073936074 with a length of 128 MB bytes from 172.19.105.101:50010 to
> 172.19.105.103:50010 using proxy source 172.19.105.101:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
> 1073961733 with a length of 128 MB bytes from 172.19.105.30:50010 to
> 172.19.105.116:50010 using proxy source 172.19.105.101:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
> 1073922224 with a length of 128 MB bytes from 172.19.105.23:50010 to
> 172.19.105.111:50010 using proxy source 172.19.105.102:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
> 1074018599 with a length of 26.12 MB bytes from 172.19.105.27:50010 to
> 172.19.105.108:50010 using proxy source 172.19.105.102:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
> 1074013794 with a length of 128 MB bytes from 172.19.105.26:50010 to
> 172.19.105.113:50010 using proxy source 172.19.105.101:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074013803 from
> 172.19.105.102:50010 to 172.19.105.113:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
> 1074013798 with a length of 128 MB bytes from 172.19.105.26:50010 to
> 172.19.105.113:50010 using proxy source 172.19.105.101:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074013794 from
> 172.19.105.101:50010 to 172.19.105.113:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
> 1074018603 with a length of 128 MB bytes from 172.19.105.27:50010 to
> 172.19.105.108:50010 using proxy source 172.19.105.102:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074018599 from
> 172.19.105.102:50010 to 172.19.105.108:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
> 1074008476 with a length of 128 MB bytes from 172.19.105.25:50010 to
> 172.19.105.106:50010 using proxy source 172.19.105.107:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
> 1073922231 with a length of 128 MB bytes from 172.19.105.23:50010 to
> 172.19.105.111:50010 using proxy source 172.19.105.114:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073922224 from
> 172.19.105.102:50010 to 172.19.105.111:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
> 1073812199 with a length of 64.75 MB bytes from 172.19.105.102:50010 to
> 172.19.105.28:50010 using proxy source 172.19.105.24:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
> 1073991855 with a length of 73.88 MB bytes from 172.19.105.24:50010 to
> 172.19.105.110:50010 using proxy source 172.19.105.102:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
> 1074035591 with a length of 128 MB bytes from 172.19.105.29:50010 to
> 172.19.105.112:50010 using proxy source 172.19.105.101:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
> 1074008479 with a length of 128 MB bytes from 172.19.105.25:50010 to
> 172.19.105.106:50010 using proxy source 172.19.105.107:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074008476 from
> 172.19.105.107:50010 to 172.19.105.106:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
> 1074035589 with a length of 6.51 KB bytes from 172.19.105.29:50010 to
> 172.19.105.112:50010 using proxy source 172.19.105.103:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074035591 from
> 172.19.105.101:50010 to 172.19.105.112:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
> 1073997260 with a length of 85.24 KB bytes from 172.19.105.101:50010 to
> 172.19.105.103:50010 using proxy source 172.19.105.106:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
> 1073992320 with a length of 128 MB bytes from 172.19.105.30:50010 to
> 172.19.105.116:50010 using proxy source 172.19.105.105:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073961733 from
> 172.19.105.101:50010 to 172.19.105.116:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073936074 from
> 172.19.105.101:50010 to 172.19.105.103:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
> 1074009436 with a length of 128 MB bytes from 172.19.105.101:50010 to
> 172.19.105.103:50010 using proxy source 172.19.105.28:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073997260 from
> 172.19.105.106:50010 to 172.19.105.103:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073991855 from
> 172.19.105.102:50010 to 172.19.105.110:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
> 1074009439 with a length of 128 MB bytes from 172.19.105.101:50010 to
> 172.19.105.103:50010 using proxy source 172.19.105.27:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074009436 from
> 172.19.105.28:50010 to 172.19.105.103:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073812199 from
> 172.19.105.24:50010 to 172.19.105.28:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
> 1073883114 with a length of 128 MB bytes from 172.19.105.102:50010 to
> 172.19.105.28:50010 using proxy source 172.19.105.27:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
> 1074018604 with a length of 128 MB bytes from 172.19.105.27:50010 to
> 172.19.105.108:50010 using proxy source 172.19.105.24:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074018603 from
> 172.19.105.102:50010 to 172.19.105.108:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
> 1074013709 with a length of 128 MB bytes from 172.19.105.26:50010 to
> 172.19.105.113:50010 using proxy source 172.19.105.27:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074013798 from
> 172.19.105.101:50010 to 172.19.105.113:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074009439 from
> 172.19.105.27:50010 to 172.19.105.103:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
> 1073987402 with a length of 75.99 MB bytes from 172.19.105.24:50010 to
> 172.19.105.110:50010 using proxy source 172.19.105.116:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
> 1073987410 with a length of 128 MB bytes from 172.19.105.24:50010 to
> 172.19.105.110:50010 using proxy source 172.19.105.24:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073987402 from
> 172.19.105.116:50010 to 172.19.105.110:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
> 1073987422 with a length of 128 MB bytes from 172.19.105.24:50010 to
> 172.19.105.110:50010 using proxy source 172.19.105.30:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
> 1073922228 with a length of 128 MB bytes from 172.19.105.23:50010 to
> 172.19.105.111:50010 using proxy source 172.19.105.23:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073987410 from
> 172.19.105.24:50010 to 172.19.105.110:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073922231 from
> 172.19.105.114:50010 to 172.19.105.111:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
> 1073922442 with a length of 128 MB bytes from 172.19.105.30:50010 to
> 172.19.105.116:50010 using proxy source 172.19.105.26:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073992320 from
> 172.19.105.105:50010 to 172.19.105.116:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
> 1073992340 with a length of 128 MB bytes from 172.19.105.30:50010 to
> 172.19.105.116:50010 using proxy source 172.19.105.111:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073922442 from
> 172.19.105.26:50010 to 172.19.105.116:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073992340 from
> 172.19.105.111:50010 to 172.19.105.116:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
> 1074008469 with a length of 128 MB bytes from 172.19.105.25:50010 to
> 172.19.105.106:50010 using proxy source 172.19.105.107:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074008479 from
> 172.19.105.107:50010 to 172.19.105.106:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
> 1074035711 with a length of 128 MB bytes from 172.19.105.29:50010 to
> 172.19.105.112:50010 using proxy source 172.19.105.29:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074035589 from
> 172.19.105.103:50010 to 172.19.105.112:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
> 1073992638 with a length of 128 MB bytes from 172.19.105.23:50010 to
> 172.19.105.111:50010 using proxy source 172.19.105.23:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073922228 from
> 172.19.105.23:50010 to 172.19.105.111:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
> 1073917025 with a length of 128 MB bytes from 172.19.105.24:50010 to
> 172.19.105.110:50010 using proxy source 172.19.105.26:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
> 1073892376 with a length of 128 MB bytes from 172.19.105.26:50010 to
> 172.19.105.113:50010 using proxy source 172.19.105.106:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073883114 from
> 172.19.105.27:50010 to 172.19.105.28:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
> 1073917544 with a length of 128 MB bytes from 172.19.105.102:50010 to
> 172.19.105.28:50010 using proxy source 172.19.105.108:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
> 1073990086 with a length of 128 MB bytes from 172.19.105.27:50010 to
> 172.19.105.108:50010 using proxy source 172.19.105.27:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073987422 from
> 172.19.105.30:50010 to 172.19.105.110:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074018604 from
> 172.19.105.24:50010 to 172.19.105.108:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074013709 from
> 172.19.105.27:50010 to 172.19.105.113:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073992638 from
> 172.19.105.23:50010 to 172.19.105.111:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
> 1074035705 with a length of 128 MB bytes from 172.19.105.29:50010 to
> 172.19.105.112:50010 using proxy source 172.19.105.23:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074035711 from
> 172.19.105.29:50010 to 172.19.105.112:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074008469 from
> 172.19.105.107:50010 to 172.19.105.106:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073917544 from
> 172.19.105.108:50010 to 172.19.105.28:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073990086 from
> 172.19.105.27:50010 to 172.19.105.108:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
> 1073911911 with a length of 128 MB bytes from 172.19.105.29:50010 to
> 172.19.105.112:50010 using proxy source 172.19.105.29:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073917025 from
> 172.19.105.26:50010 to 172.19.105.110:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073892376 from
> 172.19.105.106:50010 to 172.19.105.113:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074035705 from
> 172.19.105.23:50010 to 172.19.105.112:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073911911 from
> 172.19.105.29:50010 to 172.19.105.112:50010
> 14/02/20 14:38:26 INFO balancer.Balancer: Moving block 1074035589 from
> 172.19.105.29:50010 to 172.19.105.112:50010 through 172.19.105.103:50010is succeeded.
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
> 1074035693 with a length of 128 MB bytes from 172.19.105.29:50010 to
> 172.19.105.112:50010 using proxy source 172.19.105.29:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
> 1074009432 with a length of 128 MB bytes from 172.19.105.101:50010 to
> 172.19.105.103:50010 using proxy source 172.19.105.29:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074035693 from
> 172.19.105.29:50010 to 172.19.105.112:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074009432 from
> 172.19.105.29:50010 to 172.19.105.103:50010
> 14/02/20 14:38:26 INFO balancer.Balancer: Moving block 1073997260 from
> 172.19.105.101:50010 to 172.19.105.103:50010 through 172.19.105.106:50010is succeeded.
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
> 1074009435 with a length of 128 MB bytes from 172.19.105.101:50010 to
> 172.19.105.103:50010 using proxy source 172.19.105.27:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
> 1074008471 with a length of 128 MB bytes from 172.19.105.25:50010 to
> 172.19.105.106:50010 using proxy source 172.19.105.107:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074009435 from
> 172.19.105.27:50010 to 172.19.105.103:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074008471 from
> 172.19.105.107:50010 to 172.19.105.106:50010
> 14/02/20 14:38:32 INFO balancer.Balancer: Moving block 1074018599 from
> 172.19.105.27:50010 to 172.19.105.108:50010 through 172.19.105.102:50010is succeeded.
> 14/02/20 14:38:32 INFO balancer.Balancer: Moving block 1073812199 from
> 172.19.105.102:50010 to 172.19.105.28:50010 through 172.19.105.24:50010is succeeded.
> 14/02/20 14:38:33 INFO balancer.Balancer: Moving block 1073987402 from
> 172.19.105.24:50010 to 172.19.105.110:50010 through 172.19.105.116:50010is succeeded.
> 14/02/20 14:38:37 INFO balancer.Balancer: Moving block 1073922231 from
> 172.19.105.23:50010 to 172.19.105.111:50010 through 172.19.105.114:50010is succeeded.
> 14/02/20 14:38:38 INFO balancer.Balancer: Moving block 1073917544 from
> 172.19.105.102:50010 to 172.19.105.28:50010 through 172.19.105.108:50010is succeeded.
> 14/02/20 14:38:39 INFO balancer.Balancer: Moving block 1073922224 from
> 172.19.105.23:50010 to 172.19.105.111:50010 through 172.19.105.102:50010is succeeded.
> 14/02/20 14:38:39 INFO balancer.Balancer: Moving block 1074013803 from
> 172.19.105.26:50010 to 172.19.105.113:50010 through 172.19.105.102:50010is succeeded.
> 14/02/20 14:38:40 INFO balancer.Balancer: Moving block 1074035591 from
> 172.19.105.29:50010 to 172.19.105.112:50010 through 172.19.105.101:50010is succeeded.
> 14/02/20 14:38:40 INFO balancer.Balancer: Moving block 1073992320 from
> 172.19.105.30:50010 to 172.19.105.116:50010 through 172.19.105.105:50010is succeeded.
> 14/02/20 14:38:41 INFO balancer.Balancer: Moving block 1073961733 from
> 172.19.105.30:50010 to 172.19.105.116:50010 through 172.19.105.101:50010is succeeded.
> 14/02/20 14:38:42 INFO balancer.Balancer: Moving block 1073991855 from
> 172.19.105.24:50010 to 172.19.105.110:50010 through 172.19.105.102:50010is succeeded.
> 14/02/20 14:38:42 INFO balancer.Balancer: Moving block 1074008479 from
> 172.19.105.25:50010 to 172.19.105.106:50010 through 172.19.105.107:50010is succeeded.
> 14/02/20 14:38:43 INFO balancer.Balancer: Moving block 1074009436 from
> 172.19.105.101:50010 to 172.19.105.103:50010 through 172.19.105.28:50010is succeeded.
> 14/02/20 14:38:44 INFO balancer.Balancer: Moving block 1074008469 from
> 172.19.105.25:50010 to 172.19.105.106:50010 through 172.19.105.107:50010is succeeded.
> 14/02/20 14:38:44 INFO balancer.Balancer: Moving block 1073883114 from
> 172.19.105.102:50010 to 172.19.105.28:50010 through 172.19.105.27:50010is succeeded.
> 14/02/20 14:38:44 INFO balancer.Balancer: Moving block 1073911911 from
> 172.19.105.29:50010 to 172.19.105.112:50010 through 172.19.105.29:50010is succeeded.
> 14/02/20 14:38:44 INFO balancer.Balancer: Moving block 1073990086 from
> 172.19.105.27:50010 to 172.19.105.108:50010 through 172.19.105.27:50010is succeeded.
> 14/02/20 14:38:44 INFO balancer.Balancer: Moving block 1074008476 from
> 172.19.105.25:50010 to 172.19.105.106:50010 through 172.19.105.107:50010is succeeded.
> 14/02/20 14:38:45 INFO balancer.Balancer: Moving block 1073987422 from
> 172.19.105.24:50010 to 172.19.105.110:50010 through 172.19.105.30:50010is succeeded.
> 14/02/20 14:38:46 INFO balancer.Balancer: Moving block 1074013794 from
> 172.19.105.26:50010 to 172.19.105.113:50010 through 172.19.105.101:50010is succeeded.
> 14/02/20 14:38:46 INFO balancer.Balancer: Moving block 1074009432 from
> 172.19.105.101:50010 to 172.19.105.103:50010 through 172.19.105.29:50010is succeeded.
> 14/02/20 14:38:46 INFO balancer.Balancer: Moving block 1073936074 from
> 172.19.105.101:50010 to 172.19.105.103:50010 through 172.19.105.101:50010is succeeded.
> 14/02/20 14:38:47 INFO balancer.Balancer: Moving block 1074035711 from
> 172.19.105.29:50010 to 172.19.105.112:50010 through 172.19.105.29:50010is succeeded.
> 14/02/20 14:38:47 INFO balancer.Balancer: Moving block 1074013709 from
> 172.19.105.26:50010 to 172.19.105.113:50010 through 172.19.105.27:50010is succeeded.
> 14/02/20 14:38:47 INFO balancer.Balancer: Moving block 1074008471 from
> 172.19.105.25:50010 to 172.19.105.106:50010 through 172.19.105.107:50010is succeeded.
> 14/02/20 14:38:47 INFO balancer.Balancer: Moving block 1073992340 from
> 172.19.105.30:50010 to 172.19.105.116:50010 through 172.19.105.111:50010is succeeded.
> 14/02/20 14:38:47 INFO balancer.Balancer: Moving block 1074018604 from
> 172.19.105.27:50010 to 172.19.105.108:50010 through 172.19.105.24:50010is succeeded.
> 14/02/20 14:38:47 INFO balancer.Balancer: Moving block 1073917025 from
> 172.19.105.24:50010 to 172.19.105.110:50010 through 172.19.105.26:50010is succeeded.
> 14/02/20 14:38:48 INFO balancer.Balancer: Moving block 1074009439 from
> 172.19.105.101:50010 to 172.19.105.103:50010 through 172.19.105.27:50010is succeeded.
> 14/02/20 14:38:48 INFO balancer.Balancer: Moving block 1074013798 from
> 172.19.105.26:50010 to 172.19.105.113:50010 through 172.19.105.101:50010is succeeded.
> 14/02/20 14:38:48 INFO balancer.Balancer: Moving block 1074018603 from
> 172.19.105.27:50010 to 172.19.105.108:50010 through 172.19.105.102:50010is succeeded.
> 14/02/20 14:38:48 INFO balancer.Balancer: Moving block 1074035693 from
> 172.19.105.29:50010 to 172.19.105.112:50010 through 172.19.105.29:50010is succeeded.
> 14/02/20 14:38:48 INFO balancer.Balancer: Moving block 1073922442 from
> 172.19.105.30:50010 to 172.19.105.116:50010 through 172.19.105.26:50010is succeeded.
> 14/02/20 14:38:49 INFO balancer.Balancer: Moving block 1073892376 from
> 172.19.105.26:50010 to 172.19.105.113:50010 through 172.19.105.106:50010is succeeded.
> 14/02/20 14:38:50 INFO balancer.Balancer: Moving block 1073987410 from
> 172.19.105.24:50010 to 172.19.105.110:50010 through 172.19.105.24:50010is succeeded.
> 14/02/20 14:38:50 INFO balancer.Balancer: Moving block 1073922228 from
> 172.19.105.23:50010 to 172.19.105.111:50010 through 172.19.105.23:50010is succeeded.
> 14/02/20 14:38:50 INFO balancer.Balancer: Moving block 1073992638 from
> 172.19.105.23:50010 to 172.19.105.111:50010 through 172.19.105.23:50010is succeeded.
> 14/02/20 14:38:50 INFO balancer.Balancer: Moving block 1074009435 from
> 172.19.105.101:50010 to 172.19.105.103:50010 through 172.19.105.27:50010is succeeded.
> 14/02/20 14:38:52 INFO balancer.Balancer: Moving block 1074035705 from
> 172.19.105.29:50010 to 172.19.105.112:50010 through 172.19.105.23:50010is succeeded.

Re: A question about Balancer in HDFS

Posted by praveenesh kumar <pr...@gmail.com>.
Have you noticed any balancing in your datanodes by looking at the namenode
url (where it shows data-node level usage). You should have.
I think its not stuck, it just waiting for looking for more data blocks
which it can move around to balance the cluster.
There is threshold limit (if I remember it correctly), which you can also
set while running the balancer (otherwise it uses some default value) and
keeps on running till it reaches that threshold. I think it would be safe
to just stop the process, as it has done balancing.

Sorry I don't have the exact answer, just sharing my experiences. Someone
else might throw better light in here.

Regards
Prav


On Thu, Feb 20, 2014 at 7:09 AM, QShu <sh...@gmail.com> wrote:

> Hi, Guys:
>
>         The HDFS Balancer will stuck after running a while in my cluster
> now, the following is the log, after 14:38:52, the Balancer stuccoed and
> never continue balancing.
>         Can anybody give me some help, thank you!
>
>
> ==============================================log==============================================
> 14/02/20 14:38:25 INFO balancer.Balancer: namenodes = [hdfs://hstore]
> 14/02/20 14:38:25 INFO balancer.Balancer: p         =
> Balancer.Parameters[BalancingPolicy.Node, threshold=10.0]
> Time Stamp               Iteration#  Bytes Already Moved  Bytes Left To
> Move  Bytes Being Moved
> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/
> 172.19.105.111:50010
> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/
> 172.19.105.103:50010
> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/
> 172.19.105.108:50010
> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/
> 172.19.105.102:50010
> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack1/
> 172.19.105.23:50010
> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/
> 172.19.105.112:50010
> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/
> 172.19.105.106:50010
> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/
> 172.19.105.101:50010
> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack1/
> 172.19.105.28:50010
> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/
> 172.19.105.113:50010
> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/
> 172.19.105.110:50010
> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack1/
> 172.19.105.27:50010
> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/
> 172.19.105.116:50010
> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/
> 172.19.105.105:50010
> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack1/
> 172.19.105.29:50010
> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack1/
> 172.19.105.25:50010
> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack1/
> 172.19.105.26:50010
> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack1/
> 172.19.105.24:50010
> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/
> 172.19.105.114:50010
> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/
> 172.19.105.107:50010
> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/
> 172.19.105.109:50010
> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack1/
> 172.19.105.30:50010
> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/
> 172.19.105.115:50010
> 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/
> 172.19.105.104:50010
> 14/02/20 14:38:26 INFO balancer.Balancer: 9 over-utilized: [Source[
> 172.19.105.102:50010, utilization=42.58400812492816], Source[
> 172.19.105.23:50010, utilization=41.94449751804609], Source[
> 172.19.105.101:50010, utilization=49.25646255747681], Source[
> 172.19.105.27:50010, utilization=42.13637544454659], Source[
> 172.19.105.29:50010, utilization=42.253685824453484], Source[
> 172.19.105.25:50010, utilization=41.29288653593026], Source[
> 172.19.105.26:50010, utilization=42.58689306059978], Source[
> 172.19.105.24:50010, utilization=42.681174402528306], Source[
> 172.19.105.30:50010, utilization=42.111934753109566]]
> 14/02/20 14:38:26 INFO balancer.Balancer: 1 underutilized:
> [BalancerDatanode[172.19.105.28:50010, utilization=5.073501546034823]]
> 14/02/20 14:38:26 INFO balancer.Balancer: Need to move 1.23 TB to make the
> cluster balanced.
> 14/02/20 14:38:26 INFO balancer.Balancer: Decided to move 10 GB bytes from
> 172.19.105.102:50010 to 172.19.105.28:50010
> 14/02/20 14:38:26 INFO balancer.Balancer: Decided to move 10 GB bytes from
> 172.19.105.23:50010 to 172.19.105.111:50010
> 14/02/20 14:38:26 INFO balancer.Balancer: Decided to move 10 GB bytes from
> 172.19.105.101:50010 to 172.19.105.103:50010
> 14/02/20 14:38:26 INFO balancer.Balancer: Decided to move 10 GB bytes from
> 172.19.105.27:50010 to 172.19.105.108:50010
> 14/02/20 14:38:26 INFO balancer.Balancer: Decided to move 10 GB bytes from
> 172.19.105.29:50010 to 172.19.105.112:50010
> 14/02/20 14:38:26 INFO balancer.Balancer: Decided to move 10 GB bytes from
> 172.19.105.25:50010 to 172.19.105.106:50010
> 14/02/20 14:38:26 INFO balancer.Balancer: Decided to move 10 GB bytes from
> 172.19.105.26:50010 to 172.19.105.113:50010
> 14/02/20 14:38:26 INFO balancer.Balancer: Decided to move 10 GB bytes from
> 172.19.105.24:50010 to 172.19.105.110:50010
> 14/02/20 14:38:26 INFO balancer.Balancer: Decided to move 10 GB bytes from
> 172.19.105.30:50010 to 172.19.105.116:50010
> 14/02/20 14:38:26 INFO balancer.Balancer: Will move 90 GB in this iteration
> 2014-2-20 14:38:26                0                  0 B             1.23
> TB              90 GB
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
> 1074013803 with a length of 128 MB bytes from 172.19.105.26:50010 to
> 172.19.105.113:50010 using proxy source 172.19.105.102:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
> 1073936074 with a length of 128 MB bytes from 172.19.105.101:50010 to
> 172.19.105.103:50010 using proxy source 172.19.105.101:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
> 1073961733 with a length of 128 MB bytes from 172.19.105.30:50010 to
> 172.19.105.116:50010 using proxy source 172.19.105.101:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
> 1073922224 with a length of 128 MB bytes from 172.19.105.23:50010 to
> 172.19.105.111:50010 using proxy source 172.19.105.102:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
> 1074018599 with a length of 26.12 MB bytes from 172.19.105.27:50010 to
> 172.19.105.108:50010 using proxy source 172.19.105.102:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
> 1074013794 with a length of 128 MB bytes from 172.19.105.26:50010 to
> 172.19.105.113:50010 using proxy source 172.19.105.101:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074013803 from
> 172.19.105.102:50010 to 172.19.105.113:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
> 1074013798 with a length of 128 MB bytes from 172.19.105.26:50010 to
> 172.19.105.113:50010 using proxy source 172.19.105.101:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074013794 from
> 172.19.105.101:50010 to 172.19.105.113:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
> 1074018603 with a length of 128 MB bytes from 172.19.105.27:50010 to
> 172.19.105.108:50010 using proxy source 172.19.105.102:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074018599 from
> 172.19.105.102:50010 to 172.19.105.108:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
> 1074008476 with a length of 128 MB bytes from 172.19.105.25:50010 to
> 172.19.105.106:50010 using proxy source 172.19.105.107:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
> 1073922231 with a length of 128 MB bytes from 172.19.105.23:50010 to
> 172.19.105.111:50010 using proxy source 172.19.105.114:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073922224 from
> 172.19.105.102:50010 to 172.19.105.111:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
> 1073812199 with a length of 64.75 MB bytes from 172.19.105.102:50010 to
> 172.19.105.28:50010 using proxy source 172.19.105.24:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
> 1073991855 with a length of 73.88 MB bytes from 172.19.105.24:50010 to
> 172.19.105.110:50010 using proxy source 172.19.105.102:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
> 1074035591 with a length of 128 MB bytes from 172.19.105.29:50010 to
> 172.19.105.112:50010 using proxy source 172.19.105.101:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
> 1074008479 with a length of 128 MB bytes from 172.19.105.25:50010 to
> 172.19.105.106:50010 using proxy source 172.19.105.107:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074008476 from
> 172.19.105.107:50010 to 172.19.105.106:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
> 1074035589 with a length of 6.51 KB bytes from 172.19.105.29:50010 to
> 172.19.105.112:50010 using proxy source 172.19.105.103:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074035591 from
> 172.19.105.101:50010 to 172.19.105.112:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
> 1073997260 with a length of 85.24 KB bytes from 172.19.105.101:50010 to
> 172.19.105.103:50010 using proxy source 172.19.105.106:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
> 1073992320 with a length of 128 MB bytes from 172.19.105.30:50010 to
> 172.19.105.116:50010 using proxy source 172.19.105.105:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073961733 from
> 172.19.105.101:50010 to 172.19.105.116:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073936074 from
> 172.19.105.101:50010 to 172.19.105.103:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
> 1074009436 with a length of 128 MB bytes from 172.19.105.101:50010 to
> 172.19.105.103:50010 using proxy source 172.19.105.28:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073997260 from
> 172.19.105.106:50010 to 172.19.105.103:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073991855 from
> 172.19.105.102:50010 to 172.19.105.110:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
> 1074009439 with a length of 128 MB bytes from 172.19.105.101:50010 to
> 172.19.105.103:50010 using proxy source 172.19.105.27:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074009436 from
> 172.19.105.28:50010 to 172.19.105.103:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073812199 from
> 172.19.105.24:50010 to 172.19.105.28:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
> 1073883114 with a length of 128 MB bytes from 172.19.105.102:50010 to
> 172.19.105.28:50010 using proxy source 172.19.105.27:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
> 1074018604 with a length of 128 MB bytes from 172.19.105.27:50010 to
> 172.19.105.108:50010 using proxy source 172.19.105.24:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074018603 from
> 172.19.105.102:50010 to 172.19.105.108:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
> 1074013709 with a length of 128 MB bytes from 172.19.105.26:50010 to
> 172.19.105.113:50010 using proxy source 172.19.105.27:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074013798 from
> 172.19.105.101:50010 to 172.19.105.113:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074009439 from
> 172.19.105.27:50010 to 172.19.105.103:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
> 1073987402 with a length of 75.99 MB bytes from 172.19.105.24:50010 to
> 172.19.105.110:50010 using proxy source 172.19.105.116:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
> 1073987410 with a length of 128 MB bytes from 172.19.105.24:50010 to
> 172.19.105.110:50010 using proxy source 172.19.105.24:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073987402 from
> 172.19.105.116:50010 to 172.19.105.110:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
> 1073987422 with a length of 128 MB bytes from 172.19.105.24:50010 to
> 172.19.105.110:50010 using proxy source 172.19.105.30:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
> 1073922228 with a length of 128 MB bytes from 172.19.105.23:50010 to
> 172.19.105.111:50010 using proxy source 172.19.105.23:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073987410 from
> 172.19.105.24:50010 to 172.19.105.110:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073922231 from
> 172.19.105.114:50010 to 172.19.105.111:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
> 1073922442 with a length of 128 MB bytes from 172.19.105.30:50010 to
> 172.19.105.116:50010 using proxy source 172.19.105.26:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073992320 from
> 172.19.105.105:50010 to 172.19.105.116:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
> 1073992340 with a length of 128 MB bytes from 172.19.105.30:50010 to
> 172.19.105.116:50010 using proxy source 172.19.105.111:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073922442 from
> 172.19.105.26:50010 to 172.19.105.116:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073992340 from
> 172.19.105.111:50010 to 172.19.105.116:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
> 1074008469 with a length of 128 MB bytes from 172.19.105.25:50010 to
> 172.19.105.106:50010 using proxy source 172.19.105.107:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074008479 from
> 172.19.105.107:50010 to 172.19.105.106:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
> 1074035711 with a length of 128 MB bytes from 172.19.105.29:50010 to
> 172.19.105.112:50010 using proxy source 172.19.105.29:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074035589 from
> 172.19.105.103:50010 to 172.19.105.112:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
> 1073992638 with a length of 128 MB bytes from 172.19.105.23:50010 to
> 172.19.105.111:50010 using proxy source 172.19.105.23:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073922228 from
> 172.19.105.23:50010 to 172.19.105.111:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
> 1073917025 with a length of 128 MB bytes from 172.19.105.24:50010 to
> 172.19.105.110:50010 using proxy source 172.19.105.26:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
> 1073892376 with a length of 128 MB bytes from 172.19.105.26:50010 to
> 172.19.105.113:50010 using proxy source 172.19.105.106:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073883114 from
> 172.19.105.27:50010 to 172.19.105.28:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
> 1073917544 with a length of 128 MB bytes from 172.19.105.102:50010 to
> 172.19.105.28:50010 using proxy source 172.19.105.108:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
> 1073990086 with a length of 128 MB bytes from 172.19.105.27:50010 to
> 172.19.105.108:50010 using proxy source 172.19.105.27:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073987422 from
> 172.19.105.30:50010 to 172.19.105.110:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074018604 from
> 172.19.105.24:50010 to 172.19.105.108:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074013709 from
> 172.19.105.27:50010 to 172.19.105.113:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073992638 from
> 172.19.105.23:50010 to 172.19.105.111:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
> 1074035705 with a length of 128 MB bytes from 172.19.105.29:50010 to
> 172.19.105.112:50010 using proxy source 172.19.105.23:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074035711 from
> 172.19.105.29:50010 to 172.19.105.112:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074008469 from
> 172.19.105.107:50010 to 172.19.105.106:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073917544 from
> 172.19.105.108:50010 to 172.19.105.28:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073990086 from
> 172.19.105.27:50010 to 172.19.105.108:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
> 1073911911 with a length of 128 MB bytes from 172.19.105.29:50010 to
> 172.19.105.112:50010 using proxy source 172.19.105.29:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073917025 from
> 172.19.105.26:50010 to 172.19.105.110:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073892376 from
> 172.19.105.106:50010 to 172.19.105.113:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074035705 from
> 172.19.105.23:50010 to 172.19.105.112:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073911911 from
> 172.19.105.29:50010 to 172.19.105.112:50010
> 14/02/20 14:38:26 INFO balancer.Balancer: Moving block 1074035589 from
> 172.19.105.29:50010 to 172.19.105.112:50010 through 172.19.105.103:50010is succeeded.
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
> 1074035693 with a length of 128 MB bytes from 172.19.105.29:50010 to
> 172.19.105.112:50010 using proxy source 172.19.105.29:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
> 1074009432 with a length of 128 MB bytes from 172.19.105.101:50010 to
> 172.19.105.103:50010 using proxy source 172.19.105.29:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074035693 from
> 172.19.105.29:50010 to 172.19.105.112:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074009432 from
> 172.19.105.29:50010 to 172.19.105.103:50010
> 14/02/20 14:38:26 INFO balancer.Balancer: Moving block 1073997260 from
> 172.19.105.101:50010 to 172.19.105.103:50010 through 172.19.105.106:50010is succeeded.
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
> 1074009435 with a length of 128 MB bytes from 172.19.105.101:50010 to
> 172.19.105.103:50010 using proxy source 172.19.105.27:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block
> 1074008471 with a length of 128 MB bytes from 172.19.105.25:50010 to
> 172.19.105.106:50010 using proxy source 172.19.105.107:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074009435 from
> 172.19.105.27:50010 to 172.19.105.103:50010
> 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074008471 from
> 172.19.105.107:50010 to 172.19.105.106:50010
> 14/02/20 14:38:32 INFO balancer.Balancer: Moving block 1074018599 from
> 172.19.105.27:50010 to 172.19.105.108:50010 through 172.19.105.102:50010is succeeded.
> 14/02/20 14:38:32 INFO balancer.Balancer: Moving block 1073812199 from
> 172.19.105.102:50010 to 172.19.105.28:50010 through 172.19.105.24:50010is succeeded.
> 14/02/20 14:38:33 INFO balancer.Balancer: Moving block 1073987402 from
> 172.19.105.24:50010 to 172.19.105.110:50010 through 172.19.105.116:50010is succeeded.
> 14/02/20 14:38:37 INFO balancer.Balancer: Moving block 1073922231 from
> 172.19.105.23:50010 to 172.19.105.111:50010 through 172.19.105.114:50010is succeeded.
> 14/02/20 14:38:38 INFO balancer.Balancer: Moving block 1073917544 from
> 172.19.105.102:50010 to 172.19.105.28:50010 through 172.19.105.108:50010is succeeded.
> 14/02/20 14:38:39 INFO balancer.Balancer: Moving block 1073922224 from
> 172.19.105.23:50010 to 172.19.105.111:50010 through 172.19.105.102:50010is succeeded.
> 14/02/20 14:38:39 INFO balancer.Balancer: Moving block 1074013803 from
> 172.19.105.26:50010 to 172.19.105.113:50010 through 172.19.105.102:50010is succeeded.
> 14/02/20 14:38:40 INFO balancer.Balancer: Moving block 1074035591 from
> 172.19.105.29:50010 to 172.19.105.112:50010 through 172.19.105.101:50010is succeeded.
> 14/02/20 14:38:40 INFO balancer.Balancer: Moving block 1073992320 from
> 172.19.105.30:50010 to 172.19.105.116:50010 through 172.19.105.105:50010is succeeded.
> 14/02/20 14:38:41 INFO balancer.Balancer: Moving block 1073961733 from
> 172.19.105.30:50010 to 172.19.105.116:50010 through 172.19.105.101:50010is succeeded.
> 14/02/20 14:38:42 INFO balancer.Balancer: Moving block 1073991855 from
> 172.19.105.24:50010 to 172.19.105.110:50010 through 172.19.105.102:50010is succeeded.
> 14/02/20 14:38:42 INFO balancer.Balancer: Moving block 1074008479 from
> 172.19.105.25:50010 to 172.19.105.106:50010 through 172.19.105.107:50010is succeeded.
> 14/02/20 14:38:43 INFO balancer.Balancer: Moving block 1074009436 from
> 172.19.105.101:50010 to 172.19.105.103:50010 through 172.19.105.28:50010is succeeded.
> 14/02/20 14:38:44 INFO balancer.Balancer: Moving block 1074008469 from
> 172.19.105.25:50010 to 172.19.105.106:50010 through 172.19.105.107:50010is succeeded.
> 14/02/20 14:38:44 INFO balancer.Balancer: Moving block 1073883114 from
> 172.19.105.102:50010 to 172.19.105.28:50010 through 172.19.105.27:50010is succeeded.
> 14/02/20 14:38:44 INFO balancer.Balancer: Moving block 1073911911 from
> 172.19.105.29:50010 to 172.19.105.112:50010 through 172.19.105.29:50010is succeeded.
> 14/02/20 14:38:44 INFO balancer.Balancer: Moving block 1073990086 from
> 172.19.105.27:50010 to 172.19.105.108:50010 through 172.19.105.27:50010is succeeded.
> 14/02/20 14:38:44 INFO balancer.Balancer: Moving block 1074008476 from
> 172.19.105.25:50010 to 172.19.105.106:50010 through 172.19.105.107:50010is succeeded.
> 14/02/20 14:38:45 INFO balancer.Balancer: Moving block 1073987422 from
> 172.19.105.24:50010 to 172.19.105.110:50010 through 172.19.105.30:50010is succeeded.
> 14/02/20 14:38:46 INFO balancer.Balancer: Moving block 1074013794 from
> 172.19.105.26:50010 to 172.19.105.113:50010 through 172.19.105.101:50010is succeeded.
> 14/02/20 14:38:46 INFO balancer.Balancer: Moving block 1074009432 from
> 172.19.105.101:50010 to 172.19.105.103:50010 through 172.19.105.29:50010is succeeded.
> 14/02/20 14:38:46 INFO balancer.Balancer: Moving block 1073936074 from
> 172.19.105.101:50010 to 172.19.105.103:50010 through 172.19.105.101:50010is succeeded.
> 14/02/20 14:38:47 INFO balancer.Balancer: Moving block 1074035711 from
> 172.19.105.29:50010 to 172.19.105.112:50010 through 172.19.105.29:50010is succeeded.
> 14/02/20 14:38:47 INFO balancer.Balancer: Moving block 1074013709 from
> 172.19.105.26:50010 to 172.19.105.113:50010 through 172.19.105.27:50010is succeeded.
> 14/02/20 14:38:47 INFO balancer.Balancer: Moving block 1074008471 from
> 172.19.105.25:50010 to 172.19.105.106:50010 through 172.19.105.107:50010is succeeded.
> 14/02/20 14:38:47 INFO balancer.Balancer: Moving block 1073992340 from
> 172.19.105.30:50010 to 172.19.105.116:50010 through 172.19.105.111:50010is succeeded.
> 14/02/20 14:38:47 INFO balancer.Balancer: Moving block 1074018604 from
> 172.19.105.27:50010 to 172.19.105.108:50010 through 172.19.105.24:50010is succeeded.
> 14/02/20 14:38:47 INFO balancer.Balancer: Moving block 1073917025 from
> 172.19.105.24:50010 to 172.19.105.110:50010 through 172.19.105.26:50010is succeeded.
> 14/02/20 14:38:48 INFO balancer.Balancer: Moving block 1074009439 from
> 172.19.105.101:50010 to 172.19.105.103:50010 through 172.19.105.27:50010is succeeded.
> 14/02/20 14:38:48 INFO balancer.Balancer: Moving block 1074013798 from
> 172.19.105.26:50010 to 172.19.105.113:50010 through 172.19.105.101:50010is succeeded.
> 14/02/20 14:38:48 INFO balancer.Balancer: Moving block 1074018603 from
> 172.19.105.27:50010 to 172.19.105.108:50010 through 172.19.105.102:50010is succeeded.
> 14/02/20 14:38:48 INFO balancer.Balancer: Moving block 1074035693 from
> 172.19.105.29:50010 to 172.19.105.112:50010 through 172.19.105.29:50010is succeeded.
> 14/02/20 14:38:48 INFO balancer.Balancer: Moving block 1073922442 from
> 172.19.105.30:50010 to 172.19.105.116:50010 through 172.19.105.26:50010is succeeded.
> 14/02/20 14:38:49 INFO balancer.Balancer: Moving block 1073892376 from
> 172.19.105.26:50010 to 172.19.105.113:50010 through 172.19.105.106:50010is succeeded.
> 14/02/20 14:38:50 INFO balancer.Balancer: Moving block 1073987410 from
> 172.19.105.24:50010 to 172.19.105.110:50010 through 172.19.105.24:50010is succeeded.
> 14/02/20 14:38:50 INFO balancer.Balancer: Moving block 1073922228 from
> 172.19.105.23:50010 to 172.19.105.111:50010 through 172.19.105.23:50010is succeeded.
> 14/02/20 14:38:50 INFO balancer.Balancer: Moving block 1073992638 from
> 172.19.105.23:50010 to 172.19.105.111:50010 through 172.19.105.23:50010is succeeded.
> 14/02/20 14:38:50 INFO balancer.Balancer: Moving block 1074009435 from
> 172.19.105.101:50010 to 172.19.105.103:50010 through 172.19.105.27:50010is succeeded.
> 14/02/20 14:38:52 INFO balancer.Balancer: Moving block 1074035705 from
> 172.19.105.29:50010 to 172.19.105.112:50010 through 172.19.105.23:50010is succeeded.