You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-dev@hadoop.apache.org by "J.Andreina (JIRA)" <ji...@apache.org> on 2012/05/07 06:48:51 UTC
[jira] [Created] (HDFS-3377) While Balancing more than 10 Blocks
are being moved from one DN even though the maximum number of blocks to be
moved in an iterations is hard coded to 5
J.Andreina created HDFS-3377:
--------------------------------
Summary: While Balancing more than 10 Blocks are being moved from one DN even though the maximum number of blocks to be moved in an iterations is hard coded to 5
Key: HDFS-3377
URL: https://issues.apache.org/jira/browse/HDFS-3377
Project: Hadoop HDFS
Issue Type: Bug
Components: balancer
Affects Versions: 2.0.0
Reporter: J.Andreina
Fix For: 2.0.0, 3.0.0
Replication factor= 1,block size is default value
Step 1: Start NN,DN1
Step 2: Pump 5 GB of data.
Step 3: Start DN2 and issue balancer with threshold value 1
In the balancer report and the NN logs displays that more than 8 blocks are being moved from DN1 to DN2 in one iterations But MAX_NUM_CONCURRENT_MOVES in one iterations is hard coded to 5.
Balancer report for 1st iteration:
=================================
{noformat}
HOST-XX-XX-XX-XX:/home/Andreina/NewHadoop2nd/hadoop-2.0.0-SNAPSHOT/bin # ./hdfs balancer -threshold 1
12/05/03 17:31:28 INFO balancer.Balancer: Using a threshold of 1.0
12/05/03 17:31:28 INFO balancer.Balancer: namenodes = [hdfs://HOST-XX-XX-XX-XX:9002]
12/05/03 17:31:28 INFO balancer.Balancer: p = Balancer.Parameters[BalancingPolicy.Node, threshold=1.0]
Time Stamp Iteration# Bytes Already Moved Bytes Left To Move Bytes Being Moved
12/05/03 17:31:30 INFO net.NetworkTopology: Adding a new node: /datacenter1/rack1/YY.YY.YY.YY:50176
12/05/03 17:31:30 INFO net.NetworkTopology: Adding a new node: /datacenter1/rack1/XX.XX.XX.XX:50076
12/05/03 17:31:30 INFO balancer.Balancer: 1 over-utilized: [Source[XX.XX.XX.XX:50076, utilization=5.018416429773605]]
12/05/03 17:31:30 INFO balancer.Balancer: 1 underutilized: [BalancerDatanode[YY.YY.YY.YY:50176, utilization=3.272819804269012E-5]]
12/05/03 17:31:30 INFO balancer.Balancer: Need to move 1.06 GB to make the cluster balanced.
12/05/03 17:31:30 INFO balancer.Balancer: Decided to move 716.13 MB bytes from XX.XX.XX.XX:50076 to YY.YY.YY.YY:50176
12/05/03 17:31:30 INFO balancer.Balancer: Will move 716.13 MB in this iteration
May 3, 2012 5:31:30 PM 0 0 KB 1.06 GB 716.13 MB
12/05/03 17:35:29 INFO balancer.Balancer: Moving block -5275260117334749945 from XX.XX.XX.XX:50076 to YY.YY.YY.YY:50176 through XX.XX.XX.XX:50076 is succeeded.
12/05/03 17:36:31 INFO balancer.Balancer: Moving block -8079758341763366944 from XX.XX.XX.XX:50076 to YY.YY.YY.YY:50176 through XX.XX.XX.XX:50076 is succeeded.
12/05/03 17:37:12 INFO balancer.Balancer: Moving block -7395554712490186313 from XX.XX.XX.XX:50076 to YY.YY.YY.YY:50176 through XX.XX.XX.XX:50076 is succeeded.
12/05/03 17:37:45 INFO balancer.Balancer: Moving block 7805443002654525130 from XX.XX.XX.XX:50076 to YY.YY.YY.YY:50176 through XX.XX.XX.XX:50076 is succeeded.
12/05/03 17:38:15 INFO balancer.Balancer: Moving block 1864290085256894184 from XX.XX.XX.XX:50076 to YY.YY.YY.YY:50176 through XX.XX.XX.XX:50076 is succeeded.
12/05/03 17:40:30 INFO balancer.Balancer: Moving block 23322655230037442 from XX.XX.XX.XX:50076 to YY.YY.YY.YY:50176 through XX.XX.XX.XX:50076 is succeeded.
12/05/03 17:41:24 INFO balancer.Balancer: Moving block -8839566903692469634 from XX.XX.XX.XX:50076 to YY.YY.YY.YY:50176 through XX.XX.XX.XX:50076 is succeeded.
12/05/03 17:43:03 INFO balancer.Balancer: Moving block 7304385435779271887 from XX.XX.XX.XX:50076 to YY.YY.YY.YY:50176 through XX.XX.XX.XX:50076 is succeeded.
12/05/03 17:43:48 INFO balancer.Balancer: Moving block -7242009026552182303 from XX.XX.XX.XX:50076 to YY.YY.YY.YY:50176 through XX.XX.XX.XX:50076 is succeeded.
12/05/03 17:44:06 INFO balancer.Balancer: Moving block -2449309138254106767 from XX.XX.XX.XX:50076 to YY.YY.YY.YY:50176 through XX.XX.XX.XX:50076 is succeeded.
12/05/03 17:44:55 INFO balancer.Balancer: Moving block 500930296233438046 from XX.XX.XX.XX:50076 to YY.YY.YY.YY:50176 through XX.XX.XX.XX:50076 is succeeded.
12/05/03 17:45:04 INFO balancer.Balancer: Moving block 2642725820310610865 from XX.XX.XX.XX:50076 to YY.YY.YY.YY:50176 through XX.XX.XX.XX:50076 is succeeded.{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HDFS-3377) While Balancing more than 10 Blocks
are being moved from one DN even though the maximum number of blocks to be
moved in an iterations is hard coded to 5
Posted by "Ashish Singhi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3377?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ashish Singhi resolved HDFS-3377.
---------------------------------
Resolution: Invalid
> While Balancing more than 10 Blocks are being moved from one DN even though the maximum number of blocks to be moved in an iterations is hard coded to 5
> --------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: HDFS-3377
> URL: https://issues.apache.org/jira/browse/HDFS-3377
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: balancer
> Affects Versions: 2.0.0-alpha
> Reporter: J.Andreina
>
> Replication factor= 1,block size is default value
> Step 1: Start NN,DN1
> Step 2: Pump 5 GB of data.
> Step 3: Start DN2 and issue balancer with threshold value 1
> In the balancer report and the NN logs displays that more than 8 blocks are being moved from DN1 to DN2 in one iterations But MAX_NUM_CONCURRENT_MOVES in one iterations is hard coded to 5.
> Balancer report for 1st iteration:
> =================================
> {noformat}
> HOST-XX-XX-XX-XX:/home/Andreina/NewHadoop2nd/hadoop-2.0.0-SNAPSHOT/bin # ./hdfs balancer -threshold 1
> 12/05/03 17:31:28 INFO balancer.Balancer: Using a threshold of 1.0
> 12/05/03 17:31:28 INFO balancer.Balancer: namenodes = [hdfs://HOST-XX-XX-XX-XX:9002]
> 12/05/03 17:31:28 INFO balancer.Balancer: p = Balancer.Parameters[BalancingPolicy.Node, threshold=1.0]
> Time Stamp Iteration# Bytes Already Moved Bytes Left To Move Bytes Being Moved
> 12/05/03 17:31:30 INFO net.NetworkTopology: Adding a new node: /datacenter1/rack1/YY.YY.YY.YY:50176
> 12/05/03 17:31:30 INFO net.NetworkTopology: Adding a new node: /datacenter1/rack1/XX.XX.XX.XX:50076
> 12/05/03 17:31:30 INFO balancer.Balancer: 1 over-utilized: [Source[XX.XX.XX.XX:50076, utilization=5.018416429773605]]
> 12/05/03 17:31:30 INFO balancer.Balancer: 1 underutilized: [BalancerDatanode[YY.YY.YY.YY:50176, utilization=3.272819804269012E-5]]
> 12/05/03 17:31:30 INFO balancer.Balancer: Need to move 1.06 GB to make the cluster balanced.
> 12/05/03 17:31:30 INFO balancer.Balancer: Decided to move 716.13 MB bytes from XX.XX.XX.XX:50076 to YY.YY.YY.YY:50176
> 12/05/03 17:31:30 INFO balancer.Balancer: Will move 716.13 MB in this iteration
> May 3, 2012 5:31:30 PM 0 0 KB 1.06 GB 716.13 MB
> 12/05/03 17:35:29 INFO balancer.Balancer: Moving block -5275260117334749945 from XX.XX.XX.XX:50076 to YY.YY.YY.YY:50176 through XX.XX.XX.XX:50076 is succeeded.
> 12/05/03 17:36:31 INFO balancer.Balancer: Moving block -8079758341763366944 from XX.XX.XX.XX:50076 to YY.YY.YY.YY:50176 through XX.XX.XX.XX:50076 is succeeded.
> 12/05/03 17:37:12 INFO balancer.Balancer: Moving block -7395554712490186313 from XX.XX.XX.XX:50076 to YY.YY.YY.YY:50176 through XX.XX.XX.XX:50076 is succeeded.
> 12/05/03 17:37:45 INFO balancer.Balancer: Moving block 7805443002654525130 from XX.XX.XX.XX:50076 to YY.YY.YY.YY:50176 through XX.XX.XX.XX:50076 is succeeded.
> 12/05/03 17:38:15 INFO balancer.Balancer: Moving block 1864290085256894184 from XX.XX.XX.XX:50076 to YY.YY.YY.YY:50176 through XX.XX.XX.XX:50076 is succeeded.
> 12/05/03 17:40:30 INFO balancer.Balancer: Moving block 23322655230037442 from XX.XX.XX.XX:50076 to YY.YY.YY.YY:50176 through XX.XX.XX.XX:50076 is succeeded.
> 12/05/03 17:41:24 INFO balancer.Balancer: Moving block -8839566903692469634 from XX.XX.XX.XX:50076 to YY.YY.YY.YY:50176 through XX.XX.XX.XX:50076 is succeeded.
> 12/05/03 17:43:03 INFO balancer.Balancer: Moving block 7304385435779271887 from XX.XX.XX.XX:50076 to YY.YY.YY.YY:50176 through XX.XX.XX.XX:50076 is succeeded.
> 12/05/03 17:43:48 INFO balancer.Balancer: Moving block -7242009026552182303 from XX.XX.XX.XX:50076 to YY.YY.YY.YY:50176 through XX.XX.XX.XX:50076 is succeeded.
> 12/05/03 17:44:06 INFO balancer.Balancer: Moving block -2449309138254106767 from XX.XX.XX.XX:50076 to YY.YY.YY.YY:50176 through XX.XX.XX.XX:50076 is succeeded.
> 12/05/03 17:44:55 INFO balancer.Balancer: Moving block 500930296233438046 from XX.XX.XX.XX:50076 to YY.YY.YY.YY:50176 through XX.XX.XX.XX:50076 is succeeded.
> 12/05/03 17:45:04 INFO balancer.Balancer: Moving block 2642725820310610865 from XX.XX.XX.XX:50076 to YY.YY.YY.YY:50176 through XX.XX.XX.XX:50076 is succeeded.{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira