You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "Jieshan Bean (JIRA)" <ji...@apache.org> on 2011/06/14 02:50:47 UTC
[jira] [Created] (HBASE-3985) Same Region could be picked out twice
in LoadBalancer
Same Region could be picked out twice in LoadBalancer
-----------------------------------------------------
Key: HBASE-3985
URL: https://issues.apache.org/jira/browse/HBASE-3985
Project: HBase
Issue Type: Bug
Components: master
Affects Versions: 0.90.3
Reporter: Jieshan Bean
Assignee: Jieshan Bean
Fix For: 0.90.4
>From the HMaster logs, I found something weird:
2011-05-24 11:12:11,152 INFO org.apache.hadoop.hbase.master.HMaster: balance hri=hello,122130,1305944329350.7d6c96428e2563c3d8676474d0a9f814., src=158-1-101-202,20020,1306205409671, dest=158-1-101-222,20020,1306205940117
2011-05-24 11:12:31,536 INFO org.apache.hadoop.hbase.master.HMaster: balance hri=hello,122130,1305944329350.7d6c96428e2563c3d8676474d0a9f814., src=158-1-101-202,20020,1306205409671, dest=158-1-101-222,20020,1306205940117
We can see that, the same region was balanced twice.
To describe the problem, I give out one simple example:
1. Suppose regions count is 10 in RegionServer A.
Max: 5 Min:4
2. So the regions count need to move is: 5.
3. Before the movement of calculate, the list was shuffled.
4. The 5 moving region was picked out from the back.
5. The nextRegionForUnload value is 5.
6. So if the neededRegions is not zero. Maybe there's still one region should be picked out from RegionServer A.
This time , the picked Index is 5 which has been picked once!!!!!
|<-----5-------|
------------*--*--*--*--*--*--*--*--*--*----
|
getNextRegionForUnload
Here's the analysis from code:
1. Walk down most loaded, pruning each to the max. Picked region from back of the list(by reverse order)
Map<HServerInfo,BalanceInfo> serverBalanceInfo =
new TreeMap<HServerInfo,BalanceInfo>();
for(Map.Entry<HServerInfo, List<HRegionInfo>> server :
serversByLoad.descendingMap().entrySet()) {
HServerInfo serverInfo = server.getKey();
int regionCount = serverInfo.getLoad().getNumberOfRegions();
if(regionCount <= max) {
serverBalanceInfo.put(serverInfo, new BalanceInfo(0, 0));
break;
}
serversOverloaded++;
List<HRegionInfo> regions = randomize(server.getValue());
int numToOffload = Math.min(regionCount - max, regions.size());
int numTaken = 0;
for (int i = regions.size() - 1; i >= 0; i--) {
HRegionInfo hri = regions.get(i);
// Don't rebalance meta regions.
if (hri.isMetaRegion()) continue;
regionsToMove.add(new RegionPlan(hri, serverInfo, null));
numTaken++;
if (numTaken >= numToOffload) break;
}
/**********************************************************/
/***set the nextRegionForUnload value is numToOffload ****/
/**********************************************************/
serverBalanceInfo.put(serverInfo,
new BalanceInfo(numToOffload, (-1)*numTaken));
}
2. The second pass of picked one region from the Max regionserver by order.
if (neededRegions != 0) {
// Walk down most loaded, grabbing one from each until we get enough
for(Map.Entry<HServerInfo, List<HRegionInfo>> server :
serversByLoad.descendingMap().entrySet()) {
BalanceInfo balanceInfo = serverBalanceInfo.get(server.getKey());
int idx =
balanceInfo == null ? 0 : balanceInfo.getNextRegionForUnload();
if (idx >= server.getValue().size()) break;
HRegionInfo region = server.getValue().get(idx);
if (region.isMetaRegion()) continue; // Don't move meta regions.
regionsToMove.add(new RegionPlan(region, server.getKey(), null));
if(--neededRegions == 0) {
// No more regions needed, done shedding
break;
}
}
}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3985) Same Region could be picked out
twice in LoadBalancer
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13049300#comment-13049300 ]
stack commented on HBASE-3985:
------------------------------
Jieshan: Patch looks good. Minor things are that I'd make the new balance info DEBUG rather than INFO level and I'd test for length >= 2 before I went about subtracting 2 (I can make these changes on commit). In general more info, if 'quality', is always better.
One question. Does it work? This seems like a pretty big change:
{code}
- for (int i = regions.size() - 1; i >= 0; i--) {
+
+ for (int i = 0; i < regions.size(); i++) {
{code}
This method is in bad need of rework. Its way too long and should be standalone testable; currently it is not. Making it so is not in scope w/ this fix but before commit, unless you are going to provide a test, I need evidence this fix does not break existing balancing.
> Same Region could be picked out twice in LoadBalancer
> -----------------------------------------------------
>
> Key: HBASE-3985
> URL: https://issues.apache.org/jira/browse/HBASE-3985
> Project: HBase
> Issue Type: Bug
> Components: master
> Affects Versions: 0.90.3
> Reporter: Jieshan Bean
> Assignee: Jieshan Bean
> Fix For: 0.90.4
>
> Attachments: HBASE-3985-LoadBalancer.patch, HBASE-3985-LoadBalancer_(NoneServerLog).patch
>
>
> From the HMaster logs, I found something weird:
> 2011-05-24 11:12:11,152 INFO org.apache.hadoop.hbase.master.HMaster: balance hri=hello,122130,1305944329350.7d6c96428e2563c3d8676474d0a9f814., src=158-1-101-202,20020,1306205409671, dest=158-1-101-222,20020,1306205940117
> 2011-05-24 11:12:31,536 INFO org.apache.hadoop.hbase.master.HMaster: balance hri=hello,122130,1305944329350.7d6c96428e2563c3d8676474d0a9f814., src=158-1-101-202,20020,1306205409671, dest=158-1-101-222,20020,1306205940117
> We can see that, the same region was balanced twice.
> To describe the problem, I give out one simple example:
> 1. Suppose regions count is 10 in RegionServer A.
> Max: 5 Min:4
> 2. So the regions count need to move is: 5.
> 3. Before the movement of calculate, the list was shuffled.
> 4. The 5 moving region was picked out from the back.
> 5. The nextRegionForUnload value is 5.
> 6. So if the neededRegions is not zero. Maybe there's still one region should be picked out from RegionServer A.
> This time , the picked Index is 5 which has been picked once!!!!!
>
> |<-----5-------|
> ------------*--*--*--*--*--*--*--*--*--*----
> |
> getNextRegionForUnload
> Here's the analysis from code:
> 1. Walk down most loaded, pruning each to the max. Picked region from back of the list(by reverse order)
> Map<HServerInfo,BalanceInfo> serverBalanceInfo =
> new TreeMap<HServerInfo,BalanceInfo>();
> for(Map.Entry<HServerInfo, List<HRegionInfo>> server :
> serversByLoad.descendingMap().entrySet()) {
> HServerInfo serverInfo = server.getKey();
> int regionCount = serverInfo.getLoad().getNumberOfRegions();
> if(regionCount <= max) {
> serverBalanceInfo.put(serverInfo, new BalanceInfo(0, 0));
> break;
> }
> serversOverloaded++;
> List<HRegionInfo> regions = randomize(server.getValue());
> int numToOffload = Math.min(regionCount - max, regions.size());
> int numTaken = 0;
> for (int i = regions.size() - 1; i >= 0; i--) {
> HRegionInfo hri = regions.get(i);
> // Don't rebalance meta regions.
> if (hri.isMetaRegion()) continue;
> regionsToMove.add(new RegionPlan(hri, serverInfo, null));
> numTaken++;
> if (numTaken >= numToOffload) break;
> }
> /**********************************************************/
> /***set the nextRegionForUnload value is numToOffload ****/
> /**********************************************************/
> serverBalanceInfo.put(serverInfo,
> new BalanceInfo(numToOffload, (-1)*numTaken));
> }
> 2. The second pass of picked one region from the Max regionserver by order.
> if (neededRegions != 0) {
> // Walk down most loaded, grabbing one from each until we get enough
> for(Map.Entry<HServerInfo, List<HRegionInfo>> server :
> serversByLoad.descendingMap().entrySet()) {
> BalanceInfo balanceInfo = serverBalanceInfo.get(server.getKey());
> int idx =
> balanceInfo == null ? 0 : balanceInfo.getNextRegionForUnload();
> if (idx >= server.getValue().size()) break;
> HRegionInfo region = server.getValue().get(idx);
> if (region.isMetaRegion()) continue; // Don't move meta regions.
> regionsToMove.add(new RegionPlan(region, server.getKey(), null));
> if(--neededRegions == 0) {
> // No more regions needed, done shedding
> break;
> }
> }
> }
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3985) Same Region could be picked out
twice in LoadBalancer
Posted by "Jieshan Bean (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13049580#comment-13049580 ]
Jieshan Bean commented on HBASE-3985:
-------------------------------------
Thanks stack!
I'll modify the patch as you suggested. And I'll take more test on it. If it possile, I will submit the test result.
But about your comments, I have little doubt, for I can't understand of "in scope w/". Pardon?:(
> Same Region could be picked out twice in LoadBalancer
> -----------------------------------------------------
>
> Key: HBASE-3985
> URL: https://issues.apache.org/jira/browse/HBASE-3985
> Project: HBase
> Issue Type: Bug
> Components: master
> Affects Versions: 0.90.3
> Reporter: Jieshan Bean
> Assignee: Jieshan Bean
> Fix For: 0.90.4
>
> Attachments: HBASE-3985-LoadBalancer.patch, HBASE-3985-LoadBalancer_(NoneServerLog).patch
>
>
> From the HMaster logs, I found something weird:
> 2011-05-24 11:12:11,152 INFO org.apache.hadoop.hbase.master.HMaster: balance hri=hello,122130,1305944329350.7d6c96428e2563c3d8676474d0a9f814., src=158-1-101-202,20020,1306205409671, dest=158-1-101-222,20020,1306205940117
> 2011-05-24 11:12:31,536 INFO org.apache.hadoop.hbase.master.HMaster: balance hri=hello,122130,1305944329350.7d6c96428e2563c3d8676474d0a9f814., src=158-1-101-202,20020,1306205409671, dest=158-1-101-222,20020,1306205940117
> We can see that, the same region was balanced twice.
> To describe the problem, I give out one simple example:
> 1. Suppose regions count is 10 in RegionServer A.
> Max: 5 Min:4
> 2. So the regions count need to move is: 5.
> 3. Before the movement of calculate, the list was shuffled.
> 4. The 5 moving region was picked out from the back.
> 5. The nextRegionForUnload value is 5.
> 6. So if the neededRegions is not zero. Maybe there's still one region should be picked out from RegionServer A.
> This time , the picked Index is 5 which has been picked once!!!!!
>
> |<-----5-------|
> ------------*--*--*--*--*--*--*--*--*--*----
> |
> getNextRegionForUnload
> Here's the analysis from code:
> 1. Walk down most loaded, pruning each to the max. Picked region from back of the list(by reverse order)
> Map<HServerInfo,BalanceInfo> serverBalanceInfo =
> new TreeMap<HServerInfo,BalanceInfo>();
> for(Map.Entry<HServerInfo, List<HRegionInfo>> server :
> serversByLoad.descendingMap().entrySet()) {
> HServerInfo serverInfo = server.getKey();
> int regionCount = serverInfo.getLoad().getNumberOfRegions();
> if(regionCount <= max) {
> serverBalanceInfo.put(serverInfo, new BalanceInfo(0, 0));
> break;
> }
> serversOverloaded++;
> List<HRegionInfo> regions = randomize(server.getValue());
> int numToOffload = Math.min(regionCount - max, regions.size());
> int numTaken = 0;
> for (int i = regions.size() - 1; i >= 0; i--) {
> HRegionInfo hri = regions.get(i);
> // Don't rebalance meta regions.
> if (hri.isMetaRegion()) continue;
> regionsToMove.add(new RegionPlan(hri, serverInfo, null));
> numTaken++;
> if (numTaken >= numToOffload) break;
> }
> /**********************************************************/
> /***set the nextRegionForUnload value is numToOffload ****/
> /**********************************************************/
> serverBalanceInfo.put(serverInfo,
> new BalanceInfo(numToOffload, (-1)*numTaken));
> }
> 2. The second pass of picked one region from the Max regionserver by order.
> if (neededRegions != 0) {
> // Walk down most loaded, grabbing one from each until we get enough
> for(Map.Entry<HServerInfo, List<HRegionInfo>> server :
> serversByLoad.descendingMap().entrySet()) {
> BalanceInfo balanceInfo = serverBalanceInfo.get(server.getKey());
> int idx =
> balanceInfo == null ? 0 : balanceInfo.getNextRegionForUnload();
> if (idx >= server.getValue().size()) break;
> HRegionInfo region = server.getValue().get(idx);
> if (region.isMetaRegion()) continue; // Don't move meta regions.
> regionsToMove.add(new RegionPlan(region, server.getKey(), null));
> if(--neededRegions == 0) {
> // No more regions needed, done shedding
> break;
> }
> }
> }
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3985) Same Region could be picked out
twice in LoadBalancer
Posted by "Ted Yu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13049585#comment-13049585 ]
Ted Yu commented on HBASE-3985:
-------------------------------
Stack meant that making the method shorter is outside the scope of this JIRA.
> Same Region could be picked out twice in LoadBalancer
> -----------------------------------------------------
>
> Key: HBASE-3985
> URL: https://issues.apache.org/jira/browse/HBASE-3985
> Project: HBase
> Issue Type: Bug
> Components: master
> Affects Versions: 0.90.3
> Reporter: Jieshan Bean
> Assignee: Jieshan Bean
> Fix For: 0.90.4
>
> Attachments: HBASE-3985-LoadBalancer.patch, HBASE-3985-LoadBalancer_(NoneServerLog).patch
>
>
> From the HMaster logs, I found something weird:
> 2011-05-24 11:12:11,152 INFO org.apache.hadoop.hbase.master.HMaster: balance hri=hello,122130,1305944329350.7d6c96428e2563c3d8676474d0a9f814., src=158-1-101-202,20020,1306205409671, dest=158-1-101-222,20020,1306205940117
> 2011-05-24 11:12:31,536 INFO org.apache.hadoop.hbase.master.HMaster: balance hri=hello,122130,1305944329350.7d6c96428e2563c3d8676474d0a9f814., src=158-1-101-202,20020,1306205409671, dest=158-1-101-222,20020,1306205940117
> We can see that, the same region was balanced twice.
> To describe the problem, I give out one simple example:
> 1. Suppose regions count is 10 in RegionServer A.
> Max: 5 Min:4
> 2. So the regions count need to move is: 5.
> 3. Before the movement of calculate, the list was shuffled.
> 4. The 5 moving region was picked out from the back.
> 5. The nextRegionForUnload value is 5.
> 6. So if the neededRegions is not zero. Maybe there's still one region should be picked out from RegionServer A.
> This time , the picked Index is 5 which has been picked once!!!!!
>
> |<-----5-------|
> ------------*--*--*--*--*--*--*--*--*--*----
> |
> getNextRegionForUnload
> Here's the analysis from code:
> 1. Walk down most loaded, pruning each to the max. Picked region from back of the list(by reverse order)
> Map<HServerInfo,BalanceInfo> serverBalanceInfo =
> new TreeMap<HServerInfo,BalanceInfo>();
> for(Map.Entry<HServerInfo, List<HRegionInfo>> server :
> serversByLoad.descendingMap().entrySet()) {
> HServerInfo serverInfo = server.getKey();
> int regionCount = serverInfo.getLoad().getNumberOfRegions();
> if(regionCount <= max) {
> serverBalanceInfo.put(serverInfo, new BalanceInfo(0, 0));
> break;
> }
> serversOverloaded++;
> List<HRegionInfo> regions = randomize(server.getValue());
> int numToOffload = Math.min(regionCount - max, regions.size());
> int numTaken = 0;
> for (int i = regions.size() - 1; i >= 0; i--) {
> HRegionInfo hri = regions.get(i);
> // Don't rebalance meta regions.
> if (hri.isMetaRegion()) continue;
> regionsToMove.add(new RegionPlan(hri, serverInfo, null));
> numTaken++;
> if (numTaken >= numToOffload) break;
> }
> /**********************************************************/
> /***set the nextRegionForUnload value is numToOffload ****/
> /**********************************************************/
> serverBalanceInfo.put(serverInfo,
> new BalanceInfo(numToOffload, (-1)*numTaken));
> }
> 2. The second pass of picked one region from the Max regionserver by order.
> if (neededRegions != 0) {
> // Walk down most loaded, grabbing one from each until we get enough
> for(Map.Entry<HServerInfo, List<HRegionInfo>> server :
> serversByLoad.descendingMap().entrySet()) {
> BalanceInfo balanceInfo = serverBalanceInfo.get(server.getKey());
> int idx =
> balanceInfo == null ? 0 : balanceInfo.getNextRegionForUnload();
> if (idx >= server.getValue().size()) break;
> HRegionInfo region = server.getValue().get(idx);
> if (region.isMetaRegion()) continue; // Don't move meta regions.
> regionsToMove.add(new RegionPlan(region, server.getKey(), null));
> if(--neededRegions == 0) {
> // No more regions needed, done shedding
> break;
> }
> }
> }
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3985) Same Region could be picked out
twice in LoadBalancer
Posted by "Jieshan Bean (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13048920#comment-13048920 ]
Jieshan Bean commented on HBASE-3985:
-------------------------------------
The patch will be submitted later:)
> Same Region could be picked out twice in LoadBalancer
> -----------------------------------------------------
>
> Key: HBASE-3985
> URL: https://issues.apache.org/jira/browse/HBASE-3985
> Project: HBase
> Issue Type: Bug
> Components: master
> Affects Versions: 0.90.3
> Reporter: Jieshan Bean
> Assignee: Jieshan Bean
> Fix For: 0.90.4
>
>
> From the HMaster logs, I found something weird:
> 2011-05-24 11:12:11,152 INFO org.apache.hadoop.hbase.master.HMaster: balance hri=hello,122130,1305944329350.7d6c96428e2563c3d8676474d0a9f814., src=158-1-101-202,20020,1306205409671, dest=158-1-101-222,20020,1306205940117
> 2011-05-24 11:12:31,536 INFO org.apache.hadoop.hbase.master.HMaster: balance hri=hello,122130,1305944329350.7d6c96428e2563c3d8676474d0a9f814., src=158-1-101-202,20020,1306205409671, dest=158-1-101-222,20020,1306205940117
> We can see that, the same region was balanced twice.
> To describe the problem, I give out one simple example:
> 1. Suppose regions count is 10 in RegionServer A.
> Max: 5 Min:4
> 2. So the regions count need to move is: 5.
> 3. Before the movement of calculate, the list was shuffled.
> 4. The 5 moving region was picked out from the back.
> 5. The nextRegionForUnload value is 5.
> 6. So if the neededRegions is not zero. Maybe there's still one region should be picked out from RegionServer A.
> This time , the picked Index is 5 which has been picked once!!!!!
>
> |<-----5-------|
> ------------*--*--*--*--*--*--*--*--*--*----
> |
> getNextRegionForUnload
> Here's the analysis from code:
> 1. Walk down most loaded, pruning each to the max. Picked region from back of the list(by reverse order)
> Map<HServerInfo,BalanceInfo> serverBalanceInfo =
> new TreeMap<HServerInfo,BalanceInfo>();
> for(Map.Entry<HServerInfo, List<HRegionInfo>> server :
> serversByLoad.descendingMap().entrySet()) {
> HServerInfo serverInfo = server.getKey();
> int regionCount = serverInfo.getLoad().getNumberOfRegions();
> if(regionCount <= max) {
> serverBalanceInfo.put(serverInfo, new BalanceInfo(0, 0));
> break;
> }
> serversOverloaded++;
> List<HRegionInfo> regions = randomize(server.getValue());
> int numToOffload = Math.min(regionCount - max, regions.size());
> int numTaken = 0;
> for (int i = regions.size() - 1; i >= 0; i--) {
> HRegionInfo hri = regions.get(i);
> // Don't rebalance meta regions.
> if (hri.isMetaRegion()) continue;
> regionsToMove.add(new RegionPlan(hri, serverInfo, null));
> numTaken++;
> if (numTaken >= numToOffload) break;
> }
> /**********************************************************/
> /***set the nextRegionForUnload value is numToOffload ****/
> /**********************************************************/
> serverBalanceInfo.put(serverInfo,
> new BalanceInfo(numToOffload, (-1)*numTaken));
> }
> 2. The second pass of picked one region from the Max regionserver by order.
> if (neededRegions != 0) {
> // Walk down most loaded, grabbing one from each until we get enough
> for(Map.Entry<HServerInfo, List<HRegionInfo>> server :
> serversByLoad.descendingMap().entrySet()) {
> BalanceInfo balanceInfo = serverBalanceInfo.get(server.getKey());
> int idx =
> balanceInfo == null ? 0 : balanceInfo.getNextRegionForUnload();
> if (idx >= server.getValue().size()) break;
> HRegionInfo region = server.getValue().get(idx);
> if (region.isMetaRegion()) continue; // Don't move meta regions.
> regionsToMove.add(new RegionPlan(region, server.getKey(), null));
> if(--neededRegions == 0) {
> // No more regions needed, done shedding
> break;
> }
> }
> }
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3985) Same Region could be picked out
twice in LoadBalancer
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13050675#comment-13050675 ]
stack commented on HBASE-3985:
------------------------------
@Jieshan This is better:
{code}
- for (int i = regions.size() - 1; i >= 0; i--) { |~
- HRegionInfo hri = regions.get(i); |~
+ |~
+ for (HRegionInfo hri: regions) {
{code}
Thank you.
Let me commit. I changed one of the messages to DEBUG.
> Same Region could be picked out twice in LoadBalancer
> -----------------------------------------------------
>
> Key: HBASE-3985
> URL: https://issues.apache.org/jira/browse/HBASE-3985
> Project: HBase
> Issue Type: Bug
> Components: master
> Affects Versions: 0.90.3
> Reporter: Jieshan Bean
> Assignee: Jieshan Bean
> Fix For: 0.90.4
>
> Attachments: AllProjectTestResults.txt, HBASE-2985-LoadBalancer-V2.patch, HBASE-3985-LoadBalancer.patch, HBASE-3985-LoadBalancer_(NoneServerLog).patch, org.apache.hadoop.hbase.master.TestLoadBalancer.txt
>
>
> From the HMaster logs, I found something weird:
> 2011-05-24 11:12:11,152 INFO org.apache.hadoop.hbase.master.HMaster: balance hri=hello,122130,1305944329350.7d6c96428e2563c3d8676474d0a9f814., src=158-1-101-202,20020,1306205409671, dest=158-1-101-222,20020,1306205940117
> 2011-05-24 11:12:31,536 INFO org.apache.hadoop.hbase.master.HMaster: balance hri=hello,122130,1305944329350.7d6c96428e2563c3d8676474d0a9f814., src=158-1-101-202,20020,1306205409671, dest=158-1-101-222,20020,1306205940117
> We can see that, the same region was balanced twice.
> To describe the problem, I give out one simple example:
> 1. Suppose regions count is 10 in RegionServer A.
> Max: 5 Min:4
> 2. So the regions count need to move is: 5.
> 3. Before the movement of calculate, the list was shuffled.
> 4. The 5 moving region was picked out from the back.
> 5. The nextRegionForUnload value is 5.
> 6. So if the neededRegions is not zero. Maybe there's still one region should be picked out from RegionServer A.
> This time , the picked Index is 5 which has been picked once!!!!!
>
> |<-----5-------|
> ------------*--*--*--*--*--*--*--*--*--*----
> |
> getNextRegionForUnload
> Here's the analysis from code:
> 1. Walk down most loaded, pruning each to the max. Picked region from back of the list(by reverse order)
> Map<HServerInfo,BalanceInfo> serverBalanceInfo =
> new TreeMap<HServerInfo,BalanceInfo>();
> for(Map.Entry<HServerInfo, List<HRegionInfo>> server :
> serversByLoad.descendingMap().entrySet()) {
> HServerInfo serverInfo = server.getKey();
> int regionCount = serverInfo.getLoad().getNumberOfRegions();
> if(regionCount <= max) {
> serverBalanceInfo.put(serverInfo, new BalanceInfo(0, 0));
> break;
> }
> serversOverloaded++;
> List<HRegionInfo> regions = randomize(server.getValue());
> int numToOffload = Math.min(regionCount - max, regions.size());
> int numTaken = 0;
> for (int i = regions.size() - 1; i >= 0; i--) {
> HRegionInfo hri = regions.get(i);
> // Don't rebalance meta regions.
> if (hri.isMetaRegion()) continue;
> regionsToMove.add(new RegionPlan(hri, serverInfo, null));
> numTaken++;
> if (numTaken >= numToOffload) break;
> }
> /**********************************************************/
> /***set the nextRegionForUnload value is numToOffload ****/
> /**********************************************************/
> serverBalanceInfo.put(serverInfo,
> new BalanceInfo(numToOffload, (-1)*numTaken));
> }
> 2. The second pass of picked one region from the Max regionserver by order.
> if (neededRegions != 0) {
> // Walk down most loaded, grabbing one from each until we get enough
> for(Map.Entry<HServerInfo, List<HRegionInfo>> server :
> serversByLoad.descendingMap().entrySet()) {
> BalanceInfo balanceInfo = serverBalanceInfo.get(server.getKey());
> int idx =
> balanceInfo == null ? 0 : balanceInfo.getNextRegionForUnload();
> if (idx >= server.getValue().size()) break;
> HRegionInfo region = server.getValue().get(idx);
> if (region.isMetaRegion()) continue; // Don't move meta regions.
> regionsToMove.add(new RegionPlan(region, server.getKey(), null));
> if(--neededRegions == 0) {
> // No more regions needed, done shedding
> break;
> }
> }
> }
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-3985) Same Region could be picked out twice
in LoadBalancer
Posted by "Jieshan Bean (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3985?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jieshan Bean updated HBASE-3985:
--------------------------------
Attachment: AllProjectTestResults.txt
> Same Region could be picked out twice in LoadBalancer
> -----------------------------------------------------
>
> Key: HBASE-3985
> URL: https://issues.apache.org/jira/browse/HBASE-3985
> Project: HBase
> Issue Type: Bug
> Components: master
> Affects Versions: 0.90.3
> Reporter: Jieshan Bean
> Assignee: Jieshan Bean
> Fix For: 0.90.4
>
> Attachments: AllProjectTestResults.txt, HBASE-2985-LoadBalancer-V2.patch, HBASE-3985-LoadBalancer.patch, HBASE-3985-LoadBalancer_(NoneServerLog).patch, org.apache.hadoop.hbase.master.TestLoadBalancer.txt
>
>
> From the HMaster logs, I found something weird:
> 2011-05-24 11:12:11,152 INFO org.apache.hadoop.hbase.master.HMaster: balance hri=hello,122130,1305944329350.7d6c96428e2563c3d8676474d0a9f814., src=158-1-101-202,20020,1306205409671, dest=158-1-101-222,20020,1306205940117
> 2011-05-24 11:12:31,536 INFO org.apache.hadoop.hbase.master.HMaster: balance hri=hello,122130,1305944329350.7d6c96428e2563c3d8676474d0a9f814., src=158-1-101-202,20020,1306205409671, dest=158-1-101-222,20020,1306205940117
> We can see that, the same region was balanced twice.
> To describe the problem, I give out one simple example:
> 1. Suppose regions count is 10 in RegionServer A.
> Max: 5 Min:4
> 2. So the regions count need to move is: 5.
> 3. Before the movement of calculate, the list was shuffled.
> 4. The 5 moving region was picked out from the back.
> 5. The nextRegionForUnload value is 5.
> 6. So if the neededRegions is not zero. Maybe there's still one region should be picked out from RegionServer A.
> This time , the picked Index is 5 which has been picked once!!!!!
>
> |<-----5-------|
> ------------*--*--*--*--*--*--*--*--*--*----
> |
> getNextRegionForUnload
> Here's the analysis from code:
> 1. Walk down most loaded, pruning each to the max. Picked region from back of the list(by reverse order)
> Map<HServerInfo,BalanceInfo> serverBalanceInfo =
> new TreeMap<HServerInfo,BalanceInfo>();
> for(Map.Entry<HServerInfo, List<HRegionInfo>> server :
> serversByLoad.descendingMap().entrySet()) {
> HServerInfo serverInfo = server.getKey();
> int regionCount = serverInfo.getLoad().getNumberOfRegions();
> if(regionCount <= max) {
> serverBalanceInfo.put(serverInfo, new BalanceInfo(0, 0));
> break;
> }
> serversOverloaded++;
> List<HRegionInfo> regions = randomize(server.getValue());
> int numToOffload = Math.min(regionCount - max, regions.size());
> int numTaken = 0;
> for (int i = regions.size() - 1; i >= 0; i--) {
> HRegionInfo hri = regions.get(i);
> // Don't rebalance meta regions.
> if (hri.isMetaRegion()) continue;
> regionsToMove.add(new RegionPlan(hri, serverInfo, null));
> numTaken++;
> if (numTaken >= numToOffload) break;
> }
> /**********************************************************/
> /***set the nextRegionForUnload value is numToOffload ****/
> /**********************************************************/
> serverBalanceInfo.put(serverInfo,
> new BalanceInfo(numToOffload, (-1)*numTaken));
> }
> 2. The second pass of picked one region from the Max regionserver by order.
> if (neededRegions != 0) {
> // Walk down most loaded, grabbing one from each until we get enough
> for(Map.Entry<HServerInfo, List<HRegionInfo>> server :
> serversByLoad.descendingMap().entrySet()) {
> BalanceInfo balanceInfo = serverBalanceInfo.get(server.getKey());
> int idx =
> balanceInfo == null ? 0 : balanceInfo.getNextRegionForUnload();
> if (idx >= server.getValue().size()) break;
> HRegionInfo region = server.getValue().get(idx);
> if (region.isMetaRegion()) continue; // Don't move meta regions.
> regionsToMove.add(new RegionPlan(region, server.getKey(), null));
> if(--neededRegions == 0) {
> // No more regions needed, done shedding
> break;
> }
> }
> }
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3985) Same Region could be picked out
twice in LoadBalancer
Posted by "Jieshan Bean (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13050168#comment-13050168 ]
Jieshan Bean commented on HBASE-3985:
-------------------------------------
I have modified the patch, and take some tests on that(For the added logs, I think it's useful for analysis the balance process, so I keeped them in the patch. But the log level, I have lower it in debug.).
I have submitted the test results.
Any problem and the patch, please tell me if I still have some work to do.
Thanks!
> Same Region could be picked out twice in LoadBalancer
> -----------------------------------------------------
>
> Key: HBASE-3985
> URL: https://issues.apache.org/jira/browse/HBASE-3985
> Project: HBase
> Issue Type: Bug
> Components: master
> Affects Versions: 0.90.3
> Reporter: Jieshan Bean
> Assignee: Jieshan Bean
> Fix For: 0.90.4
>
> Attachments: AllProjectTestResults.txt, HBASE-2985-LoadBalancer-V2.patch, HBASE-3985-LoadBalancer.patch, HBASE-3985-LoadBalancer_(NoneServerLog).patch, org.apache.hadoop.hbase.master.TestLoadBalancer.txt
>
>
> From the HMaster logs, I found something weird:
> 2011-05-24 11:12:11,152 INFO org.apache.hadoop.hbase.master.HMaster: balance hri=hello,122130,1305944329350.7d6c96428e2563c3d8676474d0a9f814., src=158-1-101-202,20020,1306205409671, dest=158-1-101-222,20020,1306205940117
> 2011-05-24 11:12:31,536 INFO org.apache.hadoop.hbase.master.HMaster: balance hri=hello,122130,1305944329350.7d6c96428e2563c3d8676474d0a9f814., src=158-1-101-202,20020,1306205409671, dest=158-1-101-222,20020,1306205940117
> We can see that, the same region was balanced twice.
> To describe the problem, I give out one simple example:
> 1. Suppose regions count is 10 in RegionServer A.
> Max: 5 Min:4
> 2. So the regions count need to move is: 5.
> 3. Before the movement of calculate, the list was shuffled.
> 4. The 5 moving region was picked out from the back.
> 5. The nextRegionForUnload value is 5.
> 6. So if the neededRegions is not zero. Maybe there's still one region should be picked out from RegionServer A.
> This time , the picked Index is 5 which has been picked once!!!!!
>
> |<-----5-------|
> ------------*--*--*--*--*--*--*--*--*--*----
> |
> getNextRegionForUnload
> Here's the analysis from code:
> 1. Walk down most loaded, pruning each to the max. Picked region from back of the list(by reverse order)
> Map<HServerInfo,BalanceInfo> serverBalanceInfo =
> new TreeMap<HServerInfo,BalanceInfo>();
> for(Map.Entry<HServerInfo, List<HRegionInfo>> server :
> serversByLoad.descendingMap().entrySet()) {
> HServerInfo serverInfo = server.getKey();
> int regionCount = serverInfo.getLoad().getNumberOfRegions();
> if(regionCount <= max) {
> serverBalanceInfo.put(serverInfo, new BalanceInfo(0, 0));
> break;
> }
> serversOverloaded++;
> List<HRegionInfo> regions = randomize(server.getValue());
> int numToOffload = Math.min(regionCount - max, regions.size());
> int numTaken = 0;
> for (int i = regions.size() - 1; i >= 0; i--) {
> HRegionInfo hri = regions.get(i);
> // Don't rebalance meta regions.
> if (hri.isMetaRegion()) continue;
> regionsToMove.add(new RegionPlan(hri, serverInfo, null));
> numTaken++;
> if (numTaken >= numToOffload) break;
> }
> /**********************************************************/
> /***set the nextRegionForUnload value is numToOffload ****/
> /**********************************************************/
> serverBalanceInfo.put(serverInfo,
> new BalanceInfo(numToOffload, (-1)*numTaken));
> }
> 2. The second pass of picked one region from the Max regionserver by order.
> if (neededRegions != 0) {
> // Walk down most loaded, grabbing one from each until we get enough
> for(Map.Entry<HServerInfo, List<HRegionInfo>> server :
> serversByLoad.descendingMap().entrySet()) {
> BalanceInfo balanceInfo = serverBalanceInfo.get(server.getKey());
> int idx =
> balanceInfo == null ? 0 : balanceInfo.getNextRegionForUnload();
> if (idx >= server.getValue().size()) break;
> HRegionInfo region = server.getValue().get(idx);
> if (region.isMetaRegion()) continue; // Don't move meta regions.
> regionsToMove.add(new RegionPlan(region, server.getKey(), null));
> if(--neededRegions == 0) {
> // No more regions needed, done shedding
> break;
> }
> }
> }
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HBASE-3985) Same Region could be picked out
twice in LoadBalancer
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3985?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack resolved HBASE-3985.
--------------------------
Resolution: Fixed
Hadoop Flags: [Reviewed]
Committed branch and trunk (though all that went into trunk is the extra logging detail -- it doesn't seem to have this issue)
> Same Region could be picked out twice in LoadBalancer
> -----------------------------------------------------
>
> Key: HBASE-3985
> URL: https://issues.apache.org/jira/browse/HBASE-3985
> Project: HBase
> Issue Type: Bug
> Components: master
> Affects Versions: 0.90.3
> Reporter: Jieshan Bean
> Assignee: Jieshan Bean
> Fix For: 0.90.4
>
> Attachments: AllProjectTestResults.txt, HBASE-2985-LoadBalancer-V2.patch, HBASE-3985-LoadBalancer.patch, HBASE-3985-LoadBalancer_(NoneServerLog).patch, org.apache.hadoop.hbase.master.TestLoadBalancer.txt
>
>
> From the HMaster logs, I found something weird:
> 2011-05-24 11:12:11,152 INFO org.apache.hadoop.hbase.master.HMaster: balance hri=hello,122130,1305944329350.7d6c96428e2563c3d8676474d0a9f814., src=158-1-101-202,20020,1306205409671, dest=158-1-101-222,20020,1306205940117
> 2011-05-24 11:12:31,536 INFO org.apache.hadoop.hbase.master.HMaster: balance hri=hello,122130,1305944329350.7d6c96428e2563c3d8676474d0a9f814., src=158-1-101-202,20020,1306205409671, dest=158-1-101-222,20020,1306205940117
> We can see that, the same region was balanced twice.
> To describe the problem, I give out one simple example:
> 1. Suppose regions count is 10 in RegionServer A.
> Max: 5 Min:4
> 2. So the regions count need to move is: 5.
> 3. Before the movement of calculate, the list was shuffled.
> 4. The 5 moving region was picked out from the back.
> 5. The nextRegionForUnload value is 5.
> 6. So if the neededRegions is not zero. Maybe there's still one region should be picked out from RegionServer A.
> This time , the picked Index is 5 which has been picked once!!!!!
>
> |<-----5-------|
> ------------*--*--*--*--*--*--*--*--*--*----
> |
> getNextRegionForUnload
> Here's the analysis from code:
> 1. Walk down most loaded, pruning each to the max. Picked region from back of the list(by reverse order)
> Map<HServerInfo,BalanceInfo> serverBalanceInfo =
> new TreeMap<HServerInfo,BalanceInfo>();
> for(Map.Entry<HServerInfo, List<HRegionInfo>> server :
> serversByLoad.descendingMap().entrySet()) {
> HServerInfo serverInfo = server.getKey();
> int regionCount = serverInfo.getLoad().getNumberOfRegions();
> if(regionCount <= max) {
> serverBalanceInfo.put(serverInfo, new BalanceInfo(0, 0));
> break;
> }
> serversOverloaded++;
> List<HRegionInfo> regions = randomize(server.getValue());
> int numToOffload = Math.min(regionCount - max, regions.size());
> int numTaken = 0;
> for (int i = regions.size() - 1; i >= 0; i--) {
> HRegionInfo hri = regions.get(i);
> // Don't rebalance meta regions.
> if (hri.isMetaRegion()) continue;
> regionsToMove.add(new RegionPlan(hri, serverInfo, null));
> numTaken++;
> if (numTaken >= numToOffload) break;
> }
> /**********************************************************/
> /***set the nextRegionForUnload value is numToOffload ****/
> /**********************************************************/
> serverBalanceInfo.put(serverInfo,
> new BalanceInfo(numToOffload, (-1)*numTaken));
> }
> 2. The second pass of picked one region from the Max regionserver by order.
> if (neededRegions != 0) {
> // Walk down most loaded, grabbing one from each until we get enough
> for(Map.Entry<HServerInfo, List<HRegionInfo>> server :
> serversByLoad.descendingMap().entrySet()) {
> BalanceInfo balanceInfo = serverBalanceInfo.get(server.getKey());
> int idx =
> balanceInfo == null ? 0 : balanceInfo.getNextRegionForUnload();
> if (idx >= server.getValue().size()) break;
> HRegionInfo region = server.getValue().get(idx);
> if (region.isMetaRegion()) continue; // Don't move meta regions.
> regionsToMove.add(new RegionPlan(region, server.getKey(), null));
> if(--neededRegions == 0) {
> // No more regions needed, done shedding
> break;
> }
> }
> }
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3985) Same Region could be picked out
twice in LoadBalancer
Posted by "Jieshan Bean (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13050834#comment-13050834 ]
Jieshan Bean commented on HBASE-3985:
-------------------------------------
Thanks stack!
Yes, this issue doesn't exist in trunk.
Next time, if the issue related to both 0.90 and trunk, I will remember to make a patch for trunk also.
> Same Region could be picked out twice in LoadBalancer
> -----------------------------------------------------
>
> Key: HBASE-3985
> URL: https://issues.apache.org/jira/browse/HBASE-3985
> Project: HBase
> Issue Type: Bug
> Components: master
> Affects Versions: 0.90.3
> Reporter: Jieshan Bean
> Assignee: Jieshan Bean
> Fix For: 0.90.4
>
> Attachments: AllProjectTestResults.txt, HBASE-2985-LoadBalancer-V2.patch, HBASE-3985-LoadBalancer.patch, HBASE-3985-LoadBalancer_(NoneServerLog).patch, org.apache.hadoop.hbase.master.TestLoadBalancer.txt
>
>
> From the HMaster logs, I found something weird:
> 2011-05-24 11:12:11,152 INFO org.apache.hadoop.hbase.master.HMaster: balance hri=hello,122130,1305944329350.7d6c96428e2563c3d8676474d0a9f814., src=158-1-101-202,20020,1306205409671, dest=158-1-101-222,20020,1306205940117
> 2011-05-24 11:12:31,536 INFO org.apache.hadoop.hbase.master.HMaster: balance hri=hello,122130,1305944329350.7d6c96428e2563c3d8676474d0a9f814., src=158-1-101-202,20020,1306205409671, dest=158-1-101-222,20020,1306205940117
> We can see that, the same region was balanced twice.
> To describe the problem, I give out one simple example:
> 1. Suppose regions count is 10 in RegionServer A.
> Max: 5 Min:4
> 2. So the regions count need to move is: 5.
> 3. Before the movement of calculate, the list was shuffled.
> 4. The 5 moving region was picked out from the back.
> 5. The nextRegionForUnload value is 5.
> 6. So if the neededRegions is not zero. Maybe there's still one region should be picked out from RegionServer A.
> This time , the picked Index is 5 which has been picked once!!!!!
>
> |<-----5-------|
> ------------*--*--*--*--*--*--*--*--*--*----
> |
> getNextRegionForUnload
> Here's the analysis from code:
> 1. Walk down most loaded, pruning each to the max. Picked region from back of the list(by reverse order)
> Map<HServerInfo,BalanceInfo> serverBalanceInfo =
> new TreeMap<HServerInfo,BalanceInfo>();
> for(Map.Entry<HServerInfo, List<HRegionInfo>> server :
> serversByLoad.descendingMap().entrySet()) {
> HServerInfo serverInfo = server.getKey();
> int regionCount = serverInfo.getLoad().getNumberOfRegions();
> if(regionCount <= max) {
> serverBalanceInfo.put(serverInfo, new BalanceInfo(0, 0));
> break;
> }
> serversOverloaded++;
> List<HRegionInfo> regions = randomize(server.getValue());
> int numToOffload = Math.min(regionCount - max, regions.size());
> int numTaken = 0;
> for (int i = regions.size() - 1; i >= 0; i--) {
> HRegionInfo hri = regions.get(i);
> // Don't rebalance meta regions.
> if (hri.isMetaRegion()) continue;
> regionsToMove.add(new RegionPlan(hri, serverInfo, null));
> numTaken++;
> if (numTaken >= numToOffload) break;
> }
> /**********************************************************/
> /***set the nextRegionForUnload value is numToOffload ****/
> /**********************************************************/
> serverBalanceInfo.put(serverInfo,
> new BalanceInfo(numToOffload, (-1)*numTaken));
> }
> 2. The second pass of picked one region from the Max regionserver by order.
> if (neededRegions != 0) {
> // Walk down most loaded, grabbing one from each until we get enough
> for(Map.Entry<HServerInfo, List<HRegionInfo>> server :
> serversByLoad.descendingMap().entrySet()) {
> BalanceInfo balanceInfo = serverBalanceInfo.get(server.getKey());
> int idx =
> balanceInfo == null ? 0 : balanceInfo.getNextRegionForUnload();
> if (idx >= server.getValue().size()) break;
> HRegionInfo region = server.getValue().get(idx);
> if (region.isMetaRegion()) continue; // Don't move meta regions.
> regionsToMove.add(new RegionPlan(region, server.getKey(), null));
> if(--neededRegions == 0) {
> // No more regions needed, done shedding
> break;
> }
> }
> }
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3985) Same Region could be picked out
twice in LoadBalancer
Posted by "Jieshan Bean (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13049589#comment-13049589 ]
Jieshan Bean commented on HBASE-3985:
-------------------------------------
Thanks Ted Yu, now I get it.
> Same Region could be picked out twice in LoadBalancer
> -----------------------------------------------------
>
> Key: HBASE-3985
> URL: https://issues.apache.org/jira/browse/HBASE-3985
> Project: HBase
> Issue Type: Bug
> Components: master
> Affects Versions: 0.90.3
> Reporter: Jieshan Bean
> Assignee: Jieshan Bean
> Fix For: 0.90.4
>
> Attachments: HBASE-3985-LoadBalancer.patch, HBASE-3985-LoadBalancer_(NoneServerLog).patch
>
>
> From the HMaster logs, I found something weird:
> 2011-05-24 11:12:11,152 INFO org.apache.hadoop.hbase.master.HMaster: balance hri=hello,122130,1305944329350.7d6c96428e2563c3d8676474d0a9f814., src=158-1-101-202,20020,1306205409671, dest=158-1-101-222,20020,1306205940117
> 2011-05-24 11:12:31,536 INFO org.apache.hadoop.hbase.master.HMaster: balance hri=hello,122130,1305944329350.7d6c96428e2563c3d8676474d0a9f814., src=158-1-101-202,20020,1306205409671, dest=158-1-101-222,20020,1306205940117
> We can see that, the same region was balanced twice.
> To describe the problem, I give out one simple example:
> 1. Suppose regions count is 10 in RegionServer A.
> Max: 5 Min:4
> 2. So the regions count need to move is: 5.
> 3. Before the movement of calculate, the list was shuffled.
> 4. The 5 moving region was picked out from the back.
> 5. The nextRegionForUnload value is 5.
> 6. So if the neededRegions is not zero. Maybe there's still one region should be picked out from RegionServer A.
> This time , the picked Index is 5 which has been picked once!!!!!
>
> |<-----5-------|
> ------------*--*--*--*--*--*--*--*--*--*----
> |
> getNextRegionForUnload
> Here's the analysis from code:
> 1. Walk down most loaded, pruning each to the max. Picked region from back of the list(by reverse order)
> Map<HServerInfo,BalanceInfo> serverBalanceInfo =
> new TreeMap<HServerInfo,BalanceInfo>();
> for(Map.Entry<HServerInfo, List<HRegionInfo>> server :
> serversByLoad.descendingMap().entrySet()) {
> HServerInfo serverInfo = server.getKey();
> int regionCount = serverInfo.getLoad().getNumberOfRegions();
> if(regionCount <= max) {
> serverBalanceInfo.put(serverInfo, new BalanceInfo(0, 0));
> break;
> }
> serversOverloaded++;
> List<HRegionInfo> regions = randomize(server.getValue());
> int numToOffload = Math.min(regionCount - max, regions.size());
> int numTaken = 0;
> for (int i = regions.size() - 1; i >= 0; i--) {
> HRegionInfo hri = regions.get(i);
> // Don't rebalance meta regions.
> if (hri.isMetaRegion()) continue;
> regionsToMove.add(new RegionPlan(hri, serverInfo, null));
> numTaken++;
> if (numTaken >= numToOffload) break;
> }
> /**********************************************************/
> /***set the nextRegionForUnload value is numToOffload ****/
> /**********************************************************/
> serverBalanceInfo.put(serverInfo,
> new BalanceInfo(numToOffload, (-1)*numTaken));
> }
> 2. The second pass of picked one region from the Max regionserver by order.
> if (neededRegions != 0) {
> // Walk down most loaded, grabbing one from each until we get enough
> for(Map.Entry<HServerInfo, List<HRegionInfo>> server :
> serversByLoad.descendingMap().entrySet()) {
> BalanceInfo balanceInfo = serverBalanceInfo.get(server.getKey());
> int idx =
> balanceInfo == null ? 0 : balanceInfo.getNextRegionForUnload();
> if (idx >= server.getValue().size()) break;
> HRegionInfo region = server.getValue().get(idx);
> if (region.isMetaRegion()) continue; // Don't move meta regions.
> regionsToMove.add(new RegionPlan(region, server.getKey(), null));
> if(--neededRegions == 0) {
> // No more regions needed, done shedding
> break;
> }
> }
> }
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3985) Same Region could be picked out
twice in LoadBalancer
Posted by "Jieshan Bean (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13049074#comment-13049074 ]
Jieshan Bean commented on HBASE-3985:
-------------------------------------
I think, while running LoadBalancer, it's better to log more info about each server. So in the patch, It seems I modified more places.
If it can't be accept, and I also make another patch with less modify of logs.
So there's two patches.
> Same Region could be picked out twice in LoadBalancer
> -----------------------------------------------------
>
> Key: HBASE-3985
> URL: https://issues.apache.org/jira/browse/HBASE-3985
> Project: HBase
> Issue Type: Bug
> Components: master
> Affects Versions: 0.90.3
> Reporter: Jieshan Bean
> Assignee: Jieshan Bean
> Fix For: 0.90.4
>
> Attachments: HBASE-3985-LoadBalancer.patch, HBASE-3985-LoadBalancer_(NoneServerLog).patch
>
>
> From the HMaster logs, I found something weird:
> 2011-05-24 11:12:11,152 INFO org.apache.hadoop.hbase.master.HMaster: balance hri=hello,122130,1305944329350.7d6c96428e2563c3d8676474d0a9f814., src=158-1-101-202,20020,1306205409671, dest=158-1-101-222,20020,1306205940117
> 2011-05-24 11:12:31,536 INFO org.apache.hadoop.hbase.master.HMaster: balance hri=hello,122130,1305944329350.7d6c96428e2563c3d8676474d0a9f814., src=158-1-101-202,20020,1306205409671, dest=158-1-101-222,20020,1306205940117
> We can see that, the same region was balanced twice.
> To describe the problem, I give out one simple example:
> 1. Suppose regions count is 10 in RegionServer A.
> Max: 5 Min:4
> 2. So the regions count need to move is: 5.
> 3. Before the movement of calculate, the list was shuffled.
> 4. The 5 moving region was picked out from the back.
> 5. The nextRegionForUnload value is 5.
> 6. So if the neededRegions is not zero. Maybe there's still one region should be picked out from RegionServer A.
> This time , the picked Index is 5 which has been picked once!!!!!
>
> |<-----5-------|
> ------------*--*--*--*--*--*--*--*--*--*----
> |
> getNextRegionForUnload
> Here's the analysis from code:
> 1. Walk down most loaded, pruning each to the max. Picked region from back of the list(by reverse order)
> Map<HServerInfo,BalanceInfo> serverBalanceInfo =
> new TreeMap<HServerInfo,BalanceInfo>();
> for(Map.Entry<HServerInfo, List<HRegionInfo>> server :
> serversByLoad.descendingMap().entrySet()) {
> HServerInfo serverInfo = server.getKey();
> int regionCount = serverInfo.getLoad().getNumberOfRegions();
> if(regionCount <= max) {
> serverBalanceInfo.put(serverInfo, new BalanceInfo(0, 0));
> break;
> }
> serversOverloaded++;
> List<HRegionInfo> regions = randomize(server.getValue());
> int numToOffload = Math.min(regionCount - max, regions.size());
> int numTaken = 0;
> for (int i = regions.size() - 1; i >= 0; i--) {
> HRegionInfo hri = regions.get(i);
> // Don't rebalance meta regions.
> if (hri.isMetaRegion()) continue;
> regionsToMove.add(new RegionPlan(hri, serverInfo, null));
> numTaken++;
> if (numTaken >= numToOffload) break;
> }
> /**********************************************************/
> /***set the nextRegionForUnload value is numToOffload ****/
> /**********************************************************/
> serverBalanceInfo.put(serverInfo,
> new BalanceInfo(numToOffload, (-1)*numTaken));
> }
> 2. The second pass of picked one region from the Max regionserver by order.
> if (neededRegions != 0) {
> // Walk down most loaded, grabbing one from each until we get enough
> for(Map.Entry<HServerInfo, List<HRegionInfo>> server :
> serversByLoad.descendingMap().entrySet()) {
> BalanceInfo balanceInfo = serverBalanceInfo.get(server.getKey());
> int idx =
> balanceInfo == null ? 0 : balanceInfo.getNextRegionForUnload();
> if (idx >= server.getValue().size()) break;
> HRegionInfo region = server.getValue().get(idx);
> if (region.isMetaRegion()) continue; // Don't move meta regions.
> regionsToMove.add(new RegionPlan(region, server.getKey(), null));
> if(--neededRegions == 0) {
> // No more regions needed, done shedding
> break;
> }
> }
> }
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-3985) Same Region could be picked out twice
in LoadBalancer
Posted by "Jieshan Bean (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3985?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jieshan Bean updated HBASE-3985:
--------------------------------
Attachment: HBASE-3985-LoadBalancer_(NoneServerLog).patch
> Same Region could be picked out twice in LoadBalancer
> -----------------------------------------------------
>
> Key: HBASE-3985
> URL: https://issues.apache.org/jira/browse/HBASE-3985
> Project: HBase
> Issue Type: Bug
> Components: master
> Affects Versions: 0.90.3
> Reporter: Jieshan Bean
> Assignee: Jieshan Bean
> Fix For: 0.90.4
>
> Attachments: HBASE-3985-LoadBalancer.patch, HBASE-3985-LoadBalancer_(NoneServerLog).patch
>
>
> From the HMaster logs, I found something weird:
> 2011-05-24 11:12:11,152 INFO org.apache.hadoop.hbase.master.HMaster: balance hri=hello,122130,1305944329350.7d6c96428e2563c3d8676474d0a9f814., src=158-1-101-202,20020,1306205409671, dest=158-1-101-222,20020,1306205940117
> 2011-05-24 11:12:31,536 INFO org.apache.hadoop.hbase.master.HMaster: balance hri=hello,122130,1305944329350.7d6c96428e2563c3d8676474d0a9f814., src=158-1-101-202,20020,1306205409671, dest=158-1-101-222,20020,1306205940117
> We can see that, the same region was balanced twice.
> To describe the problem, I give out one simple example:
> 1. Suppose regions count is 10 in RegionServer A.
> Max: 5 Min:4
> 2. So the regions count need to move is: 5.
> 3. Before the movement of calculate, the list was shuffled.
> 4. The 5 moving region was picked out from the back.
> 5. The nextRegionForUnload value is 5.
> 6. So if the neededRegions is not zero. Maybe there's still one region should be picked out from RegionServer A.
> This time , the picked Index is 5 which has been picked once!!!!!
>
> |<-----5-------|
> ------------*--*--*--*--*--*--*--*--*--*----
> |
> getNextRegionForUnload
> Here's the analysis from code:
> 1. Walk down most loaded, pruning each to the max. Picked region from back of the list(by reverse order)
> Map<HServerInfo,BalanceInfo> serverBalanceInfo =
> new TreeMap<HServerInfo,BalanceInfo>();
> for(Map.Entry<HServerInfo, List<HRegionInfo>> server :
> serversByLoad.descendingMap().entrySet()) {
> HServerInfo serverInfo = server.getKey();
> int regionCount = serverInfo.getLoad().getNumberOfRegions();
> if(regionCount <= max) {
> serverBalanceInfo.put(serverInfo, new BalanceInfo(0, 0));
> break;
> }
> serversOverloaded++;
> List<HRegionInfo> regions = randomize(server.getValue());
> int numToOffload = Math.min(regionCount - max, regions.size());
> int numTaken = 0;
> for (int i = regions.size() - 1; i >= 0; i--) {
> HRegionInfo hri = regions.get(i);
> // Don't rebalance meta regions.
> if (hri.isMetaRegion()) continue;
> regionsToMove.add(new RegionPlan(hri, serverInfo, null));
> numTaken++;
> if (numTaken >= numToOffload) break;
> }
> /**********************************************************/
> /***set the nextRegionForUnload value is numToOffload ****/
> /**********************************************************/
> serverBalanceInfo.put(serverInfo,
> new BalanceInfo(numToOffload, (-1)*numTaken));
> }
> 2. The second pass of picked one region from the Max regionserver by order.
> if (neededRegions != 0) {
> // Walk down most loaded, grabbing one from each until we get enough
> for(Map.Entry<HServerInfo, List<HRegionInfo>> server :
> serversByLoad.descendingMap().entrySet()) {
> BalanceInfo balanceInfo = serverBalanceInfo.get(server.getKey());
> int idx =
> balanceInfo == null ? 0 : balanceInfo.getNextRegionForUnload();
> if (idx >= server.getValue().size()) break;
> HRegionInfo region = server.getValue().get(idx);
> if (region.isMetaRegion()) continue; // Don't move meta regions.
> regionsToMove.add(new RegionPlan(region, server.getKey(), null));
> if(--neededRegions == 0) {
> // No more regions needed, done shedding
> break;
> }
> }
> }
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-3985) Same Region could be picked out twice
in LoadBalancer
Posted by "Jieshan Bean (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3985?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jieshan Bean updated HBASE-3985:
--------------------------------
Attachment: HBASE-3985-LoadBalancer.patch
> Same Region could be picked out twice in LoadBalancer
> -----------------------------------------------------
>
> Key: HBASE-3985
> URL: https://issues.apache.org/jira/browse/HBASE-3985
> Project: HBase
> Issue Type: Bug
> Components: master
> Affects Versions: 0.90.3
> Reporter: Jieshan Bean
> Assignee: Jieshan Bean
> Fix For: 0.90.4
>
> Attachments: HBASE-3985-LoadBalancer.patch, HBASE-3985-LoadBalancer_(NoneServerLog).patch
>
>
> From the HMaster logs, I found something weird:
> 2011-05-24 11:12:11,152 INFO org.apache.hadoop.hbase.master.HMaster: balance hri=hello,122130,1305944329350.7d6c96428e2563c3d8676474d0a9f814., src=158-1-101-202,20020,1306205409671, dest=158-1-101-222,20020,1306205940117
> 2011-05-24 11:12:31,536 INFO org.apache.hadoop.hbase.master.HMaster: balance hri=hello,122130,1305944329350.7d6c96428e2563c3d8676474d0a9f814., src=158-1-101-202,20020,1306205409671, dest=158-1-101-222,20020,1306205940117
> We can see that, the same region was balanced twice.
> To describe the problem, I give out one simple example:
> 1. Suppose regions count is 10 in RegionServer A.
> Max: 5 Min:4
> 2. So the regions count need to move is: 5.
> 3. Before the movement of calculate, the list was shuffled.
> 4. The 5 moving region was picked out from the back.
> 5. The nextRegionForUnload value is 5.
> 6. So if the neededRegions is not zero. Maybe there's still one region should be picked out from RegionServer A.
> This time , the picked Index is 5 which has been picked once!!!!!
>
> |<-----5-------|
> ------------*--*--*--*--*--*--*--*--*--*----
> |
> getNextRegionForUnload
> Here's the analysis from code:
> 1. Walk down most loaded, pruning each to the max. Picked region from back of the list(by reverse order)
> Map<HServerInfo,BalanceInfo> serverBalanceInfo =
> new TreeMap<HServerInfo,BalanceInfo>();
> for(Map.Entry<HServerInfo, List<HRegionInfo>> server :
> serversByLoad.descendingMap().entrySet()) {
> HServerInfo serverInfo = server.getKey();
> int regionCount = serverInfo.getLoad().getNumberOfRegions();
> if(regionCount <= max) {
> serverBalanceInfo.put(serverInfo, new BalanceInfo(0, 0));
> break;
> }
> serversOverloaded++;
> List<HRegionInfo> regions = randomize(server.getValue());
> int numToOffload = Math.min(regionCount - max, regions.size());
> int numTaken = 0;
> for (int i = regions.size() - 1; i >= 0; i--) {
> HRegionInfo hri = regions.get(i);
> // Don't rebalance meta regions.
> if (hri.isMetaRegion()) continue;
> regionsToMove.add(new RegionPlan(hri, serverInfo, null));
> numTaken++;
> if (numTaken >= numToOffload) break;
> }
> /**********************************************************/
> /***set the nextRegionForUnload value is numToOffload ****/
> /**********************************************************/
> serverBalanceInfo.put(serverInfo,
> new BalanceInfo(numToOffload, (-1)*numTaken));
> }
> 2. The second pass of picked one region from the Max regionserver by order.
> if (neededRegions != 0) {
> // Walk down most loaded, grabbing one from each until we get enough
> for(Map.Entry<HServerInfo, List<HRegionInfo>> server :
> serversByLoad.descendingMap().entrySet()) {
> BalanceInfo balanceInfo = serverBalanceInfo.get(server.getKey());
> int idx =
> balanceInfo == null ? 0 : balanceInfo.getNextRegionForUnload();
> if (idx >= server.getValue().size()) break;
> HRegionInfo region = server.getValue().get(idx);
> if (region.isMetaRegion()) continue; // Don't move meta regions.
> regionsToMove.add(new RegionPlan(region, server.getKey(), null));
> if(--neededRegions == 0) {
> // No more regions needed, done shedding
> break;
> }
> }
> }
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-3985) Same Region could be picked out twice
in LoadBalancer
Posted by "Jieshan Bean (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3985?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jieshan Bean updated HBASE-3985:
--------------------------------
Attachment: org.apache.hadoop.hbase.master.TestLoadBalancer.txt
> Same Region could be picked out twice in LoadBalancer
> -----------------------------------------------------
>
> Key: HBASE-3985
> URL: https://issues.apache.org/jira/browse/HBASE-3985
> Project: HBase
> Issue Type: Bug
> Components: master
> Affects Versions: 0.90.3
> Reporter: Jieshan Bean
> Assignee: Jieshan Bean
> Fix For: 0.90.4
>
> Attachments: AllProjectTestResults.txt, HBASE-2985-LoadBalancer-V2.patch, HBASE-3985-LoadBalancer.patch, HBASE-3985-LoadBalancer_(NoneServerLog).patch, org.apache.hadoop.hbase.master.TestLoadBalancer.txt
>
>
> From the HMaster logs, I found something weird:
> 2011-05-24 11:12:11,152 INFO org.apache.hadoop.hbase.master.HMaster: balance hri=hello,122130,1305944329350.7d6c96428e2563c3d8676474d0a9f814., src=158-1-101-202,20020,1306205409671, dest=158-1-101-222,20020,1306205940117
> 2011-05-24 11:12:31,536 INFO org.apache.hadoop.hbase.master.HMaster: balance hri=hello,122130,1305944329350.7d6c96428e2563c3d8676474d0a9f814., src=158-1-101-202,20020,1306205409671, dest=158-1-101-222,20020,1306205940117
> We can see that, the same region was balanced twice.
> To describe the problem, I give out one simple example:
> 1. Suppose regions count is 10 in RegionServer A.
> Max: 5 Min:4
> 2. So the regions count need to move is: 5.
> 3. Before the movement of calculate, the list was shuffled.
> 4. The 5 moving region was picked out from the back.
> 5. The nextRegionForUnload value is 5.
> 6. So if the neededRegions is not zero. Maybe there's still one region should be picked out from RegionServer A.
> This time , the picked Index is 5 which has been picked once!!!!!
>
> |<-----5-------|
> ------------*--*--*--*--*--*--*--*--*--*----
> |
> getNextRegionForUnload
> Here's the analysis from code:
> 1. Walk down most loaded, pruning each to the max. Picked region from back of the list(by reverse order)
> Map<HServerInfo,BalanceInfo> serverBalanceInfo =
> new TreeMap<HServerInfo,BalanceInfo>();
> for(Map.Entry<HServerInfo, List<HRegionInfo>> server :
> serversByLoad.descendingMap().entrySet()) {
> HServerInfo serverInfo = server.getKey();
> int regionCount = serverInfo.getLoad().getNumberOfRegions();
> if(regionCount <= max) {
> serverBalanceInfo.put(serverInfo, new BalanceInfo(0, 0));
> break;
> }
> serversOverloaded++;
> List<HRegionInfo> regions = randomize(server.getValue());
> int numToOffload = Math.min(regionCount - max, regions.size());
> int numTaken = 0;
> for (int i = regions.size() - 1; i >= 0; i--) {
> HRegionInfo hri = regions.get(i);
> // Don't rebalance meta regions.
> if (hri.isMetaRegion()) continue;
> regionsToMove.add(new RegionPlan(hri, serverInfo, null));
> numTaken++;
> if (numTaken >= numToOffload) break;
> }
> /**********************************************************/
> /***set the nextRegionForUnload value is numToOffload ****/
> /**********************************************************/
> serverBalanceInfo.put(serverInfo,
> new BalanceInfo(numToOffload, (-1)*numTaken));
> }
> 2. The second pass of picked one region from the Max regionserver by order.
> if (neededRegions != 0) {
> // Walk down most loaded, grabbing one from each until we get enough
> for(Map.Entry<HServerInfo, List<HRegionInfo>> server :
> serversByLoad.descendingMap().entrySet()) {
> BalanceInfo balanceInfo = serverBalanceInfo.get(server.getKey());
> int idx =
> balanceInfo == null ? 0 : balanceInfo.getNextRegionForUnload();
> if (idx >= server.getValue().size()) break;
> HRegionInfo region = server.getValue().get(idx);
> if (region.isMetaRegion()) continue; // Don't move meta regions.
> regionsToMove.add(new RegionPlan(region, server.getKey(), null));
> if(--neededRegions == 0) {
> // No more regions needed, done shedding
> break;
> }
> }
> }
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-3985) Same Region could be picked out twice
in LoadBalancer
Posted by "Jieshan Bean (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3985?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jieshan Bean updated HBASE-3985:
--------------------------------
Attachment: HBASE-2985-LoadBalancer-V2.patch
> Same Region could be picked out twice in LoadBalancer
> -----------------------------------------------------
>
> Key: HBASE-3985
> URL: https://issues.apache.org/jira/browse/HBASE-3985
> Project: HBase
> Issue Type: Bug
> Components: master
> Affects Versions: 0.90.3
> Reporter: Jieshan Bean
> Assignee: Jieshan Bean
> Fix For: 0.90.4
>
> Attachments: AllProjectTestResults.txt, HBASE-2985-LoadBalancer-V2.patch, HBASE-3985-LoadBalancer.patch, HBASE-3985-LoadBalancer_(NoneServerLog).patch, org.apache.hadoop.hbase.master.TestLoadBalancer.txt
>
>
> From the HMaster logs, I found something weird:
> 2011-05-24 11:12:11,152 INFO org.apache.hadoop.hbase.master.HMaster: balance hri=hello,122130,1305944329350.7d6c96428e2563c3d8676474d0a9f814., src=158-1-101-202,20020,1306205409671, dest=158-1-101-222,20020,1306205940117
> 2011-05-24 11:12:31,536 INFO org.apache.hadoop.hbase.master.HMaster: balance hri=hello,122130,1305944329350.7d6c96428e2563c3d8676474d0a9f814., src=158-1-101-202,20020,1306205409671, dest=158-1-101-222,20020,1306205940117
> We can see that, the same region was balanced twice.
> To describe the problem, I give out one simple example:
> 1. Suppose regions count is 10 in RegionServer A.
> Max: 5 Min:4
> 2. So the regions count need to move is: 5.
> 3. Before the movement of calculate, the list was shuffled.
> 4. The 5 moving region was picked out from the back.
> 5. The nextRegionForUnload value is 5.
> 6. So if the neededRegions is not zero. Maybe there's still one region should be picked out from RegionServer A.
> This time , the picked Index is 5 which has been picked once!!!!!
>
> |<-----5-------|
> ------------*--*--*--*--*--*--*--*--*--*----
> |
> getNextRegionForUnload
> Here's the analysis from code:
> 1. Walk down most loaded, pruning each to the max. Picked region from back of the list(by reverse order)
> Map<HServerInfo,BalanceInfo> serverBalanceInfo =
> new TreeMap<HServerInfo,BalanceInfo>();
> for(Map.Entry<HServerInfo, List<HRegionInfo>> server :
> serversByLoad.descendingMap().entrySet()) {
> HServerInfo serverInfo = server.getKey();
> int regionCount = serverInfo.getLoad().getNumberOfRegions();
> if(regionCount <= max) {
> serverBalanceInfo.put(serverInfo, new BalanceInfo(0, 0));
> break;
> }
> serversOverloaded++;
> List<HRegionInfo> regions = randomize(server.getValue());
> int numToOffload = Math.min(regionCount - max, regions.size());
> int numTaken = 0;
> for (int i = regions.size() - 1; i >= 0; i--) {
> HRegionInfo hri = regions.get(i);
> // Don't rebalance meta regions.
> if (hri.isMetaRegion()) continue;
> regionsToMove.add(new RegionPlan(hri, serverInfo, null));
> numTaken++;
> if (numTaken >= numToOffload) break;
> }
> /**********************************************************/
> /***set the nextRegionForUnload value is numToOffload ****/
> /**********************************************************/
> serverBalanceInfo.put(serverInfo,
> new BalanceInfo(numToOffload, (-1)*numTaken));
> }
> 2. The second pass of picked one region from the Max regionserver by order.
> if (neededRegions != 0) {
> // Walk down most loaded, grabbing one from each until we get enough
> for(Map.Entry<HServerInfo, List<HRegionInfo>> server :
> serversByLoad.descendingMap().entrySet()) {
> BalanceInfo balanceInfo = serverBalanceInfo.get(server.getKey());
> int idx =
> balanceInfo == null ? 0 : balanceInfo.getNextRegionForUnload();
> if (idx >= server.getValue().size()) break;
> HRegionInfo region = server.getValue().get(idx);
> if (region.isMetaRegion()) continue; // Don't move meta regions.
> regionsToMove.add(new RegionPlan(region, server.getKey(), null));
> if(--neededRegions == 0) {
> // No more regions needed, done shedding
> break;
> }
> }
> }
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira