You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "Jean-Daniel Cryans (JIRA)" <ji...@apache.org> on 2011/03/22 00:35:06 UTC
[jira] [Created] (HBASE-3681) Check the sloppiness of the region
load before balancing
Check the sloppiness of the region load before balancing
--------------------------------------------------------
Key: HBASE-3681
URL: https://issues.apache.org/jira/browse/HBASE-3681
Project: HBase
Issue Type: Improvement
Affects Versions: 0.90.1
Reporter: Jean-Daniel Cryans
Fix For: 0.92.0
Per our discussion at the hackathon today, it seems that it would be more helpful to add a sloppiness check before doing the normal balancing.
The current situation is that the balancer always tries to get the region load even, meaning that there can be some very frequent regions movement.
Setting the balancer to run less often (like every 4 hours) isn't much better since the load could get out of whack easily.
This is why running the normal balancer frequently, but first checking for some sloppiness in the region load across the RS, seems like a more viable option.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3681) Check the sloppiness of the region
load before balancing
Posted by "Jean-Daniel Cryans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13011004#comment-13011004 ]
Jean-Daniel Cryans commented on HBASE-3681:
-------------------------------------------
Some comments:
- I think the ifs in the new constructor aren't written consistently with the rest of the code. Usually we either do brackets or one-line ifs and this is neither.
- I wouldn't reuse the min/max since they don't have the same meaning.
- The comment "so that behavior is consistent with that of 0.90.1." is true but I don't think it belongs in there, since it will go stale very fast. Just saying that here is enough IMO.
- I think for 0.92 we could set the slop to something like 0.1 or 0.2, but keep it at 0 for 0.90.1. Regarding that, Stack moved this issue to 0.92 but he agreed we can move it back into 0.90.2 since 1) we need it at SU and 2) it's going to be disabled by default.
> Check the sloppiness of the region load before balancing
> --------------------------------------------------------
>
> Key: HBASE-3681
> URL: https://issues.apache.org/jira/browse/HBASE-3681
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.90.1
> Reporter: Jean-Daniel Cryans
> Assignee: Ted Yu
> Fix For: 0.92.0
>
> Attachments: hbase-3681-v2.txt, hbase-3681.txt
>
>
> Per our discussion at the hackathon today, it seems that it would be more helpful to add a sloppiness check before doing the normal balancing.
> The current situation is that the balancer always tries to get the region load even, meaning that there can be some very frequent regions movement.
> Setting the balancer to run less often (like every 4 hours) isn't much better since the load could get out of whack easily.
> This is why running the normal balancer frequently, but first checking for some sloppiness in the region load across the RS, seems like a more viable option.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3681) Check the sloppiness of the region
load before balancing
Posted by "Jean-Daniel Cryans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13009520#comment-13009520 ]
Jean-Daniel Cryans commented on HBASE-3681:
-------------------------------------------
Sounds good Ted.
> Check the sloppiness of the region load before balancing
> --------------------------------------------------------
>
> Key: HBASE-3681
> URL: https://issues.apache.org/jira/browse/HBASE-3681
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.90.1
> Reporter: Jean-Daniel Cryans
> Assignee: Ted Yu
> Fix For: 0.92.0
>
>
> Per our discussion at the hackathon today, it seems that it would be more helpful to add a sloppiness check before doing the normal balancing.
> The current situation is that the balancer always tries to get the region load even, meaning that there can be some very frequent regions movement.
> Setting the balancer to run less often (like every 4 hours) isn't much better since the load could get out of whack easily.
> This is why running the normal balancer frequently, but first checking for some sloppiness in the region load across the RS, seems like a more viable option.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3681) Check the sloppiness of the region
load before balancing
Posted by "Jean-Daniel Cryans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13011035#comment-13011035 ]
Jean-Daniel Cryans commented on HBASE-3681:
-------------------------------------------
+1 I like this new version, I'm going to fix the following nits on commit if you don't mind:
- "balance if regionserver has average + (average * slop) regions" should actually refer to all region servers (it sounds like it is on a per RS basis)
- the default value in the code isn't the same as in hbase-default.xml, I'm going to put a different value both branches anyways
> Check the sloppiness of the region load before balancing
> --------------------------------------------------------
>
> Key: HBASE-3681
> URL: https://issues.apache.org/jira/browse/HBASE-3681
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.90.1
> Reporter: Jean-Daniel Cryans
> Assignee: Ted Yu
> Fix For: 0.92.0
>
> Attachments: hbase-3681-v2.txt, hbase-3681-v3.txt, hbase-3681.txt
>
>
> Per our discussion at the hackathon today, it seems that it would be more helpful to add a sloppiness check before doing the normal balancing.
> The current situation is that the balancer always tries to get the region load even, meaning that there can be some very frequent regions movement.
> Setting the balancer to run less often (like every 4 hours) isn't much better since the load could get out of whack easily.
> This is why running the normal balancer frequently, but first checking for some sloppiness in the region load across the RS, seems like a more viable option.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3681) Check the sloppiness of the region
load before balancing
Posted by "Jean-Daniel Cryans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13009828#comment-13009828 ]
Jean-Daniel Cryans commented on HBASE-3681:
-------------------------------------------
This is the behavior we're trying to get to. The current problem is that every time the master balances you'll have some disruption. You could instead set the balancer to run less often, and have one big batch of regions moving at the same time while some region server may get really unbalanced. Since the time to rebalance 5 regions can easily be the same as rebalancing 50 regions, the disruption ends up being almost the same.
To optimize that situation, we want to keep running the balancer often but first check against some sloppiness.
> Check the sloppiness of the region load before balancing
> --------------------------------------------------------
>
> Key: HBASE-3681
> URL: https://issues.apache.org/jira/browse/HBASE-3681
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.90.1
> Reporter: Jean-Daniel Cryans
> Assignee: Ted Yu
> Fix For: 0.92.0
>
> Attachments: hbase-3681.txt
>
>
> Per our discussion at the hackathon today, it seems that it would be more helpful to add a sloppiness check before doing the normal balancing.
> The current situation is that the balancer always tries to get the region load even, meaning that there can be some very frequent regions movement.
> Setting the balancer to run less often (like every 4 hours) isn't much better since the load could get out of whack easily.
> This is why running the normal balancer frequently, but first checking for some sloppiness in the region load across the RS, seems like a more viable option.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3681) Check the sloppiness of the region
load before balancing
Posted by "Ted Yu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13009774#comment-13009774 ]
Ted Yu commented on HBASE-3681:
-------------------------------
LoadBalancer currently doesn't have access to Configuration. That's why slop config is passed from HMaster.
I will submit another patch which checks against sloppiness envelope. If any of the servers is outside the envelope, the current balancing code will run. See following:
{code}
float average = (float)numRegions / numServers; // for logging
int min = (int) Math.floor(average * (1 - slop));
if (min > 0) min -= 1;
int max = (int) Math.ceil(average * (1 + slop));
if(serversByLoad.lastKey().getLoad().getNumberOfRegions() <= max &&
serversByLoad.firstKey().getLoad().getNumberOfRegions() >= min) {
// Skipped because no server outside (min,max) range
LOG.info("Skipping load balancing. servers=" + numServers + " " +
"regions=" + numRegions + " average=" + average + " " +
"mostloaded=" + serversByLoad.lastKey().getLoad().getNumberOfRegions() +
" leastloaded=" + serversByLoad.firstKey().getLoad().getNumberOfRegions());
return null;
}
min = numRegions / numServers;
max = numRegions % numServers == 0 ? min : min + 1;
{code}
> Check the sloppiness of the region load before balancing
> --------------------------------------------------------
>
> Key: HBASE-3681
> URL: https://issues.apache.org/jira/browse/HBASE-3681
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.90.1
> Reporter: Jean-Daniel Cryans
> Assignee: Ted Yu
> Fix For: 0.92.0
>
> Attachments: hbase-3681.txt
>
>
> Per our discussion at the hackathon today, it seems that it would be more helpful to add a sloppiness check before doing the normal balancing.
> The current situation is that the balancer always tries to get the region load even, meaning that there can be some very frequent regions movement.
> Setting the balancer to run less often (like every 4 hours) isn't much better since the load could get out of whack easily.
> This is why running the normal balancer frequently, but first checking for some sloppiness in the region load across the RS, seems like a more viable option.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-3681) Check the sloppiness of the region
load before balancing
Posted by "Ted Yu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3681?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ted Yu updated HBASE-3681:
--------------------------
Attachment: hbase-3681-v3.txt
Made changes according to review comments.
> Check the sloppiness of the region load before balancing
> --------------------------------------------------------
>
> Key: HBASE-3681
> URL: https://issues.apache.org/jira/browse/HBASE-3681
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.90.1
> Reporter: Jean-Daniel Cryans
> Assignee: Ted Yu
> Fix For: 0.92.0
>
> Attachments: hbase-3681-v2.txt, hbase-3681-v3.txt, hbase-3681.txt
>
>
> Per our discussion at the hackathon today, it seems that it would be more helpful to add a sloppiness check before doing the normal balancing.
> The current situation is that the balancer always tries to get the region load even, meaning that there can be some very frequent regions movement.
> Setting the balancer to run less often (like every 4 hours) isn't much better since the load could get out of whack easily.
> This is why running the normal balancer frequently, but first checking for some sloppiness in the region load across the RS, seems like a more viable option.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3681) Check the sloppiness of the region
load before balancing
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13011491#comment-13011491 ]
Hudson commented on HBASE-3681:
-------------------------------
Integrated in HBase-TRUNK #1814 (See [https://hudson.apache.org/hudson/job/HBase-TRUNK/1814/])
> Check the sloppiness of the region load before balancing
> --------------------------------------------------------
>
> Key: HBASE-3681
> URL: https://issues.apache.org/jira/browse/HBASE-3681
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.90.1
> Reporter: Jean-Daniel Cryans
> Assignee: Ted Yu
> Fix For: 0.90.2
>
> Attachments: hbase-3681-v2.txt, hbase-3681-v3.txt, hbase-3681.txt
>
>
> Per our discussion at the hackathon today, it seems that it would be more helpful to add a sloppiness check before doing the normal balancing.
> The current situation is that the balancer always tries to get the region load even, meaning that there can be some very frequent regions movement.
> Setting the balancer to run less often (like every 4 hours) isn't much better since the load could get out of whack easily.
> This is why running the normal balancer frequently, but first checking for some sloppiness in the region load across the RS, seems like a more viable option.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-3681) Check the sloppiness of the region
load before balancing
Posted by "Ted Yu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3681?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ted Yu updated HBASE-3681:
--------------------------
Attachment: hbase-3681-v2.txt
Second version.
TestLoadBalancer, TestRegionRebalancing, TestMultiParallel and TestMasterObserver pass.
Changed default slop to 0 so that behavior is consistent with 0.90.1
> Check the sloppiness of the region load before balancing
> --------------------------------------------------------
>
> Key: HBASE-3681
> URL: https://issues.apache.org/jira/browse/HBASE-3681
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.90.1
> Reporter: Jean-Daniel Cryans
> Assignee: Ted Yu
> Fix For: 0.92.0
>
> Attachments: hbase-3681-v2.txt, hbase-3681.txt
>
>
> Per our discussion at the hackathon today, it seems that it would be more helpful to add a sloppiness check before doing the normal balancing.
> The current situation is that the balancer always tries to get the region load even, meaning that there can be some very frequent regions movement.
> Setting the balancer to run less often (like every 4 hours) isn't much better since the load could get out of whack easily.
> This is why running the normal balancer frequently, but first checking for some sloppiness in the region load across the RS, seems like a more viable option.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HBASE-3681) Check the sloppiness of the region
load before balancing
Posted by "Ted Yu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3681?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ted Yu reassigned HBASE-3681:
-----------------------------
Assignee: Ted Yu
> Check the sloppiness of the region load before balancing
> --------------------------------------------------------
>
> Key: HBASE-3681
> URL: https://issues.apache.org/jira/browse/HBASE-3681
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.90.1
> Reporter: Jean-Daniel Cryans
> Assignee: Ted Yu
> Fix For: 0.92.0
>
>
> Per our discussion at the hackathon today, it seems that it would be more helpful to add a sloppiness check before doing the normal balancing.
> The current situation is that the balancer always tries to get the region load even, meaning that there can be some very frequent regions movement.
> Setting the balancer to run less often (like every 4 hours) isn't much better since the load could get out of whack easily.
> This is why running the normal balancer frequently, but first checking for some sloppiness in the region load across the RS, seems like a more viable option.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-3681) Check the sloppiness of the region
load before balancing
Posted by "Jean-Daniel Cryans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3681?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jean-Daniel Cryans updated HBASE-3681:
--------------------------------------
Resolution: Fixed
Fix Version/s: (was: 0.92.0)
0.90.2
Hadoop Flags: [Reviewed]
Status: Resolved (was: Patch Available)
Committed to branch (slop=0) and trunk (slop=0.2), thanks for the good work Ted!
> Check the sloppiness of the region load before balancing
> --------------------------------------------------------
>
> Key: HBASE-3681
> URL: https://issues.apache.org/jira/browse/HBASE-3681
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.90.1
> Reporter: Jean-Daniel Cryans
> Assignee: Ted Yu
> Fix For: 0.90.2
>
> Attachments: hbase-3681-v2.txt, hbase-3681-v3.txt, hbase-3681.txt
>
>
> Per our discussion at the hackathon today, it seems that it would be more helpful to add a sloppiness check before doing the normal balancing.
> The current situation is that the balancer always tries to get the region load even, meaning that there can be some very frequent regions movement.
> Setting the balancer to run less often (like every 4 hours) isn't much better since the load could get out of whack easily.
> This is why running the normal balancer frequently, but first checking for some sloppiness in the region load across the RS, seems like a more viable option.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3681) Check the sloppiness of the region
load before balancing
Posted by "Ted Yu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13011037#comment-13011037 ]
Ted Yu commented on HBASE-3681:
-------------------------------
Thanks for review J-D
> Check the sloppiness of the region load before balancing
> --------------------------------------------------------
>
> Key: HBASE-3681
> URL: https://issues.apache.org/jira/browse/HBASE-3681
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.90.1
> Reporter: Jean-Daniel Cryans
> Assignee: Ted Yu
> Fix For: 0.92.0
>
> Attachments: hbase-3681-v2.txt, hbase-3681-v3.txt, hbase-3681.txt
>
>
> Per our discussion at the hackathon today, it seems that it would be more helpful to add a sloppiness check before doing the normal balancing.
> The current situation is that the balancer always tries to get the region load even, meaning that there can be some very frequent regions movement.
> Setting the balancer to run less often (like every 4 hours) isn't much better since the load could get out of whack easily.
> This is why running the normal balancer frequently, but first checking for some sloppiness in the region load across the RS, seems like a more viable option.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3681) Check the sloppiness of the region
load before balancing
Posted by "Ted Yu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13009823#comment-13009823 ]
Ted Yu commented on HBASE-3681:
-------------------------------
The above approach may surprise some users who would expect sloppiness envelope to be respected after balancing is done.
It also leads to burst of region re-assignment from one balancing to the next.
It also makes unit testing a little complicated.
> Check the sloppiness of the region load before balancing
> --------------------------------------------------------
>
> Key: HBASE-3681
> URL: https://issues.apache.org/jira/browse/HBASE-3681
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.90.1
> Reporter: Jean-Daniel Cryans
> Assignee: Ted Yu
> Fix For: 0.92.0
>
> Attachments: hbase-3681.txt
>
>
> Per our discussion at the hackathon today, it seems that it would be more helpful to add a sloppiness check before doing the normal balancing.
> The current situation is that the balancer always tries to get the region load even, meaning that there can be some very frequent regions movement.
> Setting the balancer to run less often (like every 4 hours) isn't much better since the load could get out of whack easily.
> This is why running the normal balancer frequently, but first checking for some sloppiness in the region load across the RS, seems like a more viable option.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-3681) Check the sloppiness of the region
load before balancing
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3681?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HBASE-3681:
-------------------------
Status: Patch Available (was: Open)
Marking patch available
> Check the sloppiness of the region load before balancing
> --------------------------------------------------------
>
> Key: HBASE-3681
> URL: https://issues.apache.org/jira/browse/HBASE-3681
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.90.1
> Reporter: Jean-Daniel Cryans
> Assignee: Ted Yu
> Fix For: 0.92.0
>
> Attachments: hbase-3681-v2.txt, hbase-3681.txt
>
>
> Per our discussion at the hackathon today, it seems that it would be more helpful to add a sloppiness check before doing the normal balancing.
> The current situation is that the balancer always tries to get the region load even, meaning that there can be some very frequent regions movement.
> Setting the balancer to run less often (like every 4 hours) isn't much better since the load could get out of whack easily.
> This is why running the normal balancer frequently, but first checking for some sloppiness in the region load across the RS, seems like a more viable option.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3681) Check the sloppiness of the region
load before balancing
Posted by "Jonathan Gray (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13009840#comment-13009840 ]
Jonathan Gray commented on HBASE-3681:
--------------------------------------
For unit tests which expect the old behavior, they should set the sloppiness configuration to 0 (and we might make that the default value as well, since this change in behavior could be seen as less optimal in some cases).
Also, why do you do an additional -1 for the min? If it was left as floor, then setting sloppiness to 0 would yield exactly the same behavior as we have today.
> Check the sloppiness of the region load before balancing
> --------------------------------------------------------
>
> Key: HBASE-3681
> URL: https://issues.apache.org/jira/browse/HBASE-3681
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.90.1
> Reporter: Jean-Daniel Cryans
> Assignee: Ted Yu
> Fix For: 0.92.0
>
> Attachments: hbase-3681.txt
>
>
> Per our discussion at the hackathon today, it seems that it would be more helpful to add a sloppiness check before doing the normal balancing.
> The current situation is that the balancer always tries to get the region load even, meaning that there can be some very frequent regions movement.
> Setting the balancer to run less often (like every 4 hours) isn't much better since the load could get out of whack easily.
> This is why running the normal balancer frequently, but first checking for some sloppiness in the region load across the RS, seems like a more viable option.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3681) Check the sloppiness of the region
load before balancing
Posted by "Jean-Daniel Cryans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13009762#comment-13009762 ]
Jean-Daniel Cryans commented on HBASE-3681:
-------------------------------------------
I think there's a small misunderstanding of what the intention of this jira is.
The Balancer should only run if any of the servers are outside of the sloppiness envelope, it's not on a per region server basis else we would just be delaying the problem. So, this patch really only needs to add a check before everything else to see where each region server is with regards to the slop factor, then make the decision of doing the balance to even out everything.
Also I think the slop should be configured in AssignmentManager, since HMaster in your patch is really just passing it down.
> Check the sloppiness of the region load before balancing
> --------------------------------------------------------
>
> Key: HBASE-3681
> URL: https://issues.apache.org/jira/browse/HBASE-3681
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.90.1
> Reporter: Jean-Daniel Cryans
> Assignee: Ted Yu
> Fix For: 0.92.0
>
> Attachments: hbase-3681.txt
>
>
> Per our discussion at the hackathon today, it seems that it would be more helpful to add a sloppiness check before doing the normal balancing.
> The current situation is that the balancer always tries to get the region load even, meaning that there can be some very frequent regions movement.
> Setting the balancer to run less often (like every 4 hours) isn't much better since the load could get out of whack easily.
> This is why running the normal balancer frequently, but first checking for some sloppiness in the region load across the RS, seems like a more viable option.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-3681) Check the sloppiness of the region
load before balancing
Posted by "Ted Yu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3681?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ted Yu updated HBASE-3681:
--------------------------
Attachment: hbase-3681.txt
Initial attempt.
If slop is greater than 0, some tests fail.
e.g.
{code}
2011-03-21 22:18:44,886 INFO [main] master.TestLoadBalancer(186): Mock Cluster : { 1 , 1 , 1123 , 133 , 138 , 12 , 1444 , 0 , 0 , 144 , 1 , 1 } [srvr=12 rgns=2998 avg=249.83333 max=250 min=249]
2011-03-21 22:18:44,887 WARN [main] master.LoadBalancer(322): regionidx=2027, regionsToMove=2027, numServers=12, serversOverloaded=2, serversUnderloaded=10
2011-03-21 22:18:44,887 WARN [main] master.LoadBalancer(332): Input 127.0.0.1,14173,-4503661418262151144 1, 127.0.0.1,16709,5660298165359772741 1, 127.0.0.1,16753,6033378928668609711 1123, 127.0.0.1,21997,-5339821095735763805 133, 127.0.0.1,25701,-2236437849271081337 138, 127.0.0.1,2735,5768301853710328270 12, 127.0.0.1,28795,1395900819118748885 1444, 127.0.0.1,41820,-7080037238717148768 0, 127.0.0.1,42315,-7850691226193331732 0, 127.0.0.1,55335,-5765170823713071701 144, 127.0.0.1,55580,-8175369880198350855 1, 127.0.0.1,8395,850167680932222284 1
2011-03-21 22:18:44,887 INFO [main] master.LoadBalancer(336): Calculated a load balance in 1ms. Moving 2027 regions off of 2 overloaded servers onto 10 less loaded servers
2011-03-21 22:18:44,889 INFO [main] master.TestLoadBalancer(189): Mock Balance : { 248 , 248 , 274 , 248 , 248 , 248 , 274 , 249 , 249 , 216 , 248 , 248 }
{code}
> Check the sloppiness of the region load before balancing
> --------------------------------------------------------
>
> Key: HBASE-3681
> URL: https://issues.apache.org/jira/browse/HBASE-3681
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.90.1
> Reporter: Jean-Daniel Cryans
> Assignee: Ted Yu
> Fix For: 0.92.0
>
> Attachments: hbase-3681.txt
>
>
> Per our discussion at the hackathon today, it seems that it would be more helpful to add a sloppiness check before doing the normal balancing.
> The current situation is that the balancer always tries to get the region load even, meaning that there can be some very frequent regions movement.
> Setting the balancer to run less often (like every 4 hours) isn't much better since the load could get out of whack easily.
> This is why running the normal balancer frequently, but first checking for some sloppiness in the region load across the RS, seems like a more viable option.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3681) Check the sloppiness of the region
load before balancing
Posted by "Jean-Daniel Cryans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13009781#comment-13009781 ]
Jean-Daniel Cryans commented on HBASE-3681:
-------------------------------------------
Ah I see, then I think it would be better to create a constructor for LoadBalancer and pass the HBC.
> Check the sloppiness of the region load before balancing
> --------------------------------------------------------
>
> Key: HBASE-3681
> URL: https://issues.apache.org/jira/browse/HBASE-3681
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.90.1
> Reporter: Jean-Daniel Cryans
> Assignee: Ted Yu
> Fix For: 0.92.0
>
> Attachments: hbase-3681.txt
>
>
> Per our discussion at the hackathon today, it seems that it would be more helpful to add a sloppiness check before doing the normal balancing.
> The current situation is that the balancer always tries to get the region load even, meaning that there can be some very frequent regions movement.
> Setting the balancer to run less often (like every 4 hours) isn't much better since the load could get out of whack easily.
> This is why running the normal balancer frequently, but first checking for some sloppiness in the region load across the RS, seems like a more viable option.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3681) Check the sloppiness of the region
load before balancing
Posted by "Ted Yu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13009511#comment-13009511 ]
Ted Yu commented on HBASE-3681:
-------------------------------
I plan to put in this change:
{code}
min = (int) Math.floor(average * (1 - slop)) - 1;
max = (int) Math.ceil(average * (1 + slop));
{code}
> Check the sloppiness of the region load before balancing
> --------------------------------------------------------
>
> Key: HBASE-3681
> URL: https://issues.apache.org/jira/browse/HBASE-3681
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 0.90.1
> Reporter: Jean-Daniel Cryans
> Assignee: Ted Yu
> Fix For: 0.92.0
>
>
> Per our discussion at the hackathon today, it seems that it would be more helpful to add a sloppiness check before doing the normal balancing.
> The current situation is that the balancer always tries to get the region load even, meaning that there can be some very frequent regions movement.
> Setting the balancer to run less often (like every 4 hours) isn't much better since the load could get out of whack easily.
> This is why running the normal balancer frequently, but first checking for some sloppiness in the region load across the RS, seems like a more viable option.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira