You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hbase.apache.org by "Jim Kellerman (JIRA)" <ji...@apache.org> on 2008/05/06 03:33:55 UTC
[jira] Created: (HBASE-615) Region balancer oscillates during
cluster startup
Region balancer oscillates during cluster startup
-------------------------------------------------
Key: HBASE-615
URL: https://issues.apache.org/jira/browse/HBASE-615
Project: Hadoop HBase
Issue Type: Bug
Components: master
Affects Versions: 0.2.0
Reporter: Jim Kellerman
Assignee: Bryan Duxbury
When starting a cluster with four region servers and a large table (49 regions) (+root +meta) = 51 total regions, the region balancer oscillates for a very long time and does not seem to reach a steady state.
Additionally, for whatever reason, it seems reluctant to assign regions to the first of four region servers, which may be the root cause. In my test, the first server had 10 regions assigned, the second and fourth had 13 regions assigned, and the master would continually assign and deassign 2 regions to the third server, which oscillated between 13 and 15 regions. If it assigned the two fluctuating regions to the first server, it would achieve the best balance possible: 12, 13, 13, 13.
After 20 minutes, it had not stopped oscillating. An application trying to work against this cluster would run very slowly as it would be continually re-finding the two regions in flux.
When the table was being created, regions were nicely balanced. On restart, however, it just would not settle down.
Perhaps the balancer should set a target number of regions for each server which when the server achieved +/- 1 regions, the rebalancer would not try to change unless the number of regions changed.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-615) Region balancer oscillates during
cluster startup
Posted by "Jim Kellerman (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12606438#action_12606438 ]
Jim Kellerman commented on HBASE-615:
-------------------------------------
I had applied a similar change to my cluster and and it worked just fine. Reviewed patch. +1
> Region balancer oscillates during cluster startup
> -------------------------------------------------
>
> Key: HBASE-615
> URL: https://issues.apache.org/jira/browse/HBASE-615
> Project: Hadoop HBase
> Issue Type: Bug
> Components: master
> Affects Versions: 0.2.0
> Reporter: Jim Kellerman
> Assignee: Bryan Duxbury
> Priority: Blocker
> Fix For: 0.2.0
>
> Attachments: 615-lite.patch, 615-v2.patch
>
>
> When starting a cluster with four region servers and a large table (49 regions) (+root +meta) = 51 total regions, the region balancer oscillates for a very long time and does not seem to reach a steady state.
> Additionally, for whatever reason, it seems reluctant to assign regions to the first of four region servers, which may be the root cause. In my test, the first server had 10 regions assigned, the second and fourth had 13 regions assigned, and the master would continually assign and deassign 2 regions to the third server, which oscillated between 13 and 15 regions. If it assigned the two fluctuating regions to the first server, it would achieve the best balance possible: 12, 13, 13, 13.
> After 20 minutes, it had not stopped oscillating. An application trying to work against this cluster would run very slowly as it would be continually re-finding the two regions in flux.
> When the table was being created, regions were nicely balanced. On restart, however, it just would not settle down.
> Perhaps the balancer should set a target number of regions for each server which when the server achieved +/- 1 regions, the rebalancer would not try to change unless the number of regions changed.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-615) Region balancer oscillates during
cluster startup
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12606120#action_12606120 ]
stack commented on HBASE-615:
-----------------------------
Would that mean no balancing? Or just that balancing would be every server has ~ same number of regions? If latter, lets do that for now. It would still be massive improvement over what we have in 0.1 branch. We can make a new issue for making balancing code consider loading.
> Region balancer oscillates during cluster startup
> -------------------------------------------------
>
> Key: HBASE-615
> URL: https://issues.apache.org/jira/browse/HBASE-615
> Project: Hadoop HBase
> Issue Type: Bug
> Components: master
> Affects Versions: 0.2.0
> Reporter: Jim Kellerman
> Assignee: Bryan Duxbury
> Priority: Blocker
> Fix For: 0.2.0
>
> Attachments: 615-lite.patch
>
>
> When starting a cluster with four region servers and a large table (49 regions) (+root +meta) = 51 total regions, the region balancer oscillates for a very long time and does not seem to reach a steady state.
> Additionally, for whatever reason, it seems reluctant to assign regions to the first of four region servers, which may be the root cause. In my test, the first server had 10 regions assigned, the second and fourth had 13 regions assigned, and the master would continually assign and deassign 2 regions to the third server, which oscillated between 13 and 15 regions. If it assigned the two fluctuating regions to the first server, it would achieve the best balance possible: 12, 13, 13, 13.
> After 20 minutes, it had not stopped oscillating. An application trying to work against this cluster would run very slowly as it would be continually re-finding the two regions in flux.
> When the table was being created, regions were nicely balanced. On restart, however, it just would not settle down.
> Perhaps the balancer should set a target number of regions for each server which when the server achieved +/- 1 regions, the rebalancer would not try to change unless the number of regions changed.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (HBASE-615) Region balancer oscillates during
cluster startup
Posted by "Bryan Duxbury (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-615?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Bryan Duxbury resolved HBASE-615.
---------------------------------
Resolution: Fixed
I just committed this.
> Region balancer oscillates during cluster startup
> -------------------------------------------------
>
> Key: HBASE-615
> URL: https://issues.apache.org/jira/browse/HBASE-615
> Project: Hadoop HBase
> Issue Type: Bug
> Components: master
> Affects Versions: 0.2.0
> Reporter: Jim Kellerman
> Assignee: Bryan Duxbury
> Priority: Blocker
> Fix For: 0.2.0
>
> Attachments: 615-lite.patch, 615-v2.patch
>
>
> When starting a cluster with four region servers and a large table (49 regions) (+root +meta) = 51 total regions, the region balancer oscillates for a very long time and does not seem to reach a steady state.
> Additionally, for whatever reason, it seems reluctant to assign regions to the first of four region servers, which may be the root cause. In my test, the first server had 10 regions assigned, the second and fourth had 13 regions assigned, and the master would continually assign and deassign 2 regions to the third server, which oscillated between 13 and 15 regions. If it assigned the two fluctuating regions to the first server, it would achieve the best balance possible: 12, 13, 13, 13.
> After 20 minutes, it had not stopped oscillating. An application trying to work against this cluster would run very slowly as it would be continually re-finding the two regions in flux.
> When the table was being created, regions were nicely balanced. On restart, however, it just would not settle down.
> Perhaps the balancer should set a target number of regions for each server which when the server achieved +/- 1 regions, the rebalancer would not try to change unless the number of regions changed.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-615) Region balancer oscillates during
cluster startup
Posted by "Bryan Duxbury (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-615?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Bryan Duxbury updated HBASE-615:
--------------------------------
Attachment: 615-v2.patch
Here's a patch that removes the number of requests from the load considerations. It passes unit tests (except TestMetaUtils, which times out, but appears unrelated).
Jim, can you test this on your cluster and see if it reduces oscillation?
> Region balancer oscillates during cluster startup
> -------------------------------------------------
>
> Key: HBASE-615
> URL: https://issues.apache.org/jira/browse/HBASE-615
> Project: Hadoop HBase
> Issue Type: Bug
> Components: master
> Affects Versions: 0.2.0
> Reporter: Jim Kellerman
> Assignee: Bryan Duxbury
> Priority: Blocker
> Fix For: 0.2.0
>
> Attachments: 615-lite.patch, 615-v2.patch
>
>
> When starting a cluster with four region servers and a large table (49 regions) (+root +meta) = 51 total regions, the region balancer oscillates for a very long time and does not seem to reach a steady state.
> Additionally, for whatever reason, it seems reluctant to assign regions to the first of four region servers, which may be the root cause. In my test, the first server had 10 regions assigned, the second and fourth had 13 regions assigned, and the master would continually assign and deassign 2 regions to the third server, which oscillated between 13 and 15 regions. If it assigned the two fluctuating regions to the first server, it would achieve the best balance possible: 12, 13, 13, 13.
> After 20 minutes, it had not stopped oscillating. An application trying to work against this cluster would run very slowly as it would be continually re-finding the two regions in flux.
> When the table was being created, regions were nicely balanced. On restart, however, it just would not settle down.
> Perhaps the balancer should set a target number of regions for each server which when the server achieved +/- 1 regions, the rebalancer would not try to change unless the number of regions changed.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-615) Region balancer oscillates during
cluster startup
Posted by "Bryan Duxbury (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-615?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Bryan Duxbury updated HBASE-615:
--------------------------------
Attachment: 615-lite.patch
Here's a patch for the easiest of the ideas. Jim, can you apply this and try it out with your 51-region table?
> Region balancer oscillates during cluster startup
> -------------------------------------------------
>
> Key: HBASE-615
> URL: https://issues.apache.org/jira/browse/HBASE-615
> Project: Hadoop HBase
> Issue Type: Bug
> Components: master
> Affects Versions: 0.2.0
> Reporter: Jim Kellerman
> Assignee: Bryan Duxbury
> Fix For: 0.2.0
>
> Attachments: 615-lite.patch
>
>
> When starting a cluster with four region servers and a large table (49 regions) (+root +meta) = 51 total regions, the region balancer oscillates for a very long time and does not seem to reach a steady state.
> Additionally, for whatever reason, it seems reluctant to assign regions to the first of four region servers, which may be the root cause. In my test, the first server had 10 regions assigned, the second and fourth had 13 regions assigned, and the master would continually assign and deassign 2 regions to the third server, which oscillated between 13 and 15 regions. If it assigned the two fluctuating regions to the first server, it would achieve the best balance possible: 12, 13, 13, 13.
> After 20 minutes, it had not stopped oscillating. An application trying to work against this cluster would run very slowly as it would be continually re-finding the two regions in flux.
> When the table was being created, regions were nicely balanced. On restart, however, it just would not settle down.
> Perhaps the balancer should set a target number of regions for each server which when the server achieved +/- 1 regions, the rebalancer would not try to change unless the number of regions changed.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-615) Region balancer oscillates during
cluster startup
Posted by "Bryan Duxbury (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12606110#action_12606110 ]
Bryan Duxbury commented on HBASE-615:
-------------------------------------
Should we just disable factoring in requests in a region server's load for the moment? It might lead to a worse distribution of regions, but it might also make no difference, if on average all regions are equally busy, which would be the case if you're running big time map/reduce jobs. If you have hot rows/regions, then this could hurt you, but it's hard to say by how much.
> Region balancer oscillates during cluster startup
> -------------------------------------------------
>
> Key: HBASE-615
> URL: https://issues.apache.org/jira/browse/HBASE-615
> Project: Hadoop HBase
> Issue Type: Bug
> Components: master
> Affects Versions: 0.2.0
> Reporter: Jim Kellerman
> Assignee: Bryan Duxbury
> Priority: Blocker
> Fix For: 0.2.0
>
> Attachments: 615-lite.patch
>
>
> When starting a cluster with four region servers and a large table (49 regions) (+root +meta) = 51 total regions, the region balancer oscillates for a very long time and does not seem to reach a steady state.
> Additionally, for whatever reason, it seems reluctant to assign regions to the first of four region servers, which may be the root cause. In my test, the first server had 10 regions assigned, the second and fourth had 13 regions assigned, and the master would continually assign and deassign 2 regions to the third server, which oscillated between 13 and 15 regions. If it assigned the two fluctuating regions to the first server, it would achieve the best balance possible: 12, 13, 13, 13.
> After 20 minutes, it had not stopped oscillating. An application trying to work against this cluster would run very slowly as it would be continually re-finding the two regions in flux.
> When the table was being created, regions were nicely balanced. On restart, however, it just would not settle down.
> Perhaps the balancer should set a target number of regions for each server which when the server achieved +/- 1 regions, the rebalancer would not try to change unless the number of regions changed.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-615) Region balancer oscillates during
cluster startup
Posted by "Jim Kellerman (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12606077#action_12606077 ]
Jim Kellerman commented on HBASE-615:
-------------------------------------
> Rong-En Fan - 11/Jun/08 09:43 AM
> Please correct me if I am wrong or overlook something.
>
> During startup, META will be requested more than other regions. Therefore,
> the RegionServer that serves META will be considered more "loaded" than
> others. So, we tends not to assign more regions to that one. However,
> our rebalance algo currently considers only # of loaded regions as the "load"
> for region servers. That's the cause of oscillation at startup.
>
> I'm thinking of the possibility that during startup, we just use assign
> evenly to all region servers. Once this is stabilized, we start to consider
>
> 1. of requests as part of the server load. Moreover, the "# of requests" here
> should be calculated from a period of time, otherwise, we may moving regions
> just because some spikes.
You are absolutely correct. During startup, the server hosting the meta region gets all the requests,
so its one region gets multiplied by the number of requests giving a "load" that is far greater than all
the other servers which are getting no requests and consequently their "load" == number of regions
they are serving.
Should be a fairly easy fix to ignore requests during startup.
BTW, I verified this by changing HServerLoad.getLoad() to just return the number of regions. The
cluster had all regions on-line within a couple of minutes and they were balanced. When the
number of requests was factored in during startup, the cluster did not achieve a steady state
for 1/2 hour (after which I gave up).
just my 2 cents.
> Region balancer oscillates during cluster startup
> -------------------------------------------------
>
> Key: HBASE-615
> URL: https://issues.apache.org/jira/browse/HBASE-615
> Project: Hadoop HBase
> Issue Type: Bug
> Components: master
> Affects Versions: 0.2.0
> Reporter: Jim Kellerman
> Assignee: Bryan Duxbury
> Priority: Blocker
> Fix For: 0.2.0
>
> Attachments: 615-lite.patch
>
>
> When starting a cluster with four region servers and a large table (49 regions) (+root +meta) = 51 total regions, the region balancer oscillates for a very long time and does not seem to reach a steady state.
> Additionally, for whatever reason, it seems reluctant to assign regions to the first of four region servers, which may be the root cause. In my test, the first server had 10 regions assigned, the second and fourth had 13 regions assigned, and the master would continually assign and deassign 2 regions to the third server, which oscillated between 13 and 15 regions. If it assigned the two fluctuating regions to the first server, it would achieve the best balance possible: 12, 13, 13, 13.
> After 20 minutes, it had not stopped oscillating. An application trying to work against this cluster would run very slowly as it would be continually re-finding the two regions in flux.
> When the table was being created, regions were nicely balanced. On restart, however, it just would not settle down.
> Perhaps the balancer should set a target number of regions for each server which when the server achieved +/- 1 regions, the rebalancer would not try to change unless the number of regions changed.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-615) Region balancer oscillates during
cluster startup
Posted by "Rong-En Fan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12606227#action_12606227 ]
Rong-En Fan commented on HBASE-615:
-----------------------------------
I believe Bryan is saying the latter way.
> Region balancer oscillates during cluster startup
> -------------------------------------------------
>
> Key: HBASE-615
> URL: https://issues.apache.org/jira/browse/HBASE-615
> Project: Hadoop HBase
> Issue Type: Bug
> Components: master
> Affects Versions: 0.2.0
> Reporter: Jim Kellerman
> Assignee: Bryan Duxbury
> Priority: Blocker
> Fix For: 0.2.0
>
> Attachments: 615-lite.patch
>
>
> When starting a cluster with four region servers and a large table (49 regions) (+root +meta) = 51 total regions, the region balancer oscillates for a very long time and does not seem to reach a steady state.
> Additionally, for whatever reason, it seems reluctant to assign regions to the first of four region servers, which may be the root cause. In my test, the first server had 10 regions assigned, the second and fourth had 13 regions assigned, and the master would continually assign and deassign 2 regions to the third server, which oscillated between 13 and 15 regions. If it assigned the two fluctuating regions to the first server, it would achieve the best balance possible: 12, 13, 13, 13.
> After 20 minutes, it had not stopped oscillating. An application trying to work against this cluster would run very slowly as it would be continually re-finding the two regions in flux.
> When the table was being created, regions were nicely balanced. On restart, however, it just would not settle down.
> Perhaps the balancer should set a target number of regions for each server which when the server achieved +/- 1 regions, the rebalancer would not try to change unless the number of regions changed.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-615) Region balancer oscillates during
cluster startup
Posted by "Jim Kellerman (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-615?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jim Kellerman updated HBASE-615:
--------------------------------
Priority: Blocker (was: Major)
> Region balancer oscillates during cluster startup
> -------------------------------------------------
>
> Key: HBASE-615
> URL: https://issues.apache.org/jira/browse/HBASE-615
> Project: Hadoop HBase
> Issue Type: Bug
> Components: master
> Affects Versions: 0.2.0
> Reporter: Jim Kellerman
> Assignee: Bryan Duxbury
> Priority: Blocker
> Fix For: 0.2.0
>
> Attachments: 615-lite.patch
>
>
> When starting a cluster with four region servers and a large table (49 regions) (+root +meta) = 51 total regions, the region balancer oscillates for a very long time and does not seem to reach a steady state.
> Additionally, for whatever reason, it seems reluctant to assign regions to the first of four region servers, which may be the root cause. In my test, the first server had 10 regions assigned, the second and fourth had 13 regions assigned, and the master would continually assign and deassign 2 regions to the third server, which oscillated between 13 and 15 regions. If it assigned the two fluctuating regions to the first server, it would achieve the best balance possible: 12, 13, 13, 13.
> After 20 minutes, it had not stopped oscillating. An application trying to work against this cluster would run very slowly as it would be continually re-finding the two regions in flux.
> When the table was being created, regions were nicely balanced. On restart, however, it just would not settle down.
> Perhaps the balancer should set a target number of regions for each server which when the server achieved +/- 1 regions, the rebalancer would not try to change unless the number of regions changed.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-615) Region balancer oscillates during
cluster startup
Posted by "Jim Kellerman (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12599225#action_12599225 ]
Jim Kellerman commented on HBASE-615:
-------------------------------------
Well now it oscillates moving three regions around. There are 40 regions including root and meta, 4 region servers, but the master refuses to give more regions to the region server hosting the meta region. It has 4 regions and the rest have 11, 12 or 13
> Region balancer oscillates during cluster startup
> -------------------------------------------------
>
> Key: HBASE-615
> URL: https://issues.apache.org/jira/browse/HBASE-615
> Project: Hadoop HBase
> Issue Type: Bug
> Components: master
> Affects Versions: 0.2.0
> Reporter: Jim Kellerman
> Assignee: Bryan Duxbury
> Fix For: 0.2.0
>
> Attachments: 615-lite.patch
>
>
> When starting a cluster with four region servers and a large table (49 regions) (+root +meta) = 51 total regions, the region balancer oscillates for a very long time and does not seem to reach a steady state.
> Additionally, for whatever reason, it seems reluctant to assign regions to the first of four region servers, which may be the root cause. In my test, the first server had 10 regions assigned, the second and fourth had 13 regions assigned, and the master would continually assign and deassign 2 regions to the third server, which oscillated between 13 and 15 regions. If it assigned the two fluctuating regions to the first server, it would achieve the best balance possible: 12, 13, 13, 13.
> After 20 minutes, it had not stopped oscillating. An application trying to work against this cluster would run very slowly as it would be continually re-finding the two regions in flux.
> When the table was being created, regions were nicely balanced. On restart, however, it just would not settle down.
> Perhaps the balancer should set a target number of regions for each server which when the server achieved +/- 1 regions, the rebalancer would not try to change unless the number of regions changed.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-615) Region balancer oscillates during
cluster startup
Posted by "Jim Kellerman (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-615?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jim Kellerman updated HBASE-615:
--------------------------------
Fix Version/s: 0.2.0
> Region balancer oscillates during cluster startup
> -------------------------------------------------
>
> Key: HBASE-615
> URL: https://issues.apache.org/jira/browse/HBASE-615
> Project: Hadoop HBase
> Issue Type: Bug
> Components: master
> Affects Versions: 0.2.0
> Reporter: Jim Kellerman
> Assignee: Bryan Duxbury
> Fix For: 0.2.0
>
>
> When starting a cluster with four region servers and a large table (49 regions) (+root +meta) = 51 total regions, the region balancer oscillates for a very long time and does not seem to reach a steady state.
> Additionally, for whatever reason, it seems reluctant to assign regions to the first of four region servers, which may be the root cause. In my test, the first server had 10 regions assigned, the second and fourth had 13 regions assigned, and the master would continually assign and deassign 2 regions to the third server, which oscillated between 13 and 15 regions. If it assigned the two fluctuating regions to the first server, it would achieve the best balance possible: 12, 13, 13, 13.
> After 20 minutes, it had not stopped oscillating. An application trying to work against this cluster would run very slowly as it would be continually re-finding the two regions in flux.
> When the table was being created, regions were nicely balanced. On restart, however, it just would not settle down.
> Perhaps the balancer should set a target number of regions for each server which when the server achieved +/- 1 regions, the rebalancer would not try to change unless the number of regions changed.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-615) Region balancer oscillates during
cluster startup
Posted by "Bryan Duxbury (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12599021#action_12599021 ]
Bryan Duxbury commented on HBASE-615:
-------------------------------------
There's two ways this could go. Either we can fix it by widening the margin by which you have to be "over" the load average before we start rebalancing region assignments, or we have to redo this code to be more sensitive to the load as it is being computed on the assignment side of things.
I'll do the first no matter what as a test because it will be very simple. However, if the problem is that the definitions of "overloaded" from the perspective of load reported by the regionserver (this is some combo of regions and requests right now, yes?) and the math done by the rebalancing code are inherently different, then we'll need to either make the existing load balancing on assignment dumber or the rebalancing smarter. In the long run, we're definitely going to want to do the latter, but it requires us to start tracking requests and anything else that goes into the balancing computation at the region level, as well as actually reporting that information when the workers check in with their short list of reassignable regions. That way, when we're deciding how many regions to unassign, we can make informed decisions, rather than just trying to equalize averages.
> Region balancer oscillates during cluster startup
> -------------------------------------------------
>
> Key: HBASE-615
> URL: https://issues.apache.org/jira/browse/HBASE-615
> Project: Hadoop HBase
> Issue Type: Bug
> Components: master
> Affects Versions: 0.2.0
> Reporter: Jim Kellerman
> Assignee: Bryan Duxbury
> Fix For: 0.2.0
>
>
> When starting a cluster with four region servers and a large table (49 regions) (+root +meta) = 51 total regions, the region balancer oscillates for a very long time and does not seem to reach a steady state.
> Additionally, for whatever reason, it seems reluctant to assign regions to the first of four region servers, which may be the root cause. In my test, the first server had 10 regions assigned, the second and fourth had 13 regions assigned, and the master would continually assign and deassign 2 regions to the third server, which oscillated between 13 and 15 regions. If it assigned the two fluctuating regions to the first server, it would achieve the best balance possible: 12, 13, 13, 13.
> After 20 minutes, it had not stopped oscillating. An application trying to work against this cluster would run very slowly as it would be continually re-finding the two regions in flux.
> When the table was being created, regions were nicely balanced. On restart, however, it just would not settle down.
> Perhaps the balancer should set a target number of regions for each server which when the server achieved +/- 1 regions, the rebalancer would not try to change unless the number of regions changed.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-615) Region balancer oscillates during
cluster startup
Posted by "Rong-En Fan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12604268#action_12604268 ]
Rong-En Fan commented on HBASE-615:
-----------------------------------
Please correct me if I am wrong or overlook something.
During startup, META will be requested more than other regions. Therefore,
the RegionServer that serves META will be considered more "loaded" than
others. So, we tends not to assign more regions to that one. However,
our rebalance algo currently considers only # of loaded regions as the "load"
for region servers. That's the cause of oscillation at startup.
I'm thinking of the possibility that during startup, we just use assign
evenly to all region servers. Once this is stabilized, we start to consider
# of requests as part of the server load. Moreover, the "# of requests" here
should be calculated from a period of time, otherwise, we may moving regions
just because some spikes.
just my 2 cents.
> Region balancer oscillates during cluster startup
> -------------------------------------------------
>
> Key: HBASE-615
> URL: https://issues.apache.org/jira/browse/HBASE-615
> Project: Hadoop HBase
> Issue Type: Bug
> Components: master
> Affects Versions: 0.2.0
> Reporter: Jim Kellerman
> Assignee: Bryan Duxbury
> Priority: Blocker
> Fix For: 0.2.0
>
> Attachments: 615-lite.patch
>
>
> When starting a cluster with four region servers and a large table (49 regions) (+root +meta) = 51 total regions, the region balancer oscillates for a very long time and does not seem to reach a steady state.
> Additionally, for whatever reason, it seems reluctant to assign regions to the first of four region servers, which may be the root cause. In my test, the first server had 10 regions assigned, the second and fourth had 13 regions assigned, and the master would continually assign and deassign 2 regions to the third server, which oscillated between 13 and 15 regions. If it assigned the two fluctuating regions to the first server, it would achieve the best balance possible: 12, 13, 13, 13.
> After 20 minutes, it had not stopped oscillating. An application trying to work against this cluster would run very slowly as it would be continually re-finding the two regions in flux.
> When the table was being created, regions were nicely balanced. On restart, however, it just would not settle down.
> Perhaps the balancer should set a target number of regions for each server which when the server achieved +/- 1 regions, the rebalancer would not try to change unless the number of regions changed.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-615) Region balancer oscillates during
cluster startup
Posted by "Rong-En Fan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12606414#action_12606414 ]
Rong-En Fan commented on HBASE-615:
-----------------------------------
I actually have the same patch applied locally, the cluster will start in a balance state
within few minutes.
> Region balancer oscillates during cluster startup
> -------------------------------------------------
>
> Key: HBASE-615
> URL: https://issues.apache.org/jira/browse/HBASE-615
> Project: Hadoop HBase
> Issue Type: Bug
> Components: master
> Affects Versions: 0.2.0
> Reporter: Jim Kellerman
> Assignee: Bryan Duxbury
> Priority: Blocker
> Fix For: 0.2.0
>
> Attachments: 615-lite.patch, 615-v2.patch
>
>
> When starting a cluster with four region servers and a large table (49 regions) (+root +meta) = 51 total regions, the region balancer oscillates for a very long time and does not seem to reach a steady state.
> Additionally, for whatever reason, it seems reluctant to assign regions to the first of four region servers, which may be the root cause. In my test, the first server had 10 regions assigned, the second and fourth had 13 regions assigned, and the master would continually assign and deassign 2 regions to the third server, which oscillated between 13 and 15 regions. If it assigned the two fluctuating regions to the first server, it would achieve the best balance possible: 12, 13, 13, 13.
> After 20 minutes, it had not stopped oscillating. An application trying to work against this cluster would run very slowly as it would be continually re-finding the two regions in flux.
> When the table was being created, regions were nicely balanced. On restart, however, it just would not settle down.
> Perhaps the balancer should set a target number of regions for each server which when the server achieved +/- 1 regions, the rebalancer would not try to change unless the number of regions changed.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.