You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hbase.apache.org by "stack (JIRA)" <ji...@apache.org> on 2008/09/02 23:09:44 UTC
[jira] Created: (HBASE-862) region balancing is clumsy
region balancing is clumsy
--------------------------
Key: HBASE-862
URL: https://issues.apache.org/jira/browse/HBASE-862
Project: Hadoop HBase
Issue Type: Bug
Reporter: stack
Daniel Leffel has an install of 500 regions on 4 nodes. He's running 0.2.0.
On restart, load balancing is running while the 600 regions are being initially opened. Makes for churn. Load balancing should wait before it cuts in.
Have also seen on occasion that it will not find equilibrium after a restart.
Adding a node is catastrophic. >20% of the regions were closed and were taking the longest time to show up on the new server. I would think that the region balancing would work in more sophisticated and gradual manner.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-862) region balancing is clumsy
Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12629008#action_12629008 ]
Andrew Purtell commented on HBASE-862:
--------------------------------------
One problem with simple RR is that not all regionservers may come up at the same time. Still need to consider region counts (but not load) in the beginning to try to smooth assignments out.
Also I will try modifying bin/hbase-daemons.sh to issue commands via SSH in parallel rather than sequentially, and have start-hbase.sh bring up the regionservers ahead of the master.
> region balancing is clumsy
> --------------------------
>
> Key: HBASE-862
> URL: https://issues.apache.org/jira/browse/HBASE-862
> Project: Hadoop HBase
> Issue Type: Bug
> Reporter: stack
>
> Daniel Leffel has an install of 500 regions on 4 nodes. He's running 0.2.0.
> On restart, load balancing is running while the 600 regions are being initially opened. Makes for churn. Load balancing should wait before it cuts in.
> Have also seen on occasion that it will not find equilibrium after a restart.
> Adding a node is catastrophic. >20% of the regions were closed and were taking the longest time to show up on the new server. I would think that the region balancing would work in more sophisticated and gradual manner.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (HBASE-862) region balancing is clumsy
Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Andrew Purtell reassigned HBASE-862:
------------------------------------
Assignee: (was: Andrew Purtell)
> region balancing is clumsy
> --------------------------
>
> Key: HBASE-862
> URL: https://issues.apache.org/jira/browse/HBASE-862
> Project: Hadoop HBase
> Issue Type: Bug
> Reporter: stack
>
> Daniel Leffel has an install of 500 regions on 4 nodes. He's running 0.2.0.
> On restart, load balancing is running while the 600 regions are being initially opened. Makes for churn. Load balancing should wait before it cuts in.
> Have also seen on occasion that it will not find equilibrium after a restart.
> Adding a node is catastrophic. >20% of the regions were closed and were taking the longest time to show up on the new server. I would think that the region balancing would work in more sophisticated and gradual manner.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-862) region balancing is clumsy
Posted by "Ryan Smith (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12630035#action_12630035 ]
Ryan Smith commented on HBASE-862:
----------------------------------
+1
Ive seen this when starting up my test cluster with about 300 regions on 4 nodes. It usually subsides after a few minutes.
> region balancing is clumsy
> --------------------------
>
> Key: HBASE-862
> URL: https://issues.apache.org/jira/browse/HBASE-862
> Project: Hadoop HBase
> Issue Type: Bug
> Reporter: stack
>
> Daniel Leffel has an install of 500 regions on 4 nodes. He's running 0.2.0.
> On restart, load balancing is running while the 600 regions are being initially opened. Makes for churn. Load balancing should wait before it cuts in.
> Have also seen on occasion that it will not find equilibrium after a restart.
> Adding a node is catastrophic. >20% of the regions were closed and were taking the longest time to show up on the new server. I would think that the region balancing would work in more sophisticated and gradual manner.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (HBASE-862) region balancing is clumsy
Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Andrew Purtell reassigned HBASE-862:
------------------------------------
Assignee: Andrew Purtell
> region balancing is clumsy
> --------------------------
>
> Key: HBASE-862
> URL: https://issues.apache.org/jira/browse/HBASE-862
> Project: Hadoop HBase
> Issue Type: Bug
> Reporter: stack
> Assignee: Andrew Purtell
>
> Daniel Leffel has an install of 500 regions on 4 nodes. He's running 0.2.0.
> On restart, load balancing is running while the 600 regions are being initially opened. Makes for churn. Load balancing should wait before it cuts in.
> Have also seen on occasion that it will not find equilibrium after a restart.
> Adding a node is catastrophic. >20% of the regions were closed and were taking the longest time to show up on the new server. I would think that the region balancing would work in more sophisticated and gradual manner.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-862) region balancing is clumsy
Posted by "Billy Pearson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12628940#action_12628940 ]
Billy Pearson commented on HBASE-862:
-------------------------------------
I thank a simple way to do startups is using a round ribbon assignment of regions
Just a thought.
> region balancing is clumsy
> --------------------------
>
> Key: HBASE-862
> URL: https://issues.apache.org/jira/browse/HBASE-862
> Project: Hadoop HBase
> Issue Type: Bug
> Reporter: stack
>
> Daniel Leffel has an install of 500 regions on 4 nodes. He's running 0.2.0.
> On restart, load balancing is running while the 600 regions are being initially opened. Makes for churn. Load balancing should wait before it cuts in.
> Have also seen on occasion that it will not find equilibrium after a restart.
> Adding a node is catastrophic. >20% of the regions were closed and were taking the longest time to show up on the new server. I would think that the region balancing would work in more sophisticated and gradual manner.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-862) region balancing is clumsy
Posted by "Billy Pearson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12629071#action_12629071 ]
Billy Pearson commented on HBASE-862:
-------------------------------------
I added some ideas to HBASE-678 that might be helpful for balancing on start up later down the road when we get a safe mode.
> region balancing is clumsy
> --------------------------
>
> Key: HBASE-862
> URL: https://issues.apache.org/jira/browse/HBASE-862
> Project: Hadoop HBase
> Issue Type: Bug
> Reporter: stack
>
> Daniel Leffel has an install of 500 regions on 4 nodes. He's running 0.2.0.
> On restart, load balancing is running while the 600 regions are being initially opened. Makes for churn. Load balancing should wait before it cuts in.
> Have also seen on occasion that it will not find equilibrium after a restart.
> Adding a node is catastrophic. >20% of the regions were closed and were taking the longest time to show up on the new server. I would think that the region balancing would work in more sophisticated and gradual manner.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-862) region balancing is clumsy
Posted by "Jean-Daniel Cryans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12627812#action_12627812 ]
Jean-Daniel Cryans commented on HBASE-862:
------------------------------------------
+1, I see it too.
> region balancing is clumsy
> --------------------------
>
> Key: HBASE-862
> URL: https://issues.apache.org/jira/browse/HBASE-862
> Project: Hadoop HBase
> Issue Type: Bug
> Reporter: stack
>
> Daniel Leffel has an install of 500 regions on 4 nodes. He's running 0.2.0.
> On restart, load balancing is running while the 600 regions are being initially opened. Makes for churn. Load balancing should wait before it cuts in.
> Have also seen on occasion that it will not find equilibrium after a restart.
> Adding a node is catastrophic. >20% of the regions were closed and were taking the longest time to show up on the new server. I would think that the region balancing would work in more sophisticated and gradual manner.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-862) region balancing is clumsy
Posted by "Billy Pearson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12627907#action_12627907 ]
Billy Pearson commented on HBASE-862:
-------------------------------------
+1 I see this also.
I also see MR jobs fail often if I add a region server to the cluster while the job is running.
I thank this is sometimes from closing regions that are running a timely compaction and will not close for a while to be redeployed.
What about when we send the request to close a region make it different from normal close call and give the region server a option to decline the request
example say the master sends a request to close a small group of regions to redeploy and the region server have 1 or more of the regions queued up for compaction
let the region server send a request back to the master declining the regions that are in the compaction queue or if they have a open scanner on them etc...
also I would slow down the redeploy of the regions to 1-3 in a cycle where we wait until all the regions are open again before moving more.
We also might build in some give in the numbers per server to make it less likely to move a region if one of the servers is 1-3 regions or 1-5% out of balance.
I would like to see the balancer keep everything even but I would be ok with it leavening it a little out of balance.
Maybe we can use something like the lease timeout var from the config to define how often the balancer runs a cycle.
My down the road wish list is one day be able report back to the master in the heartbeat the load on the regions that a region server has and generate a read/write load numbers per region/table/server/cluster/etc..
With this data we could be more sophisticated on what regions to move and when.
> region balancing is clumsy
> --------------------------
>
> Key: HBASE-862
> URL: https://issues.apache.org/jira/browse/HBASE-862
> Project: Hadoop HBase
> Issue Type: Bug
> Reporter: stack
>
> Daniel Leffel has an install of 500 regions on 4 nodes. He's running 0.2.0.
> On restart, load balancing is running while the 600 regions are being initially opened. Makes for churn. Load balancing should wait before it cuts in.
> Have also seen on occasion that it will not find equilibrium after a restart.
> Adding a node is catastrophic. >20% of the regions were closed and were taking the longest time to show up on the new server. I would think that the region balancing would work in more sophisticated and gradual manner.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-862) region balancing is clumsy
Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12628285#action_12628285 ]
Andrew Purtell commented on HBASE-862:
--------------------------------------
Let me take a look at this one if nobody else is working on it. Will incorporate stack and Billy's thoughts into more sophisticated strategy that balances balancing against regionserver load and region availability considerations, ready for 0.2.2. We'll be deploying 10 additional nodes into a running 15 node cluster with 1000s of regions running a continuous cycle of TableMap/TableReduce tasks so there will be ample opportunity for testing.
> region balancing is clumsy
> --------------------------
>
> Key: HBASE-862
> URL: https://issues.apache.org/jira/browse/HBASE-862
> Project: Hadoop HBase
> Issue Type: Bug
> Reporter: stack
>
> Daniel Leffel has an install of 500 regions on 4 nodes. He's running 0.2.0.
> On restart, load balancing is running while the 600 regions are being initially opened. Makes for churn. Load balancing should wait before it cuts in.
> Have also seen on occasion that it will not find equilibrium after a restart.
> Adding a node is catastrophic. >20% of the regions were closed and were taking the longest time to show up on the new server. I would think that the region balancing would work in more sophisticated and gradual manner.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-862) region balancing is clumsy
Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12659849#action_12659849 ]
Andrew Purtell commented on HBASE-862:
--------------------------------------
+1
Seeing this on our cluster as well. Master starts up in safe mode. All HRS start up and check in. Then initial assignments start. Assignment is lumpy -- average load should be e.g. 40, some HRS get e.g. 80 initial assignments. They report as overloaded, so then regions are closed and moved almost as soon as they are opened. Compaction/split load is increased unnecessarily.
Also seeing asserts about illegal state transitions from the Master.
All regions do come up assigned and HRS are evenly loaded after a while, unless compaction/split load overwhelms DFS. Then usually there are a few regions that did not deploy correctly that have to be manually (re)deployed with close_region from the shell.
> region balancing is clumsy
> --------------------------
>
> Key: HBASE-862
> URL: https://issues.apache.org/jira/browse/HBASE-862
> Project: Hadoop HBase
> Issue Type: Bug
> Reporter: stack
>
> Daniel Leffel has an install of 500 regions on 4 nodes. He's running 0.2.0.
> On restart, load balancing is running while the 600 regions are being initially opened. Makes for churn. Load balancing should wait before it cuts in.
> Have also seen on occasion that it will not find equilibrium after a restart.
> Adding a node is catastrophic. >20% of the regions were closed and were taking the longest time to show up on the new server. I would think that the region balancing would work in more sophisticated and gradual manner.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-862) region balancing is clumsy
Posted by "Jim Kellerman (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12629038#action_12629038 ]
Jim Kellerman commented on HBASE-862:
-------------------------------------
When a cluster starts, it might help if the master knew how many region servers to expect to report in. It could then divide the number of regions by n servers, and if n servers didn't report in by the time the root and meta were assigned (for example), or after some configurable interval, the master could assume that the servers were not going to report in and assign the remaining regions over the n - (missing servers)
> region balancing is clumsy
> --------------------------
>
> Key: HBASE-862
> URL: https://issues.apache.org/jira/browse/HBASE-862
> Project: Hadoop HBase
> Issue Type: Bug
> Reporter: stack
>
> Daniel Leffel has an install of 500 regions on 4 nodes. He's running 0.2.0.
> On restart, load balancing is running while the 600 regions are being initially opened. Makes for churn. Load balancing should wait before it cuts in.
> Have also seen on occasion that it will not find equilibrium after a restart.
> Adding a node is catastrophic. >20% of the regions were closed and were taking the longest time to show up on the new server. I would think that the region balancing would work in more sophisticated and gradual manner.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Issue Comment Edited: (HBASE-862) region balancing is clumsy
Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12659849#action_12659849 ]
apurtell edited comment on HBASE-862 at 12/30/08 9:00 AM:
----------------------------------------------------------------
Seeing this on our cluster now. Master starts up in safe mode. All HRS start up and check in. Then initial assignments start. Assignment is lumpy -- average load should be e.g. 40, some HRS get e.g. 80 initial assignments. They report as overloaded, so then regions are closed and moved almost as soon as they are opened. Compaction/split load is increased unnecessarily.
Also seeing asserts about illegal state transitions from the Master.
All regions do come up assigned and HRS are evenly loaded after a while, unless compaction/split load overwhelms DFS. Then usually there are a few regions that did not deploy correctly that have to be manually (re)deployed with close_region from the shell.
was (Author: apurtell):
+1
Seeing this on our cluster as well. Master starts up in safe mode. All HRS start up and check in. Then initial assignments start. Assignment is lumpy -- average load should be e.g. 40, some HRS get e.g. 80 initial assignments. They report as overloaded, so then regions are closed and moved almost as soon as they are opened. Compaction/split load is increased unnecessarily.
Also seeing asserts about illegal state transitions from the Master.
All regions do come up assigned and HRS are evenly loaded after a while, unless compaction/split load overwhelms DFS. Then usually there are a few regions that did not deploy correctly that have to be manually (re)deployed with close_region from the shell.
> region balancing is clumsy
> --------------------------
>
> Key: HBASE-862
> URL: https://issues.apache.org/jira/browse/HBASE-862
> Project: Hadoop HBase
> Issue Type: Bug
> Reporter: stack
>
> Daniel Leffel has an install of 500 regions on 4 nodes. He's running 0.2.0.
> On restart, load balancing is running while the 600 regions are being initially opened. Makes for churn. Load balancing should wait before it cuts in.
> Have also seen on occasion that it will not find equilibrium after a restart.
> Adding a node is catastrophic. >20% of the regions were closed and were taking the longest time to show up on the new server. I would think that the region balancing would work in more sophisticated and gradual manner.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-862) region balancing is clumsy
Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12629060#action_12629060 ]
Andrew Purtell commented on HBASE-862:
--------------------------------------
For cluster startup, how about:
* Master waits a short interval for regionserver start messages to arrive.
* After the initial waiting period has elapsed, begin assigning regions based on the current count of announced regionservers.
* If additional servers report in, try to get their region count up to the current average before assigning additional regions to earlier reporters.
* Once all regions have been assigned, wait a dampening period before starting the balancer.
Combine the above with changes to start-hbase.sh and hbase-daemons.sh to start the regionservers ahead of the master and start the regionservers in parallel rather than serially, and I think the startup behavior will improve.
Balancing regions in steady state, especially if there are late or new arrivals to the cluster, is a different proposition I think. Billy had some ideas about that in an earlier comment. Also some special case handling of META is in order.
Concerning balancing in steady state, 'load balancing' attempts to ensure that the workload on each host is within a small degree of the workload present on every other host in the system. 'Load leveling' on the other hand is a more relaxed approach that only seeks to avoid congestion on any one host. Balancing is proactive. Leveling is reactive. I think both achieve the same end over time (with balancing "trying harder") and since leveling is simpler and requires little work or coordination on the part of the master, I thought I'd try that first.
> region balancing is clumsy
> --------------------------
>
> Key: HBASE-862
> URL: https://issues.apache.org/jira/browse/HBASE-862
> Project: Hadoop HBase
> Issue Type: Bug
> Reporter: stack
>
> Daniel Leffel has an install of 500 regions on 4 nodes. He's running 0.2.0.
> On restart, load balancing is running while the 600 regions are being initially opened. Makes for churn. Load balancing should wait before it cuts in.
> Have also seen on occasion that it will not find equilibrium after a restart.
> Adding a node is catastrophic. >20% of the regions were closed and were taking the longest time to show up on the new server. I would think that the region balancing would work in more sophisticated and gradual manner.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-862) region balancing is clumsy
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12664881#action_12664881 ]
stack commented on HBASE-862:
-----------------------------
HBASE-1124 should have helped this issue as well as others that made balancing sloppier.
> region balancing is clumsy
> --------------------------
>
> Key: HBASE-862
> URL: https://issues.apache.org/jira/browse/HBASE-862
> Project: Hadoop HBase
> Issue Type: Bug
> Reporter: stack
>
> Daniel Leffel has an install of 500 regions on 4 nodes. He's running 0.2.0.
> On restart, load balancing is running while the 600 regions are being initially opened. Makes for churn. Load balancing should wait before it cuts in.
> Have also seen on occasion that it will not find equilibrium after a restart.
> Adding a node is catastrophic. >20% of the regions were closed and were taking the longest time to show up on the new server. I would think that the region balancing would work in more sophisticated and gradual manner.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-862) region balancing is clumsy
Posted by "Jim Kellerman (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12628366#action_12628366 ]
Jim Kellerman commented on HBASE-862:
-------------------------------------
If you factor in load, you'll need a more sophisticated algorithm than we had in the past. Because the META region gets so much traffic, the previous algorithm would never assign any other regions to the server handling META (or rarely).
Since we've already discussed not doing balancing until all regions are loaded, maybe factoring (a more sophisticated) load at that point would be ok.
> region balancing is clumsy
> --------------------------
>
> Key: HBASE-862
> URL: https://issues.apache.org/jira/browse/HBASE-862
> Project: Hadoop HBase
> Issue Type: Bug
> Reporter: stack
>
> Daniel Leffel has an install of 500 regions on 4 nodes. He's running 0.2.0.
> On restart, load balancing is running while the 600 regions are being initially opened. Makes for churn. Load balancing should wait before it cuts in.
> Have also seen on occasion that it will not find equilibrium after a restart.
> Adding a node is catastrophic. >20% of the regions were closed and were taking the longest time to show up on the new server. I would think that the region balancing would work in more sophisticated and gradual manner.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-862) region balancing is clumsy
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12629213#action_12629213 ]
stack commented on HBASE-862:
-----------------------------
Sounds great Andrew.
> region balancing is clumsy
> --------------------------
>
> Key: HBASE-862
> URL: https://issues.apache.org/jira/browse/HBASE-862
> Project: Hadoop HBase
> Issue Type: Bug
> Reporter: stack
>
> Daniel Leffel has an install of 500 regions on 4 nodes. He's running 0.2.0.
> On restart, load balancing is running while the 600 regions are being initially opened. Makes for churn. Load balancing should wait before it cuts in.
> Have also seen on occasion that it will not find equilibrium after a restart.
> Adding a node is catastrophic. >20% of the regions were closed and were taking the longest time to show up on the new server. I would think that the region balancing would work in more sophisticated and gradual manner.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.