[jira] Updated: (HBASE-2699) Reimplement load balancing to be a background process and to not use heartbeats

["Jonathan Gray (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/HBASE-2699?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jonathan Gray updated HBASE-2699:
---------------------------------

    Attachment: HBASE-2699-LoadBalancer-v2.patch

Adds new {{LoadBalancer}} class to the master package.  Class defines three primary methods.

The new load balancing algorithm drops the notion of slop and fully balances the cluster (this will be a background process so no need to use slop to prevent churn).  The method is a bit long but I did my best to document it... at least it's in one method and testable now :)  Balancing uses the following method which takes a map of all the servers to their regions and returns a list of regions to be moved with their source and destination servers:
{noformat}
public List<RegionPlan> balanceCluster(Map<HServerInfo,List<HRegionInfo>> serverInfo)
{noformat}

For cluster startup, there is a bulk assignment method.  Currently this is implemented as round-robin but should eventually be extended to use block locations (there are some methods in this class to help but not integrated yet):
{noformat}
public Map<HServerInfo,List<HRegionInfo>> bulkAssignment(List<HRegionInfo> regions, List<HServerInfo> servers)
{noformat}

The other time we may need to determine assignments of regions is after master failover.  A new master may find regions who have been closed but no longer has in-memory state (a {{RegionPlan}}) to say where it should be opened.  In this case, the master will use an immediate assignment method, the goal of which is to just come up with an assignment as fast as possible.  Currently it randomly picks a server for each input region.
{noformat}
public Map<HRegionInfo,HServerInfo> immediateAssignment(List<HRegionInfo> regions, List<HServerInfo> servers)
{noformat}

Patch also adds a nice fat TestLoadBalancer unit test with tests for each of the three methods.  It has input arrays for a bunch of mock cluster states and methods to verify the balancing invariants to ensure correctness.

If anyone has more good edge cases let's get em added!

Yay, a testable load balancer!

> Reimplement load balancing to be a background process and to not use heartbeats
> -------------------------------------------------------------------------------
>
>                 Key: HBASE-2699
>                 URL: https://issues.apache.org/jira/browse/HBASE-2699
>             Project: HBase
>          Issue Type: Sub-task
>          Components: master, regionserver
>            Reporter: Jonathan Gray
>            Assignee: Jonathan Gray
>            Priority: Critical
>             Fix For: 0.21.0
>
>         Attachments: HBASE-2699-LoadBalancer-v2.patch
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.