[jira] [Commented] (HBASE-22349) Stochastic Load Balancer skips balancing when node is replaced in cluster

["David Manning (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HBASE-22349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17522585#comment-17522585 ] 

David Manning commented on HBASE-22349:
---------------------------------------

The scenario as originally described is fixed by HBASE-24139. However, I would like to propose using this to track other cases where we should execute the balancer, like one server with much fewer regions, or much more regions, than the average server in the cluster. (Take the original scenario, and instead of having 0 regions on the server, have only 1 region on the server.)

I can think of a few options:
# Use some hot/cold threshold like 50%. Compute the average regions per server. If a server has a region count which is >150% or <50% of this average, allow the balancer to run (short-circuit in {{needsBalance}})
# Find outliers using some type of standard deviation, and short-circuit run in {{needsBalance}} if one is found.
# Introduce a "force run" of the balancer on some timed interval.

I'm inclined to try option 1. Option 3 sounds appealing to me, because it is a backstop to catch all of the cases which are ignored by {{minCostNeedBalance}}. However, other operators may find it too interrupting, if they need little to no region movement in the cluster.

For reference, one scenario where we find ourselves in this undesirable state is by running {{region_mover}} at the same time as the load balancer. As stated in the {{region_mover}} comments, those two operations will conflict. The result can be one regionserver which has double the regions of any other server in the cluster. And if {{minCostNeedBalance}} is not exceeded, which is not difficult in a sizable cluster, one regionserver will run with double the load indefinitely.

> Stochastic Load Balancer skips balancing when node is replaced in cluster
> -------------------------------------------------------------------------
>
>                 Key: HBASE-22349
>                 URL: https://issues.apache.org/jira/browse/HBASE-22349
>             Project: HBase
>          Issue Type: Bug
>    Affects Versions: 3.0.0-alpha-1, 1.3.0, 1.4.4, 2.0.0
>            Reporter: Suthan Phillips
>            Priority: Major
>         Attachments: Hbase-22349.pdf
>
>
> In EMR cluster, whenever I replace one of the nodes, the regions never get rebalanced.
> The default minCostNeedBalance set to 0.05 is too high.
> The region count on the servers were: 21, 21, 20, 20, 20, 20, 21, 20, 20, 20 = 203
> Once a node(region server) got replaced with a new node (terminated and EMR recreated a node), the region count on the servers became: 23, 0, 23, 22, 22, 22, 22, 23, 23, 23 = 203
> From hbase-master-logs, I can see the below WARN which indicates that the default minCostNeedBalance does not hold good for these scenarios.
> ##
> 2019-04-29 09:31:37,027 WARN  [ip-172-31-35-122.ec2.internal,16000,1556524892897_ChoreService_1] cleaner.CleanerChore: WALs outstanding under hdfs://ip-172-31-35-122.ec2.internal:8020/user/hbase/oldWALs2019-04-29 09:31:42,920 INFO  [ip-172-31-35-122.ec2.internal,16000,1556524892897_ChoreService_1] balancer.StochasticLoadBalancer: Skipping load balancing because balanced cluster; total cost is 52.041826194833405, sum multiplier is 1102.0 min cost which need balance is 0.05
> ##
> To mitigate this, I had to modify the default minCostNeedBalance to lower value like 0.01f and restart Region Servers and Hbase Master. After modifying this value to 0.01f I could see the regions getting re-balanced.
> This has led me to the following questions which I would like to get it answered from the HBase experts.
> 1)What are the factors that affect the value of total cost and sum multiplier? How could we determine the right minCostNeedBalance value for any cluster?
> 2)How did Hbase arrive at setting the default value to 0.05f? Is it optimal value? If yes, then what is the recommended way to mitigate this scenario? 
> Attached: Steps to reproduce
>  
> Note: HBase-17565 patch is already applied.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)