[jira] [Work started] (HBASE-30445) (branch-2) RegionStates may has some expired serverinfo and make regions do not balance.

["David Manning (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/HBASE-30445?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Work on HBASE-30445 started by David Manning.
---------------------------------------------
> (branch-2) RegionStates may has some expired serverinfo and make regions do not balance.
> ----------------------------------------------------------------------------------------
>
>                 Key: HBASE-30445
>                 URL: https://issues.apache.org/jira/browse/HBASE-30445
>             Project: HBase
>          Issue Type: Bug
>          Components: Balancer
>    Affects Versions: 2.6.0, 2.5.4, 2.7.0
>            Reporter: David Manning
>            Assignee: David Manning
>            Priority: Minor
>
> It is possible for the balancer to get into a state where it indefinitely seeks balance but cannot obtain it. LoadBalancer logs will show
> {{balancer.StochasticLoadBalancer - Running balancer because cluster has sloppy server(s).}}
> This happens indefinitely because the {{BalancerClusterState}} construction adds a key twice for the same server. From here, the balancer thinks there is always a server that has 0 regions, which is "sloppy", and forces a balancer run even if things are otherwise balanced. The logs may look like below (notice server 23 is listed twice):
> {{balancer.BalancerClusterState - server 24 is on rack 0}}
> {{balancer.BalancerClusterState - server 23 is on rack 0}}
> {{balancer.BalancerClusterState - server 23 is on rack 0}}
> {{balancer.BalancerClusterState - server 22 is on rack 0}}
> {{balancer.BalancerClusterState - server 21 is on rack 0}}
> The workaround is a restart of hmaster to clear this buggy state.
> The likely cause in this deployment, though not verified, was missing HBASE-29323. As a result, ReportProcedureDone RPCs could have been significantly delayed/deadlocked in the queue behind Move RPCs from the RegionMover, causing stale servers to reappear in the RegionStates.serverMap even (long) after their ServerCrashProcedure was complete. Ultimately, this is the same type of cause described by HBASE-23564.
> However, after investigating the issue, 3.0 branch has deviated from 2.x branch in a way that appears unintentional. The balancer should provide its baseline of available servers from {{onlineServers}} instead of {{{}serverMap.keySet(){}}}.
> 3.0 branch history:
>  # HBASE-22523 adds as {{{}serverMap.keySet(){}}}: [https://github.com/apache/hbase/commit/04e5bf96d84ff2f111b1e879c8daa5360b621922#diff-96d9ec3583f3683e032c43a8f2561e06888bf1fe5eb4e818b3fae47b379ef196R552-R556] 
>  # HBASE-23564 changes to {{{}onlineServers{}}}: [https://github.com/apache/hbase/commit/ab40b9648b20cef6cb041789e4fbb11e4d36e3c2#diff-96d9ec3583f3683e032c43a8f2561e06888bf1fe5eb4e818b3fae47b379ef196R562] 
> 2.x branch history:
>  # HBASE-22523 adds as {{{}serverMap.keySet(){}}}: [https://github.com/apache/hbase/commit/14041b39f2da17de7a654a1733f3fa136fe13f63#diff-96d9ec3583f3683e032c43a8f2561e06888bf1fe5eb4e818b3fae47b379ef196R552-R556]
>  # HBASE-23564 changes to {{{}onlineServers{}}}: [https://github.com/apache/hbase/commit/7a0e4d814059e6a26abc834770e5f66231e0400e#diff-96d9ec3583f3683e032c43a8f2561e06888bf1fe5eb4e818b3fae47b379ef196R563] 
>  # HBASE-23102 changes back to {{{}serverMap.keySet(){}}}: [https://github.com/apache/hbase/commit/3b6d8c3394a22c71b9836c4000dbd582af248aa3#diff-96d9ec3583f3683e032c43a8f2561e06888bf1fe5eb4e818b3fae47b379ef196R585] 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)