You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "David Manning (Jira)" <ji...@apache.org> on 2026/10/01 17:23:00 UTC
[jira] [Updated] (HBASE-30445) (branch-2) RegionStates may has some expired serverinfo and make regions do not balance.
[ https://issues.apache.org/jira/browse/HBASE-30445?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
David Manning updated HBASE-30445:
----------------------------------
Description:
It is possible for the balancer to get into a state where it indefinitely seeks balance but cannot obtain it. LoadBalancer logs will show
{{balancer.StochasticLoadBalancer - Running balancer because cluster has sloppy server(s).}}
This happens indefinitely because the {{BalancerClusterState}} construction adds a key twice for the same server. From here, the balancer thinks there is always a server that has 0 regions, which is "sloppy", and forces a balancer run even if things are otherwise balanced. The logs may look like below (notice server 23 is listed twice):
{{balancer.BalancerClusterState - server 24 is on rack 0}}
{{balancer.BalancerClusterState - server 23 is on rack 0}}
{{balancer.BalancerClusterState - server 23 is on rack 0}}
{{balancer.BalancerClusterState - server 22 is on rack 0}}
{{balancer.BalancerClusterState - server 21 is on rack 0}}
The workaround is a restart of hmaster to clear this buggy state.
The likely cause in this deployment, though not verified, was missing HBASE-29323. As a result, ReportProcedureDone RPCs could have been significantly delayed/deadlocked in the queue behind Move RPCs from the RegionMover, causing stale servers to reappear in the RegionStates.serverMap even (long) after their ServerCrashProcedure was complete. Ultimately, this is the same type of cause described by HBASE-23564.
However, after investigating the issue, 3.0 branch has deviated from 2.x branch in a way that appears unintentional, and lost the change from HBASE-23564. The balancer should provide its baseline of available servers from {{onlineServers}} instead of {{{}serverMap.keySet(){}}}.
was:
It is possible for the balancer to get into a state where it indefinitely seeks balance but cannot obtain it. LoadBalancer logs will show
{{balancer.StochasticLoadBalancer - Running balancer because cluster has sloppy server(s).}}
This happens indefinitely because the {{BalancerClusterState}} construction adds a key twice for the same server. From here, the balancer thinks there is always a server that has 0 regions, which is "sloppy", and forces a balancer run even if things are otherwise balanced. The logs may look like below (notice server 23 is listed twice):
{{balancer.BalancerClusterState - server 24 is on rack 0}}
{{balancer.BalancerClusterState - server 23 is on rack 0}}
{{balancer.BalancerClusterState - server 23 is on rack 0}}
{{balancer.BalancerClusterState - server 22 is on rack 0}}
{{balancer.BalancerClusterState - server 21 is on rack 0}}
The workaround is a restart of hmaster to clear this buggy state.
The likely cause in this deployment, though not verified, was missing HBASE-29323. As a result, RegionServerReport RPCs could have been significantly delayed in the queue behind Move RPCs from the RegionMover, causing stale servers to reappear in the RegionStates.serverMap even after their ServerCrashProcedure was complete.
However, after investigating the issue, 3.0 branch has deviated from 2.x branch in a way that appears unintentional. The balancer should provide its baseline of available servers from {{onlineServers}} instead of {{{}serverMap.keySet(){}}}.
> (branch-2) RegionStates may has some expired serverinfo and make regions do not balance.
> ----------------------------------------------------------------------------------------
>
> Key: HBASE-30445
> URL: https://issues.apache.org/jira/browse/HBASE-30445
> Project: HBase
> Issue Type: Bug
> Components: Balancer
> Affects Versions: 2.6.0, 2.5.4, 2.7.0
> Reporter: David Manning
> Assignee: David Manning
> Priority: Minor
>
> It is possible for the balancer to get into a state where it indefinitely seeks balance but cannot obtain it. LoadBalancer logs will show
> {{balancer.StochasticLoadBalancer - Running balancer because cluster has sloppy server(s).}}
> This happens indefinitely because the {{BalancerClusterState}} construction adds a key twice for the same server. From here, the balancer thinks there is always a server that has 0 regions, which is "sloppy", and forces a balancer run even if things are otherwise balanced. The logs may look like below (notice server 23 is listed twice):
> {{balancer.BalancerClusterState - server 24 is on rack 0}}
> {{balancer.BalancerClusterState - server 23 is on rack 0}}
> {{balancer.BalancerClusterState - server 23 is on rack 0}}
> {{balancer.BalancerClusterState - server 22 is on rack 0}}
> {{balancer.BalancerClusterState - server 21 is on rack 0}}
> The workaround is a restart of hmaster to clear this buggy state.
> The likely cause in this deployment, though not verified, was missing HBASE-29323. As a result, ReportProcedureDone RPCs could have been significantly delayed/deadlocked in the queue behind Move RPCs from the RegionMover, causing stale servers to reappear in the RegionStates.serverMap even (long) after their ServerCrashProcedure was complete. Ultimately, this is the same type of cause described by HBASE-23564.
> However, after investigating the issue, 3.0 branch has deviated from 2.x branch in a way that appears unintentional, and lost the change from HBASE-23564. The balancer should provide its baseline of available servers from {{onlineServers}} instead of {{{}serverMap.keySet(){}}}.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)