[jira] [Created] (HBASE-30405) [Balancer] `HDFSBlocksDistribution.getTopHosts()` breaks weight ties by hostname, causing the current RS to be misranked and occasional zero-gain region moves

["wenhao (Jira)" <ji...@apache.org>]

wenhao created HBASE-30405:
------------------------------

             Summary: [Balancer] `HDFSBlocksDistribution.getTopHosts()` breaks weight ties by hostname, causing the current RS to be misranked and occasional zero-gain region moves
                 Key: HBASE-30405
                 URL: https://issues.apache.org/jira/browse/HBASE-30405
             Project: HBase
          Issue Type: Bug
          Components: Balancer
    Affects Versions: 2.5.11, 2.0.0
            Reporter: wenhao
            Assignee: wenhao


h2. Summary

When multiple hosts hold an equal total weight of a region's HFiles (i.e. equal block locality), {{HDFSBlocksDistribution.getTopHosts()}} resolves the tie by hostname. The resulting "most local host" ordering is therefore arbitrary with respect to locality, and the host currently serving the region is frequently not ranked first even though its locality is identical to the top host. This contradicts the documented contract of {{BalancerClusterState.regionLocations}} ("regionIndex -> list of serverIndex sorted by locality") and, through the most-local-entity selection used by {{{}LocalityBasedCandidateGenerator{}}}, can make the stochastic balancer emit zero-locality-gain move/swap proposals.
h2. Problem

{{HDFSBlocksDistribution.getTopHostsWithWeights()}} builds a {{TreeSet}} ordered by {{HostAndWeight.WeightComparator}} and returns its {{{}descendingSet(){}}}:
 * {{{}WeightComparator.compare(l, r){}}}: primary key is weight (ascending); *when weights are equal, the tie is broken by {{l.getHost().compareTo(r.getHost())}}* (hostname, ascending).

 * {{descendingSet()}} reverses that order, so the returned list is: weight descending, and {*}among equal weights, hostname descending{*}.

Consequently the first element of {{getTopHosts()}} is the host with the maximum block weight; if several hosts share the same maximum weight, the first one is simply the lexicographically largest hostname — not the host with any locality advantage.
h2. Root cause
 # {{HDFSBlocksDistribution.java}}

 ** {{WeightComparator}} tie-break: {{return l.getHost().compareTo(r.getHost());}}

 ** {{getTopHostsWithWeights()}} returns {{{}orderedHosts.descendingSet(){}}}, so equal-weight hosts are ordered by hostname {_}descending{_}.

 # {{RegionLocationFinder.getTopBlockLocations(RegionInfo region)}} returns the host list in exactly that order.

 # {{BalancerClusterState.registerRegion()}} fills {{regionLocations[regionIndex]}} from that list. The field contract says the list is "sorted by locality", but under weight ties it is actually sorted by hostname.

 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)