[jira] [Commented] (HBASE-29933) update_all_config hangs indefinitely when balancing event is in progress

["Dev Hingu (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HBASE-29933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18064539#comment-18064539 ] 

Dev Hingu commented on HBASE-29933:
-----------------------------------

I've created a PR for this issue.


The logic behind this solution is to check if balance is happening during the update config call
 * If yes then store the snapshot of Configuration as pendingConfiguration and update it right after the balance run is complete.
 * If no then update it immedietly.


We have one important scenario to understand.
Let's take two threads, Thread 1(BalancerChore thread which calls the HMaster.balance() and Thread 2(Client RPC thread which calls the balancer.OnConfigurationChange()).

Thread 1 calls HMaster.balance() first and acquires lock on RSGroupBasedLoadBalancer object, now Thread 2 waits to acquire the lock on same object. When Thread 1 reaches [this.balancer.throttle()|https://github.com/hingu-8103/hbase/blob/11a375776f6ade02317fb26255a3c2e5efc11c87/hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java#L2221] in HMaster, it internally calls[ RSGroupBasedLoadBalancer.wait(sleepTime)|https://github.com/hingu-8103/hbase/blob/11a375776f6ade02317fb26255a3c2e5efc11c87/hbase-balancer/src/main/java/org/apache/hadoop/hbase/master/balancer/CacheAwareLoadBalancer.java#L201] which releases monitor. Now Thread 2 will acquire the lock on RSGroupBasedLoadBalancer([see here|https://github.com/hingu-8103/hbase/blob/11a375776f6ade02317fb26255a3c2e5efc11c87/hbase-server/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupBasedLoadBalancer.java#L412]) and store the Configuration as pendingConfiguration as balancing is still running. Now Thread 1 will apply that pending configuration post balance([see here|https://github.com/hingu-8103/hbase/blob/11a375776f6ade02317fb26255a3c2e5efc11c87/hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java#L2189]).

> update_all_config hangs indefinitely when balancing event is in progress
> ------------------------------------------------------------------------
>
>                 Key: HBASE-29933
>                 URL: https://issues.apache.org/jira/browse/HBASE-29933
>             Project: HBase
>          Issue Type: Bug
>            Reporter: Dev Hingu
>            Assignee: Dev Hingu
>            Priority: Major
>              Labels: pull-request-available
>
> update_all_config command hangs indefinitely if HMaster.balance() event is going on when LoadBalancer is instance of CacheAwareLoadBalancer. 
> When HMaster.balance() is running it acquires lock on CacheAwareLoadBalancer and HMaster thread goes to sleep due to throttling in CacheAwareLoadBalancer.
> Now, CacheAwareLoadBalancer.onConfigurationChange() waits to acquire the same lock 
> Attaching stack traces for both thread
> 1. HMaster Thread : 
> {code:java}
> #355 daemon prio=5 os_prio=0 cpu=2014.35ms elapsed=19554.89s tid=0x00007f3fd018b310 nid=0x5fda waiting on condition  [0x00007f3f6a3f1000]   java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep(java.base@17.0.17/Native Method)
> at org.apache.hadoop.hbase.master.balancer.CacheAwareLoadBalancer.throttle(CacheAwareLoadBalancer.java:197)
> at org.apache.hadoop.hbase.master.HMaster.executeRegionPlansWithThrottling(HMaster.java:2164)
> at org.apache.hadoop.hbase.master.HMaster.balance(HMaster.java:2122)
> - locked <0x000000070b7d1438> (a org.apache.hadoop.hbase.master.balancer.CacheAwareLoadBalancer)
> at org.apache.hadoop.hbase.master.HMaster.balance(HMaster.java:1998)
> at org.apache.hadoop.hbase.master.HMaster.balanceOrUpdateMetrics(HMaster.java:2010)
> - locked <0x000000070b7d1438> (a org.apache.hadoop.hbase.master.balancer.CacheAwareLoadBalancer)
> at org.apache.hadoop.hbase.master.balancer.BalancerChore.chore(BalancerChore.java:47)
> at org.apache.hadoop.hbase.ScheduledChore.run(ScheduledChore.java:161){code}
> 2. update configuration RPC thread : 
> {code:java}
> #96 daemon prio=5 os_prio=0 cpu=523.03ms elapsed=19854.17s tid=0x00007f3ffbb47ed0 nid=0x48a4 waiting for monitor entry  [0x00007f3f785fe000]   
> java.lang.Thread.State: BLOCKED (on object monitor)
> at org.apache.hadoop.hbase.master.balancer.BaseLoadBalancer.onConfigurationChange(BaseLoadBalancer.java:785)    
> - waiting to lock <0x000000070b7d1438> (a org.apache.hadoop.hbase.master.balancer.CacheAwareLoadBalancer)    
> at org.apache.hadoop.hbase.conf.ConfigurationManager.notifyAllObservers(ConfigurationManager.java:110)    
> - locked <0x000000070c9e2440> (a java.util.Collections$SetFromMap)    
> at org.apache.hadoop.hbase.regionserver.HRegionServer.updateConfiguration(HRegionServer.java:3927)    
> at org.apache.hadoop.hbase.regionserver.RSRpcServices.updateConfiguration(RSRpcServices.java:3902) {code}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)