[jira] [Commented] (HBASE-24545) Add backoff to SCP check on WAL split completion

["Michael Stack (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HBASE-24545?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17134366#comment-17134366 ] 

Michael Stack commented on HBASE-24545:
---------------------------------------

Just for illustration of the problem described, here is where a single thread was hanging out:
{code}
"KeepAlivePEWorker-158" #909 daemon prio=5 os_prio=0 tid=0x0000000001fb5000 nid=0x29e in Object.wait() [0x00007f73fda29000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at java.lang.Object.wait(Object.java:502)
        at org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1529)
        - locked <0x00007f7c64048020> (a org.apache.zookeeper.ClientCnxn$Packet)
        at org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1512)
        at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:2587)
        at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getChildren(RecoverableZooKeeper.java:283)
        at org.apache.hadoop.hbase.zookeeper.ZKUtil.listChildrenNoWatch(ZKUtil.java:502)
        at org.apache.hadoop.hbase.coordination.ZKSplitLogManagerCoordination.remainingTasksInCoordination(ZKSplitLogManagerCoordination.java:125)
        at org.apache.hadoop.hbase.master.SplitLogManager.waitForSplittingCompletion(SplitLogManager.java:333)
        - locked <0x00007f76381dc690> (a org.apache.hadoop.hbase.master.SplitLogManager$TaskBatch)
        at org.apache.hadoop.hbase.master.SplitLogManager.splitLogDistributed(SplitLogManager.java:262)
        at org.apache.hadoop.hbase.master.MasterWalManager.splitLog(MasterWalManager.java:350)
        at org.apache.hadoop.hbase.master.MasterWalManager.splitLog(MasterWalManager.java:335)
        at org.apache.hadoop.hbase.master.MasterWalManager.splitLog(MasterWalManager.java:272)
        at org.apache.hadoop.hbase.master.procedure.ServerCrashProcedure.splitLogs(ServerCrashProcedure.java:312)
        at org.apache.hadoop.hbase.master.procedure.ServerCrashProcedure.executeFromState(ServerCrashProcedure.java:197)
        at org.apache.hadoop.hbase.master.procedure.ServerCrashProcedure.executeFromState(ServerCrashProcedure.java:64)
        at org.apache.hadoop.hbase.procedure2.StateMachineProcedure.execute(StateMachineProcedure.java:194)
        at org.apache.hadoop.hbase.procedure2.Procedure.doExecute(Procedure.java:962)
        at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1669)
        at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1416)
        at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:79)
        at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1986)
{code}

> Add backoff to SCP check on WAL split completion
> ------------------------------------------------
>
>                 Key: HBASE-24545
>                 URL: https://issues.apache.org/jira/browse/HBASE-24545
>             Project: HBase
>          Issue Type: Bug
>            Reporter: Michael Stack
>            Assignee: Michael Stack
>            Priority: Major
>             Fix For: 3.0.0-alpha-1, 2.3.0
>
>
> Crashed cluster. Lots of backed up WALs. Startup. Recover hundreds of servers; each has a running SCP. Taking a thread dump during recovery, I noticed that there were 160 threads each in SCP waiting on split WAL completion. Each thread was scanning zk splitWAL directory every 100ms. The dir had thousands of entries in it so each check was pulling down MB from zk... * 160 (max configured PE threads (16) * 10 for the KeepAlive factor that has us do 10 * configured PEs as max for PE worker pool).
> If lots of remaining WALs to split, have the SCP backoff on its wait so it checks less frequently.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)