You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by GitBox <gi...@apache.org> on 2019/12/23 02:30:59 UTC
[GitHub] [hbase] binlijin commented on issue #955: HBASE-23597 Give high priority for meta assign procedure and ServerCr…
binlijin commented on issue #955: HBASE-23597 Give high priority for meta assign procedure and ServerCr…
URL: https://github.com/apache/hbase/pull/955#issuecomment-568332725
@Apache9 @saintstack This background is First A ServerCrashProcedure（carry meta）execute and assign meta to B, then ServerCrashProcedure A assign other region following, there are too much regions to assign and do not finish fast, meanwhile B stop and invoke a ServerCrashProcedure(carry meta), then B ServerCrashProcedure blocked by TransitRegionStateProcedure.
{code}
"KeepAlivePEWorker-21" #5162 daemon prio=5 os_prio=0 tid=0x00007f96c5aa4800 nid=0x62ed in Object.wait() [0x00007f96382e7000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:168)
- locked <0x0000000582083f58> (a java.util.concurrent.atomic.AtomicBoolean)
at org.apache.hadoop.hbase.client.HTable.put(HTable.java:540)
at org.apache.hadoop.hbase.master.assignment.RegionStateStore.updateRegionLocation(RegionStateStore.java:209)
at org.apache.hadoop.hbase.master.assignment.RegionStateStore.updateUserRegionLocation(RegionStateStore.java:203)
at org.apache.hadoop.hbase.master.assignment.RegionStateStore.updateRegionLocation(RegionStateStore.java:141)
at org.apache.hadoop.hbase.master.assignment.AssignmentManager.transitStateAndUpdate(AssignmentManager.java:1632)
at org.apache.hadoop.hbase.master.assignment.AssignmentManager.regionOpening(AssignmentManager.java:1647)
at org.apache.hadoop.hbase.master.assignment.TransitRegionStateProcedure.openRegion(TransitRegionStateProcedure.java:207)
at org.apache.hadoop.hbase.master.assignment.TransitRegionStateProcedure.executeFromState(TransitRegionStateProcedure.java:339)
at org.apache.hadoop.hbase.master.assignment.TransitRegionStateProcedure.executeFromState(TransitRegionStateProcedure.java:102)
at org.apache.hadoop.hbase.procedure2.StateMachineProcedure.execute(StateMachineProcedure.java:194)
at org.apache.hadoop.hbase.master.assignment.TransitRegionStateProcedure.execute(TransitRegionStateProcedure.java:319)
at org.apache.hadoop.hbase.master.assignment.TransitRegionStateProcedure.execute(TransitRegionStateProcedure.java:102)
at org.apache.hadoop.hbase.procedure2.Procedure.doExecute(Procedure.java:962)
at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1648)
at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1395)
at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:78)
at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1965)
{code}
you can see the log in HBASE-23597:
2019-12-18 16:15:07,212 INFO [RegionServerTracker-0] assignment.AssignmentManager: Scheduled SCP pid=69619 for 100.107.165.61,60020,1576553057082 (carryingMeta=true) 100.107.165.61,60020,1576553057082/CRASHED/regionCount=13026/lock=java.util.concurrent.locks.ReentrantReadWriteLock@68f2ee72[Write locks = 1, Read locks = 0], oldState=ONLINE.
2019-12-18 16:15:21,629 DEBUG [RpcServer.default.FPBQ.Fifo.handler=959,queue=191,port=60000] master.DeadServer: Removed 100.107.165.61,60020,1576553057082, processing=true, numProcessing=0
2019-12-18 16:33:04,728 DEBUG [PEWorker-15] procedure.MasterProcedureScheduler: Remove ServerQueue(100.107.165.61,60020,1576553057082, xlock=false sharedLock=0 size=0) from run queue because: queue is empty after polling out pid=69619, state=RUNNABLE:SERVER_CRASH_START; ServerCrashProcedure server=100.107.165.61,60020,1576553057082, splitWal=true, meta=true
The SCP in the queue wait for 18 mins.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services