You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "stack (JIRA)" <ji...@apache.org> on 2018/01/17 21:45:00 UTC
[jira] [Comment Edited] (HBASE-19794) TestZooKeeper hangs
[ https://issues.apache.org/jira/browse/HBASE-19794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16329492#comment-16329492 ]
stack edited comment on HBASE-19794 at 1/17/18 9:44 PM:
--------------------------------------------------------
I can't make this hang locally or on a test machine. I see it failing 16% of time according to [https://builds.apache.org/view/H-L/view/HBase/job/HBase-Find-Flaky-Tests-branch2.0/lastSuccessfulBuild/artifact/dashboard.html] Its a timeout.
Log has loads of threads hanging out. Some Proc workers blocked:
Thread 2268 (RS_CLOSE_REGION-asf903:58756-1): State: BLOCKED Blocked count: 12 Waited count: 17 Blocked on org.apache.hadoop.hbase.master.assignment.RegionStates$RegionStateNode@1c0991d8 Blocked by 2083 (ProcExecWrkr-6) Stack: org.apache.hadoop.hbase.master.assignment.AssignmentManager.reportTransition(AssignmentManager.java:869) org.apache.hadoop.hbase.master.assignment.AssignmentManager.updateRegionTransition(AssignmentManager.java:857) org.apache.hadoop.hbase.master.assignment.AssignmentManager.reportRegionStateTransition(AssignmentManager.java:801) org.apache.hadoop.hbase.master.MasterRpcServices.reportRegionStateTransition(MasterRpcServices.java:1561) org.apache.hadoop.hbase.regionserver.HRegionServer.reportRegionStateTransition(HRegionServer.java:2263) org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler.process(CloseRegionHandler.java:121) org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:104) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Thread 2267 (RS_CLOSE_REGION-asf903:58756-0): State: BLOCKED Blocked count: 14 Waited count: 17 Blocked on org.apache.hadoop.hbase.master.assignment.RegionStates$RegionStateNode@75cdbae3 Blocked by 2086 (ProcExecWrkr-9) Stack: org.apache.hadoop.hbase.master.assignment.AssignmentManager.reportTransition(AssignmentManager.java:869) org.apache.hadoop.hbase.master.assignment.AssignmentManager.updateRegionTransition(AssignmentManager.java:857) org.apache.hadoop.hbase.master.assignment.AssignmentManager.reportRegionStateTransition(AssignmentManager.java:801) org.apache.hadoop.hbase.master.MasterRpcServices.reportRegionStateTransition(MasterRpcServices.java:1561) org.apache.hadoop.hbase.regionserver.HRegionServer.reportRegionStateTransition(HRegionServer.java:2263) org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler.process(CloseRegionHandler.java:121) org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:104) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748)
The Proc Workers are not daemon threads. Let me change that so at least we stop timing out. See HBASE-19527
was (Author: stack):
I can't make this hang locally or on a test machine. I see it failing 16% of time according to [https://builds.apache.org/view/H-L/view/HBase/job/HBase-Find-Flaky-Tests-branch2.0/lastSuccessfulBuild/artifact/dashboard.html] Its a timeout.
Log has loads of threads hanging out. Some Proc workers blocked:
Thread 2268 (RS_CLOSE_REGION-asf903:58756-1): State: BLOCKED Blocked count: 12 Waited count: 17 Blocked on org.apache.hadoop.hbase.master.assignment.RegionStates$RegionStateNode@1c0991d8 Blocked by 2083 (ProcExecWrkr-6) Stack: org.apache.hadoop.hbase.master.assignment.AssignmentManager.reportTransition(AssignmentManager.java:869) org.apache.hadoop.hbase.master.assignment.AssignmentManager.updateRegionTransition(AssignmentManager.java:857) org.apache.hadoop.hbase.master.assignment.AssignmentManager.reportRegionStateTransition(AssignmentManager.java:801) org.apache.hadoop.hbase.master.MasterRpcServices.reportRegionStateTransition(MasterRpcServices.java:1561) org.apache.hadoop.hbase.regionserver.HRegionServer.reportRegionStateTransition(HRegionServer.java:2263) org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler.process(CloseRegionHandler.java:121) org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:104) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Thread 2267 (RS_CLOSE_REGION-asf903:58756-0): State: BLOCKED Blocked count: 14 Waited count: 17 Blocked on org.apache.hadoop.hbase.master.assignment.RegionStates$RegionStateNode@75cdbae3 Blocked by 2086 (ProcExecWrkr-9) Stack: org.apache.hadoop.hbase.master.assignment.AssignmentManager.reportTransition(AssignmentManager.java:869) org.apache.hadoop.hbase.master.assignment.AssignmentManager.updateRegionTransition(AssignmentManager.java:857) org.apache.hadoop.hbase.master.assignment.AssignmentManager.reportRegionStateTransition(AssignmentManager.java:801) org.apache.hadoop.hbase.master.MasterRpcServices.reportRegionStateTransition(MasterRpcServices.java:1561) org.apache.hadoop.hbase.regionserver.HRegionServer.reportRegionStateTransition(HRegionServer.java:2263) org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler.process(CloseRegionHandler.java:121) org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:104) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748)
The Proc Workers are not daemon threads. Let me change that so at least we stop timing out.
> TestZooKeeper hangs
> -------------------
>
> Key: HBASE-19794
> URL: https://issues.apache.org/jira/browse/HBASE-19794
> Project: HBase
> Issue Type: Bug
> Reporter: Duo Zhang
> Assignee: stack
> Priority: Critical
> Fix For: 2.0.0-beta-2
>
> Attachments: org.apache.hadoop.hbase.TestZooKeeper-output.txt
>
>
> Seems like the TestZKAsyncRegistry that hangs in shutdown.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)