You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "Jean-Daniel Cryans (JIRA)" <ji...@apache.org> on 2013/08/20 01:22:51 UTC
[jira] [Commented] (HBASE-9268) Client doesn't recover from a
stalled region server
[ https://issues.apache.org/jira/browse/HBASE-9268?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13744491#comment-13744491 ]
Jean-Daniel Cryans commented on HBASE-9268:
-------------------------------------------
It's actually MultiServerCallable that gets stuck, I don't get this issue while reading. I see all my clients stuck on:
{noformat}
"hbase-table-pool-16-thread-1" daemon prio=10 tid=0x00007f2e8487e000 nid=0x5952 waiting for monitor entry [0x00007f2e5da05000]
java.lang.Thread.State: BLOCKED (on object monitor)
at org.apache.hadoop.hbase.ipc.RpcClient$Connection.writeRequest(RpcClient.java:1036)
- waiting to lock <0x00000000c40526a0> (a java.io.DataOutputStream)
at org.apache.hadoop.hbase.ipc.RpcClient.call(RpcClient.java:1403)
at org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1630)
at org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation.callBlockingMethod(RpcClient.java:1687)
at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$BlockingStub.multi(ClientProtos.java:21274)
at org.apache.hadoop.hbase.client.MultiServerCallable.call(MultiServerCallable.java:105)
at org.apache.hadoop.hbase.client.MultiServerCallable.call(MultiServerCallable.java:43)
at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:183)
at org.apache.hadoop.hbase.client.AsyncProcess$1.run(AsyncProcess.java:420)
{noformat}
> Client doesn't recover from a stalled region server
> ---------------------------------------------------
>
> Key: HBASE-9268
> URL: https://issues.apache.org/jira/browse/HBASE-9268
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.95.2
> Reporter: Jean-Daniel Cryans
> Fix For: 0.98.0, 0.95.3
>
>
> Got this testing the 0.95.2 RC.
> I killed -STOP a region server and let it stay like that while running PE. The clients didn't find the new region locations and in the jstack were stuck doing RPC. Eventually I killed -CONT and the client printed these:
> bq. Exception in thread "TestClient-6" java.lang.RuntimeException: org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Failed 128 actions: IOException: 90 times, SocketTimeoutException: 38 times,
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira