You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by 聪聪 <17...@qq.com> on 2016/12/22 02:13:44 UTC
in hdfs, part of the last contact value increases
Recently, I have increased two disks on a single datanode . I encountered a problem, in hdfs, part of the last contact value increases, resulting in hbase the handler is occupied.
The datanode prints the log as follows:
2016-12-21 10:13:50,816 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: RemoteException in offerService
org.apache.hadoop.ipc.RemoteException(java.lang.NullPointerException): java.lang.NullPointerException
at org.apache.hadoop.ipc.Client.call(Client.java:1411)
at org.apache.hadoop.ipc.Client.call(Client.java:1364)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
at com.sun.proxy.$Proxy14.blockReport(Unknown Source)
at org.apache.hadoop.hdfs.protocolPB.DatanodeProtocolClientSideTranslatorPB.blockReport(DatanodeProtocolClientSideTranslatorPB.java:175)
at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.blockReport(BPServiceActor.java:488)
at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.offerService(BPServiceActor.java:711)
at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:846)
at java.lang.Thread.run(Thread.java:744)
2016-12-21 10:13:51,256 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /10.90.18.60:50010, dest: /10.90.18.60:39095, bytes: 24894792, op: HDFS_READ, cliID: DFSClient_hb_rs_l-hbase40.dba.cn2.qunar.com,60020,1473253709668_1716681090_37, offset: 0, srvID: b7cb495e-de56-4f4e-b089-abe20a3e0e53, blockid: BP-329468547-10.90.18.19-1444480193809:blk_1238808018_165078752, duration: 1214526037
2016-12-21 10:13:51,987 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: RemoteException in offerService
org.apache.hadoop.ipc.RemoteException(java.lang.NullPointerException): java.lang.NullPointerException
at org.apache.hadoop.ipc.Client.call(Client.java:1411)
at org.apache.hadoop.ipc.Client.call(Client.java:1364)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
at com.sun.proxy.$Proxy14.blockReport(Unknown Source)
at org.apache.hadoop.hdfs.protocolPB.DatanodeProtocolClientSideTranslatorPB.blockReport(DatanodeProtocolClientSideTranslatorPB.java:175)
at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.blockReport(BPServiceActor.java:488)
at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.offerService(BPServiceActor.java:711)
at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:846)
at java.lang.Thread.run(Thread.java:744)
Use jstack print out the contents of regionserver as follows:
"B.defaultRpcServer.handler=62,queue=2,port=60020" daemon prio=10 tid=0x0000000000e91800 nid=0x63b8 in Object.wait() [0x00007f9b83f43000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:503)
at org.apache.hadoop.hbase.regionserver.wal.FSHLog.syncer(FSHLog.java:1371)
- locked <0x00007f9cf1334468> (a java.util.concurrent.atomic.AtomicLong)
at org.apache.hadoop.hbase.regionserver.wal.FSHLog.sync(FSHLog.java:1485)
at org.apache.hadoop.hbase.regionserver.HRegion.syncOrDefer(HRegion.java:5968)
at org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutation(HRegion.java:2580)
at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2261)
at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2213)
at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2217)
at org.apache.hadoop.hbase.regionserver.HRegionServer.doBatchOp(HRegionServer.java:4257)
at org.apache.hadoop.hbase.regionserver.HRegionServer.doNonAtomicRegionMutation(HRegionServer.java:3459)
at org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3348)
at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:29593)
at qunar.tc.hbase.interceptor.support.CustomCompositeInterceptor.callMethod(CustomCompositeInterceptor.java:76)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2008)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:108)
at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:114)
at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:94)
at java.lang.Thread.run(Thread.java:744)
"B.defaultRpcServer.handler=61,queue=1,port=60020" daemon prio=10 tid=0x0000000000e8f800 nid=0x63b7 in Object.wait() [0x00007f9b84044000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:503)
at org.apache.hadoop.hbase.regionserver.wal.FSHLog.syncer(FSHLog.java:1371)
- locked <0x00007f9cf1334468> (a java.util.concurrent.atomic.AtomicLong)
at org.apache.hadoop.hbase.regionserver.wal.FSHLog.sync(FSHLog.java:1485)
at org.apache.hadoop.hbase.regionserver.HRegion.syncOrDefer(HRegion.java:5968)
at org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutation(HRegion.java:2580)
at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2261)
at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2213)
at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2217)
at org.apache.hadoop.hbase.regionserver.HRegionServer.doBatchOp(HRegionServer.java:4257)
at org.apache.hadoop.hbase.regionserver.HRegionServer.doNonAtomicRegionMutation(HRegionServer.java:3459)
at org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3348)
at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:29593)
at qunar.tc.hbase.interceptor.support.CustomCompositeInterceptor.callMethod(CustomCompositeInterceptor.java:76)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2008)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:108)
at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:114)
at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:94)
at java.lang.Thread.run(Thread.java:744)
I would like to ask what is the specific reason, and how to optimize.