You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@impala.apache.org by "Tim Armstrong (JIRA)" <ji...@apache.org> on 2017/10/23 16:00:04 UTC
[jira] [Resolved] (IMPALA-3960) Flaky HBase splitting on RHEL7:
NotServingRegionException
[ https://issues.apache.org/jira/browse/IMPALA-3960?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tim Armstrong resolved IMPALA-3960.
-----------------------------------
Resolution: Cannot Reproduce
Has not been touched for over a year.
> Flaky HBase splitting on RHEL7: NotServingRegionException
> ---------------------------------------------------------
>
> Key: IMPALA-3960
> URL: https://issues.apache.org/jira/browse/IMPALA-3960
> Project: IMPALA
> Issue Type: Bug
> Components: Infrastructure
> Affects Versions: Impala 2.7.0
> Reporter: Tim Armstrong
> Priority: Critical
> Labels: flaky
>
> Data loading failed on RHEL7 due to a HBase error that doesn't seem to have occurred before. Filing this JIRA mainly to track the issue to see if it happens again.
> {code}
> 16/08/08 11:13:27 INFO datagenerator.HBaseTestDataRegionAssigment: Split region 'functional_hbase.alltypesagg,,1470679241600.96b7d856cfe06ff59ce896d7be3ab3e4.' after 2 attempts.
> 16/08/08 11:13:27 INFO zookeeper.RecoverableZooKeeper: Process identifier=hbase-admin-on-hconnection-0x6d622548 connecting to ZooKeeper ensemble=localhost:2181
> 16/08/08 11:13:27 INFO zookeeper.ZooKeeper: Initiating client connection, connectString=localhost:2181 sessionTimeout=180000 watcher=hbase-admin-on-hconnection-0x6d6225480x0, quorum=localhost:2181, baseZNode=/hbase
> 16/08/08 11:13:27 INFO zookeeper.ClientCnxn: Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
> 16/08/08 11:13:27 INFO zookeeper.ClientCnxn: Socket connection established, initiating session, client: /127.0.0.1:51686, server: localhost/127.0.0.1:2181
> 16/08/08 11:13:27 INFO zookeeper.ClientCnxn: Session establishment complete on server localhost/127.0.0.1:2181, sessionid = 0x1566b43cb2d006e, negotiated timeout = 90000
> 16/08/08 11:13:27 INFO zookeeper.ZooKeeper: Session: 0x1566b43cb2d006e closed
> 16/08/08 11:13:27 INFO zookeeper.ClientCnxn: EventThread shut down
> 16/08/08 11:13:27 INFO zookeeper.RecoverableZooKeeper: Process identifier=hbase-admin-on-hconnection-0x6d622548 connecting to ZooKeeper ensemble=localhost:2181
> 16/08/08 11:13:27 INFO zookeeper.ZooKeeper: Initiating client connection, connectString=localhost:2181 sessionTimeout=180000 watcher=hbase-admin-on-hconnection-0x6d6225480x0, quorum=localhost:2181, baseZNode=/hbase
> 16/08/08 11:13:27 INFO zookeeper.ClientCnxn: Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
> 16/08/08 11:13:27 INFO zookeeper.ClientCnxn: Socket connection established, initiating session, client: /127.0.0.1:51688, server: localhost/127.0.0.1:2181
> 16/08/08 11:13:27 INFO zookeeper.ClientCnxn: Session establishment complete on server localhost/127.0.0.1:2181, sessionid = 0x1566b43cb2d006f, negotiated timeout = 90000
> 16/08/08 11:13:28 INFO zookeeper.ZooKeeper: Session: 0x1566b43cb2d006f closed
> 16/08/08 11:13:28 INFO zookeeper.ClientCnxn: EventThread shut down
> Exception in thread "main" org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: Region functional_hbase.alltypesagg,1,1470679998154.4b3310ff3337a96c257b11929ecbe9b6. is not online on localhost,16201,1470678423984
> at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionByEncodedName(HRegionServer.java:2924)
> at org.apache.hadoop.hbase.regionserver.RSRpcServices.getRegion(RSRpcServices.java:1053)
> at org.apache.hadoop.hbase.regionserver.RSRpcServices.splitRegion(RSRpcServices.java:1867)
> at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22247)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2170)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:109)
> at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:134)
> at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:109)
> at java.lang.Thread.run(Thread.java:745)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
> at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
> at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106)
> at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95)
> at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:327)
> at org.apache.hadoop.hbase.protobuf.ProtobufUtil.split(ProtobufUtil.java:1883)
> at org.apache.hadoop.hbase.client.HBaseAdmin.split(HBaseAdmin.java:2618)
> at org.apache.hadoop.hbase.client.HBaseAdmin.splitRegion(HBaseAdmin.java:2581)
> at org.apache.hadoop.hbase.client.HBaseAdmin.split(HBaseAdmin.java:2602)
> at org.apache.hadoop.hbase.client.HBaseAdmin.split(HBaseAdmin.java:2591)
> at com.cloudera.impala.datagenerator.HBaseTestDataRegionAssigment.performAssigment(HBaseTestDataRegionAssigment.java:97)
> at com.cloudera.impala.datagenerator.HBaseTestDataRegionAssigment.main(HBaseTestDataRegionAssigment.java:297)
> Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.NotServingRegionException): org.apache.hadoop.hbase.NotServingRegionException: Region functional_hbase.alltypesagg,1,1470679998154.4b3310ff3337a96c257b11929ecbe9b6. is not online on localhost,16201,1470678423984
> at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionByEncodedName(HRegionServer.java:2924)
> at org.apache.hadoop.hbase.regionserver.RSRpcServices.getRegion(RSRpcServices.java:1053)
> at org.apache.hadoop.hbase.regionserver.RSRpcServices.splitRegion(RSRpcServices.java:1867)
> at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22247)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2170)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:109)
> at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:134)
> at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:109)
> at java.lang.Thread.run(Thread.java:745)
> at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1269)
> at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:226)
> at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:331)
> at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.splitRegion(AdminProtos.java:23173)
> at org.apache.hadoop.hbase.protobuf.ProtobufUtil.split(ProtobufUtil.java:1881)
> ... 6 more
> MainThread: Found 3 impalad/1 statestored/1 catalogd process(es)
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)