Regarding hbase no route error

[Krishna Bandaru <Kr...@scanbuy.com>]

Hi,

I created the Kylin cluster using EMR it ran well for 1 month recently I am getting below error very frequently.

Please help me to solve this.


2019-07-09 18:15:20,827 INFO  [main-SendThread(ip-10-0-0-56.ec2.internal:2181)] zookeeper.ClientCnxn:1303 : Session establishment complete on server ip-10-0-0-56.ec2.internal/10.0.0.56:2181, sessionid = 0x100000417370043, negotiated timeout = 40000
2019-07-09 18:15:26,266 DEBUG [main] hbase.HBaseConnection:306 : HTable 'kylin_metadata' already exists
Exception in thread "main" java.lang.IllegalArgumentException: Failed to find metadata store by url: kylin_metadata@hbase
        at org.apache.kylin.common.persistence.ResourceStore.createResourceStore(ResourceStore.java:98)
        at org.apache.kylin.common.persistence.ResourceStore.getStore(ResourceStore.java:110)
        at org.apache.kylin.rest.service.AclTableMigrationTool.checkIfNeedMigrate(AclTableMigrationTool.java:98)
        at org.apache.kylin.tool.AclTableMigrationCLI.main(AclTableMigrationCLI.java:41)
Caused by: org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after attempts=1, exceptions:
Tue Jul 09 18:15:29 UTC 2019, RpcRetryingCaller{globalStartTime=1562696126690, pause=100, retries=1}, java.io.IOException: Call to ip-10-0-0-38.ec2.internal/10.0.0.38:16020 failed on local exception: java.net.NoRouteToHostException: No route to host

        at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:164)
        at org.apache.hadoop.hbase.client.HTable.get(HTable.java:867)
        at org.apache.hadoop.hbase.client.HTable.get(HTable.java:832)
        at org.apache.kylin.storage.hbase.HBaseResourceStore.internalGetFromHTable(HBaseResourceStore.java:394)
        at org.apache.kylin.storage.hbase.HBaseResourceStore.getFromHTable(HBaseResourceStore.java:372)
        at org.apache.kylin.storage.hbase.HBaseResourceStore.existsImpl(HBaseResourceStore.java:117)
        at org.apache.kylin.common.persistence.ResourceStore.exists(ResourceStore.java:165)
        at org.apache.kylin.common.persistence.ResourceStore.createResourceStore(ResourceStore.java:93)
        ... 3 more
Caused by: java.io.IOException: Call to ip-10-0-0-38.ec2.internal/10.0.0.38:16020 failed on local exception: java.net.NoRouteToHostException: No route to host
        at org.apache.hadoop.hbase.ipc.IPCUtil.wrapException(IPCUtil.java:180)
        at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:389)
        at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:94)
        at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:409)
        at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:405)
        at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:103)
        at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:118)
        at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callMethod(AbstractRpcClient.java:422)
        at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:327)
        at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$200(AbstractRpcClient.java:94)
        at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:571)
        at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$BlockingStub.get(ClientProtos.java:37035)
        at org.apache.hadoop.hbase.client.HTable$3.call(HTable.java:859)
        at org.apache.hadoop.hbase.client.HTable$3.call(HTable.java:849)
        at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:142)
        ... 10 more
Caused by: java.net.NoRouteToHostException: No route to host
        at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
        at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
        at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
        at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531)
        at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:495)
        at org.apache.hadoop.hbase.ipc.BlockingRpcConnection.setupConnection(BlockingRpcConnection.java:256)
        at org.apache.hadoop.hbase.ipc.BlockingRpcConnection.setupIOstreams(BlockingRpcConnection.java:437)
        at org.apache.hadoop.hbase.ipc.BlockingRpcConnection.writeRequest(BlockingRpcConnection.java:540)
        at org.apache.hadoop.hbase.ipc.BlockingRpcConnection.tracedWriteRequest(BlockingRpcConnection.java:520)
        at org.apache.hadoop.hbase.ipc.BlockingRpcConnection.access$200(BlockingRpcConnection.java:85)
        at org.apache.hadoop.hbase.ipc.BlockingRpcConnection$4.run(BlockingRpcConnection.java:724)
        at org.apache.hadoop.hbase.ipc.HBaseRpcControllerImpl.notifyOnCancel(HBaseRpcControllerImpl.java:240)
        at org.apache.hadoop.hbase.ipc.BlockingRpcConnection.sendRequest(BlockingRpcConnection.java:699)
        at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callMethod(AbstractRpcClient.java:420)
        ... 17 more
2019-07-09 18:15:29,800 INFO  [close-hbase-conn] hbase.HBaseConnection:136 : Closing HBase connections...
2019-07-09 18:15:29,800 INFO  [close-hbase-conn] client.ConnectionManager$HConnectionImplementation:2259 : Closing master protocol: MasterService
2019-07-09 18:15:29,801 INFO  [close-hbase-conn] client.ConnectionManager$HConnectionImplementation:1776 : Closing zookeeper sessionid=0x100000417370043
2019-07-09 18:15:29,804 INFO  [close-hbase-conn] zookeeper.ZooKeeper:693 : Session: 0x100000417370043 closed


Thanks,
krishna