You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kylin.apache.org by rahulsingh <ra...@i-link.co.in> on 2017/03/14 13:37:15 UTC
Hbase Region server goes down
Hi all,
I am facing this exception while building the cube. Whenever I build the
cube the region server goes down. I am not understanding this exception.
Please explain is anyone know about this exception.
Thank You,
Rahul Singh
java.net.SocketTimeoutException: callTimeout=60000, callDuration=87372: row
'/execute_output/266df361-5b32-4007-8164-4d8c1972616f-02' on table
'kylin_metadata' at
region=kylin_metadata,,1487943010804.31525a8d9285b5a15424c6f698838c10.,
hostname=hrdn3,60020,1489490883750, seqNum=14429
at
org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:159)
at org.apache.hadoop.hbase.client.HTable.get(HTable.java:935)
at org.apache.hadoop.hbase.client.HTable.get(HTable.java:901)
at
org.apache.kylin.storage.hbase.HBaseResourceStore.internalGetFromHTable(HBaseResourceStore.java:332)
at
org.apache.kylin.storage.hbase.HBaseResourceStore.getFromHTable(HBaseResourceStore.java:312)
at
org.apache.kylin.storage.hbase.HBaseResourceStore.getResourceImpl(HBaseResourceStore.java:224)
at
org.apache.kylin.common.persistence.ResourceStore.getResource(ResourceStore.java:140)
at
org.apache.kylin.job.dao.ExecutableDao.readJobOutputResource(ExecutableDao.java:93)
at
org.apache.kylin.job.dao.ExecutableDao.getJobOutput(ExecutableDao.java:186)
at
org.apache.kylin.job.manager.ExecutableManager.getOutput(ExecutableManager.java:123)
at
org.apache.kylin.job.execution.AbstractExecutable.isDiscarded(AbstractExecutable.java:369)
at
org.apache.kylin.engine.mr.common.MapReduceExecutable.doWork(MapReduceExecutable.java:148)
at
org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:113)
at
org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:57)
at
org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:113)
at
org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:136)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.hadoop.hbase.ipc.FailedServerException: This server is
in the failed servers list: HRDN1/192.168.101.34:60020
at
org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.setupIOstreams(RpcClientImpl.java:708)
at
org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.writeRequest(RpcClientImpl.java:907)
at
org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.tracedWriteRequest(RpcClientImpl.java:874)
at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1243)
at
org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227)
at
org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336)
at
org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$BlockingStub.get(ClientProtos.java:34070)
at
org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRowOrBefore(ProtobufUtil.java:1589)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegionInMeta(ConnectionManager.java:1398)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1199)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.relocateRegion(ConnectionManager.java:1166)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.relocateRegion(ConnectionManager.java:1150)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.getRegionLocation(ConnectionManager.java:971)
at
org.apache.hadoop.hbase.client.HRegionLocator.getRegionLocation(HRegionLocator.java:83)
at
org.apache.hadoop.hbase.client.RegionServerCallable.prepare(RegionServerCallable.java:79)
at
org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:124)
... 18 more
--
View this message in context: http://apache-kylin.74782.x6.nabble.com/Hbase-Region-server-goes-down-tp7437.html
Sent from the Apache Kylin mailing list archive at Nabble.com.
Re: Hbase Region server goes down
Posted by rahulsingh <ra...@i-link.co.in>.
Yes. My HBase RS is running on same node as YARN NodeManager is running.
But I have'nt faced this issue on previously builded cube on same cluster.
Thank You
--
View this message in context: http://apache-kylin.74782.x6.nabble.com/Hbase-Region-server-goes-down-tp7437p7441.html
Sent from the Apache Kylin mailing list archive at Nabble.com.
Re: Hbase Region server goes down
Posted by ShaoFeng Shi <sh...@apache.org>.
If HBase RS is running together with YARN NodeManager, when building Cube,
MR may impact HBase's stability.
To ensure the query's availability and performance, it is recommended to
separate HBase RS and NodeManager; further more you can setup a dedicated
HBase cluster for query, check this:
https://kylin.apache.org/blog/2016/06/10/standalone-hbase-cluster/
2017-03-14 21:37 GMT+08:00 rahulsingh <ra...@i-link.co.in>:
> Hi all,
>
> I am facing this exception while building the cube. Whenever I build the
> cube the region server goes down. I am not understanding this exception.
> Please explain is anyone know about this exception.
>
> Thank You,
> Rahul Singh
>
> java.net.SocketTimeoutException: callTimeout=60000, callDuration=87372:
> row
> '/execute_output/266df361-5b32-4007-8164-4d8c1972616f-02' on table
> 'kylin_metadata' at
> region=kylin_metadata,,1487943010804.31525a8d9285b5a15424c6f698838c10.,
> hostname=hrdn3,60020,1489490883750, seqNum=14429
> at
> org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(
> RpcRetryingCaller.java:159)
> at org.apache.hadoop.hbase.client.HTable.get(HTable.java:935)
> at org.apache.hadoop.hbase.client.HTable.get(HTable.java:901)
> at
> org.apache.kylin.storage.hbase.HBaseResourceStore.internalGetFromHTable(
> HBaseResourceStore.java:332)
> at
> org.apache.kylin.storage.hbase.HBaseResourceStore.getFromHTable(
> HBaseResourceStore.java:312)
> at
> org.apache.kylin.storage.hbase.HBaseResourceStore.getResourceImpl(
> HBaseResourceStore.java:224)
> at
> org.apache.kylin.common.persistence.ResourceStore.
> getResource(ResourceStore.java:140)
> at
> org.apache.kylin.job.dao.ExecutableDao.readJobOutputResource(
> ExecutableDao.java:93)
> at
> org.apache.kylin.job.dao.ExecutableDao.getJobOutput(
> ExecutableDao.java:186)
> at
> org.apache.kylin.job.manager.ExecutableManager.getOutput(
> ExecutableManager.java:123)
> at
> org.apache.kylin.job.execution.AbstractExecutable.isDiscarded(
> AbstractExecutable.java:369)
> at
> org.apache.kylin.engine.mr.common.MapReduceExecutable.
> doWork(MapReduceExecutable.java:148)
> at
> org.apache.kylin.job.execution.AbstractExecutable.
> execute(AbstractExecutable.java:113)
> at
> org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(
> DefaultChainedExecutable.java:57)
> at
> org.apache.kylin.job.execution.AbstractExecutable.
> execute(AbstractExecutable.java:113)
> at
> org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(
> DefaultScheduler.java:136)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.apache.hadoop.hbase.ipc.FailedServerException: This server
> is
> in the failed servers list: HRDN1/192.168.101.34:60020
> at
> org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.
> setupIOstreams(RpcClientImpl.java:708)
> at
> org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.
> writeRequest(RpcClientImpl.java:907)
> at
> org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.tracedWriteRequest(
> RpcClientImpl.java:874)
> at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(
> RpcClientImpl.java:1243)
> at
> org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(
> AbstractRpcClient.java:227)
> at
> org.apache.hadoop.hbase.ipc.AbstractRpcClient$
> BlockingRpcChannelImplementation.callBlockingMethod(
> AbstractRpcClient.java:336)
> at
> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$
> BlockingStub.get(ClientProtos.java:34070)
> at
> org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRowOrBefore(ProtobufUtil.
> java:1589)
> at
> org.apache.hadoop.hbase.client.ConnectionManager$
> HConnectionImplementation.locateRegionInMeta(ConnectionManager.java:1398)
> at
> org.apache.hadoop.hbase.client.ConnectionManager$
> HConnectionImplementation.locateRegion(ConnectionManager.java:1199)
> at
> org.apache.hadoop.hbase.client.ConnectionManager$
> HConnectionImplementation.relocateRegion(ConnectionManager.java:1166)
> at
> org.apache.hadoop.hbase.client.ConnectionManager$
> HConnectionImplementation.relocateRegion(ConnectionManager.java:1150)
> at
> org.apache.hadoop.hbase.client.ConnectionManager$
> HConnectionImplementation.getRegionLocation(ConnectionManager.java:971)
> at
> org.apache.hadoop.hbase.client.HRegionLocator.getRegionLocation(
> HRegionLocator.java:83)
> at
> org.apache.hadoop.hbase.client.RegionServerCallable.
> prepare(RegionServerCallable.java:79)
> at
> org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(
> RpcRetryingCaller.java:124)
> ... 18 more
>
> --
> View this message in context: http://apache-kylin.74782.x6.
> nabble.com/Hbase-Region-server-goes-down-tp7437.html
> Sent from the Apache Kylin mailing list archive at Nabble.com.
>
--
Best regards,
Shaofeng Shi 史少锋