You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@kylin.apache.org by 蒋慧明 <hm...@samsung.com> on 2019/05/17 07:13:03 UTC

KYLIN build cube problem

Dear

  When I tried to build cube for one day, the report error is different.

  Sometimes, the error is :

RpcRetryingCaller{globalStartTime=1558076503569, pause=100, retries=1},
java.io.IOException: Call to [HBASE IPXX.XX.XX.XXX:16020]  failed on local
exception: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=3436827,
waitTime=9999, operationTimeout=9998 expired.  
  
---  
  


Sometimes following error occurs:

org.apache.hadoop.hbase.NotServingRegionException:
org.apache.hadoop.hbase.NotServingRegionException: Region kylin_metadata is
not online on ip-XX.XX.XX.XX,16020,1555901919971  
  
---  
  


Does anyone know about this problem? Where is the timeout-9998 configured? Why
is it not online?

Many thanks!

![](cid:9INH0GAN05W5@namo.co.kr)  
  
---  
  

  

  

![](cid:XOK0LK7CT9SZ@namo.co.kr)  
  
---  
![](http://ext.samsung.net/mail/ext/v1/external/status/update?userid=hm.jiang&do=bWFpbElEPTIwMTkwNTE3MDcxMzAzZXBjbXM1cDZlNTI3YjMzYjUxMGM5MzhhM2Q2MjRmNTE3MzYyYmI0YiZyZWNpcGllbnRBZGRyZXNzPXVzZXJAa3lsaW4uYXBhY2hlLm9yZw__)

Re: Error: Region is not online

Posted by ShaoFeng Shi <sh...@apache.org>.
Hi Huiming,

Have you recovered from this bad situation? When an RS is down, HBase needs
to take a while to identify the bad RS and then move its data to another
RS. How long this will take will depends on several factors. The following
post is discussing it.

https://stackoverflow.com/questions/36579219/how-long-hbase-need-to-take-for-recovering-one-crashed-regionserver

The port may be different if RS found the default port is not available. So
you may need double check HBase's configuration.

Please share with us if you have further information.

Best regards,

Shaofeng Shi 史少锋
Apache Kylin PMC
Email: shaofengshi@apache.org

Apache Kylin FAQ: https://kylin.apache.org/docs/gettingstarted/faq.html
Join Kylin user mail group: user-subscribe@kylin.apache.org
Join Kylin dev mail group: dev-subscribe@kylin.apache.org




蒋慧明 <hm...@samsung.com> 于2019年5月23日周四 下午2:29写道：

> Dear
>
>   When I tried to run a job, it report error like this:
>
>
>  org.apache.kylin.engine.mr.exception.HadoopShellException:
> java.lang.RuntimeException:
> org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after
> attempts=1, exceptions:
> Thu May 23 14:21:30 GMT+08:00 2019,
> RpcRetryingCaller{globalStartTime=1558592490643, pause=100, retries=1},
> org.apache.hadoop.hbase.NotServingRegionException:
> org.apache.hadoop.hbase.NotServingRegionException: Region R1 is not
> online on RS1,16020
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionByEncodedName(HRegionServer.java:3008)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.getRegion(RSRpcServices.java:1144)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.newRegionScanner(RSRpcServices.java:2476)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:2757)
> at
> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:34950)
>
>
>
> From HBASE Webui, region R1 is actually located at RS2 (16030).
>
>
>
> Anyone know about：
>
> 1. Why Kylin request with the wrong region server? How to fixed it?
>
> 2. Why the requested port 16020 is different with Hbase port 16030? Is it
> normal?
>
>
>
> Thanks a lot!
>
>
>
>>
>>
>>
>>
>
>
>

Error: Region is not online

Posted by 蒋慧明 <hm...@samsung.com>.
Dear

  When I tried to run a job, it report error like this:  


 org.apache.kylin.engine.mr.exception.HadoopShellException:
java.lang.RuntimeException:
org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after
attempts=1, exceptions:  
Thu May 23 14:21:30 GMT+08:00 2019,
RpcRetryingCaller{globalStartTime=1558592490643, pause=100, retries=1},
org.apache.hadoop.hbase.NotServingRegionException:
org.apache.hadoop.hbase.NotServingRegionException: Region R1 is not online on
RS1,16020  
at
org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionByEncodedName(HRegionServer.java:3008)  
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.getRegion(RSRpcServices.java:1144)  
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.newRegionScanner(RSRpcServices.java:2476)  
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:2757)  
at
org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:34950)  
  
---  
  


From HBASE Webui, region R1 is actually located at RS2 (16030).



Anyone know about：

1\. Why Kylin request with the wrong region server? How to fixed it?

2\. Why the requested port 16020 is different with Hbase port 16030? Is it
normal?



Thanks a lot!



> ![](cid:OC322OJWTEAC@namo.co.kr)  
>  
> ---  
>  
>

>

>  
>

>

>  
>

>

> ![](cid:93BIV0UVOUUB@namo.co.kr)  
>  
> ---  
  
  

  

  

![](cid:XOK0LK7CT9SZ@namo.co.kr)  
  
---  
![](http://ext.samsung.net/mail/ext/v1/external/status/update?userid=hm.jiang&do=bWFpbElEPTIwMTkwNTIzMDYyOTQxZXBjbXM1cDM0YTljNzg3ZmQxZGQ3ZmY5MGNiMzBhODgzNWVjYWVkOSZyZWNpcGllbnRBZGRyZXNzPXVzZXJAa3lsaW4uYXBhY2hlLm9yZw__)

RE: Re: KYLIN build cube problem

Posted by 蒋慧明 <hm...@samsung.com>.
Hello JiaTao

  Thanks a lot for suggestion.

  Before running this job, "kylin.metadata.hbase-rpc-timeout" have already
been set to 50000. But operationTimeout is 9998, not 50000. It's strange





\--------- **Original Message** \---------

**Sender** : JiaTao Tao  <ta...@gmail.com>

**Date** : 2019-05-17 21:02 (GMT+9)

**Title** : Re: KYLIN build cube problem



Hi

  

You can try to adjust "kylin.metadata.hbase-rpc-timeout" to a larger value.
And then run metadata/StorageCleanup.

  

\--  

  

Regards!

Aron Tao

  

蒋慧明 <[hm.jiang@samsung.com](mailto:hm.jiang@samsung.com)> 于2019年5月17日周五
上午7:19写道：  

> Dear

>

>   When I tried to build cube for one day, the report error is different.

>

>   Sometimes, the error is :

>

> RpcRetryingCaller{globalStartTime=1558076503569, pause=100, retries=1},
java.io.IOException: Call to [HBASE IPXX.XX.XX.XXX:16020]  failed on local
exception: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=3436827,
waitTime=9999, operationTimeout=9998 expired.  
>  
> ---  
>  
> Sometimes following error occurs:

>

> org.apache.hadoop.hbase.NotServingRegionException:
org.apache.hadoop.hbase.NotServingRegionException: Region kylin_metadata is
not online on ip-XX.XX.XX.XX,16020,1555901919971  
>  
> ---  
>  
>  
>

> Does anyone know about this problem? Where is the timeout-9998 configured?
Why is it not online?

>

> Many thanks!

>

> ![](cid:XOK0LK7CT9SZ@namo.co.kr)  
>  
> ---  
>  
>

>

>  
>

>

>  
>

>

> ![](cid:LP7KBSL8PYMC@namo.co.kr)  
>  
> ---  
  
  

  

  

![](cid:20190522022817_0@epcms5p)  
  
---  
![](http://ext.samsung.net/mail/ext/v1/external/status/update?userid=hm.jiang&do=bWFpbElEPTIwMTkwNTIyMDIyODE3ZXBjbXM1cDM2OWQxYjU0MTgzMmI3MzE2ZjZmZDJiYjZhNTRiNzgwOCZyZWNpcGllbnRBZGRyZXNzPXVzZXJAa3lsaW4uYXBhY2hlLm9yZw__)

Re: KYLIN build cube problem

Posted by JiaTao Tao <ta...@gmail.com>.
Hi

You can try to adjust "kylin.metadata.hbase-rpc-timeout" to a larger value.
And then run metadata/StorageCleanup.

-- 


Regards!

Aron Tao

蒋慧明 <hm...@samsung.com> 于2019年5月17日周五 上午7:19写道：

> Dear
>
>   When I tried to build cube for one day, the report error is different.
>
>   Sometimes, the error is :
>
> RpcRetryingCaller{globalStartTime=1558076503569, pause=100, retries=1},
> java.io.IOException: Call to [HBASE IPXX.XX.XX.XXX:16020]  failed on local
> exception: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call
> id=3436827, waitTime=9999, operationTimeout=9998 expired.
>
>
>
> Sometimes following error occurs:
>
> org.apache.hadoop.hbase.NotServingRegionException:
> org.apache.hadoop.hbase.NotServingRegionException: Region kylin_metadata is
> not online on ip-XX.XX.XX.XX,16020,1555901919971
>
>
>
> Does anyone know about this problem? Where is the timeout-9998 configured?
> Why is it not online?
>
> Many thanks!
>
>
>
>
>