You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Xiong LIU <li...@gmail.com> on 2013/06/24 13:06:36 UTC
hbase.client.ipc.pool.size problem with hbase 0.95.1
Hi, all,
when I configure ipc pool in hbase-site.xml as follows:
<property>
<name>hbase.client.ipc.pool.type</name>
<value>RoundRobinPool</value>
</property>
<property>
<name>hbase.client.ipc.pool.size</name>
<value>1</value>
</property>
the start and stop processes of hbase run without any problem.
But if I set hbase.client.ipc.pool.size to 10 (20, 30, or possibly some
other values), some problems occur.
In the regionserver's log, I found that the regionserver continuously
attempted connect to master server:
2013-06-24 16:59:11,734 INFO [regionserver60020]
regionserver.HRegionServer: Attempting connect to Master server
at HOSTNAME-TEST,60000,1372064256847
2013-06-24 16:59:14,738 INFO [regionserver60020]
regionserver.HRegionServer: Attempting connect to Master server
at HOSTNAME-TEST,60000,1372064256847
2013-06-24 16:59:17,741 INFO [regionserver60020]
regionserver.HRegionServer: Attempting connect to Master server
at HOSTNAME-TEST,60000,1372064256847
2013-06-24 16:59:20,744 INFO [regionserver60020]
regionserver.HRegionServer: Attempting connect to Master server
at HOSTNAME-TEST,60000,1372064256847
......many more
About 30 minutes after the master started, in the master's log,
client.ServerCallable began to throw the following exception repeatedly:
2013-06-24 17:32:47,233 WARN
[HOSTNAME-TEST,60000,1372064256847-CatalogJanitor] client.ServerCallable:
Call exception, tries=0, numRetries=200
java.io.IOException: Call to HOSTNAME-TEST/192.168.1.35:60020 failed on
local exception: java.io.IOException: Call 31 not added as the
connection HOSTNAME-TEST/
192.168.1.35:60020/ClientService/hadoop(auth:SIMPLE)/600000 is closing
at
org.apache.hadoop.hbase.ipc.RpcClient.wrapException(RpcClient.java:1368)
at org.apache.hadoop.hbase.ipc.RpcClient.call(RpcClient.java:1340)
at
org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1540)
at
org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation.callBlockingMethod(RpcClient.java:1597)
at
org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$BlockingStub.scan(ClientProtos.java:21367)
at
org.apache.hadoop.hbase.client.ScannerCallable.openScanner(ScannerCallable.java:290)
at
org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:147)
at
org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:55)
at
org.apache.hadoop.hbase.client.ServerCallable.withRetries(ServerCallable.java:174)
at
org.apache.hadoop.hbase.client.ClientScanner.nextScanner(ClientScanner.java:216)
at
org.apache.hadoop.hbase.client.ClientScanner.<init>(ClientScanner.java:131)
at org.apache.hadoop.hbase.client.HTable.getScanner(HTable.java:585)
at
org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:164)
at
org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:102)
at
org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:80)
at
org.apache.hadoop.hbase.master.CatalogJanitor.getMergedRegionsAndSplitParents(CatalogJanitor.java:165)
at
org.apache.hadoop.hbase.master.CatalogJanitor.getMergedRegionsAndSplitParents(CatalogJanitor.java:117)
at
org.apache.hadoop.hbase.master.CatalogJanitor.scan(CatalogJanitor.java:219)
at
org.apache.hadoop.hbase.master.CatalogJanitor.chore(CatalogJanitor.java:99)
at org.apache.hadoop.hbase.Chore.run(Chore.java:80)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.io.IOException: Call 31 not added as the
connection HOSTNAME-TEST/192.168.1.35:60020/ClientService/hadoop(auth:SIMPLE)/600000
is closing
at
org.apache.hadoop.hbase.ipc.RpcClient$Connection.addCall(RpcClient.java:469)
at
org.apache.hadoop.hbase.ipc.RpcClient.getConnection(RpcClient.java:1413)
at org.apache.hadoop.hbase.ipc.RpcClient.call(RpcClient.java:1314)
... 19 more
So I want to know what is going wrong?
Extra information:
hbase issue: https://issues.apache.org/jira/browse/HBASE-8790
environment: CentOS 5.9 x86_64, java version "1.6.0_45", CDH4.3
The hbase cluster is a fresh start with only one regionserver.
----------------------------
Best Wishes
Re: hbase.client.ipc.pool.size problem with hbase 0.95.1
Posted by Snake <sn...@gmail.com>.
Finally, I found that this is the same issue as https://issues.apache.org/jira/browse/HBASE-7685. I don't know why RpcClient.java still use the code which mentioned in 7685 for connection close method. Is there anyone can tell me the reason?
在 2013-6-24，下午7:06， Xiong LIU 写道：
> Hi, all,
> when I configure ipc pool in hbase-site.xml as follows:
>
> <property>
> <name>hbase.client.ipc.pool.type</name>
> <value>RoundRobinPool</value>
> </property>
> <property>
> <name>hbase.client.ipc.pool.size</name>
> <value>1</value>
> </property>
>
> the start and stop processes of hbase run without any problem.
>
> But if I set hbase.client.ipc.pool.size to 10 (20, 30, or possibly some
> other values), some problems occur.
>
> In the regionserver's log, I found that the regionserver continuously
> attempted connect to master server:
>
> 2013-06-24 16:59:11,734 INFO [regionserver60020]
> regionserver.HRegionServer: Attempting connect to Master server
> at HOSTNAME-TEST,60000,1372064256847
> 2013-06-24 16:59:14,738 INFO [regionserver60020]
> regionserver.HRegionServer: Attempting connect to Master server
> at HOSTNAME-TEST,60000,1372064256847
> 2013-06-24 16:59:17,741 INFO [regionserver60020]
> regionserver.HRegionServer: Attempting connect to Master server
> at HOSTNAME-TEST,60000,1372064256847
> 2013-06-24 16:59:20,744 INFO [regionserver60020]
> regionserver.HRegionServer: Attempting connect to Master server
> at HOSTNAME-TEST,60000,1372064256847
> ......many more
>
> About 30 minutes after the master started, in the master's log,
> client.ServerCallable began to throw the following exception repeatedly:
>
> 2013-06-24 17:32:47,233 WARN
> [HOSTNAME-TEST,60000,1372064256847-CatalogJanitor] client.ServerCallable:
> Call exception, tries=0, numRetries=200
> java.io.IOException: Call to HOSTNAME-TEST/192.168.1.35:60020 failed on
> local exception: java.io.IOException: Call 31 not added as the
> connection HOSTNAME-TEST/
> 192.168.1.35:60020/ClientService/hadoop(auth:SIMPLE)/600000 is closing
> at
> org.apache.hadoop.hbase.ipc.RpcClient.wrapException(RpcClient.java:1368)
> at org.apache.hadoop.hbase.ipc.RpcClient.call(RpcClient.java:1340)
> at
> org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1540)
> at
> org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation.callBlockingMethod(RpcClient.java:1597)
> at
> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$BlockingStub.scan(ClientProtos.java:21367)
> at
> org.apache.hadoop.hbase.client.ScannerCallable.openScanner(ScannerCallable.java:290)
> at
> org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:147)
> at
> org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:55)
> at
> org.apache.hadoop.hbase.client.ServerCallable.withRetries(ServerCallable.java:174)
> at
> org.apache.hadoop.hbase.client.ClientScanner.nextScanner(ClientScanner.java:216)
> at
> org.apache.hadoop.hbase.client.ClientScanner.<init>(ClientScanner.java:131)
> at org.apache.hadoop.hbase.client.HTable.getScanner(HTable.java:585)
> at
> org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:164)
> at
> org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:102)
> at
> org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:80)
> at
> org.apache.hadoop.hbase.master.CatalogJanitor.getMergedRegionsAndSplitParents(CatalogJanitor.java:165)
> at
> org.apache.hadoop.hbase.master.CatalogJanitor.getMergedRegionsAndSplitParents(CatalogJanitor.java:117)
> at
> org.apache.hadoop.hbase.master.CatalogJanitor.scan(CatalogJanitor.java:219)
> at
> org.apache.hadoop.hbase.master.CatalogJanitor.chore(CatalogJanitor.java:99)
> at org.apache.hadoop.hbase.Chore.run(Chore.java:80)
> at java.lang.Thread.run(Thread.java:662)
> Caused by: java.io.IOException: Call 31 not added as the
> connection HOSTNAME-TEST/192.168.1.35:60020/ClientService/hadoop(auth:SIMPLE)/600000
> is closing
> at
> org.apache.hadoop.hbase.ipc.RpcClient$Connection.addCall(RpcClient.java:469)
> at
> org.apache.hadoop.hbase.ipc.RpcClient.getConnection(RpcClient.java:1413)
> at org.apache.hadoop.hbase.ipc.RpcClient.call(RpcClient.java:1314)
> ... 19 more
>
> So I want to know what is going wrong?
>
> Extra information:
> hbase issue: https://issues.apache.org/jira/browse/HBASE-8790
> environment: CentOS 5.9 x86_64, java version "1.6.0_45", CDH4.3
> The hbase cluster is a fresh start with only one regionserver.
>
>
> ----------------------------
> Best Wishes