You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Jeff Zhang <zj...@gmail.com> on 2010/09/17 11:13:16 UTC
How to avoid exception when do heavy insertion insertion?
HI all,
I try to do heavy insertion. but meets come exception as follows, I
guess maybe I should do some optimization do avoid such kind of
exception, anyone can give some suggestion ?
Thanks in advance.
[java] 10/09/17 17:09:03 DEBUG
client.HConnectionManager$TableServers: Reloading region
dn,,1284706796459 location because regionserver didn't accept updates;
tries=3 of max=10, waiting=2000ms
[java] 10/09/17 17:09:05 DEBUG
client.HConnectionManager$TableServers: Removed dn,,1284706796459 for
tableName=dn from cache because of
f6958b43-a0e1-414e-9cf6-7e2c02d4cfa8
[java] 10/09/17 17:09:05 DEBUG
client.HConnectionManager$TableServers: Cached location for
dn,,1284706796459 is 116.211.20.208:60020
[java] 10/09/17 17:09:05 DEBUG
client.HConnectionManager$TableServers: Reloading region
dn,,1284706796459 location because regionserver didn't accept updates;
tries=4 of max=10, waiting=2000ms
[java] 10/09/17 17:09:07 DEBUG
client.HConnectionManager$TableServers: Removed dn,,1284706796459 for
tableName=dn from cache because of
f6958b43-a0e1-414e-9cf6-7e2c02d4cfa8
[java] 10/09/17 17:09:07 DEBUG
client.HConnectionManager$TableServers: Cached location for
dn,,1284706796459 is 116.211.20.208:60020
[java] 10/09/17 17:09:07 DEBUG
client.HConnectionManager$TableServers: Reloading region
dn,,1284706796459 location because regionserver didn't accept updates;
tries=5 of max=10, waiting=4000ms
[java] 10/09/17 17:09:11 DEBUG
client.HConnectionManager$TableServers: Removed dn,,1284706796459 for
tableName=dn from cache because of
f6958b43-a0e1-414e-9cf6-7e2c02d4cfa8
[java] 10/09/17 17:09:11 DEBUG
client.HConnectionManager$TableServers: Cached location for
dn,,1284706796459 is 116.211.20.208:60020
[java] 10/09/17 17:09:11 DEBUG
client.HConnectionManager$TableServers: Reloading region
dn,,1284706796459 location because regionserver didn't accept updates;
tries=6 of max=10, waiting=4000ms
[java] 10/09/17 17:09:15 DEBUG
client.HConnectionManager$TableServers: Removed dn,,1284706796459 for
tableName=dn from cache because of
f6958b43-a0e1-414e-9cf6-7e2c02d4cfa8
[java] 10/09/17 17:09:15 DEBUG
client.HConnectionManager$TableServers: Cached location for
dn,,1284706796459 is 116.211.20.208:60020
[java] 10/09/17 17:09:15 DEBUG
client.HConnectionManager$TableServers: Reloading region
dn,,1284706796459 location because regionserver didn't accept updates;
tries=7 of max=10, waiting=8000ms
[java] 10/09/17 17:09:23 DEBUG
client.HConnectionManager$TableServers: Removed dn,,1284706796459 for
tableName=dn from cache because of
f6958b43-a0e1-414e-9cf6-7e2c02d4cfa8
[java] 10/09/17 17:09:23 DEBUG
client.HConnectionManager$TableServers: Cached location for
dn,,1284706796459 is 116.211.20.208:60020
[java] 10/09/17 17:09:23 DEBUG
client.HConnectionManager$TableServers: Reloading region
dn,,1284706796459 location because regionserver didn't accept updates;
tries=8 of max=10, waiting=16000ms
[java] 10/09/17 17:09:39 DEBUG
client.HConnectionManager$TableServers: Removed dn,,1284706796459 for
tableName=dn from cache because of
f6958b43-a0e1-414e-9cf6-7e2c02d4cfa8
[java] 10/09/17 17:09:39 DEBUG
client.HConnectionManager$TableServers: Cached location for
dn,,1284706796459 is 116.211.20.208:60020
[java] Exception in thread "main"
org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to
contact region server Some server, retryOnlyOne=true, index=0,
islastrow=false, tries=9, numtries=10, i=1319, listsize=1370,
region=dn,,1284706796459 for region dn,,1284706796459, row
'f6958b43-a0e1-414e-9cf6-7e2c02d4cfa8', but failed after 10 attempts.
[java] Exceptions:
[java]
[java] at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers$Batch.process(HConnectionManager.java:1157)
[java] at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.processBatchOfRows(HConnectionManager.java:1238)
[java] at
org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:666)
[java] at org.apache.hadoop.hbase.client.HTable.put(HTable.java:510)
[java] at
com.snda.dw.dragonnest.GenerateData.main(GenerateData.java:39)
[java] 10/09/17 17:09:39 INFO zookeeper.ZooKeeper: Closing
session: 0x32b1d9af3d9000c
[java] 10/09/17 17:09:39 INFO zookeeper.ClientCnxn: Closing
ClientCnxn for session: 0x32b1d9af3d9000c
[java] 10/09/17 17:09:39 INFO zookeeper.ClientCnxn: Exception
while closing send thread for session 0x32b1d9af3d9000c : Read error
rc = -1 java.nio.DirectByteBuffer[pos=0 lim=4 cap=4]
[java] 10/09/17 17:09:39 INFO zookeeper.ClientCnxn: Disconnecting
ClientCnxn for session: 0x32b1d9af3d9000c
[java] 10/09/17 17:09:39 INFO zookeeper.ZooKeeper: Session:
0x32b1d9af3d9000c closed
[java] 10/09/17 17:09:39 DEBUG zookeeper.ZooKeeperWrapper: Closed
connection with ZooKeeper
[java] 10/09/17 17:09:39 INFO zookeeper.ClientCnxn: EventThread shut down
[java] Java Result: 1
--
Best Regards
Jeff Zhang
Re: How to avoid exception when do heavy insertion insertion?
Posted by Jean-Daniel Cryans <jd...@apache.org>.
It looks like your region took at long time to move (the name of the
region being dn,,1284706796459), you should grep the master log to see
why it took so long, and correlate with the region server log at
116.211.20.208.
Also please give the usual infos about your environment (versions,
hardware, configuration files, relevant logs from the servers, etc).
Thanks
J-D
On Fri, Sep 17, 2010 at 2:13 AM, Jeff Zhang <zj...@gmail.com> wrote:
> HI all,
>
> I try to do heavy insertion. but meets come exception as follows, I
> guess maybe I should do some optimization do avoid such kind of
> exception, anyone can give some suggestion ?
> Thanks in advance.
>
>
> [java] 10/09/17 17:09:03 DEBUG
> client.HConnectionManager$TableServers: Reloading region
> dn,,1284706796459 location because regionserver didn't accept updates;
> tries=3 of max=10, waiting=2000ms
> [java] 10/09/17 17:09:05 DEBUG
> client.HConnectionManager$TableServers: Removed dn,,1284706796459 for
> tableName=dn from cache because of
> f6958b43-a0e1-414e-9cf6-7e2c02d4cfa8
> [java] 10/09/17 17:09:05 DEBUG
> client.HConnectionManager$TableServers: Cached location for
> dn,,1284706796459 is 116.211.20.208:60020
> [java] 10/09/17 17:09:05 DEBUG
> client.HConnectionManager$TableServers: Reloading region
> dn,,1284706796459 location because regionserver didn't accept updates;
> tries=4 of max=10, waiting=2000ms
> [java] 10/09/17 17:09:07 DEBUG
> client.HConnectionManager$TableServers: Removed dn,,1284706796459 for
> tableName=dn from cache because of
> f6958b43-a0e1-414e-9cf6-7e2c02d4cfa8
> [java] 10/09/17 17:09:07 DEBUG
> client.HConnectionManager$TableServers: Cached location for
> dn,,1284706796459 is 116.211.20.208:60020
> [java] 10/09/17 17:09:07 DEBUG
> client.HConnectionManager$TableServers: Reloading region
> dn,,1284706796459 location because regionserver didn't accept updates;
> tries=5 of max=10, waiting=4000ms
> [java] 10/09/17 17:09:11 DEBUG
> client.HConnectionManager$TableServers: Removed dn,,1284706796459 for
> tableName=dn from cache because of
> f6958b43-a0e1-414e-9cf6-7e2c02d4cfa8
> [java] 10/09/17 17:09:11 DEBUG
> client.HConnectionManager$TableServers: Cached location for
> dn,,1284706796459 is 116.211.20.208:60020
> [java] 10/09/17 17:09:11 DEBUG
> client.HConnectionManager$TableServers: Reloading region
> dn,,1284706796459 location because regionserver didn't accept updates;
> tries=6 of max=10, waiting=4000ms
> [java] 10/09/17 17:09:15 DEBUG
> client.HConnectionManager$TableServers: Removed dn,,1284706796459 for
> tableName=dn from cache because of
> f6958b43-a0e1-414e-9cf6-7e2c02d4cfa8
> [java] 10/09/17 17:09:15 DEBUG
> client.HConnectionManager$TableServers: Cached location for
> dn,,1284706796459 is 116.211.20.208:60020
> [java] 10/09/17 17:09:15 DEBUG
> client.HConnectionManager$TableServers: Reloading region
> dn,,1284706796459 location because regionserver didn't accept updates;
> tries=7 of max=10, waiting=8000ms
> [java] 10/09/17 17:09:23 DEBUG
> client.HConnectionManager$TableServers: Removed dn,,1284706796459 for
> tableName=dn from cache because of
> f6958b43-a0e1-414e-9cf6-7e2c02d4cfa8
> [java] 10/09/17 17:09:23 DEBUG
> client.HConnectionManager$TableServers: Cached location for
> dn,,1284706796459 is 116.211.20.208:60020
> [java] 10/09/17 17:09:23 DEBUG
> client.HConnectionManager$TableServers: Reloading region
> dn,,1284706796459 location because regionserver didn't accept updates;
> tries=8 of max=10, waiting=16000ms
> [java] 10/09/17 17:09:39 DEBUG
> client.HConnectionManager$TableServers: Removed dn,,1284706796459 for
> tableName=dn from cache because of
> f6958b43-a0e1-414e-9cf6-7e2c02d4cfa8
> [java] 10/09/17 17:09:39 DEBUG
> client.HConnectionManager$TableServers: Cached location for
> dn,,1284706796459 is 116.211.20.208:60020
> [java] Exception in thread "main"
> org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to
> contact region server Some server, retryOnlyOne=true, index=0,
> islastrow=false, tries=9, numtries=10, i=1319, listsize=1370,
> region=dn,,1284706796459 for region dn,,1284706796459, row
> 'f6958b43-a0e1-414e-9cf6-7e2c02d4cfa8', but failed after 10 attempts.
> [java] Exceptions:
> [java]
> [java] at
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers$Batch.process(HConnectionManager.java:1157)
> [java] at
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.processBatchOfRows(HConnectionManager.java:1238)
> [java] at
> org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:666)
> [java] at org.apache.hadoop.hbase.client.HTable.put(HTable.java:510)
> [java] at
> com.snda.dw.dragonnest.GenerateData.main(GenerateData.java:39)
> [java] 10/09/17 17:09:39 INFO zookeeper.ZooKeeper: Closing
> session: 0x32b1d9af3d9000c
> [java] 10/09/17 17:09:39 INFO zookeeper.ClientCnxn: Closing
> ClientCnxn for session: 0x32b1d9af3d9000c
> [java] 10/09/17 17:09:39 INFO zookeeper.ClientCnxn: Exception
> while closing send thread for session 0x32b1d9af3d9000c : Read error
> rc = -1 java.nio.DirectByteBuffer[pos=0 lim=4 cap=4]
> [java] 10/09/17 17:09:39 INFO zookeeper.ClientCnxn: Disconnecting
> ClientCnxn for session: 0x32b1d9af3d9000c
> [java] 10/09/17 17:09:39 INFO zookeeper.ZooKeeper: Session:
> 0x32b1d9af3d9000c closed
> [java] 10/09/17 17:09:39 DEBUG zookeeper.ZooKeeperWrapper: Closed
> connection with ZooKeeper
> [java] 10/09/17 17:09:39 INFO zookeeper.ClientCnxn: EventThread shut down
> [java] Java Result: 1
>
> --
> Best Regards
>
> Jeff Zhang
>