You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Dru Jensen <dr...@gmail.com> on 2008/11/12 22:19:31 UTC
xceiverCount 257 exceeds the limit of concurrent xcievers 256
hbase-users,
I have been running MR processes for several days against HBase with
success until recently the region servers shut themselves down.
Hadoop 0.18.1
Hbase 0.18.1
3 node cluster
Checking the region server logs, I see the following Exception before
shutdown:
2008-11-11 19:55:52,451 INFO org.apache.hadoop.dfs.DFSClient:
Exception in createBlockOutputStream java.io.IOException: Could not
read from stream
2008-11-11 19:55:52,451 INFO org.apache.hadoop.dfs.DFSClient:
Abandoning block blk_-5467014108758633036_595771
2008-11-11 19:55:58,455 WARN org.apache.hadoop.dfs.DFSClient:
DataStreamer Exception: java.io.IOException: Unable to create new block.
2008-11-11 19:55:58,455 WARN org.apache.hadoop.dfs.DFSClient: Error
Recovery for block blk_-5467014108758633036_595771 bad datanode[0]
2008-11-11 19:55:58,482 FATAL
org.apache.hadoop.hbase.regionserver.Flusher: Replay of hlog required.
Forcing server shutdown
Hadoop datanode:
ERROR org.apache.hadoop.dfs.DataNode:
DatanodeRegistration(10.10.10.53:50010,
storageID=DS-1570581820-10.10.10.53-50010-1224117842339,
infoPort=50075, ipcPort=50020):DataXceiver: java.io.IOException:
xceiverCount 258 exceeds the limit of concurrent xcievers 256
Also, I can't seem to recover from this.
At restart HBase regionserver log:
ERROR org.apache.hadoop.hbase.regionserver.HRegionServer: error
opening region table,key,1226427697843
java.io.IOException: Could not obtain block:
blk_-7717647125888341834_589063 file=/hbase/table/1691377931/title/
mapfiles/7939912624709782499/index
Hadoop datanode:
ERROR org.apache.hadoop.dfs.DataNode:
DatanodeRegistration(10.10.10.51:50010,
storageID=DS-2079833884-10.10.10.51-50010-1224117832456,
infoPort=50075, ipcPort=50020):DataXceiver: java.io.IOException:
xceiverCount 257 exceeds the limit of concurrent xcievers 256
at org.apache.hadoop.dfs.DataNode$DataXceiver.run(DataNode.java:1030)
at java.lang.Thread.run(Thread.java:595)
Any help is appreciated.
Thanks,
Dru
Re: xceiverCount 257 exceeds the limit of concurrent xcievers 256
Posted by Michael Stack <st...@duboce.net>.
Try upping the limit on your datanodes. Set dfs.datanode.max.xcievers
up to 1024 or more.
St.Ack
Dru Jensen wrote:
> hbase-users,
>
> I have been running MR processes for several days against HBase with
> success until recently the region servers shut themselves down.
> Hadoop 0.18.1
> Hbase 0.18.1
> 3 node cluster
>
> Checking the region server logs, I see the following Exception before
> shutdown:
>
> 2008-11-11 19:55:52,451 INFO org.apache.hadoop.dfs.DFSClient:
> Exception in createBlockOutputStream java.io.IOException: Could not
> read from stream
> 2008-11-11 19:55:52,451 INFO org.apache.hadoop.dfs.DFSClient:
> Abandoning block blk_-5467014108758633036_595771
> 2008-11-11 19:55:58,455 WARN org.apache.hadoop.dfs.DFSClient:
> DataStreamer Exception: java.io.IOException: Unable to create new block.
> 2008-11-11 19:55:58,455 WARN org.apache.hadoop.dfs.DFSClient: Error
> Recovery for block blk_-5467014108758633036_595771 bad datanode[0]
> 2008-11-11 19:55:58,482 FATAL
> org.apache.hadoop.hbase.regionserver.Flusher: Replay of hlog required.
> Forcing server shutdown
>
> Hadoop datanode:
>
> ERROR org.apache.hadoop.dfs.DataNode:
> DatanodeRegistration(10.10.10.53:50010,
> storageID=DS-1570581820-10.10.10.53-50010-1224117842339,
> infoPort=50075, ipcPort=50020):DataXceiver: java.io.IOException:
> xceiverCount 258 exceeds the limit of concurrent xcievers 256
>
> Also, I can't seem to recover from this.
>
> At restart HBase regionserver log:
>
> ERROR org.apache.hadoop.hbase.regionserver.HRegionServer: error
> opening region table,key,1226427697843
> java.io.IOException: Could not obtain block:
> blk_-7717647125888341834_589063
> file=/hbase/table/1691377931/title/mapfiles/7939912624709782499/index
>
> Hadoop datanode:
>
> ERROR org.apache.hadoop.dfs.DataNode:
> DatanodeRegistration(10.10.10.51:50010,
> storageID=DS-2079833884-10.10.10.51-50010-1224117832456,
> infoPort=50075, ipcPort=50020):DataXceiver: java.io.IOException:
> xceiverCount 257 exceeds the limit of concurrent xcievers 256
> at org.apache.hadoop.dfs.DataNode$DataXceiver.run(DataNode.java:1030)
> at java.lang.Thread.run(Thread.java:595)
>
> Any help is appreciated.
>
> Thanks,
> Dru
>