Re: A RS、DN of machine crash down，then hbase handler be filled，and hbase cluster request rapidly go down

[Stack <st...@duboce.net>]

On Fri, Apr 27, 2018 at 11:45 PM, Chenxi Tong <tc...@gmail.com> wrote:

>
>
> Dear all，
>
> For help, many many thanks to help。
>
> Some days ago，our production hbase cluster experience a trouble，about Apr
> 23 21:52, a machine suddenly crash down，the regionserver and datanode also
> down in the machine,
>
> then about six regionserver handler be filled, hbase cluster request
> rapidly go down. Until we stop the six regionserver the hbase cluster
> recover.
>
> we guess the hbase numCallsInGeneralQueue overstocke, but we increase
> numCallsInGeneralQueue and decrease handler and cut off network of a
> machine, we can not reappear the scenario.
>
> So , I want to ask for help how to solve the scenario gracefully and how
> we can reappear the scenario for test.
>
> Error log ：
>
> 2018-04-23 21:57:11,472 INFO [StoreFileOpenerThread-info-1]
> regionserver.StoreFile$Reader: Loaded Delete Family Bloom (CompoundBloom
>
> Filter) metadata for 1bade4c9adb6444baebf9e797a037f78
>
> 2018-04-23 21:57:14,151 WARN [StoreFileOpenerThread-info-1]
> hdfs.BlockReaderFactory: I/O error constructing remote block reader.
>
> java.net.NoRouteToHostException: No route to host
>


You need to fix this. Incidence of the above will cause havoc on your
clusters.
Thanks,
S




>
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
>
> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
>
> at org.apache.hadoop.net.SocketIOWithTimeout.connect(
> SocketIOWithTimeout.java:206)
>
> at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:529)
>
> at org.apache.hadoop.hdfs.DFSClient.newConnectedPeer(DFSClient.java:2884)
>
> at org.apache.hadoop.hdfs.BlockReaderFactory.nextTcpPeer(
> BlockReaderFactory.java:747)
>
> at org.apache.hadoop.hdfs.BlockReaderFactory.getRemoteBlockReaderFromTcp(
> BlockReaderFactory.java:662)
>
> at org.apache.hadoop.hdfs.BlockReaderFactory.build(
> BlockReaderFactory.java:326)
>
> at org.apache.hadoop.hdfs.DFSInputStream.blockSeekTo(
> DFSInputStream.java:570)
>
> at org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(
> DFSInputStream.java:793)
>
> at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:840)
>
> at java.io.DataInputStream.readFully(DataInputStream.java:195)
>
> at org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(
> FixedFileTrailer.java:390)
>
> at org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(
> HFile.java:482)
>
> at org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:525)
>
> at org.apache.hadoop.hbase.regionserver.StoreFile$Reader.
> <init>(StoreFile.java:1164)
>
> at org.apache.hadoop.hbase.regionserver.StoreFileInfo.
> open(StoreFileInfo.java:259)
>
> at org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:427)
>
> at org.apache.hadoop.hbase.regionserver.StoreFile.
> createReader(StoreFile.java:528)
>
> at org.apache.hadoop.hbase.regionserver.StoreFile.
> createReader(StoreFile.java:518)
>
> at org.apache.hadoop.hbase.regionserver.HStore.createStoreFileAndReader(
> HStore.java:667)
>
> at org.apache.hadoop.hbase.regionserver.HStore.access$000(HStore.java:119)
>
> at org.apache.hadoop.hbase.regionserver.HStore$1.call(HStore.java:534)
>
> at org.apache.hadoop.hbase.regionserver.HStore$1.call(HStore.java:531)
>
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>
> at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1142)
>
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:617)
>
> at java.lang.Thread.run(Thread.java:745)
>
> 2018-04-23 21:57:14,151 WARN [StoreFileOpenerThread-info-1]
> hdfs.BlockReaderFactory: I/O error constructing remote block reader.
>
> java.net.NoRouteToHostException: No route to host
>
>
>
>
>
>
>