You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Miao Wang <wa...@163.com> on 2023/10/30 10:49:02 UTC

How to proactively update the information of hdfs blocks in hbase cache

Hi, community


We have an HBase cluster using version 2.4.6 and Hadoop version 3.0.0 cdh6.3.2,I have two questions to ask. Could you please answer them? Thank you


1.When datanode crashes for an extended period of time and the client fails to hit the cache when requesting data, the region server will still request abnormal datanode nodes. Is this cached information only updated when the table is making a major? ,Is the information in this cache only updated when the table generates the main table? When the dn service encounters an exception, how can I proactively update this information in the regional server cache?
The error message is as follows:


Failed to connect to /10.11.34.29:9866 for file /hbase/data/default/tj_idcard_sha_02/429606d2083c287a13d0ebe43774938f/cf/dc063bb106b34039a014690f86b1c22c for block BP-1561188726-10.11.39.10-1684756341563:blk_1078186539_4445860:java.io.IOException: Connection reset by peer 


2.When datanoe encounters an exception, the regionserve will generate the following error log, which seriously affects the client's request time. Analyzing the log roughly affects the duration of ten minutes. Why are these two error messages still inconsistent? How can I avoid it?




 Connection failure: Failed to connect to /10.11.34.29:9866 for file /hbase/archive/data/default/tj_soical_v05_feature_hb_rhsh/500ea510085d36ce0efa33ac63a15a88/cf/e6003a114c5d415ba720ac7657538120 for block BP-1561188726-10.11.39.10-1684756341563:blk_1078131727_4391047:java.net.SocketTimeoutException: 60000 millis timeout while waiting for channel to be ready for read. ch : java.nio.channels.SocketChannel[connected local=/10.11.2.142:14154 remote=/10.11.34.29:9866]
java.net.SocketTimeoutException: 60000 millis timeout while waiting for channel to be ready for read. ch : java.nio.channels.SocketChannel[connected local=/10.11.2.142:14154 remote=/10.11.34.29:9866]
at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
        at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
        at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
        at sun.nio.ch.IOUtil.read(IOUtil.java:197)
        at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380)
        at org.apache.hadoop.net.SocketInputStream$Reader.performIO(SocketInputStream.java:57)
        at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:142)
        at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161)
        at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131)
        at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:118)
        at java.io.FilterInputStream.read(FilterInputStream.java:83)
        at org.apache.hadoop.hdfs.protocolPB.PBHelperClient.vintPrefixed(PBHelperClient.java:537)
        at org.apache.hadoop.hdfs.client.impl.BlockReaderRemote.newBlockReader(BlockReaderRemote.java:407)
        at org.apache.hadoop.hdfs.client.impl.BlockReaderFactory.getRemoteBlockReader(BlockReaderFactory.java:848)
        at org.apache.hadoop.hdfs.client.impl.BlockReaderFactory.getRemoteBlockReaderFromTcp(BlockReaderFactory.java:744)
        at org.apache.hadoop.hdfs.client.impl.BlockReaderFactory.build(BlockReaderFactory.java:379)
        at org.apache.hadoop.hdfs.DFSInputStream.getBlockReader(DFSInputStream.java:645)
        at org.apache.hadoop.hdfs.DFSInputStream.actualGetFromOneDataNode(DFSInputStream.java:1050)
        at org.apache.hadoop.hdfs.DFSInputStream.fetchBlockByteRange(DFSInputStream.java:1002)
        at org.apache.hadoop.hdfs.DFSInputStream.pread(DFSInputStream.java:1361)
        at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:1325)
        at org.apache.hadoop.fs.FSDataInputStream.read(FSDataInputStream.java:93)
        at org.apache.hadoop.hbase.io.util.BlockIOUtils.preadWithExtra(BlockIOUtils.java:214)
        at org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readAtOffset(HFileBlock.java:1518)
        at org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockDataInternal(HFileBlock.java:1741)
        at org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockData(HFileBlock.java:1550)
        at org.apache.hadoop.hbase.io.hfile.HFileReaderImpl.readBlock(HFileReaderImpl.java:1337)
        at org.apache.hadoop.hbase.io.hfile.CompoundBloomFilter.getBloomBlock(CompoundBloomFilter.java:122)
        at org.apache.hadoop.hbase.io.hfile.CompoundBloomFilter.contains(CompoundBloomFilter.java:102)
        at org.apache.hadoop.hbase.regionserver.StoreFileReader.checkGeneralBloomFilter(StoreFileReader.java:433)
        at org.apache.hadoop.hbase.regionserver.StoreFileReader.passesGeneralRowBloomFilter(StoreFileReader.java:322)
        at org.apache.hadoop.hbase.regionserver.StoreFileReader.passesBloomFilter(StoreFileReader.java:251)
        at org.apache.hadoop.hbase.regionserver.StoreFileScanner.shouldUseScanner(StoreFileScanner.java:491)
        at org.apache.hadoop.hbase.regionserver.StoreScanner.selectScannersFrom(StoreScanner.java:472)
        at org.apache.hadoop.hbase.regionserver.StoreScanner.<init>(StoreScanner.java:249) 

Re: How to proactively update the information of hdfs blocks in hbase cache

Posted by Miao Wang <wa...@163.com>.
Okay, I'm trying to improve the compilation and deployment of Hadoop version; Thank you for your answer​

---- Replied Message ----
| From | 张铎(Duo Zhang)<pa...@gmail.com> |
| Date | 10/30/2023 19:55 |
| To | dev<de...@hbase.apache.org> |
| Cc | user@hbase.apache.org<us...@hbase.apache.org> |
| Subject | Re: How to proactively update the information of hdfs blocks in hbase cache |
See https://issues.apache.org/jira/browse/HDFS-13571.

The problem for HBase is that the region server will keep a large
amount of DFSInputStream open, for random access. If a DN node is
completely gone, i.e, can not be reached through the network, you will
face a connection timeout, usually it will be very slow, in your case,
60 seconds, for HBase to find out the connection is broken.

It is not easy for HBase to fix this problem and in HDFS-13571, the
hdfs community introduced a way to mark a DN as dead before sending
request to it, and the dead node information is shared through all
DFSInputStream so after the first time you see a connection timeout of
a DN, soon all the DFSInputStream will give up requesting the dead DN,
which could mostly solve the above problem.

You can try to build HBase with hadoop 3.3.x and deploy it again for
fixing the problem, as the code changes are mostly at client side.

Thanks.

Miao Wang <wa...@163.com> 于2023年10月30日周一 18:49写道:
>
> Hi, community
>
>
> We have an HBase cluster using version 2.4.6 and Hadoop version 3.0.0 cdh6.3.2,I have two questions to ask. Could you please answer them? Thank you
>
>
> 1.When datanode crashes for an extended period of time and the client fails to hit the cache when requesting data, the region server will still request abnormal datanode nodes. Is this cached information only updated when the table is making a major? ,Is the information in this cache only updated when the table generates the main table? When the dn service encounters an exception, how can I proactively update this information in the regional server cache?
> The error message is as follows:
>
>
> Failed to connect to /10.11.34.29:9866 for file /hbase/data/default/tj_idcard_sha_02/429606d2083c287a13d0ebe43774938f/cf/dc063bb106b34039a014690f86b1c22c for block BP-1561188726-10.11.39.10-1684756341563:blk_1078186539_4445860:java.io.IOException: Connection reset by peer
>
>
> 2.When datanoe encounters an exception, the regionserve will generate the following error log, which seriously affects the client's request time. Analyzing the log roughly affects the duration of ten minutes. Why are these two error messages still inconsistent? How can I avoid it?
>
>
>
>
>  Connection failure: Failed to connect to /10.11.34.29:9866 for file /hbase/archive/data/default/tj_soical_v05_feature_hb_rhsh/500ea510085d36ce0efa33ac63a15a88/cf/e6003a114c5d415ba720ac7657538120 for block BP-1561188726-10.11.39.10-1684756341563:blk_1078131727_4391047:java.net.SocketTimeoutException: 60000 millis timeout while waiting for channel to be ready for read. ch : java.nio.channels.SocketChannel[connected local=/10.11.2.142:14154 remote=/10.11.34.29:9866]
> java.net.SocketTimeoutException: 60000 millis timeout while waiting for channel to be ready for read. ch : java.nio.channels.SocketChannel[connected local=/10.11.2.142:14154 remote=/10.11.34.29:9866]
> at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
>         at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
>         at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
>         at sun.nio.ch.IOUtil.read(IOUtil.java:197)
>         at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380)
>         at org.apache.hadoop.net.SocketInputStream$Reader.performIO(SocketInputStream.java:57)
>         at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:142)
>         at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161)
>         at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131)
>         at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:118)
>         at java.io.FilterInputStream.read(FilterInputStream.java:83)
>         at org.apache.hadoop.hdfs.protocolPB.PBHelperClient.vintPrefixed(PBHelperClient.java:537)
>         at org.apache.hadoop.hdfs.client.impl.BlockReaderRemote.newBlockReader(BlockReaderRemote.java:407)
>         at org.apache.hadoop.hdfs.client.impl.BlockReaderFactory.getRemoteBlockReader(BlockReaderFactory.java:848)
>         at org.apache.hadoop.hdfs.client.impl.BlockReaderFactory.getRemoteBlockReaderFromTcp(BlockReaderFactory.java:744)
>         at org.apache.hadoop.hdfs.client.impl.BlockReaderFactory.build(BlockReaderFactory.java:379)
>         at org.apache.hadoop.hdfs.DFSInputStream.getBlockReader(DFSInputStream.java:645)
>         at org.apache.hadoop.hdfs.DFSInputStream.actualGetFromOneDataNode(DFSInputStream.java:1050)
>         at org.apache.hadoop.hdfs.DFSInputStream.fetchBlockByteRange(DFSInputStream.java:1002)
>         at org.apache.hadoop.hdfs.DFSInputStream.pread(DFSInputStream.java:1361)
>         at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:1325)
>         at org.apache.hadoop.fs.FSDataInputStream.read(FSDataInputStream.java:93)
>         at org.apache.hadoop.hbase.io.util.BlockIOUtils.preadWithExtra(BlockIOUtils.java:214)
>         at org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readAtOffset(HFileBlock.java:1518)
>         at org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockDataInternal(HFileBlock.java:1741)
>         at org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockData(HFileBlock.java:1550)
>         at org.apache.hadoop.hbase.io.hfile.HFileReaderImpl.readBlock(HFileReaderImpl.java:1337)
>         at org.apache.hadoop.hbase.io.hfile.CompoundBloomFilter.getBloomBlock(CompoundBloomFilter.java:122)
>         at org.apache.hadoop.hbase.io.hfile.CompoundBloomFilter.contains(CompoundBloomFilter.java:102)
>         at org.apache.hadoop.hbase.regionserver.StoreFileReader.checkGeneralBloomFilter(StoreFileReader.java:433)
>         at org.apache.hadoop.hbase.regionserver.StoreFileReader.passesGeneralRowBloomFilter(StoreFileReader.java:322)
>         at org.apache.hadoop.hbase.regionserver.StoreFileReader.passesBloomFilter(StoreFileReader.java:251)
>         at org.apache.hadoop.hbase.regionserver.StoreFileScanner.shouldUseScanner(StoreFileScanner.java:491)
>         at org.apache.hadoop.hbase.regionserver.StoreScanner.selectScannersFrom(StoreScanner.java:472)
>         at org.apache.hadoop.hbase.regionserver.StoreScanner.<init>(StoreScanner.java:249)

Re: How to proactively update the information of hdfs blocks in hbase cache

Posted by Miao Wang <wa...@163.com>.
Okay, I'm trying to improve the compilation and deployment of Hadoop version; Thank you for your answer​

---- Replied Message ----
| From | 张铎(Duo Zhang)<pa...@gmail.com> |
| Date | 10/30/2023 19:55 |
| To | dev<de...@hbase.apache.org> |
| Cc | user@hbase.apache.org<us...@hbase.apache.org> |
| Subject | Re: How to proactively update the information of hdfs blocks in hbase cache |
See https://issues.apache.org/jira/browse/HDFS-13571.

The problem for HBase is that the region server will keep a large
amount of DFSInputStream open, for random access. If a DN node is
completely gone, i.e, can not be reached through the network, you will
face a connection timeout, usually it will be very slow, in your case,
60 seconds, for HBase to find out the connection is broken.

It is not easy for HBase to fix this problem and in HDFS-13571, the
hdfs community introduced a way to mark a DN as dead before sending
request to it, and the dead node information is shared through all
DFSInputStream so after the first time you see a connection timeout of
a DN, soon all the DFSInputStream will give up requesting the dead DN,
which could mostly solve the above problem.

You can try to build HBase with hadoop 3.3.x and deploy it again for
fixing the problem, as the code changes are mostly at client side.

Thanks.

Miao Wang <wa...@163.com> 于2023年10月30日周一 18:49写道:
>
> Hi, community
>
>
> We have an HBase cluster using version 2.4.6 and Hadoop version 3.0.0 cdh6.3.2,I have two questions to ask. Could you please answer them? Thank you
>
>
> 1.When datanode crashes for an extended period of time and the client fails to hit the cache when requesting data, the region server will still request abnormal datanode nodes. Is this cached information only updated when the table is making a major? ,Is the information in this cache only updated when the table generates the main table? When the dn service encounters an exception, how can I proactively update this information in the regional server cache?
> The error message is as follows:
>
>
> Failed to connect to /10.11.34.29:9866 for file /hbase/data/default/tj_idcard_sha_02/429606d2083c287a13d0ebe43774938f/cf/dc063bb106b34039a014690f86b1c22c for block BP-1561188726-10.11.39.10-1684756341563:blk_1078186539_4445860:java.io.IOException: Connection reset by peer
>
>
> 2.When datanoe encounters an exception, the regionserve will generate the following error log, which seriously affects the client's request time. Analyzing the log roughly affects the duration of ten minutes. Why are these two error messages still inconsistent? How can I avoid it?
>
>
>
>
>  Connection failure: Failed to connect to /10.11.34.29:9866 for file /hbase/archive/data/default/tj_soical_v05_feature_hb_rhsh/500ea510085d36ce0efa33ac63a15a88/cf/e6003a114c5d415ba720ac7657538120 for block BP-1561188726-10.11.39.10-1684756341563:blk_1078131727_4391047:java.net.SocketTimeoutException: 60000 millis timeout while waiting for channel to be ready for read. ch : java.nio.channels.SocketChannel[connected local=/10.11.2.142:14154 remote=/10.11.34.29:9866]
> java.net.SocketTimeoutException: 60000 millis timeout while waiting for channel to be ready for read. ch : java.nio.channels.SocketChannel[connected local=/10.11.2.142:14154 remote=/10.11.34.29:9866]
> at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
>         at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
>         at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
>         at sun.nio.ch.IOUtil.read(IOUtil.java:197)
>         at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380)
>         at org.apache.hadoop.net.SocketInputStream$Reader.performIO(SocketInputStream.java:57)
>         at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:142)
>         at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161)
>         at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131)
>         at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:118)
>         at java.io.FilterInputStream.read(FilterInputStream.java:83)
>         at org.apache.hadoop.hdfs.protocolPB.PBHelperClient.vintPrefixed(PBHelperClient.java:537)
>         at org.apache.hadoop.hdfs.client.impl.BlockReaderRemote.newBlockReader(BlockReaderRemote.java:407)
>         at org.apache.hadoop.hdfs.client.impl.BlockReaderFactory.getRemoteBlockReader(BlockReaderFactory.java:848)
>         at org.apache.hadoop.hdfs.client.impl.BlockReaderFactory.getRemoteBlockReaderFromTcp(BlockReaderFactory.java:744)
>         at org.apache.hadoop.hdfs.client.impl.BlockReaderFactory.build(BlockReaderFactory.java:379)
>         at org.apache.hadoop.hdfs.DFSInputStream.getBlockReader(DFSInputStream.java:645)
>         at org.apache.hadoop.hdfs.DFSInputStream.actualGetFromOneDataNode(DFSInputStream.java:1050)
>         at org.apache.hadoop.hdfs.DFSInputStream.fetchBlockByteRange(DFSInputStream.java:1002)
>         at org.apache.hadoop.hdfs.DFSInputStream.pread(DFSInputStream.java:1361)
>         at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:1325)
>         at org.apache.hadoop.fs.FSDataInputStream.read(FSDataInputStream.java:93)
>         at org.apache.hadoop.hbase.io.util.BlockIOUtils.preadWithExtra(BlockIOUtils.java:214)
>         at org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readAtOffset(HFileBlock.java:1518)
>         at org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockDataInternal(HFileBlock.java:1741)
>         at org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockData(HFileBlock.java:1550)
>         at org.apache.hadoop.hbase.io.hfile.HFileReaderImpl.readBlock(HFileReaderImpl.java:1337)
>         at org.apache.hadoop.hbase.io.hfile.CompoundBloomFilter.getBloomBlock(CompoundBloomFilter.java:122)
>         at org.apache.hadoop.hbase.io.hfile.CompoundBloomFilter.contains(CompoundBloomFilter.java:102)
>         at org.apache.hadoop.hbase.regionserver.StoreFileReader.checkGeneralBloomFilter(StoreFileReader.java:433)
>         at org.apache.hadoop.hbase.regionserver.StoreFileReader.passesGeneralRowBloomFilter(StoreFileReader.java:322)
>         at org.apache.hadoop.hbase.regionserver.StoreFileReader.passesBloomFilter(StoreFileReader.java:251)
>         at org.apache.hadoop.hbase.regionserver.StoreFileScanner.shouldUseScanner(StoreFileScanner.java:491)
>         at org.apache.hadoop.hbase.regionserver.StoreScanner.selectScannersFrom(StoreScanner.java:472)
>         at org.apache.hadoop.hbase.regionserver.StoreScanner.<init>(StoreScanner.java:249)

Re: How to proactively update the information of hdfs blocks in hbase cache

Posted by "张铎(Duo Zhang)" <pa...@gmail.com>.
See https://issues.apache.org/jira/browse/HDFS-13571.

The problem for HBase is that the region server will keep a large
amount of DFSInputStream open, for random access. If a DN node is
completely gone, i.e, can not be reached through the network, you will
face a connection timeout, usually it will be very slow, in your case,
60 seconds, for HBase to find out the connection is broken.

It is not easy for HBase to fix this problem and in HDFS-13571, the
hdfs community introduced a way to mark a DN as dead before sending
request to it, and the dead node information is shared through all
DFSInputStream so after the first time you see a connection timeout of
a DN, soon all the DFSInputStream will give up requesting the dead DN,
which could mostly solve the above problem.

You can try to build HBase with hadoop 3.3.x and deploy it again for
fixing the problem, as the code changes are mostly at client side.

Thanks.

Miao Wang <wa...@163.com> 于2023年10月30日周一 18:49写道:
>
> Hi, community
>
>
> We have an HBase cluster using version 2.4.6 and Hadoop version 3.0.0 cdh6.3.2,I have two questions to ask. Could you please answer them? Thank you
>
>
> 1.When datanode crashes for an extended period of time and the client fails to hit the cache when requesting data, the region server will still request abnormal datanode nodes. Is this cached information only updated when the table is making a major? ,Is the information in this cache only updated when the table generates the main table? When the dn service encounters an exception, how can I proactively update this information in the regional server cache?
> The error message is as follows:
>
>
> Failed to connect to /10.11.34.29:9866 for file /hbase/data/default/tj_idcard_sha_02/429606d2083c287a13d0ebe43774938f/cf/dc063bb106b34039a014690f86b1c22c for block BP-1561188726-10.11.39.10-1684756341563:blk_1078186539_4445860:java.io.IOException: Connection reset by peer
>
>
> 2.When datanoe encounters an exception, the regionserve will generate the following error log, which seriously affects the client's request time. Analyzing the log roughly affects the duration of ten minutes. Why are these two error messages still inconsistent? How can I avoid it?
>
>
>
>
>  Connection failure: Failed to connect to /10.11.34.29:9866 for file /hbase/archive/data/default/tj_soical_v05_feature_hb_rhsh/500ea510085d36ce0efa33ac63a15a88/cf/e6003a114c5d415ba720ac7657538120 for block BP-1561188726-10.11.39.10-1684756341563:blk_1078131727_4391047:java.net.SocketTimeoutException: 60000 millis timeout while waiting for channel to be ready for read. ch : java.nio.channels.SocketChannel[connected local=/10.11.2.142:14154 remote=/10.11.34.29:9866]
> java.net.SocketTimeoutException: 60000 millis timeout while waiting for channel to be ready for read. ch : java.nio.channels.SocketChannel[connected local=/10.11.2.142:14154 remote=/10.11.34.29:9866]
> at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
>         at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
>         at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
>         at sun.nio.ch.IOUtil.read(IOUtil.java:197)
>         at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380)
>         at org.apache.hadoop.net.SocketInputStream$Reader.performIO(SocketInputStream.java:57)
>         at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:142)
>         at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161)
>         at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131)
>         at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:118)
>         at java.io.FilterInputStream.read(FilterInputStream.java:83)
>         at org.apache.hadoop.hdfs.protocolPB.PBHelperClient.vintPrefixed(PBHelperClient.java:537)
>         at org.apache.hadoop.hdfs.client.impl.BlockReaderRemote.newBlockReader(BlockReaderRemote.java:407)
>         at org.apache.hadoop.hdfs.client.impl.BlockReaderFactory.getRemoteBlockReader(BlockReaderFactory.java:848)
>         at org.apache.hadoop.hdfs.client.impl.BlockReaderFactory.getRemoteBlockReaderFromTcp(BlockReaderFactory.java:744)
>         at org.apache.hadoop.hdfs.client.impl.BlockReaderFactory.build(BlockReaderFactory.java:379)
>         at org.apache.hadoop.hdfs.DFSInputStream.getBlockReader(DFSInputStream.java:645)
>         at org.apache.hadoop.hdfs.DFSInputStream.actualGetFromOneDataNode(DFSInputStream.java:1050)
>         at org.apache.hadoop.hdfs.DFSInputStream.fetchBlockByteRange(DFSInputStream.java:1002)
>         at org.apache.hadoop.hdfs.DFSInputStream.pread(DFSInputStream.java:1361)
>         at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:1325)
>         at org.apache.hadoop.fs.FSDataInputStream.read(FSDataInputStream.java:93)
>         at org.apache.hadoop.hbase.io.util.BlockIOUtils.preadWithExtra(BlockIOUtils.java:214)
>         at org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readAtOffset(HFileBlock.java:1518)
>         at org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockDataInternal(HFileBlock.java:1741)
>         at org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockData(HFileBlock.java:1550)
>         at org.apache.hadoop.hbase.io.hfile.HFileReaderImpl.readBlock(HFileReaderImpl.java:1337)
>         at org.apache.hadoop.hbase.io.hfile.CompoundBloomFilter.getBloomBlock(CompoundBloomFilter.java:122)
>         at org.apache.hadoop.hbase.io.hfile.CompoundBloomFilter.contains(CompoundBloomFilter.java:102)
>         at org.apache.hadoop.hbase.regionserver.StoreFileReader.checkGeneralBloomFilter(StoreFileReader.java:433)
>         at org.apache.hadoop.hbase.regionserver.StoreFileReader.passesGeneralRowBloomFilter(StoreFileReader.java:322)
>         at org.apache.hadoop.hbase.regionserver.StoreFileReader.passesBloomFilter(StoreFileReader.java:251)
>         at org.apache.hadoop.hbase.regionserver.StoreFileScanner.shouldUseScanner(StoreFileScanner.java:491)
>         at org.apache.hadoop.hbase.regionserver.StoreScanner.selectScannersFrom(StoreScanner.java:472)
>         at org.apache.hadoop.hbase.regionserver.StoreScanner.<init>(StoreScanner.java:249)

Re: How to proactively update the information of hdfs blocks in hbase cache

Posted by "张铎(Duo Zhang)" <pa...@gmail.com>.
See https://issues.apache.org/jira/browse/HDFS-13571.

The problem for HBase is that the region server will keep a large
amount of DFSInputStream open, for random access. If a DN node is
completely gone, i.e, can not be reached through the network, you will
face a connection timeout, usually it will be very slow, in your case,
60 seconds, for HBase to find out the connection is broken.

It is not easy for HBase to fix this problem and in HDFS-13571, the
hdfs community introduced a way to mark a DN as dead before sending
request to it, and the dead node information is shared through all
DFSInputStream so after the first time you see a connection timeout of
a DN, soon all the DFSInputStream will give up requesting the dead DN,
which could mostly solve the above problem.

You can try to build HBase with hadoop 3.3.x and deploy it again for
fixing the problem, as the code changes are mostly at client side.

Thanks.

Miao Wang <wa...@163.com> 于2023年10月30日周一 18:49写道:
>
> Hi, community
>
>
> We have an HBase cluster using version 2.4.6 and Hadoop version 3.0.0 cdh6.3.2,I have two questions to ask. Could you please answer them? Thank you
>
>
> 1.When datanode crashes for an extended period of time and the client fails to hit the cache when requesting data, the region server will still request abnormal datanode nodes. Is this cached information only updated when the table is making a major? ,Is the information in this cache only updated when the table generates the main table? When the dn service encounters an exception, how can I proactively update this information in the regional server cache?
> The error message is as follows:
>
>
> Failed to connect to /10.11.34.29:9866 for file /hbase/data/default/tj_idcard_sha_02/429606d2083c287a13d0ebe43774938f/cf/dc063bb106b34039a014690f86b1c22c for block BP-1561188726-10.11.39.10-1684756341563:blk_1078186539_4445860:java.io.IOException: Connection reset by peer
>
>
> 2.When datanoe encounters an exception, the regionserve will generate the following error log, which seriously affects the client's request time. Analyzing the log roughly affects the duration of ten minutes. Why are these two error messages still inconsistent? How can I avoid it?
>
>
>
>
>  Connection failure: Failed to connect to /10.11.34.29:9866 for file /hbase/archive/data/default/tj_soical_v05_feature_hb_rhsh/500ea510085d36ce0efa33ac63a15a88/cf/e6003a114c5d415ba720ac7657538120 for block BP-1561188726-10.11.39.10-1684756341563:blk_1078131727_4391047:java.net.SocketTimeoutException: 60000 millis timeout while waiting for channel to be ready for read. ch : java.nio.channels.SocketChannel[connected local=/10.11.2.142:14154 remote=/10.11.34.29:9866]
> java.net.SocketTimeoutException: 60000 millis timeout while waiting for channel to be ready for read. ch : java.nio.channels.SocketChannel[connected local=/10.11.2.142:14154 remote=/10.11.34.29:9866]
> at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
>         at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
>         at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
>         at sun.nio.ch.IOUtil.read(IOUtil.java:197)
>         at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380)
>         at org.apache.hadoop.net.SocketInputStream$Reader.performIO(SocketInputStream.java:57)
>         at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:142)
>         at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161)
>         at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131)
>         at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:118)
>         at java.io.FilterInputStream.read(FilterInputStream.java:83)
>         at org.apache.hadoop.hdfs.protocolPB.PBHelperClient.vintPrefixed(PBHelperClient.java:537)
>         at org.apache.hadoop.hdfs.client.impl.BlockReaderRemote.newBlockReader(BlockReaderRemote.java:407)
>         at org.apache.hadoop.hdfs.client.impl.BlockReaderFactory.getRemoteBlockReader(BlockReaderFactory.java:848)
>         at org.apache.hadoop.hdfs.client.impl.BlockReaderFactory.getRemoteBlockReaderFromTcp(BlockReaderFactory.java:744)
>         at org.apache.hadoop.hdfs.client.impl.BlockReaderFactory.build(BlockReaderFactory.java:379)
>         at org.apache.hadoop.hdfs.DFSInputStream.getBlockReader(DFSInputStream.java:645)
>         at org.apache.hadoop.hdfs.DFSInputStream.actualGetFromOneDataNode(DFSInputStream.java:1050)
>         at org.apache.hadoop.hdfs.DFSInputStream.fetchBlockByteRange(DFSInputStream.java:1002)
>         at org.apache.hadoop.hdfs.DFSInputStream.pread(DFSInputStream.java:1361)
>         at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:1325)
>         at org.apache.hadoop.fs.FSDataInputStream.read(FSDataInputStream.java:93)
>         at org.apache.hadoop.hbase.io.util.BlockIOUtils.preadWithExtra(BlockIOUtils.java:214)
>         at org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readAtOffset(HFileBlock.java:1518)
>         at org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockDataInternal(HFileBlock.java:1741)
>         at org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockData(HFileBlock.java:1550)
>         at org.apache.hadoop.hbase.io.hfile.HFileReaderImpl.readBlock(HFileReaderImpl.java:1337)
>         at org.apache.hadoop.hbase.io.hfile.CompoundBloomFilter.getBloomBlock(CompoundBloomFilter.java:122)
>         at org.apache.hadoop.hbase.io.hfile.CompoundBloomFilter.contains(CompoundBloomFilter.java:102)
>         at org.apache.hadoop.hbase.regionserver.StoreFileReader.checkGeneralBloomFilter(StoreFileReader.java:433)
>         at org.apache.hadoop.hbase.regionserver.StoreFileReader.passesGeneralRowBloomFilter(StoreFileReader.java:322)
>         at org.apache.hadoop.hbase.regionserver.StoreFileReader.passesBloomFilter(StoreFileReader.java:251)
>         at org.apache.hadoop.hbase.regionserver.StoreFileScanner.shouldUseScanner(StoreFileScanner.java:491)
>         at org.apache.hadoop.hbase.regionserver.StoreScanner.selectScannersFrom(StoreScanner.java:472)
>         at org.apache.hadoop.hbase.regionserver.StoreScanner.<init>(StoreScanner.java:249)