[jira] [Commented] (HBASE-28338) Bounded leak of FSDataInputStream buffers from checksum switching

["Hudson (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HBASE-28338?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17814702#comment-17814702 ] 

Hudson commented on HBASE-28338:
--------------------------------

Results for branch branch-2.6
	[build #49 on builds.a.o|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.6/49/]: (x) *{color:red}-1 overall{color}*
----
details (if available):

(/) {color:green}+1 general checks{color}
-- For more information [see general report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.6/49/General_20Nightly_20Build_20Report/]


(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2) report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.6/49/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]


(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3) report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.6/49/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]


(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11 report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.6/49/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]


(/) {color:green}+1 source release artifact{color}
-- See build output for details.


(x) {color:red}-1 client integration test{color}
--Failed when running client tests on top of Hadoop 3. [see log for details|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.6/49//artifact/output-integration/hadoop-3.log]. (note that this means we didn't check the Hadoop 3 shaded client)


> Bounded leak of FSDataInputStream buffers from checksum switching
> -----------------------------------------------------------------
>
>                 Key: HBASE-28338
>                 URL: https://issues.apache.org/jira/browse/HBASE-28338
>             Project: HBase
>          Issue Type: Bug
>            Reporter: Bryan Beaudreault
>            Priority: Major
>              Labels: pull-request-available
>
> In FSDataInputStreamWrapper, the unbuffer() method caches an unbuffer instance the first time it is called. When an FSDataInputStreamWrapper is initialized, it has hbase checksum disabled.
> In HFileInfo.initTrailerAndContext we get the stream, read the trailer, then call unbuffer. At this point, checksums have not been enabled yet via prepareForBlockReader. So the call to unbuffer() caches the current non-checksum stream as the unbuffer instance.
> Later, in initMetaAndIndex we do a similar thing. This time, prepareForBlockReader has been called, so we are now using hbase checksums. When initMetaAndIndex calls unbuffer(), it uses the old unbuffer instance which actually has been closed when we switched to hbase checksums. So that call does nothing, and the new no-checksum input stream is never unbuffered.
> I haven't seen this cause an issue with normal hdfs replication (though haven't gone looking). It's very problematic for Erasure Coding because DFSStripedInputStream holds a large buffer (numDataBlocks * cellSize, so 6mb for RS-6-3-1024k) that is only used for stream reads NOT pread. The FSDataInputStreamWrapper we are talking about here is only used for pread in hbase, so those 6mb buffers just hang around totally unused but unreclaimable. Since there is an input stream per StoreFile, this can add up very quickly on big servers.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)