[jira] [Created] (HBASE-28338) Bounded leak of FSDataInputStream buffers from checksum switching

["Bryan Beaudreault (Jira)" <ji...@apache.org>]

Bryan Beaudreault created HBASE-28338:
-----------------------------------------

             Summary: Bounded leak of FSDataInputStream buffers from checksum switching
                 Key: HBASE-28338
                 URL: https://issues.apache.org/jira/browse/HBASE-28338
             Project: HBase
          Issue Type: Bug
            Reporter: Bryan Beaudreault


In FSDataInputStreamWrapper, the unbuffer() method caches an unbuffer instance the first time it is called. When an FSDataInputStreamWrapper is initialized, it has hbase checksum disabled.

In HFileInfo.initTrailerAndContext we get the stream, read the trailer, then call unbuffer. At this point, checksums have not been enabled yet via prepareForBlockReader. So the call to unbuffer() caches the current non-checksum stream as the unbuffer instance.

Later, in initMetaAndIndex we do a similar thing. This time, prepareForBlockReader has been called, so we are now using hbase checksums. When initMetaAndIndex calls unbuffer(), it uses the old unbuffer instance which actually has been closed when we switched to hbase checksums. So that call does nothing, and the new no-checksum input stream is never unbuffered.

I haven't seen this cause an issue with normal hdfs replication (though haven't gone looking). It's very problematic for Erasure Coding because DFSStripedInputStream holds a large buffer (numDataBlocks * cellSize, so 6mb for RS-6-3-1024k) that is only used for stream reads NOT pread. The FSDataInputStreamWrapper we are talking about here is only used for pread in hbase, so those 6mb buffers just hang around totally unused but unreclaimable. Since there is an input stream per StoreFile, this can add up very quickly on big servers.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)