[jira] [Created] (HDFS-17364) Use WeakReferencedElasticByteBufferPool in DFSStripedInputStream

["Bryan Beaudreault (Jira)" <ji...@apache.org>]

Bryan Beaudreault created HDFS-17364:
----------------------------------------

             Summary: Use WeakReferencedElasticByteBufferPool in DFSStripedInputStream
                 Key: HDFS-17364
                 URL: https://issues.apache.org/jira/browse/HDFS-17364
             Project: Hadoop HDFS
          Issue Type: Improvement
            Reporter: Bryan Beaudreault


DFSStripedInputStream uses ElasticByteBufferPool to allocate byte buffers for the "curStripeBuf". This is used for non-positional (stateful) reads and is allocated with a size of numDataBlocks * cellSize. For RS-6-3-1024k, that means each DFSStripedInputStream could allocate a 6mb buffer. When the IS is finished, the buffer is put back in the pool. Over time and with spikes of concurrent reads, the pool grows and most of the buffers sit there unused.
 
WeakReferencedElasticByteBufferPool was introduced HADOOP-18105 and mitigates this issue because the excess buffers can be GC'd once they are no longer needed. We should use this same pool in DFSStripedInputStream



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org