You are viewing a plain text version of this content. The canonical link for it is here.

Posted to common-dev@hadoop.apache.org by "Bryan Beaudreault (Jira)" <ji...@apache.org> on 2023/01/13 18:08:00 UTC

[jira] [Created] (HADOOP-18593) Reduce byte array allocation in DecompressorStreams

Bryan Beaudreault created HADOOP-18593:
------------------------------------------

             Summary: Reduce byte array allocation in DecompressorStreams
                 Key: HADOOP-18593
                 URL: https://issues.apache.org/jira/browse/HADOOP-18593
             Project: Hadoop Common
          Issue Type: Improvement
            Reporter: Bryan Beaudreault


Compression codecs all implement CompressionCodec interface, which has a createInputStream method which takes an InputStream. These end up creating a class which extends DecompressorStream, which allocates 2 byte arrays of configurable size.

Currently its impossible to re-use these DecompressorStreams. Heavy users of decompression, can end up seeing large amounts of allocations coming from the creation of DecompressorStreams. Realtime systems like HBase have put a lot of work into reducing allocations, either through pooling and/or off-heaping. 

It would be very useful to be able to either re-use DecompressorStreams or add constructors which accept a ByteArrayManager or other source of byte array allocator. Either way could reduce allocations considerably in this case.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-dev-help@hadoop.apache.org