You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@parquet.apache.org by "zhoukang (JIRA)" <ji...@apache.org> on 2018/06/08 11:41:00 UTC
[jira] [Created] (PARQUET-1320) Fast clean unused direct memory
when decompress
zhoukang created PARQUET-1320:
---------------------------------
Summary: Fast clean unused direct memory when decompress
Key: PARQUET-1320
URL: https://issues.apache.org/jira/browse/PARQUET-1320
Project: Parquet
Issue Type: Improvement
Components: parquet-mr
Reporter: zhoukang
When use *NonBlockedDecompressorStream* which call:
*SnappyDecompressor.setInput*
{code:java}
public synchronized void setInput(byte[] buffer, int off, int len) {
SnappyUtil
public synchronized void setInput(byte[] buffer, int off, int len) {
SnappyUtil.validateBuffer(buffer, off, len);
if (inputBuffer.capacity() - inputBuffer.position() < len) {
ByteBuffer newBuffer = ByteBuffer.allocateDirect(inputBuffer.position() + len);
inputBuffer.rewind();
newBuffer.put(inputBuffer);
inputBuffer = newBuffer;
} else {
inputBuffer.limit(inputBuffer.position() + len);
}
inputBuffer.put(buffer, off, len);
}
.validateBuffer(buffer, off, len);
if (inputBuffer.capacity() - inputBuffer.position() < len) {
ByteBuffer newBuffer = ByteBuffer.allocateDirect(inputBuffer.position() + len);
inputBuffer.rewind();
newBuffer.put(inputBuffer);
inputBuffer = newBuffer;
} else {
inputBuffer.limit(inputBuffer.position() + len);
}
inputBuffer.put(buffer, off, len);
}
{code}
If we do not get any full gc for old gen.we may failed by off-heap memory leak
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)