[jira] [Commented] (PARQUET-1320) Fast clean unused direct memory when decompress

["ASF GitHub Bot (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/PARQUET-1320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16505934#comment-16505934 ] 

ASF GitHub Bot commented on PARQUET-1320:
-----------------------------------------

caneGuy opened a new pull request #492: PARQUET-1320: Fix potential direct memory leak for snappy decompressor
URL: https://github.com/apache/parquet-mr/pull/492
 
 
   When use NonBlockedDecompressorStream for decompress.
   If we do not get any full gc for old gen.we may failed by off-heap memory leak
   
   This patch fix this potential leak.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


> Fast clean unused direct memory when decompress
> -----------------------------------------------
>
>                 Key: PARQUET-1320
>                 URL: https://issues.apache.org/jira/browse/PARQUET-1320
>             Project: Parquet
>          Issue Type: Improvement
>          Components: parquet-mr
>            Reporter: zhoukang
>            Priority: Major
>
> When use *NonBlockedDecompressorStream* which call:
> *SnappyDecompressor.setInput*
> {code:java}
> public synchronized void setInput(byte[] buffer, int off, int len) {
>  SnappyUtil
> public synchronized void setInput(byte[] buffer, int off, int len) {
>  SnappyUtil.validateBuffer(buffer, off, len);
>  if (inputBuffer.capacity() - inputBuffer.position() < len) {
>  ByteBuffer newBuffer = ByteBuffer.allocateDirect(inputBuffer.position() + len);
>  inputBuffer.rewind();
>  newBuffer.put(inputBuffer);
>  inputBuffer = newBuffer; 
>  } else {
>  inputBuffer.limit(inputBuffer.position() + len);
>  }
>  inputBuffer.put(buffer, off, len);
> }
> .validateBuffer(buffer, off, len);
>  if (inputBuffer.capacity() - inputBuffer.position() < len) {
>  ByteBuffer newBuffer = ByteBuffer.allocateDirect(inputBuffer.position() + len);
>  inputBuffer.rewind();
>  newBuffer.put(inputBuffer);
>  inputBuffer = newBuffer; 
>  } else {
>  inputBuffer.limit(inputBuffer.position() + len);
>  }
>  inputBuffer.put(buffer, off, len);
> }
> {code}
> If we do not get any full gc for old gen.we may failed by off-heap memory leak
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)