[jira] [Comment Edited] (FLINK-13985) Use native memory for managed memory.

["Andrey Zagrebin (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/FLINK-13985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16929262#comment-16929262 ] 

Andrey Zagrebin edited comment on FLINK-13985 at 9/13/19 2:51 PM:
------------------------------------------------------------------

Thanks for sharing the thoughts [~StephanEwen].

I looked into the issue and rough idea atm is that _MemorySegmentFactory_ gets one more method to allocate _MemorySegment_ with _unsafe_.
 We need one more concrete class _HybridUnsafeMemorySegment_:
 MemorySegment << _HybridUnsafeMemorySegment_ << HybridMemorySegment
 _HybridUnsafeMemorySegment_ is pretty much copy of _HybridMemorySegment_, except it does not have _ByteBuffer offHeapBuffer_ and returns _ByteBuffer_ from _getOffHeapBuffer_ and wrap methods in a different way.
 There is no straightforward way to wrap unsafe memory with _ByteBuffer_. I found [netty does this|#L469] by hacking into the private constructor of _DirectByteBuffer_. This is possible if it is available in JVM (not sure why it might be not).

For the mentioned aspects:
 - {color:#57d9a3}Add {{long allocate()}} and {{void release(long)}} to a core util, like {{MemoryUtils}} to abstract over Unsafe, so that it is easier to deal with the Unsafe removal later{color}: as I see we can extend _MemorySegmentFactory._
 - {color:#57d9a3}Release the memory in the {{free()}} method of the memory segments. Good point, that was also my plan for _HybridUnsafeMemorySegment_.{color}
 - {color:#57d9a3}Add a safety net that frees the memory segments upon garbage collection. Have a look at how the DirectByteBuffer does it with the cleaners and reference queues{color}. Good point I will have a look into the details


was (Author: azagrebin):
Thanks for sharing the thoughts [~StephanEwen].

 I looked into the issue and rough idea atm is that _MemorySegmentFactory_ gets one more method to allocate _MemorySegment_ with _unsafe_.
 We need one more concrete class _HybridUnsafeMemorySegment_:
 MemorySegment << _HybridUnsafeMemorySegment_ << HybridMemorySegment
 _HybridUnsafeMemorySegment_ is pretty much copy of _HybridMemorySegment_, except it does not have _ByteBuffer offHeapBuffer_ and returns _ByteBuffer_ from _getOffHeapBuffer_ and wrap methods in a different way.
 There is no straightforward way to wrap unsafe memory with _ByteBuffer_. I found [netty does this|#L469] by hacking into the private constructor of _DirectByteBuffer_. This is possible if it is available in JVM (not sure why it might be not).

For the mentioned aspects:
 - {color:#57d9a3}Add {{long allocate()}} and {{void release(long)}} to a core util, like {{MemoryUtils}} to abstract over Unsafe, so that it is easier to deal with the Unsafe removal later{color}: as I see we can extend _MemorySegmentFactory._
 - {color:#57d9a3}Release the memory in the {{free()}} method of the memory segments.{color:#172b4d} Good point, that was also my plan for _HybridUnsafeMemorySegment_.{color}{color}
 - {color:#57d9a3}Add a safety net that frees the memory segments upon garbage collection. Have a look at how the DirectByteBuffer does it with the cleaners and reference queues.{color} Good point I will have a look into the details

> Use native memory for managed memory.
> -------------------------------------
>
>                 Key: FLINK-13985
>                 URL: https://issues.apache.org/jira/browse/FLINK-13985
>             Project: Flink
>          Issue Type: Sub-task
>            Reporter: Xintong Song
>            Assignee: Andrey Zagrebin
>            Priority: Major
>
> * Allocate memory with {{Unsafe.allocateMemory}}
>  ** {{MemoryManager}}
> Implement this issue in common code paths for the legacy / new mode. This should only affect the GC behavior.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)