[jira] [Comment Edited] (FLINK-14566) Enable ResourceSpec to get/set whether managed memory is used

["Zhu Zhu (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/FLINK-14566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16963844#comment-16963844 ] 

Zhu Zhu edited comment on FLINK-14566 at 10/31/19 10:36 AM:
------------------------------------------------------------

[~ykt836] When saying "little", I mean even "no" in the worst case. The operator can throw error in this case. 
But it might be acceptable, given that the resources is set to be UNKNOWN. Just like OOM is acceptable if the heap resource is not specified(UNKNOWN) and the TM heap memory is not properly configured.
To solve it, users need to specify the required resource.

Only counting operators really using managed memory(FLINK-14566 + FLINK-14062) is an optimization to reduce managed memory waste, but it still cannot guarantee an operator will always acquire enough managed memory to work. So technically it's not necessary.


was (Author: zhuzh):
[~ykt836] When saying "little", I mean even "no" in the worst case. The operator can throw error in this case. 
But it might be acceptable, given that the resources is set to be UNKNOWN. Just like OOM is acceptable if the heap resource is not specified(UNKNOWN) and the TM heap memory is not properly configured.
To solve it, users need to specify the required resource.

Only counting operators really using managed memory is an optimization to reduce managed memory waste, but it still cannot guarantee an operator will always acquire enough managed memory to work. So technically it's not necessary.

> Enable ResourceSpec to get/set whether managed memory is used
> -------------------------------------------------------------
>
>                 Key: FLINK-14566
>                 URL: https://issues.apache.org/jira/browse/FLINK-14566
>             Project: Flink
>          Issue Type: Sub-task
>          Components: Runtime / Coordination
>    Affects Versions: 1.10.0
>            Reporter: Zhu Zhu
>            Priority: Major
>             Fix For: 1.10.0
>
>
> To calculate managed memory fraction for an operator with UNKNOWN resources, we need to know whether the operator will use managed memory, according to FLINK-14062.
> I'd propose to define special values of onHeapManagedMemory/offHeapManagedMemory 
> in ResourceSpec(introduced in FLINK-14405) to distinguish whether managed memory is used. 
> - For UNKNOWN resource spec, onHeapManagedMemory==MemorySize.ZERO means on heap managed memory is used, while onHeapManagedMemory==null means not. For non-UNKNOWN resource spec, managed memory size larger than zero indicates it is used.
> - similar for off heap managed memory.
> getters: isUsingManagedMemoryOnHeap()/isUsingManagedMemoryOffHeap()
> setters: setUsingManagedMemoryOnHeap(boolean)/setUsingManagedMemoryOffHeap(boolean)
> (setters are only allowed to invoke on UNKNOWN, should be ignored for non-UNKNOWN for compatibility)
> {{ResourceSpec#isKnown()}} should be added to replace existing UNKNOWN usages for equality checking.
> cc [~ykt836] [~trohrmann] [~azagrebin] [~xintongsong]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)