[jira] [Closed] (FLINK-5131) Fine-grained Resource Configuration

["Till Rohrmann (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/FLINK-5131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Till Rohrmann closed FLINK-5131.
--------------------------------
    Resolution: Duplicate

Superseded by [FLIP-53|https://cwiki.apache.org/confluence/display/FLINK/FLIP-53%3A+Fine+Grained+Operator+Resource+Management]

> Fine-grained Resource Configuration
> -----------------------------------
>
>                 Key: FLINK-5131
>                 URL: https://issues.apache.org/jira/browse/FLINK-5131
>             Project: Flink
>          Issue Type: New Feature
>          Components: API / DataSet, API / DataStream, Runtime / Coordination
>            Reporter: zhijiang
>            Priority: Major
>
> Normally the UDF just creates short-life small objects and these can be recycled quickly by JVM, so most of the memory resource is controlled and managed by *TaskManager* framework.  But for some special cases, the UDF may consume much resource to create long-live big objects, so it is necessary to provide the options for professional users to define the resource usages if needed.
> The basic approach is the following:
>   - Introduce the *ResourceSpec* structure to describe the different resource factors (cpu cores, heap memory, direct memory, native memory, etc) and provide some basic construction methods for resource group.
>   - The *ResourceSpec* can be setted onto the internal transformation in DataStream and base operator in DataSet separately.
>   - The *StateBackend* should provide the method to estimate the memory usage based on hint of state size in *ResourceSpec*.
>   - In job graph generation, the *ResourceSpec* will be aggregated  for chained operators.
>   - When *JobManager* requests slot for submitting task from *ResourceManager*, the *ResourceProfile* will be expanded  to correspond with *ResourceSpec*.
>   - The *ResourceManager* requests resource for container from cluster, it should consider extra framework memory except for slot *ResourceProfile*.
>   - The framework memory is mainly used by *NetworkBufferPool* and *MemoryManager* in *TaskManager*, and it can be configured in job level.
>   - Apart from resource, The JVM options attached with container should be supported and could also be configured in job level.
> This feature will be implemented directly into flip-6 branch.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)