You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "Maximilian Michels (JIRA)" <ji...@apache.org> on 2015/09/10 19:31:45 UTC
[jira] [Comment Edited] (FLINK-2641) Integrate the off-heap memory
configuration with the TaskManager start script
[ https://issues.apache.org/jira/browse/FLINK-2641?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14738986#comment-14738986 ]
Maximilian Michels edited comment on FLINK-2641 at 9/10/15 5:30 PM:
--------------------------------------------------------------------
The new configuration paramters are
- taskmanager.memory.off-heap: Boolean
- taskmanager.memory.off-heap-ratio: Float
These have to interplay with these existing config entries:
- taskmanager.memory.size: Int
- taskmanager.memory.fraction: Float
- taskmanager.heap.mb: Int
This works as follows:
* If {{taskmanager.memory.size}} is set, the task manager allocates the number of memory either on the JVM heap or as direct memory depending on whether {{taskmanager.memory.off-heap}} is set to true.
* If {{taskmanager.memory.size}} is not set, the memory allocation is slightly more involved:
** In the standard heap mode ({{taskmanager.memory.off-heap}} set to false), the task manager gets the free heap memory during startup, multiplies it by {{taskmanager.memory.fraction}}, and allocates the resulting memory size as heap memory.
** In off-heap mode ({{taskmanager.memory.off-heap}} set to true), the task manager gets the maximum available allocatable memory (set by the -Xmx parameter), multiplies it by the {{taskmanager.memory.off-heap-ratio}}, and allocates the resulting memory size as direct (off-heap) memory size.
In order to be able to switch from heap to off-heap memory usage and not having to adjust any memory settings, the {{taskmanager.heap.mb}} setting has to be dynamically adjusted:
* If {{taskmanager.memory.off-heap}} is set and
** {{taskmanager.memory.size}} is set, then the maximum JVM memory size needs to be adjusted to {{taskmanager.heap.mb - taskmanager.memory.size}}.
** {{taskmanager.memory.size}} is not set, the maximum JVM memory size needs to be adjusted to {{taskmanager.heap.mb * (1/(taskmanager.memory.off-heap-ratio+1))}}. Note that {{taskmanager.memory.off-heap-ratio > 1.0}}.
-----
Up for discussion:
It might confuse users that {{taskmanager.heap.mb}} specifies the JVM heap size but we use it to calculate the total memory (on and off-heap combined). Therefore, I'd propose to add another config entry which let's you specify the JVM process memory: {{taskmanager.jvm.mb}} would set the maximum memory which is then split up between JVM heap and off-heap memory. The old parameter can still be used to explicitly set the heap memory. Since we introduce new config parameters anyways with the off-heap memory, I think this is a fair compromise.
{{taskmanager.heap.mb}} will be deprecated in favor of {{taskmanager.jvm.mb}}. Then we should also rename {{jobmanager.heap.mb}} to {{jobmanager.jvm.mb}}. Since I'm not entirely sure about the name and whether we should introduce additional config parameters, I will leave that up for discussion now and implement the config parameter handling with the default {{taskmanager.heap.mb}} parameter and its dynamic adjustment.
was (Author: mxm):
The new configuration paramters are
- taskmanager.memory.off-heap: Boolean
- taskmanager.memory.off-heap-ratio: Float
These have to interplay with these existing config entries:
- taskmanager.memory.size: Int
- taskmanager.memory.fraction: Float
- taskmanager.heap.mb: Int
This works as follows:
* If {{taskmanager.memory.size}} is set, the task manager allocates the number of memory either on the JVM heap or as direct memory depending on whether {{taskmanager.memory.off-heap}} is set to true.
* If {{taskmanager.memory.size}} is not set, the memory allocation is slightly more involved:
** In the standard heap mode ({{taskmanager.memory.off-heap}} set to false), the task manager gets the free heap memory during startup, multiplies it by {{taskmanager.memory.fraction}}, and allocates the resulting memory size as heap memory.
** In off-heap mode ({{taskmanager.memory.off-heap}} set to true), the task manager gets the maximum available allocatable memory (set by the -Xmx parameter), multiplies it by the {{taskmanager.memory.off-heap-ratio}}, and allocates the resulting memory size as direct (off-heap) memory size.
In order to be able to switch from heap to off-heap memory usage and not having to adjust any memory settings, the {{taskmanager.heap.mb}} setting has to be dynamically adjusted:
* If {{taskmanager.memory.off-heap}} is set and
** {{taskmanager.memory.size}} is set, then the maximum JVM memory size needs to be adjusted to {{taskmanager.heap.mb - taskmanager.memory.size}}.
** {{taskmanager.memory.size}} is not set, the maximum JVM memory size needs to be adjusted to {{taskmanager.heap.mb * (1 - (1/taskmanager.memory.off-heap-ratio))}}. Note that {{taskmanager.memory.off-heap-ratio > 1.0}}.
-----
Up for discussion:
It might confuse users that {{taskmanager.heap.mb}} specifies the JVM heap size but we use it to calculate the total memory (on and off-heap combined). Therefore, I'd propose to add another config entry which let's you specify the JVM process memory: {{taskmanager.jvm.mb}} would set the maximum memory which is then split up between JVM heap and off-heap memory. The old parameter can still be used to explicitly set the heap memory. Since we introduce new config parameters anyways with the off-heap memory, I think this is a fair compromise.
{{taskmanager.heap.mb}} will be deprecated in favor of {{taskmanager.jvm.mb}}. Then we should also rename {{jobmanager.heap.mb}} to {{jobmanager.jvm.mb}}. Since I'm not entirely sure about the name and whether we should introduce additional config parameters, I will leave that up for discussion now and implement the config parameter handling with the default {{taskmanager.heap.mb}} parameter and its dynamic adjustment.
> Integrate the off-heap memory configuration with the TaskManager start script
> -----------------------------------------------------------------------------
>
> Key: FLINK-2641
> URL: https://issues.apache.org/jira/browse/FLINK-2641
> Project: Flink
> Issue Type: New Feature
> Components: Start-Stop Scripts
> Affects Versions: 0.10
> Reporter: Stephan Ewen
> Assignee: Maximilian Michels
> Fix For: 0.10
>
>
> The TaskManager start script needs to adjust the {{-Xmx}}, {{-Xms}}, and {{-XX:MaxDirectMemorySize}} parameters according to the off-heap memory settings.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)