You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "Piotr Nowojski (JIRA)" <ji...@apache.org> on 2018/03/01 16:14:00 UTC
[jira] [Updated] (FLINK-8826) In Flip6 mode, when starting yarn
cluster, configured taskmanager.heap.mb is ignored
[ https://issues.apache.org/jira/browse/FLINK-8826?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Piotr Nowojski updated FLINK-8826:
----------------------------------
Description:
When I tried running some job on the cluster, despite setting
taskmanager.heap.mb = 3072
taskmanager.network.memory.fraction: 0.4
and reported in the console
{code:java}
Cluster specification: ClusterSpecification{masterMemoryMB=768, taskManagerMemoryMB=3072, numberTaskManagers=92, slotsPerTaskManager=1}{code}
The actual settings were:
{noformat}
2018-03-01 14:53:18,918 INFO org.apache.flink.yarn.YarnTaskExecutorRunner - --------------------------------------------------------------------------------
2018-03-01 14:53:18,921 INFO org.apache.flink.yarn.YarnTaskExecutorRunner - Starting YARN TaskExecutor runner (Version: 1.5-SNAPSHOT, Rev:e92eb39, Date:28.02.2018 @ 17:43:39 UTC)
2018-03-01 14:53:18,921 INFO org.apache.flink.yarn.YarnTaskExecutorRunner - OS current user: yarn
2018-03-01 14:53:19,780 INFO org.apache.flink.yarn.YarnTaskExecutorRunner - Current Hadoop/Kerberos user: hadoop
2018-03-01 14:53:19,781 INFO org.apache.flink.yarn.YarnTaskExecutorRunner - JVM: OpenJDK 64-Bit Server VM - Oracle Corporation - 1.8/25.161-b14
2018-03-01 14:53:19,781 INFO org.apache.flink.yarn.YarnTaskExecutorRunner - Maximum heap size: 245 MiBytes
2018-03-01 14:53:19,781 INFO org.apache.flink.yarn.YarnTaskExecutorRunner - JAVA_HOME: /usr/lib/jvm/java-openjdk
2018-03-01 14:53:19,783 INFO org.apache.flink.yarn.YarnTaskExecutorRunner - Hadoop version: 2.4.1
2018-03-01 14:53:19,783 INFO org.apache.flink.yarn.YarnTaskExecutorRunner - JVM Options:
2018-03-01 14:53:19,783 INFO org.apache.flink.yarn.YarnTaskExecutorRunner - -Xms255m
2018-03-01 14:53:19,784 INFO org.apache.flink.yarn.YarnTaskExecutorRunner - -Xmx255m
2018-03-01 14:53:19,784 INFO org.apache.flink.yarn.YarnTaskExecutorRunner - -XX:MaxDirectMemorySize=769m
2018-03-01 14:53:19,784 INFO org.apache.flink.yarn.YarnTaskExecutorRunner - -Dlog.file=/var/log/hadoop-yarn/containers/application_1516373731080_1150/container_1516373731080_1150_01_000105/taskmanager.log
2018-03-01 14:53:19,784 INFO org.apache.flink.yarn.YarnTaskExecutorRunner - -Dlogback.configurationFile=file:./logback.xml
2018-03-01 14:53:19,784 INFO org.apache.flink.yarn.YarnTaskExecutorRunner - -Dlog4j.configuration=file:./log4j.properties
2018-03-01 14:53:19,784 INFO org.apache.flink.yarn.YarnTaskExecutorRunner - Program Arguments:
2018-03-01 14:53:19,784 INFO org.apache.flink.yarn.YarnTaskExecutorRunner - --configDir{noformat}
Heap was set to 255, while with default cuts of it should be 1383. 255MB seems like coming from default taskmanager.heap.mb value of 1024.
When starting in non flip6 everything works as expected:
{noformat}
2018-03-01 14:04:49,650 INFO org.apache.flink.yarn.YarnTaskManagerRunnerFactory - --------------------------------------------------------------------------------
2018-03-01 14:04:49,700 INFO org.apache.flink.yarn.YarnTaskManagerRunnerFactory - Starting YARN TaskManager (Version: 1.5-SNAPSHOT, Rev:e92eb39, Date:28.02.2018 @ 17:43:39 UTC)
2018-03-01 14:04:49,700 INFO org.apache.flink.yarn.YarnTaskManagerRunnerFactory - OS current user: yarn
2018-03-01 14:04:53,277 INFO org.apache.flink.yarn.YarnTaskManagerRunnerFactory - Current Hadoop/Kerberos user: hadoop
2018-03-01 14:04:53,278 INFO org.apache.flink.yarn.YarnTaskManagerRunnerFactory - JVM: OpenJDK 64-Bit Server VM - Oracle Corporation - 1.8/25.161-b14
2018-03-01 14:04:53,279 INFO org.apache.flink.yarn.YarnTaskManagerRunnerFactory - Maximum heap size: 1326 MiBytes
2018-03-01 14:04:53,279 INFO org.apache.flink.yarn.YarnTaskManagerRunnerFactory - JAVA_HOME: /usr/lib/jvm/java-openjdk
2018-03-01 14:04:53,282 INFO org.apache.flink.yarn.YarnTaskManagerRunnerFactory - Hadoop version: 2.4.1
2018-03-01 14:04:53,284 INFO org.apache.flink.yarn.YarnTaskManagerRunnerFactory - JVM Options:
2018-03-01 14:04:53,284 INFO org.apache.flink.yarn.YarnTaskManagerRunnerFactory - -Xms1383m
2018-03-01 14:04:53,284 INFO org.apache.flink.yarn.YarnTaskManagerRunnerFactory - -Xmx1383m
2018-03-01 14:04:53,284 INFO org.apache.flink.yarn.YarnTaskManagerRunnerFactory - -XX:MaxDirectMemorySize=1689m
2018-03-01 14:04:53,284 INFO org.apache.flink.yarn.YarnTaskManagerRunnerFactory - -Dlog.file=/var/log/hadoop-yarn/containers/application_1516373731080_1138/container_1516373731080_1138_01_000063/taskmanager.log
2018-03-01 14:04:53,285 INFO org.apache.flink.yarn.YarnTaskManagerRunnerFactory - -Dlogback.configurationFile=file:./logback.xml
2018-03-01 14:04:53,286 INFO org.apache.flink.yarn.YarnTaskManagerRunnerFactory - -Dlog4j.configuration=file:./log4j.properties
2018-03-01 14:04:53,287 INFO org.apache.flink.yarn.YarnTaskManagerRunnerFactory - Program Arguments:
2018-03-01 14:04:53,287 INFO org.apache.flink.yarn.YarnTaskManagerRunnerFactory - --configDir
2018-03-01 14:04:53,287 INFO org.apache.flink.yarn.YarnTaskManagerRunnerFactory - .{noformat}
CC [~till.rohrmann]
was:
When I tried running some job on the cluster, despite setting
taskmanager.heap.mb = 3072
taskmanager.network.memory.fraction: 0.4
and reported in the console
{{
Cluster specification: ClusterSpecification\{masterMemoryMB=768, taskManagerMemoryMB=3072, numberTaskManagers=92, slotsPerTaskManager=1}
}}
The actual settings were:
{{
2018-03-01 14:53:18,918 INFO org.apache.flink.yarn.YarnTaskExecutorRunner - --------------------------------------------------------------------------------
2018-03-01 14:53:18,921 INFO org.apache.flink.yarn.YarnTaskExecutorRunner - Starting YARN TaskExecutor runner (Version: 1.5-SNAPSHOT, Rev:e92eb39, Date:28.02.2018 @ 17:43:39 UTC)
2018-03-01 14:53:18,921 INFO org.apache.flink.yarn.YarnTaskExecutorRunner - OS current user: yarn
2018-03-01 14:53:19,780 INFO org.apache.flink.yarn.YarnTaskExecutorRunner - Current Hadoop/Kerberos user: hadoop
2018-03-01 14:53:19,781 INFO org.apache.flink.yarn.YarnTaskExecutorRunner - JVM: OpenJDK 64-Bit Server VM - Oracle Corporation - 1.8/25.161-b14
2018-03-01 14:53:19,781 INFO org.apache.flink.yarn.YarnTaskExecutorRunner - Maximum heap size: 245 MiBytes
2018-03-01 14:53:19,781 INFO org.apache.flink.yarn.YarnTaskExecutorRunner - JAVA_HOME: /usr/lib/jvm/java-openjdk
2018-03-01 14:53:19,783 INFO org.apache.flink.yarn.YarnTaskExecutorRunner - Hadoop version: 2.4.1
2018-03-01 14:53:19,783 INFO org.apache.flink.yarn.YarnTaskExecutorRunner - JVM Options:
2018-03-01 14:53:19,783 INFO org.apache.flink.yarn.YarnTaskExecutorRunner - -Xms255m
2018-03-01 14:53:19,784 INFO org.apache.flink.yarn.YarnTaskExecutorRunner - -Xmx255m
2018-03-01 14:53:19,784 INFO org.apache.flink.yarn.YarnTaskExecutorRunner - -XX:MaxDirectMemorySize=769m
2018-03-01 14:53:19,784 INFO org.apache.flink.yarn.YarnTaskExecutorRunner - -Dlog.file=/var/log/hadoop-yarn/containers/application_1516373731080_1150/container_1516373731080_1150_01_000105/taskmanager.log
2018-03-01 14:53:19,784 INFO org.apache.flink.yarn.YarnTaskExecutorRunner - -Dlogback.configurationFile=file:./logback.xml
2018-03-01 14:53:19,784 INFO org.apache.flink.yarn.YarnTaskExecutorRunner - -Dlog4j.configuration=file:./log4j.properties
2018-03-01 14:53:19,784 INFO org.apache.flink.yarn.YarnTaskExecutorRunner - Program Arguments:
2018-03-01 14:53:19,784 INFO org.apache.flink.yarn.YarnTaskExecutorRunner - --configDir
}}
Heap was set to 255, while with default cuts of it should be 1383. 255MB seems like coming from default taskmanager.heap.mb value of 1024.
When starting in non flip6 everything works as expected:
{{
2018-03-01 14:04:49,650 INFO org.apache.flink.yarn.YarnTaskManagerRunnerFactory - --------------------------------------------------------------------------------
2018-03-01 14:04:49,700 INFO org.apache.flink.yarn.YarnTaskManagerRunnerFactory - Starting YARN TaskManager (Version: 1.5-SNAPSHOT, Rev:e92eb39, Date:28.02.2018 @ 17:43:39 UTC)
2018-03-01 14:04:49,700 INFO org.apache.flink.yarn.YarnTaskManagerRunnerFactory - OS current user: yarn
2018-03-01 14:04:53,277 INFO org.apache.flink.yarn.YarnTaskManagerRunnerFactory - Current Hadoop/Kerberos user: hadoop
2018-03-01 14:04:53,278 INFO org.apache.flink.yarn.YarnTaskManagerRunnerFactory - JVM: OpenJDK 64-Bit Server VM - Oracle Corporation - 1.8/25.161-b14
2018-03-01 14:04:53,279 INFO org.apache.flink.yarn.YarnTaskManagerRunnerFactory - Maximum heap size: 1326 MiBytes
2018-03-01 14:04:53,279 INFO org.apache.flink.yarn.YarnTaskManagerRunnerFactory - JAVA_HOME: /usr/lib/jvm/java-openjdk
2018-03-01 14:04:53,282 INFO org.apache.flink.yarn.YarnTaskManagerRunnerFactory - Hadoop version: 2.4.1
2018-03-01 14:04:53,284 INFO org.apache.flink.yarn.YarnTaskManagerRunnerFactory - JVM Options:
2018-03-01 14:04:53,284 INFO org.apache.flink.yarn.YarnTaskManagerRunnerFactory - -Xms1383m
2018-03-01 14:04:53,284 INFO org.apache.flink.yarn.YarnTaskManagerRunnerFactory - -Xmx1383m
2018-03-01 14:04:53,284 INFO org.apache.flink.yarn.YarnTaskManagerRunnerFactory - -XX:MaxDirectMemorySize=1689m
2018-03-01 14:04:53,284 INFO org.apache.flink.yarn.YarnTaskManagerRunnerFactory - -Dlog.file=/var/log/hadoop-yarn/containers/application_1516373731080_1138/container_1516373731080_1138_01_000063/taskmanager.log
2018-03-01 14:04:53,285 INFO org.apache.flink.yarn.YarnTaskManagerRunnerFactory - -Dlogback.configurationFile=file:./logback.xml
2018-03-01 14:04:53,286 INFO org.apache.flink.yarn.YarnTaskManagerRunnerFactory - -Dlog4j.configuration=file:./log4j.properties
2018-03-01 14:04:53,287 INFO org.apache.flink.yarn.YarnTaskManagerRunnerFactory - Program Arguments:
2018-03-01 14:04:53,287 INFO org.apache.flink.yarn.YarnTaskManagerRunnerFactory - --configDir
2018-03-01 14:04:53,287 INFO org.apache.flink.yarn.YarnTaskManagerRunnerFactory - .
}}
CC [~till.rohrmann]
> In Flip6 mode, when starting yarn cluster, configured taskmanager.heap.mb is ignored
> ------------------------------------------------------------------------------------
>
> Key: FLINK-8826
> URL: https://issues.apache.org/jira/browse/FLINK-8826
> Project: Flink
> Issue Type: Bug
> Components: ResourceManager, YARN
> Affects Versions: 1.5.0
> Reporter: Piotr Nowojski
> Priority: Blocker
>
> When I tried running some job on the cluster, despite setting
> taskmanager.heap.mb = 3072
> taskmanager.network.memory.fraction: 0.4
> and reported in the console
> {code:java}
> Cluster specification: ClusterSpecification{masterMemoryMB=768, taskManagerMemoryMB=3072, numberTaskManagers=92, slotsPerTaskManager=1}{code}
> The actual settings were:
> {noformat}
>
> 2018-03-01 14:53:18,918 INFO org.apache.flink.yarn.YarnTaskExecutorRunner - --------------------------------------------------------------------------------
> 2018-03-01 14:53:18,921 INFO org.apache.flink.yarn.YarnTaskExecutorRunner - Starting YARN TaskExecutor runner (Version: 1.5-SNAPSHOT, Rev:e92eb39, Date:28.02.2018 @ 17:43:39 UTC)
> 2018-03-01 14:53:18,921 INFO org.apache.flink.yarn.YarnTaskExecutorRunner - OS current user: yarn
> 2018-03-01 14:53:19,780 INFO org.apache.flink.yarn.YarnTaskExecutorRunner - Current Hadoop/Kerberos user: hadoop
> 2018-03-01 14:53:19,781 INFO org.apache.flink.yarn.YarnTaskExecutorRunner - JVM: OpenJDK 64-Bit Server VM - Oracle Corporation - 1.8/25.161-b14
> 2018-03-01 14:53:19,781 INFO org.apache.flink.yarn.YarnTaskExecutorRunner - Maximum heap size: 245 MiBytes
> 2018-03-01 14:53:19,781 INFO org.apache.flink.yarn.YarnTaskExecutorRunner - JAVA_HOME: /usr/lib/jvm/java-openjdk
> 2018-03-01 14:53:19,783 INFO org.apache.flink.yarn.YarnTaskExecutorRunner - Hadoop version: 2.4.1
> 2018-03-01 14:53:19,783 INFO org.apache.flink.yarn.YarnTaskExecutorRunner - JVM Options:
> 2018-03-01 14:53:19,783 INFO org.apache.flink.yarn.YarnTaskExecutorRunner - -Xms255m
> 2018-03-01 14:53:19,784 INFO org.apache.flink.yarn.YarnTaskExecutorRunner - -Xmx255m
> 2018-03-01 14:53:19,784 INFO org.apache.flink.yarn.YarnTaskExecutorRunner - -XX:MaxDirectMemorySize=769m
> 2018-03-01 14:53:19,784 INFO org.apache.flink.yarn.YarnTaskExecutorRunner - -Dlog.file=/var/log/hadoop-yarn/containers/application_1516373731080_1150/container_1516373731080_1150_01_000105/taskmanager.log
> 2018-03-01 14:53:19,784 INFO org.apache.flink.yarn.YarnTaskExecutorRunner - -Dlogback.configurationFile=file:./logback.xml
> 2018-03-01 14:53:19,784 INFO org.apache.flink.yarn.YarnTaskExecutorRunner - -Dlog4j.configuration=file:./log4j.properties
> 2018-03-01 14:53:19,784 INFO org.apache.flink.yarn.YarnTaskExecutorRunner - Program Arguments:
> 2018-03-01 14:53:19,784 INFO org.apache.flink.yarn.YarnTaskExecutorRunner - --configDir{noformat}
> Heap was set to 255, while with default cuts of it should be 1383. 255MB seems like coming from default taskmanager.heap.mb value of 1024.
> When starting in non flip6 everything works as expected:
> {noformat}
>
> 2018-03-01 14:04:49,650 INFO org.apache.flink.yarn.YarnTaskManagerRunnerFactory - --------------------------------------------------------------------------------
> 2018-03-01 14:04:49,700 INFO org.apache.flink.yarn.YarnTaskManagerRunnerFactory - Starting YARN TaskManager (Version: 1.5-SNAPSHOT, Rev:e92eb39, Date:28.02.2018 @ 17:43:39 UTC)
> 2018-03-01 14:04:49,700 INFO org.apache.flink.yarn.YarnTaskManagerRunnerFactory - OS current user: yarn
> 2018-03-01 14:04:53,277 INFO org.apache.flink.yarn.YarnTaskManagerRunnerFactory - Current Hadoop/Kerberos user: hadoop
> 2018-03-01 14:04:53,278 INFO org.apache.flink.yarn.YarnTaskManagerRunnerFactory - JVM: OpenJDK 64-Bit Server VM - Oracle Corporation - 1.8/25.161-b14
> 2018-03-01 14:04:53,279 INFO org.apache.flink.yarn.YarnTaskManagerRunnerFactory - Maximum heap size: 1326 MiBytes
> 2018-03-01 14:04:53,279 INFO org.apache.flink.yarn.YarnTaskManagerRunnerFactory - JAVA_HOME: /usr/lib/jvm/java-openjdk
> 2018-03-01 14:04:53,282 INFO org.apache.flink.yarn.YarnTaskManagerRunnerFactory - Hadoop version: 2.4.1
> 2018-03-01 14:04:53,284 INFO org.apache.flink.yarn.YarnTaskManagerRunnerFactory - JVM Options:
> 2018-03-01 14:04:53,284 INFO org.apache.flink.yarn.YarnTaskManagerRunnerFactory - -Xms1383m
> 2018-03-01 14:04:53,284 INFO org.apache.flink.yarn.YarnTaskManagerRunnerFactory - -Xmx1383m
> 2018-03-01 14:04:53,284 INFO org.apache.flink.yarn.YarnTaskManagerRunnerFactory - -XX:MaxDirectMemorySize=1689m
> 2018-03-01 14:04:53,284 INFO org.apache.flink.yarn.YarnTaskManagerRunnerFactory - -Dlog.file=/var/log/hadoop-yarn/containers/application_1516373731080_1138/container_1516373731080_1138_01_000063/taskmanager.log
> 2018-03-01 14:04:53,285 INFO org.apache.flink.yarn.YarnTaskManagerRunnerFactory - -Dlogback.configurationFile=file:./logback.xml
> 2018-03-01 14:04:53,286 INFO org.apache.flink.yarn.YarnTaskManagerRunnerFactory - -Dlog4j.configuration=file:./log4j.properties
> 2018-03-01 14:04:53,287 INFO org.apache.flink.yarn.YarnTaskManagerRunnerFactory - Program Arguments:
> 2018-03-01 14:04:53,287 INFO org.apache.flink.yarn.YarnTaskManagerRunnerFactory - --configDir
> 2018-03-01 14:04:53,287 INFO org.apache.flink.yarn.YarnTaskManagerRunnerFactory - .{noformat}
>
> CC [~till.rohrmann]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)