[jira] [Commented] (FLINK-7400) off-heap limits set to conservatively in cluster environments

["ASF GitHub Bot (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/FLINK-7400?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16216584#comment-16216584 ] 

ASF GitHub Bot commented on FLINK-7400:
---------------------------------------

Github user tillrohrmann commented on a diff in the pull request:

    https://github.com/apache/flink/pull/4506#discussion_r146479686
  
    --- Diff: flink-yarn-tests/src/test/java/org/apache/flink/yarn/YARNSessionCapacitySchedulerITCase.java ---
    @@ -132,6 +132,36 @@ public void perJobYarnCluster() {
     	}
     
     	/**
    +	 * Test per-job yarn cluster and memory calculations for off-heap use (see FLINK-7400).
    +	 *
    +	 * <p>This also tests the prefixed CliFrontend options for the YARN case
    +	 * We also test if the requested parallelism of 2 is passed through.
    +	 * The parallelism is requested at the YARN client (-ys).
    +	 */
    +	@Test
    +	public void perJobYarnClusterOffHeap() {
    +		LOG.info("Starting perJobYarnCluster()");
    +		addTestAppender(JobClient.class, Level.INFO);
    +		File exampleJarLocation = new File("target/programs/BatchWordCount.jar");
    +		Assert.assertNotNull("Could not find wordcount jar", exampleJarLocation);
    +		runWithArgs(new String[]{"run", "-m", "yarn-cluster",
    +				"-yj", flinkUberjar.getAbsolutePath(), "-yt", flinkLibFolder.getAbsolutePath(),
    +				"-yn", "1",
    +				"-ys", "2", //test that the job is executed with a DOP of 2
    +				"-yjm", "768",
    +				"-ytm", "1024",
    +				"-yD", "taskmanager.memory.off-heap=true",
    +				"-yD", "taskmanager.memory.size=246", // this should fit!
    --- End diff --
    
    Could you add some explanation why this fits now and not before?


> off-heap limits set to conservatively in cluster environments
> -------------------------------------------------------------
>
>                 Key: FLINK-7400
>                 URL: https://issues.apache.org/jira/browse/FLINK-7400
>             Project: Flink
>          Issue Type: Bug
>          Components: Cluster Management, Mesos, YARN
>    Affects Versions: 1.3.0, 1.3.1, 1.3.2
>            Reporter: Nico Kruber
>            Assignee: Nico Kruber
>
> Inside {{ContaineredTaskManagerParameters}}, since FLINK-6217, the {{offHeapSize}} is set to the amount of memory Flink will use off-heap which will be set as the value for {{-XX:MaxDirectMemorySize}} in various cases. This does not account for any off-heap use by other components than Flink, e.g. RocksDB, other libraries, or the JVM itself.
> We should add the {{cutoff}} from the {{CONTAINERIZED_HEAP_CUTOFF_RATIO}} configuration parameter to {{offHeapSize}} as implied by the description on what this parameter is there for.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)