[jira] [Resolved] (FLINK-132) Ozone Memory Layout

["Ufuk Celebi (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/FLINK-132?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Ufuk Celebi resolved FLINK-132.
-------------------------------

    Resolution: Not a Problem

> Ozone Memory Layout
> -------------------
>
>                 Key: FLINK-132
>                 URL: https://issues.apache.org/jira/browse/FLINK-132
>             Project: Flink
>          Issue Type: Bug
>            Reporter: GitHub Import
>              Labels: github-import
>             Fix For: pre-apache
>
>
> Hi residents of the Stratosphere,
> I found it a bit difficult to get a big picture of the memory usage of Ozone so I created this small comparison graphic (ozone vs. yarn). You can ignore yarn...
> Can someone confirm that it looks like this on the high level?
> ![memory|https://f.cloud.github.com/assets/659564/1288867/19996e3e-3018-11e3-93bb-dd6bfca4ed1e.png)
> Having such high level picture in the documentation would be great (with the relevant names of the configuration options added). I also omitted details like default values (memory manager now defaults to 0.7*(heap space) if I read correctly from source)
> #### Network Buffers
> First I thought (and heard) that the network buffers are direct buffers and not taken from the heap space. But during my experiments (cloud-11) I found out that they seem to be taken from the heap space:
> **Settings that did not work**
> DEFAULT_NEPHELE_TM_HEAP: 10GB
> taskmanager.memory.size: 8GB
> Network buffers: 4GB (numberBuffers: 32 KB bufferSize: 128 KB)
> => Task manager did not start, not enough heap space for 8+4 GB
> 	Exception in thread "main" java.lang.OutOfMemoryError: Java heap space 
> 	        at eu.stratosphere.nephele.services.memorymanager.spi.DefaultMemoryManager.<init>(DefaultMemoryManager.java:144) 
> 	        at eu.stratosphere.nephele.services.memorymanager.spi.DefaultMemoryManager.<init>(DefaultMemoryManager.java:100) 
> 	        at eu.stratosphere.nephele.taskmanager.TaskManager.<init>(TaskManager.java:322) 
> 	        at eu.stratosphere.nephele.taskmanager.TaskManager.main(TaskManager.java:374)
> **Settings that worked**
> DEFAULT_NEPHELE_TM_HEAP: 10GB
> taskmanager.memory.size 8GB
> Network buffers: 1GB (numberBuffers: 32 KB bufferSize: 32 KB]
> => worked, there is 1GB heap space left
> ---------------- Imported from GitHub ----------------
> Url: https://github.com/stratosphere/stratosphere/issues/132
> Created by: [andrehacker|https://github.com/andrehacker]
> Labels: documentation, 
> Created at: Tue Oct 08 10:05:37 CEST 2013
> State: open



--
This message was sent by Atlassian JIRA
(v6.2#6252)