OutOfMemory error (Direct buffer memory) while allocating the TaskManager off-heap memory

[Yassine MARZOUGUI <y....@mindlytix.com>]

Hi all,

I tried starting a local Flink 1.2.0 cluster using start-local.sh, with the
following settings for the taskmanager memory:

taskmanager.heap.mb: 16384
taskmanager.memory.off-heap: true
taskmanager.memory.preallocate: true

That throws and OOM error:
Caused by: java.lang.Exception: OutOfMemory error (Direct buffer memory)
while allocating the TaskManager off-heap memory (39017161219 bytes). Try
increasing the maximum direct memory (-XX:MaxDirectMemorySize)

However If I add an obsolute taskmanager.memory.size:
taskmanager.memory.size: 15360
the cluster starts successfully.

My understanding is that if taskmanager.memory.size is unspecified then it
should be equal to 0.7 * taskmanager.heap.mb. So I don't understand why it
throws an exception and it works if its larger than that fraction.

Any help is appreciated.

Best,
Yassine

Re: OutOfMemory error (Direct buffer memory) while allocating the TaskManager off-heap memory

[Nico Kruber <ni...@data-artisans.com>]

Hi Yassine,
Thanks for reporting this. The problem you run into is due to start-local.sh 
which we discourage in favour of start-cluster.sh that resembles real use case 
better.

In your case, start-local.sh starts a job manager with an embedded task 
manager but does not parse the task manager config properly to set the right 
parameters.
With start-cluster.sh (or manually via jobmanager.sh and taskmanager.sh), job 
and task manager are started separately in separate JVMs without this issue.

There's an open pull request to make start-cluster.sh better cooperate in 
certain situations in order to replace start-local.sh completely[1] but it 
hasn't been merged yet nor is start-local.sh replaced. In the future, we might 
do that though.

FYI: I created a Jira issue for us to check further code paths that may lead 
to this problem:
https://issues.apache.org/jira/browse/FLINK-5973


Regards
Nico

[1] https://github.com/apache/flink/pull/3298

On Friday, 3 March 2017 17:07:14 CET Yassine MARZOUGUI wrote:
> Hi all,
> 
> I tried starting a local Flink 1.2.0 cluster using start-local.sh, with the
> following settings for the taskmanager memory:
> 
> taskmanager.heap.mb: 16384
> taskmanager.memory.off-heap: true
> taskmanager.memory.preallocate: true
> 
> That throws and OOM error:
> Caused by: java.lang.Exception: OutOfMemory error (Direct buffer memory)
> while allocating the TaskManager off-heap memory (39017161219 bytes). Try
> increasing the maximum direct memory (-XX:MaxDirectMemorySize)
> 
> However If I add an obsolute taskmanager.memory.size:
> taskmanager.memory.size: 15360
> the cluster starts successfully.
> 
> My understanding is that if taskmanager.memory.size is unspecified then it
> should be equal to 0.7 * taskmanager.heap.mb. So I don't understand why it
> throws an exception and it works if its larger than that fraction.
> 
> Any help is appreciated.
> 
> Best,
> Yassine