Broadcast state and OutOfMemoryError: Direct buffer memory

[Dan Hill <qu...@gmail.com>]

Hi.  My team recently added broadcast state to our Flink jobs.  We've
started hitting this OOM "Direct buffer memory" error.  Is this a common
problem with broadcast state?  Or is it likely a different problem?
Thanks! - Dan

Re: Broadcast state and OutOfMemoryError: Direct buffer memory

[yanfei lei <fr...@gmail.com>]

Hi Dan,
Usually broadcast state needs more network buffers, the network buffer used
to exchange data records between tasks would request a portion of direct
memory[1],  I think it is possible to get the “Direct buffer memory” OOM
errors in this scenarios. Maybe you can try to increase
taskmanager.memory.framework.off-heap.size
<https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/config/#taskmanager-memory-framework-off-heap-size>
 and taskmanager.memory.task.off-heap.size
<https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/config/#taskmanager-memory-task-off-heap-size>
.

[1]
https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/memory/mem_setup_tm/#detailed-memory-model

Best,
Yanfei

Dan Hill <qu...@gmail.com> 于2022年10月21日周五 15:39写道：

> Hi.  My team recently added broadcast state to our Flink jobs.  We've
> started hitting this OOM "Direct buffer memory" error.  Is this a common
> problem with broadcast state?  Or is it likely a different problem?
> Thanks! - Dan
>