Task Manager Heap Memory Overflowing

[Zain Haider Nemati <za...@retailo.co>]

Hi,
I'm running a job with kafka source pulling in million of records with
parallelism 16 and Im seeing heap memory on task manager overflowing after
the job as ran for some time, no matter how much I increase the memory
allocation. Can someone help me out in this regards?

Job Stats and memory configs:

[image: image.png]

[image: image.png]

Re: Task Manager Heap Memory Overflowing

[Zhilong Hong <zh...@gmail.com>]

HI, Zain:

According to the second image you provided, the heap memory of TaskManagers
in your job is sufficient. There's no OOM at the moment. From the first
image you provided we can see the task "Flat Map" is too busy and the
upstream task "Source" is under the backpressure. Please consider
increasing the parallelism of the Flat Map task and see if the job performs
better. Increasing the parallelism can also help avoid OOM, since the data
will be distributed to more TaskManagers.

Best,
Zhilong

On Sat, May 21, 2022 at 5:19 AM Zain Haider Nemati <za...@retailo.co>
wrote:

> Hi,
> I'm running a job with kafka source pulling in million of records with
> parallelism 16 and Im seeing heap memory on task manager overflowing after
> the job as ran for some time, no matter how much I increase the memory
> allocation. Can someone help me out in this regards?
>
> Job Stats and memory configs:
>
> [image: image.png]
>
> [image: image.png]
>
>
>