JobManager Resident memory Always Increasing

[Pranjul Ahuja <ah...@gmail.com>]

Hi,

We are running the JobManager container with 1024GB out of which 512MB is allocated to the heap. We observe that the JobManager container's resident memory is always increasing and it never comes down. Heap usage remains to be constant and not rising abruptly. Can anyone help here where else this memory is going? 

Statebackend Used - HashMapStateBackend
Checkpoint Storage - JobManagerCheckpointStorage 

Re: JobManager Resident memory Always Increasing

[David Morávek <dm...@apache.org>]

Hi Pranjul,

which deployment mode are you using?

- For session cluster, I'd say it's possible that memory grows with # of
jobs.
- For application mode, there is actual user-code executed, so if you're
using some native libraries in your job driver, that may be another reason
for the growing memory.

If you want to debug the issue yourself, Yun already provided good pointers
on how to approach that.

Best,
D.


On Mon, Aug 16, 2021 at 8:38 AM Yun Tang <my...@live.com> wrote:

> Hi Pranjul,
>
> First of all, you adopted on-heap state backend: HashMapStateBackend,
> which would not use native off-heap memory. Moreover, JobManager would not
> initialize any keyed state backend instance. And if not enable high
> availability, JobManagerCheckpointStorage would also not use direct memory
> to write checkpoint stream out. Did you use some third-party native library?
>
> You could use native memory tracking [1] to see whether JVM used too much
> overhead native memory. And use memory allocator analysis tool such as
> JeMalloc [2] to see whether existed unexpected native memory usage.
>
>
> [1] https://shipilev.net/jvm/anatomy-quarks/12-native-memory-tracking/
> [2] https://gist.github.com/thomasdarimont/79b3cef01e5786210309
>
> Best
> Yun Tang
> ------------------------------
> *From:* Pranjul Ahuja <ah...@gmail.com>
> *Sent:* Monday, August 16, 2021 13:10
> *To:* user@flink.apache.org <us...@flink.apache.org>
> *Subject:* JobManager Resident memory Always Increasing
>
> Hi,
>
> We are running the JobManager container with 1024GB out of which 512MB is
> allocated to the heap. We observe that the JobManager container's resident
> memory is always increasing and it never comes down. Heap usage remains to
> be constant and not rising abruptly. Can anyone help here where else this
> memory is going?
>
> Statebackend Used - HashMapStateBackend
> Checkpoint Storage - JobManagerCheckpointStorage
>
>
>

Re: JobManager Resident memory Always Increasing

[Yun Tang <my...@live.com>]

Hi Pranjul,

First of all, you adopted on-heap state backend: HashMapStateBackend, which would not use native off-heap memory. Moreover, JobManager would not initialize any keyed state backend instance. And if not enable high availability, JobManagerCheckpointStorage would also not use direct memory to write checkpoint stream out. Did you use some third-party native library?

You could use native memory tracking [1] to see whether JVM used too much overhead native memory. And use memory allocator analysis tool such as JeMalloc [2] to see whether existed unexpected native memory usage.


[1] https://shipilev.net/jvm/anatomy-quarks/12-native-memory-tracking/
[2] https://gist.github.com/thomasdarimont/79b3cef01e5786210309

Best
Yun Tang
________________________________
From: Pranjul Ahuja <ah...@gmail.com>
Sent: Monday, August 16, 2021 13:10
To: user@flink.apache.org <us...@flink.apache.org>
Subject: JobManager Resident memory Always Increasing

Hi,

We are running the JobManager container with 1024GB out of which 512MB is allocated to the heap. We observe that the JobManager container's resident memory is always increasing and it never comes down. Heap usage remains to be constant and not rising abruptly. Can anyone help here where else this memory is going?

Statebackend Used - HashMapStateBackend
Checkpoint Storage - JobManagerCheckpointStorage