You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Trystan <en...@gmail.com> on 2022/07/12 17:49:23 UTC

Memory increase after restart

After a restart, memory usage of TaskManager pods will increase and stay
elevated. This is repeatable and grows yet again after a secondary restart.
Below you can see two restarts and the subsequent sustained memory usage
after the restarts.

I have not seen this in any memory/hashmap state based job, only rocksdb.
This state does have a TTL, if that makes a difference. This behavior can
make it very difficult for a job to recover, since it can lead to OOM
issues in Kubernetes.

Is there an obvious setting that we missed? Are there any configs we can
tune to prevent this? We are on Flink 1.13.5.

[image: Screen Shot 2022-07-12 at 12.32.06 PM.png]

Re: Memory increase after restart

Posted by Trystan <en...@gmail.com>.
Ha, I actually came across your post for another reason (related to s3
bucketing), but I didn't connect the dots.

So the issue seems to stem from the block cache being uncontrolled /
unreleased by rocksdb. And you saw no performance impact from disabling it
- I wonder that could be because you had a ton of disparate data, such that
the cache might have been constantly thrashing and missing anyway. If so,
that's our case too - the keyspace is so high that I'd be surprised if the
blocks lasted long in the cache.

I'll give it a shot!

On Thu, Jul 28, 2022, 10:10 PM yanfei lei <fr...@gmail.com> wrote:

> Flink community is trying to solve this problem by limiting RocksDB memory
> capacity strictly[1], but is blocked by some issues with rocksdb[2].
> And thanks to Yaroslav‘s great work for a workaround, this is really
> useful.
>
> [1] https://issues.apache.org/jira/browse/FLINK-15532
> [2] https://github.com/apache/flink/pull/16835
>
> Yaroslav Tkachenko <ya...@goldsky.io> 于2022年7月29日周五 00:42写道:
>
>> I worked on the issue together with the OP from the thread, and we ended
>> up keeping the block cache disabled. We even wrote a blog post about it:
>> https://shopify.engineering/optimizing-apache-flink-applications-tips
>> (last section).
>>
>> On Thu, Jul 28, 2022 at 9:33 AM Trystan <en...@gmail.com> wrote:
>>
>>> Yes, it's a very similar behavior! I am hesitant to disable to block
>>> cache. It doesn't seem like the OP followed up, though I can revive that
>>> thread and kill this one, if that's preferred.
>>>
>>> Was able to replicate this again - occasionally the memory will drop
>>> back down after some hours, but in this case it stays elevated more or less
>>> permanently.
>>>
>>> [image: Screen Shot 2022-07-28 at 11.20.11 AM.png]
>>>
>>> On Fri, Jul 15, 2022 at 11:09 AM Alexander Fedulov <
>>> alexander@ververica.com> wrote:
>>>
>>>> That sounds familiar:
>>>> https://lists.apache.org/thread/wsdtbfgzcbq5ooqljvrq45315zloys9x
>>>>
>>>> Best,
>>>> Alexander Fedulov
>>>>
>>>> On Thu, Jul 14, 2022 at 8:48 PM Yaroslav Tkachenko <ya...@goldsky.io>
>>>> wrote:
>>>>
>>>>> In this case it's probably not a classloading issue, it seems to be
>>>>> RocksDB-specific.
>>>>>
>>>>> On Thu, Jul 14, 2022 at 11:03 AM Trystan <en...@gmail.com> wrote:
>>>>>
>>>>>> We are using JobMode, but we are copying the job jar into
>>>>>> /opt/flink/lib/ and running it from there. Is that not correct, or
>>>>>> is Application Model required?
>>>>>>
>>>>>> On restart, it looks like a few hundred classes at most are loaded,
>>>>>> out of 29,000 other classes. Is that actually the likely cause? I'm also
>>>>>> unsure why this behavior would only affect rocksdb jobs and not hashmap
>>>>>> state backends.
>>>>>>
>>>>>> On Thu, Jul 14, 2022 at 12:19 PM Yaroslav Tkachenko <
>>>>>> yaroslav@goldsky.io> wrote:
>>>>>>
>>>>>>>
>>>>>>> https://nightlies.apache.org/flink/flink-docs-release-1.13/docs/ops/debugging/debugging_classloading/#avoiding-dynamic-classloading-for-user-code
>>>>>>> explain more about the classloading behaviour.
>>>>>>>
>>>>>>> You can try moving your application uber jar to the /lib folder of
>>>>>>> the Flink distribution, that will prevent classes from being reloaded on
>>>>>>> every restart. This approach assumes you use application mode for Flink.
>>>>>>>
>>>>>>> On Thu, Jul 14, 2022 at 10:13 AM Trystan <en...@gmail.com> wrote:
>>>>>>>
>>>>>>>> Yeah, this job has a value state with - according to the flink
>>>>>>>> rocksdb metrics - an estimated cardinality of 10B keys.
>>>>>>>>
>>>>>>>> I've attached screenshots with the JVM classloader metrics. These
>>>>>>>> values seem to be cumulative, not a rate. There is a slight increase on
>>>>>>>> each restart, but nothing crazy. However the constant increase seems like
>>>>>>>> something is off, I wouldn't expect it to be un/loading this much, but this
>>>>>>>> is the first time I've looked at this particular metric. Also attached is a
>>>>>>>> graph of memory usage across restarts - blue annotation lines are restart
>>>>>>>> points. In this case, it seems that most of the TMs did (eventually) drop
>>>>>>>> their memory consumption, but one particular pod hung around elevated for
>>>>>>>> nearly 12h. If I were to leave this job alone for a week or two and do a
>>>>>>>> restart, I would expect memory usage across nearly all pods to stay
>>>>>>>> permanently elevated, based on what we've seen with this behavior.
>>>>>>>>
>>>>>>>> Is the class un/loading suspicious here? Something in rocksdb not
>>>>>>>> cleaning itself up?
>>>>>>>>
>>>>>>>> [image: Screen Shot 2022-07-14 at 12.07.44 PM.png][image: Screen
>>>>>>>> Shot 2022-07-14 at 12.08.05 PM.png]
>>>>>>>>
>>>>>>>> On Wed, Jul 13, 2022 at 8:03 PM yu'an huang <h....@gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> I had seen simliar case in our prod environment. The job will
>>>>>>>>> create many small state. These state will create many class in JVM
>>>>>>>>> metaspace when restarting which may cause the Metaspace. I don't whether
>>>>>>>>> this is your case.
>>>>>>>>>
>>>>>>>>> On Wed, 13 Jul 2022 at 2:38 AM, Trystan <en...@gmail.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Thanks Yaroslav! I'll have to add this metric to our monitoring
>>>>>>>>>> pipeline. I'll redeploy and let the job's memory reach capacity before
>>>>>>>>>> forcing a restart again and will follow up.
>>>>>>>>>>
>>>>>>>>>> On Tue, Jul 12, 2022 at 1:08 PM Yaroslav Tkachenko <
>>>>>>>>>> yaroslav@goldsky.io> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi Trystan,
>>>>>>>>>>>
>>>>>>>>>>> Could you also monitor the number of loaded classes? E.g. when
>>>>>>>>>>> using DataDog we can obtain a metric like
>>>>>>>>>>> "flink.taskmanager.Status.JVM.ClassLoader.ClassesLoaded". Or you can
>>>>>>>>>>> monitor JVM Metaspace.
>>>>>>>>>>>
>>>>>>>>>>> If this is your case then refer to the recommendation here:
>>>>>>>>>>> https://nightlies.apache.org/flink/flink-docs-release-1.13/docs/ops/debugging/debugging_classloading/#avoiding-dynamic-classloading-for-user-code
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Tue, Jul 12, 2022 at 10:51 AM Trystan <en...@gmail.com>
>>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> After a restart, memory usage of TaskManager pods will increase
>>>>>>>>>>>> and stay elevated. This is repeatable and grows yet again after a secondary
>>>>>>>>>>>> restart. Below you can see two restarts and the subsequent sustained memory
>>>>>>>>>>>> usage after the restarts.
>>>>>>>>>>>>
>>>>>>>>>>>> I have not seen this in any memory/hashmap state based job,
>>>>>>>>>>>> only rocksdb. This state does have a TTL, if that makes a difference. This
>>>>>>>>>>>> behavior can make it very difficult for a job to recover, since it can lead
>>>>>>>>>>>> to OOM issues in Kubernetes.
>>>>>>>>>>>>
>>>>>>>>>>>> Is there an obvious setting that we missed? Are there any
>>>>>>>>>>>> configs we can tune to prevent this? We are on Flink 1.13.5.
>>>>>>>>>>>>
>>>>>>>>>>>> [image: Screen Shot 2022-07-12 at 12.32.06 PM.png]
>>>>>>>>>>>>
>>>>>>>>>>>

Re: Memory increase after restart

Posted by yanfei lei <fr...@gmail.com>.
Flink community is trying to solve this problem by limiting RocksDB memory
capacity strictly[1], but is blocked by some issues with rocksdb[2].
And thanks to Yaroslav‘s great work for a workaround, this is really useful.

[1] https://issues.apache.org/jira/browse/FLINK-15532
[2] https://github.com/apache/flink/pull/16835

Yaroslav Tkachenko <ya...@goldsky.io> 于2022年7月29日周五 00:42写道:

> I worked on the issue together with the OP from the thread, and we ended
> up keeping the block cache disabled. We even wrote a blog post about it:
> https://shopify.engineering/optimizing-apache-flink-applications-tips
> (last section).
>
> On Thu, Jul 28, 2022 at 9:33 AM Trystan <en...@gmail.com> wrote:
>
>> Yes, it's a very similar behavior! I am hesitant to disable to block
>> cache. It doesn't seem like the OP followed up, though I can revive that
>> thread and kill this one, if that's preferred.
>>
>> Was able to replicate this again - occasionally the memory will drop back
>> down after some hours, but in this case it stays elevated more or less
>> permanently.
>>
>> [image: Screen Shot 2022-07-28 at 11.20.11 AM.png]
>>
>> On Fri, Jul 15, 2022 at 11:09 AM Alexander Fedulov <
>> alexander@ververica.com> wrote:
>>
>>> That sounds familiar:
>>> https://lists.apache.org/thread/wsdtbfgzcbq5ooqljvrq45315zloys9x
>>>
>>> Best,
>>> Alexander Fedulov
>>>
>>> On Thu, Jul 14, 2022 at 8:48 PM Yaroslav Tkachenko <ya...@goldsky.io>
>>> wrote:
>>>
>>>> In this case it's probably not a classloading issue, it seems to be
>>>> RocksDB-specific.
>>>>
>>>> On Thu, Jul 14, 2022 at 11:03 AM Trystan <en...@gmail.com> wrote:
>>>>
>>>>> We are using JobMode, but we are copying the job jar into
>>>>> /opt/flink/lib/ and running it from there. Is that not correct, or is
>>>>> Application Model required?
>>>>>
>>>>> On restart, it looks like a few hundred classes at most are loaded,
>>>>> out of 29,000 other classes. Is that actually the likely cause? I'm also
>>>>> unsure why this behavior would only affect rocksdb jobs and not hashmap
>>>>> state backends.
>>>>>
>>>>> On Thu, Jul 14, 2022 at 12:19 PM Yaroslav Tkachenko <
>>>>> yaroslav@goldsky.io> wrote:
>>>>>
>>>>>>
>>>>>> https://nightlies.apache.org/flink/flink-docs-release-1.13/docs/ops/debugging/debugging_classloading/#avoiding-dynamic-classloading-for-user-code
>>>>>> explain more about the classloading behaviour.
>>>>>>
>>>>>> You can try moving your application uber jar to the /lib folder of
>>>>>> the Flink distribution, that will prevent classes from being reloaded on
>>>>>> every restart. This approach assumes you use application mode for Flink.
>>>>>>
>>>>>> On Thu, Jul 14, 2022 at 10:13 AM Trystan <en...@gmail.com> wrote:
>>>>>>
>>>>>>> Yeah, this job has a value state with - according to the flink
>>>>>>> rocksdb metrics - an estimated cardinality of 10B keys.
>>>>>>>
>>>>>>> I've attached screenshots with the JVM classloader metrics. These
>>>>>>> values seem to be cumulative, not a rate. There is a slight increase on
>>>>>>> each restart, but nothing crazy. However the constant increase seems like
>>>>>>> something is off, I wouldn't expect it to be un/loading this much, but this
>>>>>>> is the first time I've looked at this particular metric. Also attached is a
>>>>>>> graph of memory usage across restarts - blue annotation lines are restart
>>>>>>> points. In this case, it seems that most of the TMs did (eventually) drop
>>>>>>> their memory consumption, but one particular pod hung around elevated for
>>>>>>> nearly 12h. If I were to leave this job alone for a week or two and do a
>>>>>>> restart, I would expect memory usage across nearly all pods to stay
>>>>>>> permanently elevated, based on what we've seen with this behavior.
>>>>>>>
>>>>>>> Is the class un/loading suspicious here? Something in rocksdb not
>>>>>>> cleaning itself up?
>>>>>>>
>>>>>>> [image: Screen Shot 2022-07-14 at 12.07.44 PM.png][image: Screen
>>>>>>> Shot 2022-07-14 at 12.08.05 PM.png]
>>>>>>>
>>>>>>> On Wed, Jul 13, 2022 at 8:03 PM yu'an huang <h....@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> I had seen simliar case in our prod environment. The job will
>>>>>>>> create many small state. These state will create many class in JVM
>>>>>>>> metaspace when restarting which may cause the Metaspace. I don't whether
>>>>>>>> this is your case.
>>>>>>>>
>>>>>>>> On Wed, 13 Jul 2022 at 2:38 AM, Trystan <en...@gmail.com> wrote:
>>>>>>>>
>>>>>>>>> Thanks Yaroslav! I'll have to add this metric to our monitoring
>>>>>>>>> pipeline. I'll redeploy and let the job's memory reach capacity before
>>>>>>>>> forcing a restart again and will follow up.
>>>>>>>>>
>>>>>>>>> On Tue, Jul 12, 2022 at 1:08 PM Yaroslav Tkachenko <
>>>>>>>>> yaroslav@goldsky.io> wrote:
>>>>>>>>>
>>>>>>>>>> Hi Trystan,
>>>>>>>>>>
>>>>>>>>>> Could you also monitor the number of loaded classes? E.g. when
>>>>>>>>>> using DataDog we can obtain a metric like
>>>>>>>>>> "flink.taskmanager.Status.JVM.ClassLoader.ClassesLoaded". Or you can
>>>>>>>>>> monitor JVM Metaspace.
>>>>>>>>>>
>>>>>>>>>> If this is your case then refer to the recommendation here:
>>>>>>>>>> https://nightlies.apache.org/flink/flink-docs-release-1.13/docs/ops/debugging/debugging_classloading/#avoiding-dynamic-classloading-for-user-code
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Tue, Jul 12, 2022 at 10:51 AM Trystan <en...@gmail.com>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> After a restart, memory usage of TaskManager pods will increase
>>>>>>>>>>> and stay elevated. This is repeatable and grows yet again after a secondary
>>>>>>>>>>> restart. Below you can see two restarts and the subsequent sustained memory
>>>>>>>>>>> usage after the restarts.
>>>>>>>>>>>
>>>>>>>>>>> I have not seen this in any memory/hashmap state based job, only
>>>>>>>>>>> rocksdb. This state does have a TTL, if that makes a difference. This
>>>>>>>>>>> behavior can make it very difficult for a job to recover, since it can lead
>>>>>>>>>>> to OOM issues in Kubernetes.
>>>>>>>>>>>
>>>>>>>>>>> Is there an obvious setting that we missed? Are there any
>>>>>>>>>>> configs we can tune to prevent this? We are on Flink 1.13.5.
>>>>>>>>>>>
>>>>>>>>>>> [image: Screen Shot 2022-07-12 at 12.32.06 PM.png]
>>>>>>>>>>>
>>>>>>>>>>

Re: Memory increase after restart

Posted by Yaroslav Tkachenko <ya...@goldsky.io>.
I worked on the issue together with the OP from the thread, and we ended up
keeping the block cache disabled. We even wrote a blog post about it:
https://shopify.engineering/optimizing-apache-flink-applications-tips (last
section).

On Thu, Jul 28, 2022 at 9:33 AM Trystan <en...@gmail.com> wrote:

> Yes, it's a very similar behavior! I am hesitant to disable to block
> cache. It doesn't seem like the OP followed up, though I can revive that
> thread and kill this one, if that's preferred.
>
> Was able to replicate this again - occasionally the memory will drop back
> down after some hours, but in this case it stays elevated more or less
> permanently.
>
> [image: Screen Shot 2022-07-28 at 11.20.11 AM.png]
>
> On Fri, Jul 15, 2022 at 11:09 AM Alexander Fedulov <
> alexander@ververica.com> wrote:
>
>> That sounds familiar:
>> https://lists.apache.org/thread/wsdtbfgzcbq5ooqljvrq45315zloys9x
>>
>> Best,
>> Alexander Fedulov
>>
>> On Thu, Jul 14, 2022 at 8:48 PM Yaroslav Tkachenko <ya...@goldsky.io>
>> wrote:
>>
>>> In this case it's probably not a classloading issue, it seems to be
>>> RocksDB-specific.
>>>
>>> On Thu, Jul 14, 2022 at 11:03 AM Trystan <en...@gmail.com> wrote:
>>>
>>>> We are using JobMode, but we are copying the job jar into
>>>> /opt/flink/lib/ and running it from there. Is that not correct, or is
>>>> Application Model required?
>>>>
>>>> On restart, it looks like a few hundred classes at most are loaded, out
>>>> of 29,000 other classes. Is that actually the likely cause? I'm also unsure
>>>> why this behavior would only affect rocksdb jobs and not hashmap state
>>>> backends.
>>>>
>>>> On Thu, Jul 14, 2022 at 12:19 PM Yaroslav Tkachenko <
>>>> yaroslav@goldsky.io> wrote:
>>>>
>>>>>
>>>>> https://nightlies.apache.org/flink/flink-docs-release-1.13/docs/ops/debugging/debugging_classloading/#avoiding-dynamic-classloading-for-user-code
>>>>> explain more about the classloading behaviour.
>>>>>
>>>>> You can try moving your application uber jar to the /lib folder of the
>>>>> Flink distribution, that will prevent classes from being reloaded on every
>>>>> restart. This approach assumes you use application mode for Flink.
>>>>>
>>>>> On Thu, Jul 14, 2022 at 10:13 AM Trystan <en...@gmail.com> wrote:
>>>>>
>>>>>> Yeah, this job has a value state with - according to the flink
>>>>>> rocksdb metrics - an estimated cardinality of 10B keys.
>>>>>>
>>>>>> I've attached screenshots with the JVM classloader metrics. These
>>>>>> values seem to be cumulative, not a rate. There is a slight increase on
>>>>>> each restart, but nothing crazy. However the constant increase seems like
>>>>>> something is off, I wouldn't expect it to be un/loading this much, but this
>>>>>> is the first time I've looked at this particular metric. Also attached is a
>>>>>> graph of memory usage across restarts - blue annotation lines are restart
>>>>>> points. In this case, it seems that most of the TMs did (eventually) drop
>>>>>> their memory consumption, but one particular pod hung around elevated for
>>>>>> nearly 12h. If I were to leave this job alone for a week or two and do a
>>>>>> restart, I would expect memory usage across nearly all pods to stay
>>>>>> permanently elevated, based on what we've seen with this behavior.
>>>>>>
>>>>>> Is the class un/loading suspicious here? Something in rocksdb not
>>>>>> cleaning itself up?
>>>>>>
>>>>>> [image: Screen Shot 2022-07-14 at 12.07.44 PM.png][image: Screen
>>>>>> Shot 2022-07-14 at 12.08.05 PM.png]
>>>>>>
>>>>>> On Wed, Jul 13, 2022 at 8:03 PM yu'an huang <h....@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> I had seen simliar case in our prod environment. The job will create
>>>>>>> many small state. These state will create many class in JVM metaspace when
>>>>>>> restarting which may cause the Metaspace. I don't whether this is your case.
>>>>>>>
>>>>>>> On Wed, 13 Jul 2022 at 2:38 AM, Trystan <en...@gmail.com> wrote:
>>>>>>>
>>>>>>>> Thanks Yaroslav! I'll have to add this metric to our monitoring
>>>>>>>> pipeline. I'll redeploy and let the job's memory reach capacity before
>>>>>>>> forcing a restart again and will follow up.
>>>>>>>>
>>>>>>>> On Tue, Jul 12, 2022 at 1:08 PM Yaroslav Tkachenko <
>>>>>>>> yaroslav@goldsky.io> wrote:
>>>>>>>>
>>>>>>>>> Hi Trystan,
>>>>>>>>>
>>>>>>>>> Could you also monitor the number of loaded classes? E.g. when
>>>>>>>>> using DataDog we can obtain a metric like
>>>>>>>>> "flink.taskmanager.Status.JVM.ClassLoader.ClassesLoaded". Or you can
>>>>>>>>> monitor JVM Metaspace.
>>>>>>>>>
>>>>>>>>> If this is your case then refer to the recommendation here:
>>>>>>>>> https://nightlies.apache.org/flink/flink-docs-release-1.13/docs/ops/debugging/debugging_classloading/#avoiding-dynamic-classloading-for-user-code
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Tue, Jul 12, 2022 at 10:51 AM Trystan <en...@gmail.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> After a restart, memory usage of TaskManager pods will increase
>>>>>>>>>> and stay elevated. This is repeatable and grows yet again after a secondary
>>>>>>>>>> restart. Below you can see two restarts and the subsequent sustained memory
>>>>>>>>>> usage after the restarts.
>>>>>>>>>>
>>>>>>>>>> I have not seen this in any memory/hashmap state based job, only
>>>>>>>>>> rocksdb. This state does have a TTL, if that makes a difference. This
>>>>>>>>>> behavior can make it very difficult for a job to recover, since it can lead
>>>>>>>>>> to OOM issues in Kubernetes.
>>>>>>>>>>
>>>>>>>>>> Is there an obvious setting that we missed? Are there any configs
>>>>>>>>>> we can tune to prevent this? We are on Flink 1.13.5.
>>>>>>>>>>
>>>>>>>>>> [image: Screen Shot 2022-07-12 at 12.32.06 PM.png]
>>>>>>>>>>
>>>>>>>>>

Re: Memory increase after restart

Posted by Trystan <en...@gmail.com>.
Yes, it's a very similar behavior! I am hesitant to disable to block cache.
It doesn't seem like the OP followed up, though I can revive that thread
and kill this one, if that's preferred.

Was able to replicate this again - occasionally the memory will drop back
down after some hours, but in this case it stays elevated more or less
permanently.

[image: Screen Shot 2022-07-28 at 11.20.11 AM.png]

On Fri, Jul 15, 2022 at 11:09 AM Alexander Fedulov <al...@ververica.com>
wrote:

> That sounds familiar:
> https://lists.apache.org/thread/wsdtbfgzcbq5ooqljvrq45315zloys9x
>
> Best,
> Alexander Fedulov
>
> On Thu, Jul 14, 2022 at 8:48 PM Yaroslav Tkachenko <ya...@goldsky.io>
> wrote:
>
>> In this case it's probably not a classloading issue, it seems to be
>> RocksDB-specific.
>>
>> On Thu, Jul 14, 2022 at 11:03 AM Trystan <en...@gmail.com> wrote:
>>
>>> We are using JobMode, but we are copying the job jar into
>>> /opt/flink/lib/ and running it from there. Is that not correct, or is
>>> Application Model required?
>>>
>>> On restart, it looks like a few hundred classes at most are loaded, out
>>> of 29,000 other classes. Is that actually the likely cause? I'm also unsure
>>> why this behavior would only affect rocksdb jobs and not hashmap state
>>> backends.
>>>
>>> On Thu, Jul 14, 2022 at 12:19 PM Yaroslav Tkachenko <ya...@goldsky.io>
>>> wrote:
>>>
>>>>
>>>> https://nightlies.apache.org/flink/flink-docs-release-1.13/docs/ops/debugging/debugging_classloading/#avoiding-dynamic-classloading-for-user-code
>>>> explain more about the classloading behaviour.
>>>>
>>>> You can try moving your application uber jar to the /lib folder of the
>>>> Flink distribution, that will prevent classes from being reloaded on every
>>>> restart. This approach assumes you use application mode for Flink.
>>>>
>>>> On Thu, Jul 14, 2022 at 10:13 AM Trystan <en...@gmail.com> wrote:
>>>>
>>>>> Yeah, this job has a value state with - according to the flink rocksdb
>>>>> metrics - an estimated cardinality of 10B keys.
>>>>>
>>>>> I've attached screenshots with the JVM classloader metrics. These
>>>>> values seem to be cumulative, not a rate. There is a slight increase on
>>>>> each restart, but nothing crazy. However the constant increase seems like
>>>>> something is off, I wouldn't expect it to be un/loading this much, but this
>>>>> is the first time I've looked at this particular metric. Also attached is a
>>>>> graph of memory usage across restarts - blue annotation lines are restart
>>>>> points. In this case, it seems that most of the TMs did (eventually) drop
>>>>> their memory consumption, but one particular pod hung around elevated for
>>>>> nearly 12h. If I were to leave this job alone for a week or two and do a
>>>>> restart, I would expect memory usage across nearly all pods to stay
>>>>> permanently elevated, based on what we've seen with this behavior.
>>>>>
>>>>> Is the class un/loading suspicious here? Something in rocksdb not
>>>>> cleaning itself up?
>>>>>
>>>>> [image: Screen Shot 2022-07-14 at 12.07.44 PM.png][image: Screen Shot
>>>>> 2022-07-14 at 12.08.05 PM.png]
>>>>>
>>>>> On Wed, Jul 13, 2022 at 8:03 PM yu'an huang <h....@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> I had seen simliar case in our prod environment. The job will create
>>>>>> many small state. These state will create many class in JVM metaspace when
>>>>>> restarting which may cause the Metaspace. I don't whether this is your case.
>>>>>>
>>>>>> On Wed, 13 Jul 2022 at 2:38 AM, Trystan <en...@gmail.com> wrote:
>>>>>>
>>>>>>> Thanks Yaroslav! I'll have to add this metric to our monitoring
>>>>>>> pipeline. I'll redeploy and let the job's memory reach capacity before
>>>>>>> forcing a restart again and will follow up.
>>>>>>>
>>>>>>> On Tue, Jul 12, 2022 at 1:08 PM Yaroslav Tkachenko <
>>>>>>> yaroslav@goldsky.io> wrote:
>>>>>>>
>>>>>>>> Hi Trystan,
>>>>>>>>
>>>>>>>> Could you also monitor the number of loaded classes? E.g. when
>>>>>>>> using DataDog we can obtain a metric like
>>>>>>>> "flink.taskmanager.Status.JVM.ClassLoader.ClassesLoaded". Or you can
>>>>>>>> monitor JVM Metaspace.
>>>>>>>>
>>>>>>>> If this is your case then refer to the recommendation here:
>>>>>>>> https://nightlies.apache.org/flink/flink-docs-release-1.13/docs/ops/debugging/debugging_classloading/#avoiding-dynamic-classloading-for-user-code
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Tue, Jul 12, 2022 at 10:51 AM Trystan <en...@gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> After a restart, memory usage of TaskManager pods will increase
>>>>>>>>> and stay elevated. This is repeatable and grows yet again after a secondary
>>>>>>>>> restart. Below you can see two restarts and the subsequent sustained memory
>>>>>>>>> usage after the restarts.
>>>>>>>>>
>>>>>>>>> I have not seen this in any memory/hashmap state based job, only
>>>>>>>>> rocksdb. This state does have a TTL, if that makes a difference. This
>>>>>>>>> behavior can make it very difficult for a job to recover, since it can lead
>>>>>>>>> to OOM issues in Kubernetes.
>>>>>>>>>
>>>>>>>>> Is there an obvious setting that we missed? Are there any configs
>>>>>>>>> we can tune to prevent this? We are on Flink 1.13.5.
>>>>>>>>>
>>>>>>>>> [image: Screen Shot 2022-07-12 at 12.32.06 PM.png]
>>>>>>>>>
>>>>>>>>

Re: Memory increase after restart

Posted by Alexander Fedulov <al...@ververica.com>.
That sounds familiar:
https://lists.apache.org/thread/wsdtbfgzcbq5ooqljvrq45315zloys9x

Best,
Alexander Fedulov

On Thu, Jul 14, 2022 at 8:48 PM Yaroslav Tkachenko <ya...@goldsky.io>
wrote:

> In this case it's probably not a classloading issue, it seems to be
> RocksDB-specific.
>
> On Thu, Jul 14, 2022 at 11:03 AM Trystan <en...@gmail.com> wrote:
>
>> We are using JobMode, but we are copying the job jar into /opt/flink/lib/ and
>> running it from there. Is that not correct, or is Application Model
>> required?
>>
>> On restart, it looks like a few hundred classes at most are loaded, out
>> of 29,000 other classes. Is that actually the likely cause? I'm also unsure
>> why this behavior would only affect rocksdb jobs and not hashmap state
>> backends.
>>
>> On Thu, Jul 14, 2022 at 12:19 PM Yaroslav Tkachenko <ya...@goldsky.io>
>> wrote:
>>
>>>
>>> https://nightlies.apache.org/flink/flink-docs-release-1.13/docs/ops/debugging/debugging_classloading/#avoiding-dynamic-classloading-for-user-code
>>> explain more about the classloading behaviour.
>>>
>>> You can try moving your application uber jar to the /lib folder of the
>>> Flink distribution, that will prevent classes from being reloaded on every
>>> restart. This approach assumes you use application mode for Flink.
>>>
>>> On Thu, Jul 14, 2022 at 10:13 AM Trystan <en...@gmail.com> wrote:
>>>
>>>> Yeah, this job has a value state with - according to the flink rocksdb
>>>> metrics - an estimated cardinality of 10B keys.
>>>>
>>>> I've attached screenshots with the JVM classloader metrics. These
>>>> values seem to be cumulative, not a rate. There is a slight increase on
>>>> each restart, but nothing crazy. However the constant increase seems like
>>>> something is off, I wouldn't expect it to be un/loading this much, but this
>>>> is the first time I've looked at this particular metric. Also attached is a
>>>> graph of memory usage across restarts - blue annotation lines are restart
>>>> points. In this case, it seems that most of the TMs did (eventually) drop
>>>> their memory consumption, but one particular pod hung around elevated for
>>>> nearly 12h. If I were to leave this job alone for a week or two and do a
>>>> restart, I would expect memory usage across nearly all pods to stay
>>>> permanently elevated, based on what we've seen with this behavior.
>>>>
>>>> Is the class un/loading suspicious here? Something in rocksdb not
>>>> cleaning itself up?
>>>>
>>>> [image: Screen Shot 2022-07-14 at 12.07.44 PM.png][image: Screen Shot
>>>> 2022-07-14 at 12.08.05 PM.png]
>>>>
>>>> On Wed, Jul 13, 2022 at 8:03 PM yu'an huang <h....@gmail.com>
>>>> wrote:
>>>>
>>>>> I had seen simliar case in our prod environment. The job will create
>>>>> many small state. These state will create many class in JVM metaspace when
>>>>> restarting which may cause the Metaspace. I don't whether this is your case.
>>>>>
>>>>> On Wed, 13 Jul 2022 at 2:38 AM, Trystan <en...@gmail.com> wrote:
>>>>>
>>>>>> Thanks Yaroslav! I'll have to add this metric to our monitoring
>>>>>> pipeline. I'll redeploy and let the job's memory reach capacity before
>>>>>> forcing a restart again and will follow up.
>>>>>>
>>>>>> On Tue, Jul 12, 2022 at 1:08 PM Yaroslav Tkachenko <
>>>>>> yaroslav@goldsky.io> wrote:
>>>>>>
>>>>>>> Hi Trystan,
>>>>>>>
>>>>>>> Could you also monitor the number of loaded classes? E.g. when using
>>>>>>> DataDog we can obtain a metric like
>>>>>>> "flink.taskmanager.Status.JVM.ClassLoader.ClassesLoaded". Or you can
>>>>>>> monitor JVM Metaspace.
>>>>>>>
>>>>>>> If this is your case then refer to the recommendation here:
>>>>>>> https://nightlies.apache.org/flink/flink-docs-release-1.13/docs/ops/debugging/debugging_classloading/#avoiding-dynamic-classloading-for-user-code
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Tue, Jul 12, 2022 at 10:51 AM Trystan <en...@gmail.com> wrote:
>>>>>>>
>>>>>>>> After a restart, memory usage of TaskManager pods will increase and
>>>>>>>> stay elevated. This is repeatable and grows yet again after a secondary
>>>>>>>> restart. Below you can see two restarts and the subsequent sustained memory
>>>>>>>> usage after the restarts.
>>>>>>>>
>>>>>>>> I have not seen this in any memory/hashmap state based job, only
>>>>>>>> rocksdb. This state does have a TTL, if that makes a difference. This
>>>>>>>> behavior can make it very difficult for a job to recover, since it can lead
>>>>>>>> to OOM issues in Kubernetes.
>>>>>>>>
>>>>>>>> Is there an obvious setting that we missed? Are there any configs
>>>>>>>> we can tune to prevent this? We are on Flink 1.13.5.
>>>>>>>>
>>>>>>>> [image: Screen Shot 2022-07-12 at 12.32.06 PM.png]
>>>>>>>>
>>>>>>>

Re: Memory increase after restart

Posted by Yaroslav Tkachenko <ya...@goldsky.io>.
In this case it's probably not a classloading issue, it seems to be
RocksDB-specific.

On Thu, Jul 14, 2022 at 11:03 AM Trystan <en...@gmail.com> wrote:

> We are using JobMode, but we are copying the job jar into /opt/flink/lib/ and
> running it from there. Is that not correct, or is Application Model
> required?
>
> On restart, it looks like a few hundred classes at most are loaded, out of
> 29,000 other classes. Is that actually the likely cause? I'm also unsure
> why this behavior would only affect rocksdb jobs and not hashmap state
> backends.
>
> On Thu, Jul 14, 2022 at 12:19 PM Yaroslav Tkachenko <ya...@goldsky.io>
> wrote:
>
>>
>> https://nightlies.apache.org/flink/flink-docs-release-1.13/docs/ops/debugging/debugging_classloading/#avoiding-dynamic-classloading-for-user-code
>> explain more about the classloading behaviour.
>>
>> You can try moving your application uber jar to the /lib folder of the
>> Flink distribution, that will prevent classes from being reloaded on every
>> restart. This approach assumes you use application mode for Flink.
>>
>> On Thu, Jul 14, 2022 at 10:13 AM Trystan <en...@gmail.com> wrote:
>>
>>> Yeah, this job has a value state with - according to the flink rocksdb
>>> metrics - an estimated cardinality of 10B keys.
>>>
>>> I've attached screenshots with the JVM classloader metrics. These values
>>> seem to be cumulative, not a rate. There is a slight increase on each
>>> restart, but nothing crazy. However the constant increase seems like
>>> something is off, I wouldn't expect it to be un/loading this much, but this
>>> is the first time I've looked at this particular metric. Also attached is a
>>> graph of memory usage across restarts - blue annotation lines are restart
>>> points. In this case, it seems that most of the TMs did (eventually) drop
>>> their memory consumption, but one particular pod hung around elevated for
>>> nearly 12h. If I were to leave this job alone for a week or two and do a
>>> restart, I would expect memory usage across nearly all pods to stay
>>> permanently elevated, based on what we've seen with this behavior.
>>>
>>> Is the class un/loading suspicious here? Something in rocksdb not
>>> cleaning itself up?
>>>
>>> [image: Screen Shot 2022-07-14 at 12.07.44 PM.png][image: Screen Shot
>>> 2022-07-14 at 12.08.05 PM.png]
>>>
>>> On Wed, Jul 13, 2022 at 8:03 PM yu'an huang <h....@gmail.com> wrote:
>>>
>>>> I had seen simliar case in our prod environment. The job will create
>>>> many small state. These state will create many class in JVM metaspace when
>>>> restarting which may cause the Metaspace. I don't whether this is your case.
>>>>
>>>> On Wed, 13 Jul 2022 at 2:38 AM, Trystan <en...@gmail.com> wrote:
>>>>
>>>>> Thanks Yaroslav! I'll have to add this metric to our monitoring
>>>>> pipeline. I'll redeploy and let the job's memory reach capacity before
>>>>> forcing a restart again and will follow up.
>>>>>
>>>>> On Tue, Jul 12, 2022 at 1:08 PM Yaroslav Tkachenko <
>>>>> yaroslav@goldsky.io> wrote:
>>>>>
>>>>>> Hi Trystan,
>>>>>>
>>>>>> Could you also monitor the number of loaded classes? E.g. when using
>>>>>> DataDog we can obtain a metric like
>>>>>> "flink.taskmanager.Status.JVM.ClassLoader.ClassesLoaded". Or you can
>>>>>> monitor JVM Metaspace.
>>>>>>
>>>>>> If this is your case then refer to the recommendation here:
>>>>>> https://nightlies.apache.org/flink/flink-docs-release-1.13/docs/ops/debugging/debugging_classloading/#avoiding-dynamic-classloading-for-user-code
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Tue, Jul 12, 2022 at 10:51 AM Trystan <en...@gmail.com> wrote:
>>>>>>
>>>>>>> After a restart, memory usage of TaskManager pods will increase and
>>>>>>> stay elevated. This is repeatable and grows yet again after a secondary
>>>>>>> restart. Below you can see two restarts and the subsequent sustained memory
>>>>>>> usage after the restarts.
>>>>>>>
>>>>>>> I have not seen this in any memory/hashmap state based job, only
>>>>>>> rocksdb. This state does have a TTL, if that makes a difference. This
>>>>>>> behavior can make it very difficult for a job to recover, since it can lead
>>>>>>> to OOM issues in Kubernetes.
>>>>>>>
>>>>>>> Is there an obvious setting that we missed? Are there any configs we
>>>>>>> can tune to prevent this? We are on Flink 1.13.5.
>>>>>>>
>>>>>>> [image: Screen Shot 2022-07-12 at 12.32.06 PM.png]
>>>>>>>
>>>>>>

Re: Memory increase after restart

Posted by Trystan <en...@gmail.com>.
We are using JobMode, but we are copying the job jar into /opt/flink/lib/ and
running it from there. Is that not correct, or is Application Model
required?

On restart, it looks like a few hundred classes at most are loaded, out of
29,000 other classes. Is that actually the likely cause? I'm also unsure
why this behavior would only affect rocksdb jobs and not hashmap state
backends.

On Thu, Jul 14, 2022 at 12:19 PM Yaroslav Tkachenko <ya...@goldsky.io>
wrote:

>
> https://nightlies.apache.org/flink/flink-docs-release-1.13/docs/ops/debugging/debugging_classloading/#avoiding-dynamic-classloading-for-user-code
> explain more about the classloading behaviour.
>
> You can try moving your application uber jar to the /lib folder of the
> Flink distribution, that will prevent classes from being reloaded on every
> restart. This approach assumes you use application mode for Flink.
>
> On Thu, Jul 14, 2022 at 10:13 AM Trystan <en...@gmail.com> wrote:
>
>> Yeah, this job has a value state with - according to the flink rocksdb
>> metrics - an estimated cardinality of 10B keys.
>>
>> I've attached screenshots with the JVM classloader metrics. These values
>> seem to be cumulative, not a rate. There is a slight increase on each
>> restart, but nothing crazy. However the constant increase seems like
>> something is off, I wouldn't expect it to be un/loading this much, but this
>> is the first time I've looked at this particular metric. Also attached is a
>> graph of memory usage across restarts - blue annotation lines are restart
>> points. In this case, it seems that most of the TMs did (eventually) drop
>> their memory consumption, but one particular pod hung around elevated for
>> nearly 12h. If I were to leave this job alone for a week or two and do a
>> restart, I would expect memory usage across nearly all pods to stay
>> permanently elevated, based on what we've seen with this behavior.
>>
>> Is the class un/loading suspicious here? Something in rocksdb not
>> cleaning itself up?
>>
>> [image: Screen Shot 2022-07-14 at 12.07.44 PM.png][image: Screen Shot
>> 2022-07-14 at 12.08.05 PM.png]
>>
>> On Wed, Jul 13, 2022 at 8:03 PM yu'an huang <h....@gmail.com> wrote:
>>
>>> I had seen simliar case in our prod environment. The job will create
>>> many small state. These state will create many class in JVM metaspace when
>>> restarting which may cause the Metaspace. I don't whether this is your case.
>>>
>>> On Wed, 13 Jul 2022 at 2:38 AM, Trystan <en...@gmail.com> wrote:
>>>
>>>> Thanks Yaroslav! I'll have to add this metric to our monitoring
>>>> pipeline. I'll redeploy and let the job's memory reach capacity before
>>>> forcing a restart again and will follow up.
>>>>
>>>> On Tue, Jul 12, 2022 at 1:08 PM Yaroslav Tkachenko <ya...@goldsky.io>
>>>> wrote:
>>>>
>>>>> Hi Trystan,
>>>>>
>>>>> Could you also monitor the number of loaded classes? E.g. when using
>>>>> DataDog we can obtain a metric like
>>>>> "flink.taskmanager.Status.JVM.ClassLoader.ClassesLoaded". Or you can
>>>>> monitor JVM Metaspace.
>>>>>
>>>>> If this is your case then refer to the recommendation here:
>>>>> https://nightlies.apache.org/flink/flink-docs-release-1.13/docs/ops/debugging/debugging_classloading/#avoiding-dynamic-classloading-for-user-code
>>>>>
>>>>>
>>>>>
>>>>> On Tue, Jul 12, 2022 at 10:51 AM Trystan <en...@gmail.com> wrote:
>>>>>
>>>>>> After a restart, memory usage of TaskManager pods will increase and
>>>>>> stay elevated. This is repeatable and grows yet again after a secondary
>>>>>> restart. Below you can see two restarts and the subsequent sustained memory
>>>>>> usage after the restarts.
>>>>>>
>>>>>> I have not seen this in any memory/hashmap state based job, only
>>>>>> rocksdb. This state does have a TTL, if that makes a difference. This
>>>>>> behavior can make it very difficult for a job to recover, since it can lead
>>>>>> to OOM issues in Kubernetes.
>>>>>>
>>>>>> Is there an obvious setting that we missed? Are there any configs we
>>>>>> can tune to prevent this? We are on Flink 1.13.5.
>>>>>>
>>>>>> [image: Screen Shot 2022-07-12 at 12.32.06 PM.png]
>>>>>>
>>>>>

Re: Memory increase after restart

Posted by Yaroslav Tkachenko <ya...@goldsky.io>.
https://nightlies.apache.org/flink/flink-docs-release-1.13/docs/ops/debugging/debugging_classloading/#avoiding-dynamic-classloading-for-user-code
explain more about the classloading behaviour.

You can try moving your application uber jar to the /lib folder of the
Flink distribution, that will prevent classes from being reloaded on every
restart. This approach assumes you use application mode for Flink.

On Thu, Jul 14, 2022 at 10:13 AM Trystan <en...@gmail.com> wrote:

> Yeah, this job has a value state with - according to the flink rocksdb
> metrics - an estimated cardinality of 10B keys.
>
> I've attached screenshots with the JVM classloader metrics. These values
> seem to be cumulative, not a rate. There is a slight increase on each
> restart, but nothing crazy. However the constant increase seems like
> something is off, I wouldn't expect it to be un/loading this much, but this
> is the first time I've looked at this particular metric. Also attached is a
> graph of memory usage across restarts - blue annotation lines are restart
> points. In this case, it seems that most of the TMs did (eventually) drop
> their memory consumption, but one particular pod hung around elevated for
> nearly 12h. If I were to leave this job alone for a week or two and do a
> restart, I would expect memory usage across nearly all pods to stay
> permanently elevated, based on what we've seen with this behavior.
>
> Is the class un/loading suspicious here? Something in rocksdb not cleaning
> itself up?
>
> [image: Screen Shot 2022-07-14 at 12.07.44 PM.png][image: Screen Shot
> 2022-07-14 at 12.08.05 PM.png]
>
> On Wed, Jul 13, 2022 at 8:03 PM yu'an huang <h....@gmail.com> wrote:
>
>> I had seen simliar case in our prod environment. The job will create many
>> small state. These state will create many class in JVM metaspace when
>> restarting which may cause the Metaspace. I don't whether this is your case.
>>
>> On Wed, 13 Jul 2022 at 2:38 AM, Trystan <en...@gmail.com> wrote:
>>
>>> Thanks Yaroslav! I'll have to add this metric to our monitoring
>>> pipeline. I'll redeploy and let the job's memory reach capacity before
>>> forcing a restart again and will follow up.
>>>
>>> On Tue, Jul 12, 2022 at 1:08 PM Yaroslav Tkachenko <ya...@goldsky.io>
>>> wrote:
>>>
>>>> Hi Trystan,
>>>>
>>>> Could you also monitor the number of loaded classes? E.g. when using
>>>> DataDog we can obtain a metric like
>>>> "flink.taskmanager.Status.JVM.ClassLoader.ClassesLoaded". Or you can
>>>> monitor JVM Metaspace.
>>>>
>>>> If this is your case then refer to the recommendation here:
>>>> https://nightlies.apache.org/flink/flink-docs-release-1.13/docs/ops/debugging/debugging_classloading/#avoiding-dynamic-classloading-for-user-code
>>>>
>>>>
>>>>
>>>> On Tue, Jul 12, 2022 at 10:51 AM Trystan <en...@gmail.com> wrote:
>>>>
>>>>> After a restart, memory usage of TaskManager pods will increase and
>>>>> stay elevated. This is repeatable and grows yet again after a secondary
>>>>> restart. Below you can see two restarts and the subsequent sustained memory
>>>>> usage after the restarts.
>>>>>
>>>>> I have not seen this in any memory/hashmap state based job, only
>>>>> rocksdb. This state does have a TTL, if that makes a difference. This
>>>>> behavior can make it very difficult for a job to recover, since it can lead
>>>>> to OOM issues in Kubernetes.
>>>>>
>>>>> Is there an obvious setting that we missed? Are there any configs we
>>>>> can tune to prevent this? We are on Flink 1.13.5.
>>>>>
>>>>> [image: Screen Shot 2022-07-12 at 12.32.06 PM.png]
>>>>>
>>>>

Re: Memory increase after restart

Posted by Trystan <en...@gmail.com>.
Yeah, this job has a value state with - according to the flink rocksdb
metrics - an estimated cardinality of 10B keys.

I've attached screenshots with the JVM classloader metrics. These values
seem to be cumulative, not a rate. There is a slight increase on each
restart, but nothing crazy. However the constant increase seems like
something is off, I wouldn't expect it to be un/loading this much, but this
is the first time I've looked at this particular metric. Also attached is a
graph of memory usage across restarts - blue annotation lines are restart
points. In this case, it seems that most of the TMs did (eventually) drop
their memory consumption, but one particular pod hung around elevated for
nearly 12h. If I were to leave this job alone for a week or two and do a
restart, I would expect memory usage across nearly all pods to stay
permanently elevated, based on what we've seen with this behavior.

Is the class un/loading suspicious here? Something in rocksdb not cleaning
itself up?

[image: Screen Shot 2022-07-14 at 12.07.44 PM.png][image: Screen Shot
2022-07-14 at 12.08.05 PM.png]

On Wed, Jul 13, 2022 at 8:03 PM yu'an huang <h....@gmail.com> wrote:

> I had seen simliar case in our prod environment. The job will create many
> small state. These state will create many class in JVM metaspace when
> restarting which may cause the Metaspace. I don't whether this is your case.
>
> On Wed, 13 Jul 2022 at 2:38 AM, Trystan <en...@gmail.com> wrote:
>
>> Thanks Yaroslav! I'll have to add this metric to our monitoring pipeline.
>> I'll redeploy and let the job's memory reach capacity before forcing a
>> restart again and will follow up.
>>
>> On Tue, Jul 12, 2022 at 1:08 PM Yaroslav Tkachenko <ya...@goldsky.io>
>> wrote:
>>
>>> Hi Trystan,
>>>
>>> Could you also monitor the number of loaded classes? E.g. when using
>>> DataDog we can obtain a metric like
>>> "flink.taskmanager.Status.JVM.ClassLoader.ClassesLoaded". Or you can
>>> monitor JVM Metaspace.
>>>
>>> If this is your case then refer to the recommendation here:
>>> https://nightlies.apache.org/flink/flink-docs-release-1.13/docs/ops/debugging/debugging_classloading/#avoiding-dynamic-classloading-for-user-code
>>>
>>>
>>>
>>> On Tue, Jul 12, 2022 at 10:51 AM Trystan <en...@gmail.com> wrote:
>>>
>>>> After a restart, memory usage of TaskManager pods will increase and
>>>> stay elevated. This is repeatable and grows yet again after a secondary
>>>> restart. Below you can see two restarts and the subsequent sustained memory
>>>> usage after the restarts.
>>>>
>>>> I have not seen this in any memory/hashmap state based job, only
>>>> rocksdb. This state does have a TTL, if that makes a difference. This
>>>> behavior can make it very difficult for a job to recover, since it can lead
>>>> to OOM issues in Kubernetes.
>>>>
>>>> Is there an obvious setting that we missed? Are there any configs we
>>>> can tune to prevent this? We are on Flink 1.13.5.
>>>>
>>>> [image: Screen Shot 2022-07-12 at 12.32.06 PM.png]
>>>>
>>>

Re: Memory increase after restart

Posted by yu'an huang <h....@gmail.com>.
I had seen simliar case in our prod environment. The job will create many
small state. These state will create many class in JVM metaspace when
restarting which may cause the Metaspace. I don't whether this is your case.

On Wed, 13 Jul 2022 at 2:38 AM, Trystan <en...@gmail.com> wrote:

> Thanks Yaroslav! I'll have to add this metric to our monitoring pipeline.
> I'll redeploy and let the job's memory reach capacity before forcing a
> restart again and will follow up.
>
> On Tue, Jul 12, 2022 at 1:08 PM Yaroslav Tkachenko <ya...@goldsky.io>
> wrote:
>
>> Hi Trystan,
>>
>> Could you also monitor the number of loaded classes? E.g. when using
>> DataDog we can obtain a metric like
>> "flink.taskmanager.Status.JVM.ClassLoader.ClassesLoaded". Or you can
>> monitor JVM Metaspace.
>>
>> If this is your case then refer to the recommendation here:
>> https://nightlies.apache.org/flink/flink-docs-release-1.13/docs/ops/debugging/debugging_classloading/#avoiding-dynamic-classloading-for-user-code
>>
>>
>>
>> On Tue, Jul 12, 2022 at 10:51 AM Trystan <en...@gmail.com> wrote:
>>
>>> After a restart, memory usage of TaskManager pods will increase and stay
>>> elevated. This is repeatable and grows yet again after a secondary restart.
>>> Below you can see two restarts and the subsequent sustained memory usage
>>> after the restarts.
>>>
>>> I have not seen this in any memory/hashmap state based job, only
>>> rocksdb. This state does have a TTL, if that makes a difference. This
>>> behavior can make it very difficult for a job to recover, since it can lead
>>> to OOM issues in Kubernetes.
>>>
>>> Is there an obvious setting that we missed? Are there any configs we can
>>> tune to prevent this? We are on Flink 1.13.5.
>>>
>>> [image: Screen Shot 2022-07-12 at 12.32.06 PM.png]
>>>
>>

Re: Memory increase after restart

Posted by Trystan <en...@gmail.com>.
Thanks Yaroslav! I'll have to add this metric to our monitoring pipeline.
I'll redeploy and let the job's memory reach capacity before forcing a
restart again and will follow up.

On Tue, Jul 12, 2022 at 1:08 PM Yaroslav Tkachenko <ya...@goldsky.io>
wrote:

> Hi Trystan,
>
> Could you also monitor the number of loaded classes? E.g. when using
> DataDog we can obtain a metric like
> "flink.taskmanager.Status.JVM.ClassLoader.ClassesLoaded". Or you can
> monitor JVM Metaspace.
>
> If this is your case then refer to the recommendation here:
> https://nightlies.apache.org/flink/flink-docs-release-1.13/docs/ops/debugging/debugging_classloading/#avoiding-dynamic-classloading-for-user-code
>
>
>
> On Tue, Jul 12, 2022 at 10:51 AM Trystan <en...@gmail.com> wrote:
>
>> After a restart, memory usage of TaskManager pods will increase and stay
>> elevated. This is repeatable and grows yet again after a secondary restart.
>> Below you can see two restarts and the subsequent sustained memory usage
>> after the restarts.
>>
>> I have not seen this in any memory/hashmap state based job, only rocksdb.
>> This state does have a TTL, if that makes a difference. This behavior can
>> make it very difficult for a job to recover, since it can lead to OOM
>> issues in Kubernetes.
>>
>> Is there an obvious setting that we missed? Are there any configs we can
>> tune to prevent this? We are on Flink 1.13.5.
>>
>> [image: Screen Shot 2022-07-12 at 12.32.06 PM.png]
>>
>

Re: Memory increase after restart

Posted by Yaroslav Tkachenko <ya...@goldsky.io>.
Hi Trystan,

Could you also monitor the number of loaded classes? E.g. when using
DataDog we can obtain a metric like
"flink.taskmanager.Status.JVM.ClassLoader.ClassesLoaded". Or you can
monitor JVM Metaspace.

If this is your case then refer to the recommendation here:
https://nightlies.apache.org/flink/flink-docs-release-1.13/docs/ops/debugging/debugging_classloading/#avoiding-dynamic-classloading-for-user-code



On Tue, Jul 12, 2022 at 10:51 AM Trystan <en...@gmail.com> wrote:

> After a restart, memory usage of TaskManager pods will increase and stay
> elevated. This is repeatable and grows yet again after a secondary restart.
> Below you can see two restarts and the subsequent sustained memory usage
> after the restarts.
>
> I have not seen this in any memory/hashmap state based job, only rocksdb.
> This state does have a TTL, if that makes a difference. This behavior can
> make it very difficult for a job to recover, since it can lead to OOM
> issues in Kubernetes.
>
> Is there an obvious setting that we missed? Are there any configs we can
> tune to prevent this? We are on Flink 1.13.5.
>
> [image: Screen Shot 2022-07-12 at 12.32.06 PM.png]
>