Monitor the usage of keyed state

[Mu Kong <ko...@gmail.com>]

Hi community,

I have a Flink job running with RichMapFunction that uses keyed state.
Although the TTL is enabled, I wonder if there is a way that I can monitor
the memory usage of the keyed state. I'm using RocksDB as the state backend.

Best regards,
Mu

Re: Monitor the usage of keyed state

[Yun Tang <my...@live.com>]

Hi Mu

I want to share something more about the memory usage of RocksDB.

If you enable managed memory for rocksDB (which is enabled by default) [1], you should refer to the block cache usage as we cast index & filter into cache and counting write buffer usage in cache.
We could refer to the usage of block cache [2] to know the overall memory usage of RocksDB.
BTW, since we use the same cache for rocksDB instances within one slot, you might notice that all rocksDBs in the same slot would report the same block cache usage, please not sum them up.

[1] https://ci.apache.org/projects/flink/flink-docs-release-1.11/ops/config.html#state-backend-rocksdb-memory-managed
[2] https://ci.apache.org/projects/flink/flink-docs-release-1.11/ops/config.html#state-backend-rocksdb-metrics-block-cache-usage

Best,
Yun Tang
________________________________
From: Andrey Zagrebin <az...@apache.org>
Sent: Tuesday, August 25, 2020 22:12
To: Mu Kong <ko...@gmail.com>
Cc: flink-user@apache.org <fl...@apache.org>
Subject: Re: Monitor the usage of keyed state

Hi Mu,

I would suggest to look into RocksDB metrics which you can enable as Flink metrics [1]

Best,
Andrey

[1] https://ci.apache.org/projects/flink/flink-docs-release-1.11/ops/config.html#rocksdb-native-metrics

On Fri, Aug 21, 2020 at 4:27 AM Mu Kong <ko...@gmail.com>> wrote:
Hi community,

I have a Flink job running with RichMapFunction that uses keyed state.
Although the TTL is enabled, I wonder if there is a way that I can monitor the memory usage of the keyed state. I'm using RocksDB as the state backend.

Best regards,
Mu

Re: Monitor the usage of keyed state

[Andrey Zagrebin <az...@apache.org>]

Hi Mu,

I would suggest to look into RocksDB metrics which you can enable as Flink
metrics [1]

Best,
Andrey

[1]
https://ci.apache.org/projects/flink/flink-docs-release-1.11/ops/config.html#rocksdb-native-metrics

On Fri, Aug 21, 2020 at 4:27 AM Mu Kong <ko...@gmail.com> wrote:

> Hi community,
>
> I have a Flink job running with RichMapFunction that uses keyed state.
> Although the TTL is enabled, I wonder if there is a way that I can monitor
> the memory usage of the keyed state. I'm using RocksDB as the state backend.
>
> Best regards,
> Mu
>