Re: Extremely low performance of state operations upon large quantities of state when using RocksDBstateBackend

[Yun Tang <my...@live.com>]

Hi,

From the flame graph, we can see most of the CPU usage is consumed during get the filter. And I think you might hit the performance problem as filter block is evicted for caching data block.
Please consider to upgrade to Flink-1.13 and set "state.backend.rocksdb.memory.partitioned-index-filters" [1] as true. You could also refer to FLINK-20496<https://issues.apache.org/jira/browse/FLINK-20496> [2] for more details.


[1] https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/deployment/config/#state-backend-rocksdb-memory-partitioned-index-filters
[2] https://issues.apache.org/jira/browse/FLINK-20496

Best
Yun Tang


________________________________
From: 刘首维 <jo...@qq.com>
Sent: Sunday, August 1, 2021 1:14
To: user <us...@flink.apache.org>
Cc: myasuka <my...@live.com>
Subject: Extremely low performance of state operations upon large quantities of state when using RocksDBstateBackend


Hi all,

  I'm trying to use RocksDBstateBackend to solve the problem of flink task with large quantites of state. But unfortunately, the performance was unexpectedly low.  Well, though I tried to take lots of measures, following the flink doc[1], the performance still can not satisfy the my requirement.

Looking forward to ANY advice, thx！

@YUN TANG, would you mind giving some suggestions? thx

Here is the context:

  Flink Version: 1.12.x
  Deployment: YARN per job
  Storage: HDD
  Memory  Total 16g, managed memory 11.8g  Task heap 1.7g
  TM cpu: 6 physical cores
  Predefined Option: SPINNING_DISK_OPTIMIZED_HIGH_MEM
 Other RocksDBStatebackend options:  state.backend.rocksdb.memory.high-prio-pool-ratio=0.2

In my case, there are about 2billion keyed states in total. For every keyed state, the size is not that big. For per Record, there must be 0~2 state writing operation(rocksdb put) and 1~2 state reading operation(rocksdb get).

   The performance was good at beginning. According to Flame graph and metrics info. state related operations was only  about 7%, and cpu load was not too high.
But after processing about 100 million records. The performance became extermely low and the cpu load was high.

Here are some useful metrics, from my perspective.

[cid:A95A9032@EA32DA53.F3840561.png]

                                                     record processing in total

[cid:FEA624B4@04DC5E21.F3840561.png]

                        record processing per second

[cid:41890B8F@ED323B5B.F3840561.png]

                      tm cpu load

[cid:37DD93D1@3AB44E28.F3840561.png]

                                         flame graph when the performance was low








[1] https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/ops/state/large_state_tuning/#tuning-rocksdb-memory