[jira] [Commented] (FLINK-19293) RocksDB last_checkpoint.state_size grows endlessly until savepoint/restore

["Arvid Heise (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/FLINK-19293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17237368#comment-17237368 ] 

Arvid Heise commented on FLINK-19293:
-------------------------------------

Sorry for the late reply.

A possible reason could be that compaction in rocksDB is not triggered because the update rate is too high. See also [https://flink.apache.org/2019/05/19/state-ttl.html] .
{noformat}
Keep in mind that calling the Flink TTL filter slows down the RocksDB compaction.
{noformat}
Since normal data access takes priority over compaction in rocksDB, it might be that your cluster is underprovisioned for the TTL to work properly.

> RocksDB last_checkpoint.state_size grows endlessly until savepoint/restore
> --------------------------------------------------------------------------
>
>                 Key: FLINK-19293
>                 URL: https://issues.apache.org/jira/browse/FLINK-19293
>             Project: Flink
>          Issue Type: Bug
>          Components: Library / CEP, Runtime / Checkpointing, Runtime / State Backends
>    Affects Versions: 1.10.1
>            Reporter: Thomas Wozniakowski
>            Priority: Major
>         Attachments: Screenshot 2020-09-18 at 13.58.30.png
>
>
> Hi Guys,
> I am seeing some strange behaviour that may be a bug, or may just be intended.
> We are running a Flink job on a 1.10.1 cluster running with 1 JobManager and 2 TaskManagers, parallelism 4. The job itself is simple:
> # Source: kinesis connector reading from a single shard stream
> # CEP: ~25 CEP Keyed Pattern operators watching the event stream for different kinds of behaviour. They all have ".withinSeconds(xxxx)" applied. Nothing is set up to grow endlessly.
> # Sink: Single operator writing messages to SQS (custom code)
> We are seeing the checkpoint size grow constantly until the job is restarted using a savepoint/restore. The size continues to grow past the point that the ".withinSeconds(xxxx)" limits should cause old data to be discarded. The growth is also out of proportion to the general platform growth (which is actually trending down at the moment due to COVID).
> I've attached a snapshot from our monitoring dashboard below. You can see the huge drops in state_size on a savepoint/restore.
> Our state configuration is as follows:
> Backend: RocksDB
> Mode: EXACTLY_ONCE
> Max Concurrent: 1
> Externalised Checkpoints: RETAIN_ON_CANCELLATION
> Async: TRUE
> Incremental: TRUE
> TTL Compaction Filter enabled: TRUE
> We are worried that the CEP library may be leaking state somewhere, leaving some objects not cleaned up. Unfortunately I can't share one of these checkpoints with the community due to the sensitive nature of the data contained within, but if anyone has any suggestions for how I could analyse the checkpoints to look for leaks, please let me know.
> Thanks in advance for the help



--
This message was sent by Atlassian Jira
(v8.3.4#803005)