State backend considerations

[Nick Bendtner <bu...@gmail.com>]

Hi guys,
I have a few questions on state backends.
Is there a guideline on how big the state has to be where it makes sense to
use RocksDB rather than FsStatebackend ? Is there an analysis on
latency for a full checkpoint for FsSateBackend based on increase in state
size ?


Best,
Nick.

Re: State backend considerations

[Arvid Heise <ar...@ververica.com>]

Hi Nick,

Both questions are hard to answer given that it depends on your hardware,
access patterns (read/update), record size/structure, parallelism, and
probably a ton of other parameters.

The usual approach is to simply evaluate it in your setting. Since it's a
matter of configuration, you can do some A/B testing.

In general, you need RocksDB if you want to have incremental checkpoints,
which is recommended if you have rather few updates and big state.

On Mon, Jun 22, 2020 at 2:13 AM Nick Bendtner <bu...@gmail.com> wrote:

> Hi guys,
> I have a few questions on state backends.
> Is there a guideline on how big the state has to be where it makes sense
> to use RocksDB rather than FsStatebackend ? Is there an analysis on
> latency for a full checkpoint for FsSateBackend based on increase in state
> size ?
>
>
> Best,
> Nick.
>


-- 

Arvid Heise | Senior Java Developer

<https://www.ververica.com/>

Follow us @VervericaData

--

Join Flink Forward <https://flink-forward.org/> - The Apache Flink
Conference

Stream Processing | Event Driven | Real Time

--

Ververica GmbH | Invalidenstrasse 115, 10115 Berlin, Germany

--
Ververica GmbH
Registered at Amtsgericht Charlottenburg: HRB 158244 B
Managing Directors: Timothy Alexander Steinert, Yip Park Tung Jason, Ji
(Toni) Cheng