You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@flink.apache.org by Fritz Budiyanto <fb...@icloud.com> on 2017/06/02 17:20:39 UTC

FileSystem vs RocksDb backend

Hi All,

If my states fit in the JVM heap, should I be using filesystem backend with HDFS instead of RocksDB ?

Could someone comment on the FS vs RocksDb backend performance ?

I’ve also heard that RocksDb backend is required to support delta checkpointing. Is this true ?
Is delta checkpointing part of 1.3.0 ?


Thanks,
Fritz

Re: FileSystem vs RocksDb backend

Posted by Fabian Hueske <fh...@gmail.com>.

Hi Fritz,

the FSStateBackend is a lot faster than the RocksDBStateBackend.
I don't know exact numbers but if I remember correctly somebody mentioned a
10x difference.

Delta checkpointing is only supported by RocksDB since Flink 1.3.
Given that the FSStateBackend is only suitable for small states, this
should not be significant limitation.

Best, Fabian

2017-06-02 19:20 GMT+02:00 Fritz Budiyanto <fb...@icloud.com>:

> Hi All,
>
> If my states fit in the JVM heap, should I be using filesystem backend
> with HDFS instead of RocksDB ?
>
> Could someone comment on the FS vs RocksDb backend performance ?
>
> I’ve also heard that RocksDb backend is required to support delta
> checkpointing. Is this true ?
> Is delta checkpointing part of 1.3.0 ?
>
>
> Thanks,
> Fritz