[DISCUSS] FLIP-151: Incremental snapshots for heap-based state backend

[Khachatryan Roman <kh...@gmail.com>]

Hi devs,

I'd like to start a discussion of FLIP-151: Incremental snapshots for
heap-based state backend [1]

Heap backend, while being limited state sizes fitting into memory, also has
some advantages compared to RocksDB backend:
1. Serialization once per checkpoint, not per state modification. This
allows to “squash” updates to the same keys
2. Shorter synchronous phase (compared to RocksDB incremental)
3. No need for sorting and compaction, no IO amplification and JNI overhead
This can potentially give higher throughput and efficiency.

However, Heap backend currently lacks incremental checkpoints. This FLIP
aims to add initial support for them.

[1]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-151%3A+Incremental+snapshots+for+heap-based+state+backend


Any feedback highly appreciated.

Regards,
Roman