State backend questions

[Christophe Jolif <cj...@gmail.com>]

Hi all,

At first my state should not be "that" big and fit in memory, so
FsStateBackend could be a solution for me. However moving forward I
envision more features and more users and the state growing. With that in
mind RocksDBStateBackend might be the solution.

Is there an easy "upgrade" path from one to another? In other words is
there an easy path to "move" the state from one backend to another one and
restart the job from there if the need arise or should I definitely plan
long ahead and use RocksDB right away if I don't want to get into trouble?

If yes, how much configuration is available at Flink level to configure
memory used by RocksDB to cache data without having to go back to disk so
that I don't penalize too much the current use-cases?

Thanks,
-- 
Christophe

Re: State backend questions

[Chesnay Schepler <ch...@apache.org>]

According to this thread 
<http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Change-state-backend-tp14928p14966.html> 
it is not yet possible to switch to/from RocksDBStatebackend, so I would 
suggest to with RocksDB from the start.

For tuning RocksDB, see here 
<https://ci.apache.org/projects/flink/flink-docs-release-1.4/ops/state/large_state_tuning.html#tuning-rocksdb>.

On 17.01.2018 01:30, Christophe Jolif wrote:
> Hi all,
>
> At first my state should not be "that" big and fit in memory, so 
> FsStateBackend could be a solution for me. However moving forward I 
> envision more features and more users and the state growing. With that 
> in mind RocksDBStateBackend might be the solution.
>
> Is there an easy "upgrade" path from one to another? In other words is 
> there an easy path to "move" the state from one backend to another one 
> and restart the job from there if the need arise or should I 
> definitely plan long ahead and use RocksDB right away if I don't want 
> to get into trouble?
>
> If yes, how much configuration is available at Flink level to 
> configure memory used by RocksDB to cache data without having to go 
> back to disk so that I don't penalize too much the current use-cases?
>
> Thanks,
> -- 
> Christophe