Re: Operator variables in memory scoped by key

[gerardg <ge...@talaia.io>]

Thanks Aljoscha,

So I can think of three possible solutions:

* Use an instance dictionary to store the trie tree scoped by the same key
that the KeyedStream. That should work if the lifetime of the operator
object is tied to the keys that it processes.
* Store the trie tree in a ValueState but somehow tell flink to not
checkpoint this state. Also, it would be better if this could be in a
MemoryStateBackend and the rest of the application state in a
RocksDBStateBackend but I think backends can't be mixed in the same job.
* Forget about the MapState and just store the tree in a ValueState and
somehow control how it is checkpointed so I can manually implement
incremental checkpoints. This seems kind of complex as I would need to keep
track of the insertions and deletions to the tree from the last checkpoint.

The first one is the most straightforward but I'm not sure if it is really
feasible without knowing flink internals. Also, if I rely in how Flink
internally manages objects and at some point it changes, it could introduce
a bug difficult to detect.

Could you provide some insight? 

Thanks,

Gerard
  



--
Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/