How does custom object/data structures get mapped into rocksdb underneath?

[kant kodali <ka...@gmail.com>]

Hi All,

I want to do stateful streaming and I was wondering how Custom objects get
mapped into rocksdb?

say I have the following class that represents my state

public class MyState {
    private HashMap<String, T> map1 ; // T can be any type
    private HashMap<Integer, S> map2; // S can be any type
}

I wonder how these two maps gets mapped into rocksdb? and how does Flink
know that map1 and map2 together are part of my state but not
individual ones in isolation?

Thanks!

Re: How does custom object/data structures get mapped into rocksdb underneath?

[Aljoscha Krettek <al...@apache.org>]

Hi,

when working with state you have to create a state descriptor ([1]). 
This will have a TypeInformation or TypeSerializer that Flink uses to 
(de)serialize your state. This will be used to serialize your state to 
bytes, in the case of RocksDB we serialize both the key (the key of the 
record that is used to key the stream) and the value (your state object, 
MyState in you example) to bytes and store them in the RocksDB key-value 
store.

Best,
Aljoscha

[1] 
https://ci.apache.org/projects/flink/flink-docs-master/dev/stream/state/state.html

On 09.03.20 12:27, kant kodali wrote:
> Hi All,
> 
> I want to do stateful streaming and I was wondering how Custom objects get
> mapped into rocksdb?
> 
> say I have the following class that represents my state
> 
> public class MyState {
>      private HashMap<String, T> map1 ; // T can be any type
>      private HashMap<Integer, S> map2; // S can be any type
> }
> 
> I wonder how these two maps gets mapped into rocksdb? and how does Flink
> know that map1 and map2 together are part of my state but not
> individual ones in isolation?
> 
> Thanks!
>