[jira] [Commented] (FLINK-4164) Use only one rocksdb column family to store all states in an operator

["Aljoscha Krettek (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/FLINK-4164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15366007#comment-15366007 ] 

Aljoscha Krettek commented on FLINK-4164:
-----------------------------------------

I had a quick chat with [~StephanEwen] about this. We came to the conclusion that it might make sense to split the RocksDBStateBacked into separate backends. Then we could have one with the current behavior and one where all state is stored in one column. This would allow users to choose the right backend for their requirements. The reason I'm hesitant about changing the backend to always store all state in a single column is that this might make iteration more complex and that the key prefix will lead to more overhead.

> Use only one rocksdb column family to store all states in an operator
> ---------------------------------------------------------------------
>
>                 Key: FLINK-4164
>                 URL: https://issues.apache.org/jira/browse/FLINK-4164
>             Project: Flink
>          Issue Type: Improvement
>          Components: State Backends, Checkpointing
>            Reporter: LiuBiao
>            Priority: Minor
>
> I see now in master branch, there is only one rocksdb instance in an operator, each state will be assigned into separated column families. It's a good improvement, but I think there are still some problems. Column families in rocksdb do not share memtable and sst files(correct me if I am wrong, there is not much docs about column families in rocksdb). If user have thousands of  states, the overhead will be costly. Memory may will be exhausted, also there will be too much small sst files.
> If we use prefix in key instead of column family, the memory that rocksdb costed will be under control. The shortcoming is that we can not optimize states separately, and performance may be a little worse than before. But I think it is worth to do, it is meaning for some cases.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)