[jira] [Commented] (FLINK-3947) Provide low level access to RocksDB state backend

["Aljoscha Krettek (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/FLINK-3947?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15294810#comment-15294810 ] 

Aljoscha Krettek commented on FLINK-3947:
-----------------------------------------

There is already this: FLINK-3582 which is addressed in this PR: https://github.com/apache/flink/pull/1957

For the {{ListState}} and reading as an {{Iterable}} the only optimization that we could do is to deserialize the individual list elements as we read them from the {{Iterable}}. We still get the whole list state as a BLOB from RocksDB because it knows nothing except BLOBs for keys and values.

> Provide low level access to RocksDB state backend
> -------------------------------------------------
>
>                 Key: FLINK-3947
>                 URL: https://issues.apache.org/jira/browse/FLINK-3947
>             Project: Flink
>          Issue Type: Improvement
>          Components: state backends
>    Affects Versions: 1.0.3
>            Reporter: Elias Levy
>
> The current state API is limiting and some implementations are not as efficient as they could be, particularly when working with large states. For instance, a ListState is append only.  You cannot remove values from the list.  And the RocksDBListState get() implementation reads all list values from RocksDB instead of returning an Iterable that only reads values as needed.
> Furthermore, RocksDB is an ordered KV store, yet there is no ordered map state API with an ability to iterate over the stored values in order.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)