[jira] [Commented] (FLINK-3947) Provide low level access to RocksDB state backend

["Elias Levy (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/FLINK-3947?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15951948#comment-15951948 ] 

Elias Levy commented on FLINK-3947:
-----------------------------------

MapState looks promising. That said, I note that it is not an ordered map and does not provide a range method to iterate over an ordered range of keys, something the underlaying RocksDB implementation would support.  Nor is it a TTL cache, which would mean periodic scans of the whole map to expire entries.  But it may be sufficient to support my use case.  I'll need to spend some time sketching it out.

> Provide low level access to RocksDB state backend
> -------------------------------------------------
>
>                 Key: FLINK-3947
>                 URL: https://issues.apache.org/jira/browse/FLINK-3947
>             Project: Flink
>          Issue Type: Improvement
>          Components: State Backends, Checkpointing
>    Affects Versions: 1.0.3
>            Reporter: Elias Levy
>
> The current state API is limiting and some implementations are not as efficient as they could be, particularly when working with large states. For instance, a ListState is append only.  You cannot remove values from the list.  And the RocksDBListState get() implementation reads all list values from RocksDB instead of returning an Iterable that only reads values as needed.
> Furthermore, RocksDB is an ordered KV store, yet there is no ordered map state API with an ability to iterate over the stored values in order.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)