[jira] [Resolved] (FLINK-2808) Rework / Extend the StatehandleProvider

["Stephan Ewen (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/FLINK-2808?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Stephan Ewen resolved FLINK-2808.
---------------------------------

Implemented in 479bec0b2b44315196c1f1cddeb114c79d1717db

> Rework / Extend the StatehandleProvider
> ---------------------------------------
>
>                 Key: FLINK-2808
>                 URL: https://issues.apache.org/jira/browse/FLINK-2808
>             Project: Flink
>          Issue Type: Improvement
>          Components: Streaming
>    Affects Versions: 0.10
>            Reporter: Stephan Ewen
>            Assignee: Stephan Ewen
>             Fix For: 0.10
>
>
> I would like to make some changes (mostly additions) to the {{StateHandleProvider}}. Ideally for the upcoming release, as it is somewhat part of the public API.
> The rational behind this is to handle in a nice and extensible way the creation of key/value state backed by various implementations (FS, distributed KV store, local KV store with FS backup, ...) and various checkpointing ways (full dump, append, incremental keys, ...)
> The changes would concretely be:
> 1.  There should be a default {{StateHandleProvider}} set on the execution environment. Functions can later specify the {{StateHandleProvider}} when grabbing the {{StreamOperatorState}} from the runtime context (plus optionally a {{Checkpointer}})
> 2.  The {{StreamOperatorState}} is created from the {{StateHandleProvider}}. That way, a KeyValueStore state backend can create a {{StreamOperatorState}} that directly updates data in the KV store on every access, if that is desired (and filter accesses by timestamps to only show committed data)
> 3.  The StateHandleProvider should have methods to get an output stream that writes to the state checkpoint directly (and returns a StateHandle upon closing). That way we can convert and dump large state into the checkpoint without crating a full copy in memory before.
> Lastly, I would like to change some names
>   - {{StateHandleProvider}} to either {{StateBackend}}, {{StateStore}}, or {{StateProvider}} (simpler name).
>   - {{StreamOperatorState}} to either {{State}} or {{KVState}}.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)