[jira] [Created] (FLINK-2808) Rework / Extend the StatehandleProvider

["Stephan Ewen (JIRA)" <ji...@apache.org>]

Stephan Ewen created FLINK-2808:
-----------------------------------

             Summary: Rework / Extend the StatehandleProvider
                 Key: FLINK-2808
                 URL: https://issues.apache.org/jira/browse/FLINK-2808
             Project: Flink
          Issue Type: Improvement
          Components: Streaming
    Affects Versions: 0.10
            Reporter: Stephan Ewen
            Assignee: Stephan Ewen
             Fix For: 0.10


I would like to make some changes (mostly additions) to the {{StateHandleProvider}}. Ideally for the upcoming release, as it is somewhat part of the public API.

The rational behind this is to handle in a nice and extensible way the creation of key/value state backed by various implementations (FS, distributed KV store, local KV store with FS backup, ...) and various checkpointing ways (full dump, append, incremental keys, ...)

The changes would concretely be:

1.  There should be a default {{StateHandleProvider}} set on the execution environment. Functions can later specify the {{StateHandleProvider}} when grabbing the {{StreamOperatorState}} from the runtime context (plus optionally a {{Checkpointer}})

2.  The {{StreamOperatorState}} is created from the {{StateHandleProvider}}. That way, a KeyValueStore state backend can create a {{StreamOperatorState}} that directly updates data in the KV store on every access, if that is desired (and filter accesses by timestamps to only show committed data)

3.  The StateHandleProvider should have methods to get an output stream that writes to the state checkpoint directly (and returns a StateHandle upon closing). That way we can convert and dump large state into the checkpoint without crating a full copy in memory before.


Lastly, I would like to change some names
  - {{StateHandleProvider}} to either {{StateBackend}}, {{StateStore}}, or {{StateProvider}} (simpler name).
  - {{StreamOperatorState}} to either {{State}} or {{KVState}}.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)