You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "Kezhu Wang (Jira)" <ji...@apache.org> on 2019/11/14 15:10:00 UTC
[jira] [Updated] (FLINK-14790) Custom state backend in job graph
composition without depending on concrete state backend implementation
[ https://issues.apache.org/jira/browse/FLINK-14790?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Kezhu Wang updated FLINK-14790:
-------------------------------
Description:
In Flink 1.9 and before, if we want to custom state backend, we have to supply concrete state backend implementation in {{StreamExecutionEnvironment.setStateBackend}}. This implies that we have to depend on that concrete state backend implementation in job graph composition phase since we have to construct this concrete state backend. This may not be appropriate if we separate job graph composition and execution strictly, as job graph composition requires no state backend functions besides customization.
Suppose that we are going to build a declarative streaming platform that building plain json/yaml like textual object, which may contain custom state backend configuration, with attached dependent jars to job graph and submitting generated job graph to Flink, we apparently should avoid ad-hoc state backend construction due to code reusability and indeterministic state backends configured.
So I propose to support state backend customization in job graph composition phase to not depend on concrete state backend implementation.
To achieve this goal, a {{Configuration}} field, say {{StreamExecutionEnvironment.stateBackendConfiguration}}, should be sufficient.
was:
In Flink 1.9 and before, if we want to custom state backend, we have to supply concrete state backend implementation in {{StreamExecutionEnvironment.setStateBackend}}. This implies that we have to depend on that concrete state backend implementation in job graph composition phase since we have to construct this concrete state backend. This may not be appropriate if we separate job graph composition and execution strictly, as job graph composition requires no state backend functions besides customization.
Suppose that we are going to build a declarative streaming platform that building plain json/yaml like textual object, which may contain custom state backend configuration, with attached dependent jars to job graph and submitting generated job graph to Flink, we apparently should avoid ad-hoc state backend construction due to code reusability and indeterministic state backends configured.
So I propose to state backend customization in job graph composition phase to not depend on concrete state backend implementation.
To achieve this goal, a {{Configuration}} field, say {{StreamExecutionEnvironment.stateBackendConfiguration}}, should be sufficient.
> Custom state backend in job graph composition without depending on concrete state backend implementation
> --------------------------------------------------------------------------------------------------------
>
> Key: FLINK-14790
> URL: https://issues.apache.org/jira/browse/FLINK-14790
> Project: Flink
> Issue Type: New Feature
> Components: API / DataStream
> Reporter: Kezhu Wang
> Priority: Major
>
> In Flink 1.9 and before, if we want to custom state backend, we have to supply concrete state backend implementation in {{StreamExecutionEnvironment.setStateBackend}}. This implies that we have to depend on that concrete state backend implementation in job graph composition phase since we have to construct this concrete state backend. This may not be appropriate if we separate job graph composition and execution strictly, as job graph composition requires no state backend functions besides customization.
> Suppose that we are going to build a declarative streaming platform that building plain json/yaml like textual object, which may contain custom state backend configuration, with attached dependent jars to job graph and submitting generated job graph to Flink, we apparently should avoid ad-hoc state backend construction due to code reusability and indeterministic state backends configured.
> So I propose to support state backend customization in job graph composition phase to not depend on concrete state backend implementation.
> To achieve this goal, a {{Configuration}} field, say {{StreamExecutionEnvironment.stateBackendConfiguration}}, should be sufficient.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)