[jira] [Created] (FLINK-21578) Closeable Sink Committer/GlobalCommitter were created to function in onestep during job graph composition

["Kezhu Wang (Jira)" <ji...@apache.org>]

Kezhu Wang created FLINK-21578:
----------------------------------

             Summary: Closeable Sink Committer/GlobalCommitter were created to function in onestep during job graph composition
                 Key: FLINK-21578
                 URL: https://issues.apache.org/jira/browse/FLINK-21578
             Project: Flink
          Issue Type: Improvement
          Components: API / DataStream
    Affects Versions: 1.13.0
            Reporter: Kezhu Wang


Normally, functions/operators are created in job graph composition phase for serialization and transmission. Them are "opened" in flink cluster to function. This two steps procedure succeed in that there will be no resource-cleanup requirement in job graph composition phase.

While {{Committer}} and {{GlobalCommitter}} has no such "open" operatin but they were created in job graph composition phase.

Following are fixes I could image if we converge to "this is problematic".
 # Add {{open}} or similar method for these two classes.
 # Add {{hasCommitter}}, {{hasGlobalCommitter}} to {{Sink}} and make {{createCommitter}} and others not optional(enforce this in runtime).

Personally, I am a bit preferring second approach for possible less code path touching in job graph composition phase. But first approach has advantage that it could be an no breaking change.

There might be other approaches though.

cc [~guoweima] [~gaoyunhaii]  [~aljoscha]  [~kkl0u]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)