[jira] [Updated] (BEAM-6733) Use Flink 1.6/1.7 prepareSnapshotPreBarrier to replace BufferedOutputManager

["Kenneth Knowles (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/BEAM-6733?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Kenneth Knowles updated BEAM-6733:
----------------------------------

This Jira ticket has a pull request attached to it, but is still open. Did the pull request resolve the issue? If so, could you please mark it resolved? This will help the project have a clear view of its open issues.

> Use Flink 1.6/1.7 prepareSnapshotPreBarrier to replace BufferedOutputManager
> ----------------------------------------------------------------------------
>
>                 Key: BEAM-6733
>                 URL: https://issues.apache.org/jira/browse/BEAM-6733
>             Project: Beam
>          Issue Type: Improvement
>          Components: runner-flink
>            Reporter: Maximilian Michels
>            Priority: P3
>          Time Spent: 5h
>  Remaining Estimate: 0h
>
> Flink 1.6/1.7 provides a hook to execute an action before the snapshot barrier is emitted by the operator. At the moment (<=1.5) the Flink Runner has to buffer any elements which are emitted during a snapshot because the barrier has already been emitted. This leads to a lot of code complexity.
> We can remove the buffering in favor of finishing the current bundle in {{DoFnOperator}}'s {{prepareSnapshotPreBarrier}}. The 1.5/1.6/1.7 build setup poses a challenge to do that in a way that does not lead to much code duplication.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)