[jira] [Commented] (BEAM-6733) Use Flink 1.6/1.7 prepareSnapshotPreBarrier to replace BufferedOutputManager

["Thomas Weise (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/BEAM-6733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16936920#comment-16936920 ] 

Thomas Weise commented on BEAM-6733:
------------------------------------

This has an effect on checkpointing that is probably worth discussing. Checkpoint barriers would be blocked by finishing the bundle, impacting alignment similar to what we see in Flink under backpressure. Heavy backpressure in Flink causes checkpoint timeouts and related operational issues.

Regarding latency this should be neutral as we already wait for the bundle to finish before we let the watermark pass?

> Use Flink 1.6/1.7 prepareSnapshotPreBarrier to replace BufferedOutputManager
> ----------------------------------------------------------------------------
>
>                 Key: BEAM-6733
>                 URL: https://issues.apache.org/jira/browse/BEAM-6733
>             Project: Beam
>          Issue Type: Improvement
>          Components: runner-flink
>            Reporter: Maximilian Michels
>            Assignee: Maximilian Michels
>            Priority: Minor
>          Time Spent: 2h 20m
>  Remaining Estimate: 0h
>
> Flink 1.6/1.7 provides a hook to execute an action before the snapshot barrier is emitted by the operator. At the moment (<=1.5) the Flink Runner has to buffer any elements which are emitted during a snapshot because the barrier has already been emitted. This leads to a lot of code complexity.
> We can remove the buffering in favor of finishing the current bundle in {{DoFnOperator}}'s {{prepareSnapshotPreBarrier}}. The 1.5/1.6/1.7 build setup poses a challenge to do that in a way that does not lead to much code duplication.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)