You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "Stephan Ewen (JIRA)" <ji...@apache.org> on 2018/05/24 12:06:00 UTC
[jira] [Closed] (FLINK-9428) Allow operators to flush data on
checkpoint pre-barrier
[ https://issues.apache.org/jira/browse/FLINK-9428?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Stephan Ewen closed FLINK-9428.
-------------------------------
> Allow operators to flush data on checkpoint pre-barrier
> -------------------------------------------------------
>
> Key: FLINK-9428
> URL: https://issues.apache.org/jira/browse/FLINK-9428
> Project: Flink
> Issue Type: New Feature
> Components: State Backends, Checkpointing
> Reporter: Stephan Ewen
> Assignee: Stephan Ewen
> Priority: Major
> Fix For: 1.6.0
>
>
> Some operators maintain some small transient state that may be inefficient to checkpoint, especially when it would need to be checkpointed also in a re-scalable way.
> An example are opportunistic pre-aggregation operators, which have small the pre-aggregation state that is frequently flushed downstream.
> Rather that persisting that state in a checkpoint, it can make sense to flush the data downstream upon a checkpoint, to let it be part of the downstream operator's state.
> This feature is sensitive, because flushing state has a clean implication on the downstream operator's checkpoint alignment. However, used with care, and with the new back-pressure-based checkpoint alignment, this feature can be very useful.
> Because it is sensitive, I suggest to make this only an internal feature (accessible to operators) and NOT expose it in the public API at this point.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)