[jira] [Commented] (FLINK-3946) State API Should Support Data Expiration

["Aljoscha Krettek (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/FLINK-3946?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15294795#comment-15294795 ] 

Aljoscha Krettek commented on FLINK-3946:
-----------------------------------------

I thought about this when adding the {{StateDescriptor}} interface a while back. This would be the place to put a configuration value for the expiry. The state backend would have to be given access to the internal timer service and it would need to register timers for state that has expiry.

In order to make this pleasant we also need to abstract away the processing-time and event-time timers behind interfaces. Right now, the window operator has custom code to deal with watermark-based timers but if several components start requiring this functionality we should put it behind a good interface and let the AbstractStreamOperator handle it.

> State API Should Support Data Expiration
> ----------------------------------------
>
>                 Key: FLINK-3946
>                 URL: https://issues.apache.org/jira/browse/FLINK-3946
>             Project: Flink
>          Issue Type: Improvement
>          Components: state backends
>    Affects Versions: 1.0.3
>            Reporter: Elias Levy
>
> The state API should support data expiration.  Consider a custom data stream operator that operates on a keyed stream and maintains a cache of items in its keyed state.  The operator can expire items in the state based on event time or  processing time.  But as the state is keyed, if the operator is never invoked again for a previously observed key, the state associated with that key will never be expired, leading the overall job state to grow indefinitely.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)