[jira] [Comment Edited] (FLINK-10830) Consider making processing time provider pluggable

["Kostas Kloudas (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/FLINK-10830?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16679920#comment-16679920 ] 

Kostas Kloudas edited comment on FLINK-10830 at 11/8/18 3:43 PM:
-----------------------------------------------------------------

Although this seems like a nice idea, I have some concerns:

1) the first one has to do with the fact that the processing time timer service is also used internally by the StreamTask for scheduling different tasks for example emitting Watermarks in the case of a PeriodicWatermarkEmitter. So I am not so sure if we want to allow the user to mess with this. 

2) if we expose this, and assuming that the user uses processing time then we have to also establish a mechanism that tells Flink "what time it is", i.e. a processing time watermark. If not, then in the case that we use this service to register a timer, who will tell the service when to fire that timer? It can be that we say "fire as soon as you see an element with timestamp greater than the timer", but in this case we have to wait for the "next element" which can lead to weird behaviour especially for keyed streams where some elements may become inactive.

3) finally, we will have the same problems as with the "non-checkpointed watermark" upon restoring.



was (Author: kkl0u):
Although this seems like a nice idea, I have some concerns:

1) that have to do with the fact that the processing time timer service is also used internally by the StreamTask for scheduling different tasks for example emitting Watermarks in the case of a PeriodicWatermarkEmitter. So I am not so sure if we want to allow the user to mess with this. 

2) if we expose this, and assuming that the user uses processing time then we have to also establish a mechanism that tells Flink "what time it is", i.e. a processing time watermark. If not, then in the case that we use this service to register a timer, who will tell the service when to fire that timer? It can be that we say "fire as soon as you see an element with timestamp greater than the timer", but in this case we have to wait for the "next element" which can lead to weird behaviour especially for keyed streams where some elements may become inactive.

3) finally, we will have the same problems as with the "non-checkpointed watermark" upon restoring.


> Consider making processing time provider pluggable
> --------------------------------------------------
>
>                 Key: FLINK-10830
>                 URL: https://issues.apache.org/jira/browse/FLINK-10830
>             Project: Flink
>          Issue Type: Improvement
>            Reporter: Andrey Zagrebin
>            Priority: Major
>
> At the moment, the processing time is basically implemented in a fixed way as 
> System.currentTimeMillis() and not configurable by users.
> If this implementation does not fit application business logic for some reason there is no way for users to change it.
> Examples:
>  * The timestamp provided by currentTimeMillis is not guaranteed to be monotonically increasing. It can jump back for a while because of possible periodic synchronisation of local clock with other more accurate system. It can be a problem for application business logic if we say that the general notion of time is that it always increases.
>  * Hard to implement end-to-end tests because synchronisation between time in test and in Flink is out of control.
> We can make it configurable and let users optionally set their own factory to create processing time provider. All features which depend on querying current processing time can use this implementation. The default one can still stay System.currentTimeMillis().



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)