You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Deniz Koçak <le...@gmail.com> on 2021/06/28 06:43:16 UTC
Event Time Timers in Process functions when time characteristic is ProcessingTime
Hi,
In environment configuration when set the time characteristics to
ProcessingTime via setStreamTimeCharacteristic(...) call, I cannot see
watermarks in the Flink UI. I think this is expected, because
watermarks disabled in the source (using Kafka as source)?
Another point here is, can I use Event Time Timers even if I set the
time characteristics to ProcessingTime (via
ctx.timerService().registerEventTimeTimer(...))? As far as I
understood from the documentation, Event Time Timers needs watermarks
which sets the operator time, so I wonder if I can use event time
timers in case of ProcessingTime selected in the environment? Also
when I set the time characteristics to Event Time can I use both
processing time timers & event time timers without any problem?
Thanks,
Re: Event Time Timers in Process functions when time characteristic
is ProcessingTime
Posted by JING ZHANG <be...@gmail.com>.
Hi lenduha,
> Processing time: We can only use processing time windows & timer.
Event time: We are able to use both processing time & event time
windows & timers. Processing time is always wall/clock time depending
on the host running task manager so independent from the event time?
Yes.
> Also Kafka seems to be supporting timestamp/watermark, however is
there any source not supporting timestamp/watermark & possible
solutions to that like a custom watermark generator?
Yes, there are two places in Flink where WatermarkStrategy can be used [1]:
1) directly on sources, like Kafka connector you mentioned.
2) after non-source operation.
The first option is preferable, because it allows sources to exploit
knowledge about shards/partition/splits in the watermarking logic, which is
more accurate.
But if sources does not have such ability, you could use second option.
Please check document [1] for more details about option1 and option2.
Please note, the document[1] is based on version Flink 1.13 and 1.13+.
Please check document[2] if you use old version.
[1]
https://ci.apache.org/projects/flink/flink-docs-master/docs/dev/datastream/event-time/generating_watermarks/
[2]
https://ci.apache.org/projects/flink/flink-docs-release-1.12/dev/event_timestamps_watermarks.html
Best regards,
JING ZHANG
Deniz Koçak <le...@gmail.com> 于2021年6月28日周一 下午5:06写道：
> Thanks for the info Jing, just to clarify one point, so when we set
> the environment time characteristics to
>
> Processing time: We can only use processing time windows & timer.
> Event time: We are able to use both processing time & event time
> windows & timers. Processing time is always wall/clock time depending
> on the host running task manager so independent from the event time?
>
> Also Kafka seems to be supporting timestamp/watermark, however is
> there any source not supporting timestamp/watermark & possible
> solutions to that like a custom watermark generator?
>
> Thanks,
>
> On Mon, Jun 28, 2021 at 8:25 AM JING ZHANG <be...@gmail.com> wrote:
> >
> > Hi lenduha,
> >
> > > When set the time characteristics to ProcessingTime via
> setStreamTimeCharacteristic(...) call, I cannot see watermarks in the Flink
> UI.
> >
> > The watermark would be swallowed in the case.
> >
> > > can I use Event Time Timers even if I set the
> > time characteristics to ProcessingTime (via
> > ctx.timerService().registerEventTimeTimer(...))?
> >
> > I'm afraid not. The event time Timers could not work after setting
> ProcessingTime by setStreamTimeCharacteristic,
> > because the watermark should be swallowed in the case, so the event time
> timers would not be triggered.
> >
> > > when I set the time characteristics to Event Time can I use both
> > processing time timers & event time timers without any problem?
> >
> > Event time timers could work, of course. Besides, explicitly using
> processing-time windows and timers works in event-time mode.
> > Please note that In Flink1.12, the default stream time characteristic
> has been changed to EventTime [1].
> >
> > [1]
> https://ci.apache.org/projects/flink/flink-docs-release-1.13/release-notes/flink-1.12/#flip-134-batch-execution-for-the-datastream-api
> >
> > Best regards,
> > JING ZHANG
> >
> > Deniz Koçak <le...@gmail.com> 于2021年6月28日周一 下午2:43写道：
> >>
> >> Hi,
> >>
> >> In environment configuration when set the time characteristics to
> >> ProcessingTime via setStreamTimeCharacteristic(...) call, I cannot see
> >> watermarks in the Flink UI. I think this is expected, because
> >> watermarks disabled in the source (using Kafka as source)?
> >>
> >> Another point here is, can I use Event Time Timers even if I set the
> >> time characteristics to ProcessingTime (via
> >> ctx.timerService().registerEventTimeTimer(...))? As far as I
> >> understood from the documentation, Event Time Timers needs watermarks
> >> which sets the operator time, so I wonder if I can use event time
> >> timers in case of ProcessingTime selected in the environment? Also
> >> when I set the time characteristics to Event Time can I use both
> >> processing time timers & event time timers without any problem?
> >>
> >> Thanks,
>
Re: Event Time Timers in Process functions when time characteristic
is ProcessingTime
Posted by Deniz Koçak <le...@gmail.com>.
Thanks for the info Jing, just to clarify one point, so when we set
the environment time characteristics to
Processing time: We can only use processing time windows & timer.
Event time: We are able to use both processing time & event time
windows & timers. Processing time is always wall/clock time depending
on the host running task manager so independent from the event time?
Also Kafka seems to be supporting timestamp/watermark, however is
there any source not supporting timestamp/watermark & possible
solutions to that like a custom watermark generator?
Thanks,
On Mon, Jun 28, 2021 at 8:25 AM JING ZHANG <be...@gmail.com> wrote:
>
> Hi lenduha,
>
> > When set the time characteristics to ProcessingTime via setStreamTimeCharacteristic(...) call, I cannot see watermarks in the Flink UI.
>
> The watermark would be swallowed in the case.
>
> > can I use Event Time Timers even if I set the
> time characteristics to ProcessingTime (via
> ctx.timerService().registerEventTimeTimer(...))?
>
> I'm afraid not. The event time Timers could not work after setting ProcessingTime by setStreamTimeCharacteristic,
> because the watermark should be swallowed in the case, so the event time timers would not be triggered.
>
> > when I set the time characteristics to Event Time can I use both
> processing time timers & event time timers without any problem?
>
> Event time timers could work, of course. Besides, explicitly using processing-time windows and timers works in event-time mode.
> Please note that In Flink1.12, the default stream time characteristic has been changed to EventTime [1].
>
> [1] https://ci.apache.org/projects/flink/flink-docs-release-1.13/release-notes/flink-1.12/#flip-134-batch-execution-for-the-datastream-api
>
> Best regards,
> JING ZHANG
>
> Deniz Koçak <le...@gmail.com> 于2021年6月28日周一 下午2:43写道：
>>
>> Hi,
>>
>> In environment configuration when set the time characteristics to
>> ProcessingTime via setStreamTimeCharacteristic(...) call, I cannot see
>> watermarks in the Flink UI. I think this is expected, because
>> watermarks disabled in the source (using Kafka as source)?
>>
>> Another point here is, can I use Event Time Timers even if I set the
>> time characteristics to ProcessingTime (via
>> ctx.timerService().registerEventTimeTimer(...))? As far as I
>> understood from the documentation, Event Time Timers needs watermarks
>> which sets the operator time, so I wonder if I can use event time
>> timers in case of ProcessingTime selected in the environment? Also
>> when I set the time characteristics to Event Time can I use both
>> processing time timers & event time timers without any problem?
>>
>> Thanks,
Re: Event Time Timers in Process functions when time characteristic
is ProcessingTime
Posted by JING ZHANG <be...@gmail.com>.
Hi lenduha,
> When set the time characteristics to ProcessingTime via
setStreamTimeCharacteristic(...) call, I cannot see watermarks in the Flink
UI.
The watermark would be swallowed in the case.
> can I use Event Time Timers even if I set the
time characteristics to ProcessingTime (via
ctx.timerService().registerEventTimeTimer(...))?
I'm afraid not. The event time Timers could not work after setting
ProcessingTime by setStreamTimeCharacteristic,
because the watermark should be swallowed in the case, so the event time
timers would not be triggered.
> when I set the time characteristics to Event Time can I use both
processing time timers & event time timers without any problem?
Event time timers could work, of course. Besides, explicitly using
processing-time windows and timers works in event-time mode.
Please note that In Flink1.12, the default stream time characteristic has
been changed to EventTime [1].
[1]
https://ci.apache.org/projects/flink/flink-docs-release-1.13/release-notes/flink-1.12/#flip-134-batch-execution-for-the-datastream-api
Best regards,
JING ZHANG
Deniz Koçak <le...@gmail.com> 于2021年6月28日周一 下午2:43写道：
> Hi,
>
> In environment configuration when set the time characteristics to
> ProcessingTime via setStreamTimeCharacteristic(...) call, I cannot see
> watermarks in the Flink UI. I think this is expected, because
> watermarks disabled in the source (using Kafka as source)?
>
> Another point here is, can I use Event Time Timers even if I set the
> time characteristics to ProcessingTime (via
> ctx.timerService().registerEventTimeTimer(...))? As far as I
> understood from the documentation, Event Time Timers needs watermarks
> which sets the operator time, so I wonder if I can use event time
> timers in case of ProcessingTime selected in the environment? Also
> when I set the time characteristics to Event Time can I use both
> processing time timers & event time timers without any problem?
>
> Thanks,
>