You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by M Singh <ma...@yahoo.com> on 2019/07/09 01:42:12 UTC
Apache Flink - Relation between stream time characteristic and
timer triggers
Hi:
I have a few questions about the stream time characteristics:
1. If the time characteristic is set to TimeCharacteristic.EventTime, but the timers in a processor or trigger is set using registerProcessingTimeTimer (or vice versa), then will that timer fire ?
2. Once the time character is set on the stream environment, and changed later in the application, which one is applied, the first one or the last one ?
3. If the stream time characteristic is set to IngestionTime, then is there any adverse effect of assigning the timestamp using assignTimeStampAndWatermark to a stream later in the application ?
Thanks
Re: Apache Flink - Relation between stream time characteristic and
timer triggers
Posted by M Singh <ma...@yahoo.com>.
Thanks Fabian/Xingcan/Yun for all your help. Mans
On Thursday, July 11, 2019, 11:46:42 AM EDT, Fabian Hueske <fh...@gmail.com> wrote:
Hi,
ProcessingTime timers are always supportedEventTime timers are only supported for EventTime and IngestionTime
Best, Fabian
Am Do., 11. Juli 2019 um 17:44 Uhr schrieb M Singh <ma...@yahoo.com>:
Thanks Fabian for your response.
Just to clarify then - regardless of the time characteristics, if a processor or window trigger registers with a ProcessingTime and EventTime timers - they will all fire when the appropriate watermarks arrive.
Thanks again.
On Thursday, July 11, 2019, 05:41:54 AM EDT, Fabian Hueske <fh...@gmail.com> wrote:
Hi Mans,
IngestionTime is uses the same internal mechanisms as EventTime (record timestamps and watermarks).
The difference is that instead of extracting a timestamp from the record (using a custom timestamp extractor & wm assigner), Flink will assign timestamps based on the machine clock of the machine that runs the source task and will also automatically generate watermarks. If you ask for my opinion, IngestionTime combines the disadvantages of ProcessingTime and EventTime. You pay the latency / performance penalty of EventTime for the non-determinism of ProcessingTime.
So, if you enable IngestionTime, you can use EventTime timers and ProcessingTime timers.
Best, Fabian
Am Mi., 10. Juli 2019 um 09:37 Uhr schrieb M Singh <ma...@yahoo.com>:
Thanks for your answer Xingcan.
Just to clarify - if the characteristic is set to IngestionTime or ProcessingTime, the event time triggers will be ignored and not fire.
Mans
On Tuesday, July 9, 2019, 04:32:00 PM EDT, Xingcan Cui <xi...@gmail.com> wrote:
Yes, Mans. You can use both processing-time and event-time timers if you set the time characteristic to event-time. They'll be triggered by their own time semantics, separately. (actually there’s no watermark for processing time)
Cheers,Xingcan
On Jul 9, 2019, at 11:40 AM, M Singh <ma...@yahoo.com> wrote:
Thanks Yun for your answers.
Does this mean that we can use processing and event timers (in processors or triggers) regardless of the time characteristic ? Also, is possible to use both together and will they both fire at the appropriate watermarks for processing and event times ?
Mans
On Tuesday, July 9, 2019, 12:18:30 AM EDT, Yun Gao <yu...@aliyun.com> wrote:
Hi, For the three questions, 1. The processing time timer will be trigger. IMO you may think the processing time timer as in parallel with the event time timer. They are processed separately underlying. The processing time timer will be triggered according to the realistic time. 2. I'am not very clear on how to changed later in the application. Do you mean call `StreamExecutionEnvironment#setStreamTimeCharacteristics` multiple times ? If so, then the last call will take effect for all the operators before or after the last call, since the setting will only take effect in `StreamExecutionEnvironment#execute`. 3. 'assignTimeStampAndWatermark' will change the timestamp of the record. IMO you may think each record contains a timestamp field, and the filed is set when ingesting, but 'assignTimeStampAndWatermark' will change the value of this field, so the following operators relying on the timestamp will see the updated value.
Best,Yun
------------------------------------------------------------------From:M Singh <ma...@yahoo.com>Send Time:2019 Jul. 9 (Tue.) 09:42To:User <us...@flink.apache.org>Subject:Apache Flink - Relation between stream time characteristic and timer triggers
Hi:
I have a few questions about the stream time characteristics:
1. If the time characteristic is set to TimeCharacteristic.EventTime, but the timers in a processor or trigger is set using registerProcessingTimeTimer (or vice versa), then will that timer fire ?
2. Once the time character is set on the stream environment, and changed later in the application, which one is applied, the first one or the last one ?
3. If the stream time characteristic is set to IngestionTime, then is there any adverse effect of assigning the timestamp using assignTimeStampAndWatermark to a stream later in the application ?
Thanks
Re: Apache Flink - Relation between stream time characteristic and
timer triggers
Posted by Fabian Hueske <fh...@gmail.com>.
Hi,
ProcessingTime timers are always supported
EventTime timers are only supported for EventTime and IngestionTime
Best, Fabian
Am Do., 11. Juli 2019 um 17:44 Uhr schrieb M Singh <ma...@yahoo.com>:
> Thanks Fabian for your response.
>
> Just to clarify then - regardless of the time characteristics, if a
> processor or window trigger registers with a ProcessingTime and EventTime
> timers - they will all fire when the appropriate watermarks arrive.
>
> Thanks again.
>
> On Thursday, July 11, 2019, 05:41:54 AM EDT, Fabian Hueske <
> fhueske@gmail.com> wrote:
>
>
> Hi Mans,
>
> IngestionTime is uses the same internal mechanisms as EventTime (record
> timestamps and watermarks).
>
> The difference is that instead of extracting a timestamp from the record
> (using a custom timestamp extractor & wm assigner), Flink will assign
> timestamps based on the machine clock of the machine that runs the source
> task and will also automatically generate watermarks. If you ask for my
> opinion, IngestionTime combines the disadvantages of ProcessingTime and
> EventTime. You pay the latency / performance penalty of EventTime for the
> non-determinism of ProcessingTime.
>
> So, if you enable IngestionTime, you can use EventTime timers and
> ProcessingTime timers.
>
> Best, Fabian
>
> Am Mi., 10. Juli 2019 um 09:37 Uhr schrieb M Singh <ma...@yahoo.com>:
>
> Thanks for your answer Xingcan.
>
> Just to clarify - if the characteristic is set to IngestionTime or
> ProcessingTime, the event time triggers will be ignored and not fire.
>
> Mans
>
> On Tuesday, July 9, 2019, 04:32:00 PM EDT, Xingcan Cui <xi...@gmail.com>
> wrote:
>
>
> Yes, Mans. You can use both processing-time and event-time timers if you
> set the time characteristic to event-time. They'll be triggered by their
> own time semantics, separately. (actually there’s no watermark for
> processing time)
>
> Cheers,
> Xingcan
>
> On Jul 9, 2019, at 11:40 AM, M Singh <ma...@yahoo.com> wrote:
>
> Thanks Yun for your answers.
>
> Does this mean that we can use processing and event timers (in processors
> or triggers) regardless of the time characteristic ? Also, is possible to
> use both together and will they both fire at the appropriate watermarks for
> processing and event times ?
>
> Mans
>
> On Tuesday, July 9, 2019, 12:18:30 AM EDT, Yun Gao <yu...@aliyun.com>
> wrote:
>
>
> Hi,
> For the three questions,
> 1. The processing time timer will be trigger. IMO you may think the
> processing time timer as in parallel with the event time timer. They are
> processed separately underlying. The processing time timer will be
> triggered according to the realistic time.
> 2. I'am not very clear on how to changed later in the application. Do
> you mean call `StreamExecutionEnvironment#setStreamTimeCharacteristics`
> multiple times ? If so, then the last call will take effect for all the
> operators before or after the last call, since the setting will only take
> effect in `StreamExecutionEnvironment#execute`.
> 3. 'assignTimeStampAndWatermark' will change the timestamp of the
> record. IMO you may think each record contains a timestamp field, and the
> filed is set when ingesting, but 'assignTimeStampAndWatermark' will change
> the value of this field, so the following operators relying on the
> timestamp will see the updated value.
>
> Best,
> Yun
>
>
>
> ------------------------------------------------------------------
> From:M Singh <ma...@yahoo.com>
> Send Time:2019 Jul. 9 (Tue.) 09:42
> To:User <us...@flink.apache.org>
> Subject:Apache Flink - Relation between stream time characteristic and
> timer triggers
>
> Hi:
>
> I have a few questions about the stream time characteristics:
>
> 1. If the time characteristic is set to TimeCharacteristic.EventTime, but
> the timers in a processor or trigger is set using
> registerProcessingTimeTimer (or vice versa), then will that timer fire ?
>
> 2. Once the time character is set on the stream environment, and changed
> later in the application, which one is applied, the first one or the last
> one ?
>
> 3. If the stream time characteristic is set to IngestionTime, then is
> there any adverse effect of assigning the timestamp using
> assignTimeStampAndWatermark to a stream later in the application ?
>
> Thanks
>
>
>
>
Re: Apache Flink - Relation between stream time characteristic and
timer triggers
Posted by M Singh <ma...@yahoo.com>.
Thanks Fabian for your response.
Just to clarify then - regardless of the time characteristics, if a processor or window trigger registers with a ProcessingTime and EventTime timers - they will all fire when the appropriate watermarks arrive.
Thanks again.
On Thursday, July 11, 2019, 05:41:54 AM EDT, Fabian Hueske <fh...@gmail.com> wrote:
Hi Mans,
IngestionTime is uses the same internal mechanisms as EventTime (record timestamps and watermarks).
The difference is that instead of extracting a timestamp from the record (using a custom timestamp extractor & wm assigner), Flink will assign timestamps based on the machine clock of the machine that runs the source task and will also automatically generate watermarks. If you ask for my opinion, IngestionTime combines the disadvantages of ProcessingTime and EventTime. You pay the latency / performance penalty of EventTime for the non-determinism of ProcessingTime.
So, if you enable IngestionTime, you can use EventTime timers and ProcessingTime timers.
Best, Fabian
Am Mi., 10. Juli 2019 um 09:37 Uhr schrieb M Singh <ma...@yahoo.com>:
Thanks for your answer Xingcan.
Just to clarify - if the characteristic is set to IngestionTime or ProcessingTime, the event time triggers will be ignored and not fire.
Mans
On Tuesday, July 9, 2019, 04:32:00 PM EDT, Xingcan Cui <xi...@gmail.com> wrote:
Yes, Mans. You can use both processing-time and event-time timers if you set the time characteristic to event-time. They'll be triggered by their own time semantics, separately. (actually there’s no watermark for processing time)
Cheers,Xingcan
On Jul 9, 2019, at 11:40 AM, M Singh <ma...@yahoo.com> wrote:
Thanks Yun for your answers.
Does this mean that we can use processing and event timers (in processors or triggers) regardless of the time characteristic ? Also, is possible to use both together and will they both fire at the appropriate watermarks for processing and event times ?
Mans
On Tuesday, July 9, 2019, 12:18:30 AM EDT, Yun Gao <yu...@aliyun.com> wrote:
Hi, For the three questions, 1. The processing time timer will be trigger. IMO you may think the processing time timer as in parallel with the event time timer. They are processed separately underlying. The processing time timer will be triggered according to the realistic time. 2. I'am not very clear on how to changed later in the application. Do you mean call `StreamExecutionEnvironment#setStreamTimeCharacteristics` multiple times ? If so, then the last call will take effect for all the operators before or after the last call, since the setting will only take effect in `StreamExecutionEnvironment#execute`. 3. 'assignTimeStampAndWatermark' will change the timestamp of the record. IMO you may think each record contains a timestamp field, and the filed is set when ingesting, but 'assignTimeStampAndWatermark' will change the value of this field, so the following operators relying on the timestamp will see the updated value.
Best,Yun
------------------------------------------------------------------From:M Singh <ma...@yahoo.com>Send Time:2019 Jul. 9 (Tue.) 09:42To:User <us...@flink.apache.org>Subject:Apache Flink - Relation between stream time characteristic and timer triggers
Hi:
I have a few questions about the stream time characteristics:
1. If the time characteristic is set to TimeCharacteristic.EventTime, but the timers in a processor or trigger is set using registerProcessingTimeTimer (or vice versa), then will that timer fire ?
2. Once the time character is set on the stream environment, and changed later in the application, which one is applied, the first one or the last one ?
3. If the stream time characteristic is set to IngestionTime, then is there any adverse effect of assigning the timestamp using assignTimeStampAndWatermark to a stream later in the application ?
Thanks
Re: Apache Flink - Relation between stream time characteristic and
timer triggers
Posted by Fabian Hueske <fh...@gmail.com>.
Hi Mans,
IngestionTime is uses the same internal mechanisms as EventTime (record
timestamps and watermarks).
The difference is that instead of extracting a timestamp from the record
(using a custom timestamp extractor & wm assigner), Flink will assign
timestamps based on the machine clock of the machine that runs the source
task and will also automatically generate watermarks. If you ask for my
opinion, IngestionTime combines the disadvantages of ProcessingTime and
EventTime. You pay the latency / performance penalty of EventTime for the
non-determinism of ProcessingTime.
So, if you enable IngestionTime, you can use EventTime timers and
ProcessingTime timers.
Best, Fabian
Am Mi., 10. Juli 2019 um 09:37 Uhr schrieb M Singh <ma...@yahoo.com>:
> Thanks for your answer Xingcan.
>
> Just to clarify - if the characteristic is set to IngestionTime or
> ProcessingTime, the event time triggers will be ignored and not fire.
>
> Mans
>
> On Tuesday, July 9, 2019, 04:32:00 PM EDT, Xingcan Cui <xi...@gmail.com>
> wrote:
>
>
> Yes, Mans. You can use both processing-time and event-time timers if you
> set the time characteristic to event-time. They'll be triggered by their
> own time semantics, separately. (actually there’s no watermark for
> processing time)
>
> Cheers,
> Xingcan
>
> On Jul 9, 2019, at 11:40 AM, M Singh <ma...@yahoo.com> wrote:
>
> Thanks Yun for your answers.
>
> Does this mean that we can use processing and event timers (in processors
> or triggers) regardless of the time characteristic ? Also, is possible to
> use both together and will they both fire at the appropriate watermarks for
> processing and event times ?
>
> Mans
>
> On Tuesday, July 9, 2019, 12:18:30 AM EDT, Yun Gao <yu...@aliyun.com>
> wrote:
>
>
> Hi,
> For the three questions,
> 1. The processing time timer will be trigger. IMO you may think the
> processing time timer as in parallel with the event time timer. They are
> processed separately underlying. The processing time timer will be
> triggered according to the realistic time.
> 2. I'am not very clear on how to changed later in the application. Do
> you mean call `StreamExecutionEnvironment#setStreamTimeCharacteristics`
> multiple times ? If so, then the last call will take effect for all the
> operators before or after the last call, since the setting will only take
> effect in `StreamExecutionEnvironment#execute`.
> 3. 'assignTimeStampAndWatermark' will change the timestamp of the
> record. IMO you may think each record contains a timestamp field, and the
> filed is set when ingesting, but 'assignTimeStampAndWatermark' will change
> the value of this field, so the following operators relying on the
> timestamp will see the updated value.
>
> Best,
> Yun
>
>
>
> ------------------------------------------------------------------
> From:M Singh <ma...@yahoo.com>
> Send Time:2019 Jul. 9 (Tue.) 09:42
> To:User <us...@flink.apache.org>
> Subject:Apache Flink - Relation between stream time characteristic and
> timer triggers
>
> Hi:
>
> I have a few questions about the stream time characteristics:
>
> 1. If the time characteristic is set to TimeCharacteristic.EventTime, but
> the timers in a processor or trigger is set using
> registerProcessingTimeTimer (or vice versa), then will that timer fire ?
>
> 2. Once the time character is set on the stream environment, and changed
> later in the application, which one is applied, the first one or the last
> one ?
>
> 3. If the stream time characteristic is set to IngestionTime, then is
> there any adverse effect of assigning the timestamp using
> assignTimeStampAndWatermark to a stream later in the application ?
>
> Thanks
>
>
>
>
Re: Apache Flink - Relation between stream time characteristic and
timer triggers
Posted by M Singh <ma...@yahoo.com>.
Thanks for your answer Xingcan.
Just to clarify - if the characteristic is set to IngestionTime or ProcessingTime, the event time triggers will be ignored and not fire.
Mans
On Tuesday, July 9, 2019, 04:32:00 PM EDT, Xingcan Cui <xi...@gmail.com> wrote:
Yes, Mans. You can use both processing-time and event-time timers if you set the time characteristic to event-time. They'll be triggered by their own time semantics, separately. (actually there’s no watermark for processing time)
Cheers,Xingcan
On Jul 9, 2019, at 11:40 AM, M Singh <ma...@yahoo.com> wrote:
Thanks Yun for your answers.
Does this mean that we can use processing and event timers (in processors or triggers) regardless of the time characteristic ? Also, is possible to use both together and will they both fire at the appropriate watermarks for processing and event times ?
Mans
On Tuesday, July 9, 2019, 12:18:30 AM EDT, Yun Gao <yu...@aliyun.com> wrote:
Hi, For the three questions, 1. The processing time timer will be trigger. IMO you may think the processing time timer as in parallel with the event time timer. They are processed separately underlying. The processing time timer will be triggered according to the realistic time. 2. I'am not very clear on how to changed later in the application. Do you mean call `StreamExecutionEnvironment#setStreamTimeCharacteristics` multiple times ? If so, then the last call will take effect for all the operators before or after the last call, since the setting will only take effect in `StreamExecutionEnvironment#execute`. 3. 'assignTimeStampAndWatermark' will change the timestamp of the record. IMO you may think each record contains a timestamp field, and the filed is set when ingesting, but 'assignTimeStampAndWatermark' will change the value of this field, so the following operators relying on the timestamp will see the updated value.
Best,Yun
------------------------------------------------------------------From:M Singh <ma...@yahoo.com>Send Time:2019 Jul. 9 (Tue.) 09:42To:User <us...@flink.apache.org>Subject:Apache Flink - Relation between stream time characteristic and timer triggers
Hi:
I have a few questions about the stream time characteristics:
1. If the time characteristic is set to TimeCharacteristic.EventTime, but the timers in a processor or trigger is set using registerProcessingTimeTimer (or vice versa), then will that timer fire ?
2. Once the time character is set on the stream environment, and changed later in the application, which one is applied, the first one or the last one ?
3. If the stream time characteristic is set to IngestionTime, then is there any adverse effect of assigning the timestamp using assignTimeStampAndWatermark to a stream later in the application ?
Thanks
Re: Apache Flink - Relation between stream time characteristic and
timer triggers
Posted by Xingcan Cui <xi...@gmail.com>.
Yes, Mans. You can use both processing-time and event-time timers if you set the time characteristic to event-time. They'll be triggered by their own time semantics, separately. (actually there’s no watermark for processing time)
Cheers,
Xingcan
> On Jul 9, 2019, at 11:40 AM, M Singh <ma...@yahoo.com> wrote:
>
> Thanks Yun for your answers.
>
> Does this mean that we can use processing and event timers (in processors or triggers) regardless of the time characteristic ? Also, is possible to use both together and will they both fire at the appropriate watermarks for processing and event times ?
>
> Mans
>
> On Tuesday, July 9, 2019, 12:18:30 AM EDT, Yun Gao <yu...@aliyun.com> wrote:
>
>
> Hi,
> For the three questions,
> 1. The processing time timer will be trigger. IMO you may think the processing time timer as in parallel with the event time timer. They are processed separately underlying. The processing time timer will be triggered according to the realistic time.
> 2. I'am not very clear on how to changed later in the application. Do you mean call `StreamExecutionEnvironment#setStreamTimeCharacteristics` multiple times ? If so, then the last call will take effect for all the operators before or after the last call, since the setting will only take effect in `StreamExecutionEnvironment#execute`.
> 3. 'assignTimeStampAndWatermark' will change the timestamp of the record. IMO you may think each record contains a timestamp field, and the filed is set when ingesting, but 'assignTimeStampAndWatermark' will change the value of this field, so the following operators relying on the timestamp will see the updated value.
>
> Best,
> Yun
>
>
>
> ------------------------------------------------------------------
> From:M Singh <ma...@yahoo.com>
> Send Time:2019 Jul. 9 (Tue.) 09:42
> To:User <us...@flink.apache.org>
> Subject:Apache Flink - Relation between stream time characteristic and timer triggers
>
> Hi:
>
> I have a few questions about the stream time characteristics:
>
> 1. If the time characteristic is set to TimeCharacteristic.EventTime, but the timers in a processor or trigger is set using registerProcessingTimeTimer (or vice versa), then will that timer fire ?
>
> 2. Once the time character is set on the stream environment, and changed later in the application, which one is applied, the first one or the last one ?
>
> 3. If the stream time characteristic is set to IngestionTime, then is there any adverse effect of assigning the timestamp using assignTimeStampAndWatermark to a stream later in the application ?
>
> Thanks
>
Re: Apache Flink - Relation between stream time characteristic and
timer triggers
Posted by M Singh <ma...@yahoo.com>.
Thanks Yun for your answers.
Does this mean that we can use processing and event timers (in processors or triggers) regardless of the time characteristic ? Also, is possible to use both together and will they both fire at the appropriate watermarks for processing and event times ?
Mans
On Tuesday, July 9, 2019, 12:18:30 AM EDT, Yun Gao <yu...@aliyun.com> wrote:
Hi, For the three questions, 1. The processing time timer will be trigger. IMO you may think the processing time timer as in parallel with the event time timer. They are processed separately underlying. The processing time timer will be triggered according to the realistic time. 2. I'am not very clear on how to changed later in the application. Do you mean call `StreamExecutionEnvironment#setStreamTimeCharacteristics` multiple times ? If so, then the last call will take effect for all the operators before or after the last call, since the setting will only take effect in `StreamExecutionEnvironment#execute`. 3. 'assignTimeStampAndWatermark' will change the timestamp of the record. IMO you may think each record contains a timestamp field, and the filed is set when ingesting, but 'assignTimeStampAndWatermark' will change the value of this field, so the following operators relying on the timestamp will see the updated value.
Best,Yun
------------------------------------------------------------------From:M Singh <ma...@yahoo.com>Send Time:2019 Jul. 9 (Tue.) 09:42To:User <us...@flink.apache.org>Subject:Apache Flink - Relation between stream time characteristic and timer triggers
Hi:
I have a few questions about the stream time characteristics:
1. If the time characteristic is set to TimeCharacteristic.EventTime, but the timers in a processor or trigger is set using registerProcessingTimeTimer (or vice versa), then will that timer fire ?
2. Once the time character is set on the stream environment, and changed later in the application, which one is applied, the first one or the last one ?
3. If the stream time characteristic is set to IngestionTime, then is there any adverse effect of assigning the timestamp using assignTimeStampAndWatermark to a stream later in the application ?
Thanks
Re: Apache Flink - Relation between stream time characteristic and timer triggers
Posted by Yun Gao <yu...@aliyun.com>.
Hi,
For the three questions,
1. The processing time timer will be trigger. IMO you may think the processing time timer as in parallel with the event time timer. They are processed separately underlying. The processing time timer will be triggered according to the realistic time.
2. I'am not very clear on how to changed later in the application. Do you mean call `StreamExecutionEnvironment#setStreamTimeCharacteristics` multiple times ? If so, then the last call will take effect for all the operators before or after the last call, since the setting will only take effect in `StreamExecutionEnvironment#execute`.
3. 'assignTimeStampAndWatermark' will change the timestamp of the record. IMO you may think each record contains a timestamp field, and the filed is set when ingesting, but 'assignTimeStampAndWatermark' will change the value of this field, so the following operators relying on the timestamp will see the updated value.
Best,
Yun
------------------------------------------------------------------
From:M Singh <ma...@yahoo.com>
Send Time:2019 Jul. 9 (Tue.) 09:42
To:User <us...@flink.apache.org>
Subject:Apache Flink - Relation between stream time characteristic and timer triggers
Hi:
I have a few questions about the stream time characteristics:
1. If the time characteristic is set to TimeCharacteristic.EventTime, but the timers in a processor or trigger is set using registerProcessingTimeTimer (or vice versa), then will that timer fire ?
2. Once the time character is set on the stream environment, and changed later in the application, which one is applied, the first one or the last one ?
3. If the stream time characteristic is set to IngestionTime, then is there any adverse effect of assigning the timestamp using assignTimeStampAndWatermark to a stream later in the application ?
Thanks