Custom Watermarks with Flink

[Wyatt Frelot <wy...@gmail.com>]

I originally posted to Stack Overflow because I was trying to figure out he
to do this in Flink.

Wondering how to implement something of the sort:
(1) *write up:  *
https://drive.google.com/file/d/0Bw69DO1tid2_SzVVendtUV9WMVdIUXptQ1hHSl9KNjAyMTBn/view?usp=drivesdk
(2) *original post: *
https://stackoverflow.com/questions/50949083/custom-watermarks-with-apache-flink
-- 
Wyatt J. Frelot

Re: Custom Watermarks with Flink

[Fabian Hueske <fh...@gmail.com>]

Hi,

I would not encode this information in watermarks. Watermarks are rather an
internal mechanism to reason about event-time.
Flink also generates watermarks internally. This makes the behavior less
predictive.

You could either inject special meta data records (which Flink handles just
like other annotate each record with the meta data.
Flink 1.5.0 introduced Broadcast State [1] which can be used to apply rules
to a full stream, e.g., to set meta data.

Best,
Fabian

[1]
https://ci.apache.org/projects/flink/flink-docs-release-1.5/dev/stream/state/broadcast_state.html

2018-06-23 23:45 GMT+02:00 Wyatt Frelot <wy...@gmail.com>:

> I originally posted to Stack Overflow because I was trying to figure out
> he to do this in Flink.
>
> Wondering how to implement something of the sort:
> (1) *write up:  *https://drive.google.com/file/d/0Bw69DO1tid2_
> SzVVendtUV9WMVdIUXptQ1hHSl9KNjAyMTBn/view?usp=drivesdk
> (2) *original post: *
> https://stackoverflow.com/questions/50949083/custom-
> watermarks-with-apache-flink
> --
> Wyatt J. Frelot
>