Flink stream data ordering/sequence

[Sane Lee <le...@gmail.com>]

Hi all,

Do flink have mechanism for dealing with ordered streams?
As far as I know , flink attaches timestamps to data items once they are
processed. What about the timestamps are already in data items?
For example, if some data item is missing from particular window (according
to its ordering/sequence number), how does flink react to it?
Waits or generates missing data according to in-hand data or just ignores
 it?

-best
Sana

Re: Flink stream data ordering/sequence

[Fabian Hueske <fh...@gmail.com>]

Hi Sana,

The feature you are looking for is called event time processing in Flink.
These blog posts should help you to become familiar with the concepts:

1) Event-Time concepts:
http://data-artisans.com/how-apache-flink-enables-new-streaming-applications-part-1/
2) Windows in Flink:
http://flink.apache.org/news/2015/12/04/Introducing-windows.html
3) Event-Time example use-case:
https://www.elastic.co/blog/building-real-time-dashboard-applications-with-apache-flink-elasticsearch-and-kibana
4) Code for example: https://github.com/dataArtisans/flink-streaming-demo

You can implement several strategies for dealing with late-arriving events.

Best, Fabian

2016-01-29 10:58 GMT+01:00 Sane Lee <le...@gmail.com>:

> Hi all,
>
> Do flink have mechanism for dealing with ordered streams?
> As far as I know , flink attaches timestamps to data items once they are
> processed. What about the timestamps are already in data items?
> For example, if some data item is missing from particular window
> (according to its ordering/sequence number), how does flink react to it?
> Waits or generates missing data according to in-hand data or just ignores
>  it?
>
> -best
> Sana
>