How to generate a sequential watermark which increases by one unit each time

[Averell <lv...@gmail.com>]

Hi everyone,

I have a stream of files, each file has multiple records. A record has one
Long field named `ts`, which holds the file creation time (so it increases
over time, and one same value for all records in each file). However, I need
to have a watermark that increases by exactly one unit for each file.

I thought of extending the AssignerWithPeriodicWatermarks interface with a
member variable holding that sequence value. However, it seems to me that it
is not possible to persist that value during checkpoints.

Are there any options for me?

Thanks and best regards,
Averell



--
Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Re: How to generate a sequential watermark which increases by one unit each time

[Paul Lam <pa...@gmail.com>]

Hi Averell,

IMHO, a simple approach would be adding a rich map that holds the sequence value (backed by states) and attach it to the records before the assigner operator.

Best,
Paul Lam

> 在 2019年5月21日，20:37，Averell <lv...@gmail.com> 写道：
> 
> Hi everyone,
> 
> I have a stream of files, each file has multiple records. A record has one
> Long field named `ts`, which holds the file creation time (so it increases
> over time, and one same value for all records in each file). However, I need
> to have a watermark that increases by exactly one unit for each file.
> 
> I thought of extending the AssignerWithPeriodicWatermarks interface with a
> member variable holding that sequence value. However, it seems to me that it
> is not possible to persist that value during checkpoints.
> 
> Are there any options for me?
> 
> Thanks and best regards,
> Averell
> 
> 
> 
> --
> Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/