Sampling rate higher than 1Khz

[Nicholas Walton <nw...@me.com>]

Flinks watermarks are in milliseconds. I have time sampled off a sensor at a rate exceeding 1Khz or 1 per millisecond. Is there a way to handle timestamp granularity below milliseconds, or will I have to generate timestamp for the millisecond value preceding that associated with the sensor reading, which IUC will introduce latency into the processing pipeline. 

TIA

Re: Sampling rate higher than 1Khz

[Piotr Nowojski <pi...@da-platform.com>]

Hi,

Maybe stupid idea, but does anything prevents a user from pretending that watermarks/event times are in different unit, for example microseconds? Of course assuming using row/event time and not using processing time for anything?

Piotrek 

> On 28 Jan 2019, at 14:58, Tzu-Li (Gordon) Tai <tz...@apache.org> wrote:
> 
> Hi!
> 
> Yes, Flink's watermark timestamps are in milliseconds, which means that time-based operators such as time window operators will be fired at a per-millisecond granularity.
> Whether or not this introduces "latency" in the pipeline depends on the granularity of your time window operations; if you need to have window durations shorter than 1 millisecond, then yes, having only millisecond watermarks will introduce latency.
> Currently in Flink, time-based operations such as windows / registering timers are all done at millisecond accuracy.
> 
> Cheers,
> Gordon
> 
> On Mon, Jan 28, 2019 at 7:55 PM Nicholas Walton <nwalton@me.com <ma...@me.com>> wrote:
> Flinks watermarks are in milliseconds. I have time sampled off a sensor at a rate exceeding 1Khz or 1 per millisecond. Is there a way to handle timestamp granularity below milliseconds, or will I have to generate timestamp for the millisecond value preceding that associated with the sensor reading, which IUC will introduce latency into the processing pipeline. 
> 
> TIA

Re: Sampling rate higher than 1Khz

["Tzu-Li (Gordon) Tai" <tz...@apache.org>]

Hi!

Yes, Flink's watermark timestamps are in milliseconds, which means that
time-based operators such as time window operators will be fired at a
per-millisecond granularity.
Whether or not this introduces "latency" in the pipeline depends on the
granularity of your time window operations; if you need to have window
durations shorter than 1 millisecond, then yes, having only millisecond
watermarks will introduce latency.
Currently in Flink, time-based operations such as windows / registering
timers are all done at millisecond accuracy.

Cheers,
Gordon

On Mon, Jan 28, 2019 at 7:55 PM Nicholas Walton <nw...@me.com> wrote:

> Flinks watermarks are in milliseconds. I have time sampled off a sensor at
> a rate exceeding 1Khz or 1 per millisecond. Is there a way to handle
> timestamp granularity below milliseconds, or will I have to generate
> timestamp for the millisecond value preceding that associated with the
> sensor reading, which IUC will introduce latency into the processing
> pipeline.
>
> TIA