StreamSQL procTime granularity

[Stefano Bortoli <st...@huawei.com>]

Hi guys,

Based on the discussion about time management in https://github.com/apache/flink/pull/3641 , does it make sense to use nanoTime for procTime semantic aggregation processing? This way we will not have the possibility of events falling in the same "millisecond" processing bucket and keep a consistent aggregation sorting (also in the state). I understand that event-time cannot be managed in nanosecond as java does not give wall-clock time in nanoseconds, but for the procTime within the JVM we should be safe.

Please let me know what you think.

Best,
Stefano

RE: StreamSQL procTime granularity

[Stefano Bortoli <st...@huawei.com>]

The main reason would be to avoid issues related to co-occurrence in processing time. However, your last comment shows that we can neglect the issue for the moment. More work is not an issue per se. But more work for nothing is another story. ;-) In fact, as you underlined, the procTime semantics kinds of makes the time-related sorting relative to the implementation choices. Fair enough.

Thanks for clarifiying.

Best,
Stefano

-----Original Message-----
From: Fabian Hueske [mailto:fhueske@gmail.com] 
Sent: Thursday, April 06, 2017 1:17 PM
To: dev@flink.apache.org
Subject: Re: StreamSQL procTime granularity

Hi Stefano,

thanks for starting this discussion. I think treating processing time with a nanosecond resolution might be difficult for a few reasons:
- We would need to treat it in all operators as nanotime (joins, etc.)
- Flink does not provide tooling for nanotime resolution. The internal timer service is on milliseconds.
- We could not use the testing facilities that the DataStream API offers because they work on milliseconds.

I'm not sure what the benefits of switching to nanoseconds would be.
Sure, higher resolution sounds good, but on the other hand processing time is somewhat arbitrary anyway.

Best, Fabian

2017-04-04 10:23 GMT+02:00 Stefano Bortoli <st...@huawei.com>:

> Hi guys,
>
> Based on the discussion about time management in
> https://github.com/apache/flink/pull/3641 , does it make sense to use 
> nanoTime for procTime semantic aggregation processing? This way we 
> will not have the possibility of events falling in the same 
> "millisecond" processing bucket and keep a consistent aggregation 
> sorting (also in the state). I understand that event-time cannot be 
> managed in nanosecond as java does not give wall-clock time in 
> nanoseconds, but for the procTime within the JVM we should be safe.
>
> Please let me know what you think.
>
> Best,
> Stefano
>
>

Re: StreamSQL procTime granularity

[Fabian Hueske <fh...@gmail.com>]

Hi Stefano,

thanks for starting this discussion. I think treating processing time with
a nanosecond resolution might be difficult for a few reasons:
- We would need to treat it in all operators as nanotime (joins, etc.)
- Flink does not provide tooling for nanotime resolution. The internal
timer service is on milliseconds.
- We could not use the testing facilities that the DataStream API offers
because they work on milliseconds.

I'm not sure what the benefits of switching to nanoseconds would be.
Sure, higher resolution sounds good, but on the other hand processing time
is somewhat arbitrary anyway.

Best, Fabian

2017-04-04 10:23 GMT+02:00 Stefano Bortoli <st...@huawei.com>:

> Hi guys,
>
> Based on the discussion about time management in
> https://github.com/apache/flink/pull/3641 , does it make sense to use
> nanoTime for procTime semantic aggregation processing? This way we will not
> have the possibility of events falling in the same "millisecond" processing
> bucket and keep a consistent aggregation sorting (also in the state). I
> understand that event-time cannot be managed in nanosecond as java does not
> give wall-clock time in nanoseconds, but for the procTime within the JVM we
> should be safe.
>
> Please let me know what you think.
>
> Best,
> Stefano
>
>