You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@flink.apache.org by Maciej Bryński <ma...@brynski.pl> on 2021/08/10 13:21:10 UTC

Allowed lateness in Flink SQL

Hi Guys,
I was checking if anything changed recently with allowed lateness
support in Flink SQL and I found this PR:
https://github.com/apache/flink/pull/16022

Is there any documentation for table.exec.emit.allow-lateness ?
Is this option working only in window agregation?

Regards,
-- 
Maciek Bryński

Re: Allowed lateness in Flink SQL

Posted by JING ZHANG <be...@gmail.com>.

Hi Maciej,
The pr is related to FLINK-21301
<https://issues.apache.org/jira/browse/FLINK-21301> [1].

Sets the time by which elements are allowed to be late. Elements that
arrive behind the watermark by more than the specified time " +
    "will be dropped. " +
    "Note: use the value if it is set, else use
'minIdleStateRetentionTime' in table config." +
    "< 0 is illegal configuration. " +
    "0 means disable allow lateness. " +
    "> 0 means allow-lateness.

First of all, the pr is merged to 1.14 which is not released yet.
Secondly, `table.exec.emit.allow-lateness` is experimental
<https://github.com/experimental> configuration which only works for Group
window aggregation [2], it would not effect the Window TVF aggregation.
Thirdly, `table.exec.emit.allow-lateness` configuration is back forward
compatible, it specifies allow-lateness for Group window aggregation and
state ttl can still take effect when the new config is not set `
table.exec.emit.allow-lateness`

[1] https://issues.apache.org/jira/browse/FLINK-21301
[2]
https://ci.apache.org/projects/flink/flink-docs-master/docs/dev/table/sql/queries/window-agg/#group-window-aggregation

Maciej Bryński <ma...@brynski.pl> 于2021年8月10日周二 下午9:21写道：

> Hi Guys,
> I was checking if anything changed recently with allowed lateness
> support in Flink SQL and I found this PR:
> https://github.com/apache/flink/pull/16022
>
> Is there any documentation for table.exec.emit.allow-lateness ?
> Is this option working only in window agregation?
>
> Regards,
> --
> Maciek Bryński
>

Re: Allowed lateness in Flink SQL

Posted by Maciej Bryński <ma...@brynski.pl>.

Thanks Ingo.
I will look into CURRENT_WATERMARK.

About allow-lateness. I can imagine the following situation.
Let's have a left interval join between two streams.
I want to wait 15 minutes for events from the right stream to arrive
(this can be done by setting a watermark on the right stream).
Also I want to update the join result if events from the right stream
are more than 15 minutes late.  (this can be possibly handled by
allow-lateness).

Regards,
Maciek



wt., 10 sie 2021 o 15:38 Ingo Bürk <in...@ververica.com> napisał(a):
>
> Hi Maciej,
>
> there is no documentation for it (besides in the code itself) because it's an experimental flag. What would you expect allow-lateness to do outside the context of a window? Maybe you'd also be interested in CURRENT_WATERMARK()[1][2] which will be released with 1.14 and allows some level of late data handling.
>
> [1] https://issues.apache.org/jira/browse/FLINK-22737
> [2] https://ci.apache.org/projects/flink/flink-docs-master/docs/dev/table/functions/systemfunctions/#temporal-functions
>
>
> Best
> Ingo
>
> On Tue, Aug 10, 2021 at 3:21 PM Maciej Bryński <ma...@brynski.pl> wrote:
>>
>> Hi Guys,
>> I was checking if anything changed recently with allowed lateness
>> support in Flink SQL and I found this PR:
>> https://github.com/apache/flink/pull/16022
>>
>> Is there any documentation for table.exec.emit.allow-lateness ?
>> Is this option working only in window agregation?
>>
>> Regards,
>> --
>> Maciek Bryński



-- 
Maciek Bryński

Re: Allowed lateness in Flink SQL

Posted by Ingo Bürk <in...@ververica.com>.

Hi Maciej,

there is no documentation for it (besides in the code itself) because it's
an experimental flag. What would you expect allow-lateness to do outside
the context of a window? Maybe you'd also be interested in
CURRENT_WATERMARK()[1][2] which will be released with 1.14 and allows some
level of late data handling.

[1] https://issues.apache.org/jira/browse/FLINK-22737
[2]
https://ci.apache.org/projects/flink/flink-docs-master/docs/dev/table/functions/systemfunctions/#temporal-functions

Best
Ingo

On Tue, Aug 10, 2021 at 3:21 PM Maciej Bryński <ma...@brynski.pl> wrote:

> Hi Guys,
> I was checking if anything changed recently with allowed lateness
> support in Flink SQL and I found this PR:
> https://github.com/apache/flink/pull/16022
>
> Is there any documentation for table.exec.emit.allow-lateness ?
> Is this option working only in window agregation?
>
> Regards,
> --
> Maciek Bryński
>