You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by liuxiangcao <xi...@gmail.com> on 2022/03/30 21:38:43 UTC
how to achieve sideOutputLateData() in FlinkSQL?
Hi Flink community,
In Flink DataStream Java API, user can get get data that was discarded as
late using WindowedStream.sideOutputLateData(OutputTag) (see [1]). I'm
wondering what is the best way for user to achieve this in Flink SQL?
For background, we are providing pure sql deployment to our internal users
which means user won't be using Flink Table API directly. They will write
Flink SQL script, with Java only used for UDF.
Would appreciate if any one here can share your insights or experiences.
Thanks!
[1]
https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/operators/windows/#getting-late-data-as-a-side-output
Re: how to achieve sideOutputLateData() in FlinkSQL?
Posted by liuxiangcao <xi...@gmail.com>.
Thanks a lot for the help! Yu'an and Martijn.
To share and confirm my understanding, the recipe using CURRENT_WATERMARK
to get late data will return all data arriving later than the defined
bounded out-of-orderness, without consideration of window closing time.
In comparison, WindowedStream.sideOutputLateData(OutputTag) will only
return data that arrives after its window is already closed.
On Thu, Mar 31, 2022 at 2:26 AM Martijn Visser <ma...@apache.org>
wrote:
> Hi,
>
> The only thing you currently can do is filter out late data using the
> CURRENT_WATERMARK function since Flink 1.14. There's a SQL Cookbook recipe
> on this function which can be found at
> https://github.com/ververica/flink-sql-cookbook/blob/main/other-builtin-functions/03_current_watermark/03_current_watermark.md
>
> Best regards,
>
> Martijn Visser
> https://twitter.com/MartijnVisser82
> https://github.com/MartijnVisser
>
>
> On Thu, 31 Mar 2022 at 11:00, yu'an huang <h....@gmail.com> wrote:
>
>> Hi, in my understanding, Flink only support to get late data by side
>> output in data stream api currently. For Table API/SQL, unfortunately, late
>> events will always be dropped.
>>
>> You can see this link as reference:
>> https://stackoverflow.com/questions/60218235/using-event-time-with-lateness-in-flink-sql-windows
>>
>>
>>
>> > On 31 Mar 2022, at 5:38 AM, liuxiangcao <xi...@gmail.com>
>> wrote:
>> >
>> > Hi Flink community,
>> >
>> > In Flink DataStream Java API, user can get get data that was discarded
>> as late using WindowedStream.sideOutputLateData(OutputTag) (see [1]). I'm
>> wondering what is the best way for user to achieve this in Flink SQL?
>> >
>> > For background, we are providing pure sql deployment to our internal
>> users which means user won't be using Flink Table API directly. They will
>> write Flink SQL script, with Java only used for UDF.
>> >
>> > Would appreciate if any one here can share your insights or
>> experiences. Thanks!
>> >
>> > [1]
>> https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/operators/windows/#getting-late-data-as-a-side-output
>> >
>> >
>> >
>>
>>
--
Best Wishes & Regards
Xiangcao Liu
Re: how to achieve sideOutputLateData() in FlinkSQL?
Posted by Martijn Visser <ma...@apache.org>.
Hi,
The only thing you currently can do is filter out late data using the
CURRENT_WATERMARK function since Flink 1.14. There's a SQL Cookbook recipe
on this function which can be found at
https://github.com/ververica/flink-sql-cookbook/blob/main/other-builtin-functions/03_current_watermark/03_current_watermark.md
Best regards,
Martijn Visser
https://twitter.com/MartijnVisser82
https://github.com/MartijnVisser
On Thu, 31 Mar 2022 at 11:00, yu'an huang <h....@gmail.com> wrote:
> Hi, in my understanding, Flink only support to get late data by side
> output in data stream api currently. For Table API/SQL, unfortunately, late
> events will always be dropped.
>
> You can see this link as reference:
> https://stackoverflow.com/questions/60218235/using-event-time-with-lateness-in-flink-sql-windows
>
>
>
> > On 31 Mar 2022, at 5:38 AM, liuxiangcao <xi...@gmail.com> wrote:
> >
> > Hi Flink community,
> >
> > In Flink DataStream Java API, user can get get data that was discarded
> as late using WindowedStream.sideOutputLateData(OutputTag) (see [1]). I'm
> wondering what is the best way for user to achieve this in Flink SQL?
> >
> > For background, we are providing pure sql deployment to our internal
> users which means user won't be using Flink Table API directly. They will
> write Flink SQL script, with Java only used for UDF.
> >
> > Would appreciate if any one here can share your insights or experiences.
> Thanks!
> >
> > [1]
> https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/operators/windows/#getting-late-data-as-a-side-output
> >
> >
> >
>
>
Re: how to achieve sideOutputLateData() in FlinkSQL?
Posted by yu'an huang <h....@gmail.com>.
Hi, in my understanding, Flink only support to get late data by side output in data stream api currently. For Table API/SQL, unfortunately, late events will always be dropped.
You can see this link as reference: https://stackoverflow.com/questions/60218235/using-event-time-with-lateness-in-flink-sql-windows
> On 31 Mar 2022, at 5:38 AM, liuxiangcao <xi...@gmail.com> wrote:
>
> Hi Flink community,
>
> In Flink DataStream Java API, user can get get data that was discarded as late using WindowedStream.sideOutputLateData(OutputTag) (see [1]). I'm wondering what is the best way for user to achieve this in Flink SQL?
>
> For background, we are providing pure sql deployment to our internal users which means user won't be using Flink Table API directly. They will write Flink SQL script, with Java only used for UDF.
>
> Would appreciate if any one here can share your insights or experiences. Thanks!
>
> [1] https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/operators/windows/#getting-late-data-as-a-side-output
>
>
>