Barriers in Flink SQL

[Ralph Matthias Debusmann <ma...@gmail.com>]

Hi,

one question - it seems that "barriers" are perfectly supported by Flink,
but not yet supported in Flink SQL.

When I e.g. do a UNION of two views derived from one source table fed by
Kafka, I get thousands of intermediate results which are incorrect (the
example I am using is this one:
https://www.scattered-thoughts.net/writing/internal-consistency-in-streaming-systems/),
and I can only be sure to get the correct result if I stop producing new
messages into the source table.

1) Is my understanding correct that barriers are not implemented for Flink
SQL?
2) Why is it not implemented/is this on the roadmap?

Best, Ralph

Re: Barriers in Flink SQL

[Giannis Polyzos <ip...@gmail.com>]

Hi Ralph,
can you explain a bit more? When you say "barriers" you should be referring
to the checkpoints, but from your description seems more like watermarks.
What functionality is supported in Flink and not Flink SQL? In terms of
watermarks, there were a few shortcomings between the two APIs which are
addressed in the upcoming 1.18 release
https://cwiki.apache.org/confluence/display/FLINK/FLIP-296%3A+Extend+watermark-related+features+for+SQL
What operations are you running and you see incorrect results?

Best

On Wed, Oct 25, 2023 at 9:51 PM Ralph Matthias Debusmann <
matthias.debusmann@gmail.com> wrote:

> Hi,
>
> one question - it seems that "barriers" are perfectly supported by Flink,
> but not yet supported in Flink SQL.
>
> When I e.g. do a UNION of two views derived from one source table fed by
> Kafka, I get thousands of intermediate results which are incorrect (the
> example I am using is this one:
> https://www.scattered-thoughts.net/writing/internal-consistency-in-streaming-systems/),
> and I can only be sure to get the correct result if I stop producing new
> messages into the source table.
>
> 1) Is my understanding correct that barriers are not implemented for Flink
> SQL?
> 2) Why is it not implemented/is this on the roadmap?
>
> Best, Ralph
>
>