You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Krzysztof Chmielewski <kr...@gmail.com> on 2021/12/14 21:46:59 UTC
UDF and Broadcast State Pattern
Hi,
Is there a way to build an UDF [1] for FLink SQL that can be used with
Broadcast State Pattern [2]?
I have a use case, where I would like to be able to use broadcast control
stream to change logic in UDF.
Regards,
Krzysztof Chmielewski
[1]
https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/functions/udfs/#user-defined-functions
[2]
https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/fault-tolerance/broadcast_state/#the-broadcast-state-pattern
Re: UDF and Broadcast State Pattern
Posted by Krzysztof Chmielewski <kr...@gmail.com>.
Thank you very much for the clarification Seth.
Best Regards
Krzysztof Chmielewski
śr., 15 gru 2021, 16:12 użytkownik Seth Wiesman <sj...@gmail.com>
napisał:
> Hi Krzysztof,
>
> There is a difference in semantics here between yourself and Caizhi. SQL
> UDFs can be used statefully - see AggregateFunction and
> TableAggregateFunction for examples. You even have access to ListView and
> MapView which are backed by ListState and MapState accordingly. These
> functions contain aggregates which do participate in checkpointing and are
> strongly consistent.
>
> What is not supported are more low level process function type operations
> (custom state registration, user access to timers, broadcast state as
> you've discovered). There have been some discussions about how to add this
> sort of functionality in a SQL compliant manner but nothing concrete.
>
> In the meantime, Flink SQL has strong interop with the DataStream API. You
> can always transform a Table into a DataStream, do some low level
> processing, and then transform it back into a table to run further SQL.
>
> Seth
>
> On Wed, Dec 15, 2021 at 3:52 AM Krzysztof Chmielewski <
> krzysiek.chmielewski@gmail.com> wrote:
>
>> Thank you,
>> yes I was thinking about simply running my own thread in UDF and consume
>> some queue something like that.
>> Having some background with DataStreamAPI i was hoping that I can reuse
>> same mechanisms (like Broadcast State Pattern or CoProcessFunction) in
>> Flink SQL.
>> However it seems there is a quite noticeable gap between what you can do
>> with SQL and what you can control comparing to DataStreamAPI.
>>
>> Regarding UDF being stateless. I assume you mean that UDF does not
>> participate in checkpoint mechanism and I cannot initialize a Flink state
>> in UDF right?
>> I'm wondering why it is not possible. Seems like an "obligatory" feature
>> for Statefull stream processing platform that supports SQL. In a time when
>> there is a huge interest with Flink SQL and everyone is talking about it,
>> thing like state support is not available in SQL UDF is suprissing.
>>
>> Are there any plans, maybe FLIP to change it?
>>
>> Regards,
>> Krzysztof Chmielewski
>>
>> śr., 15 gru 2021 o 02:36 Caizhi Weng <ts...@gmail.com> napisał(a):
>>
>>> Hi!
>>>
>>> Currently you can't use broadcast state in Flink SQL UDF because UDFs
>>> are all stateless.
>>>
>>> However you mentioned your use case that you want to control the logic
>>> in UDF with some information. If that is the case, you can just run a
>>> thread in your UDF to read that information and change the behavior of the
>>> eval method accordingly.
>>>
>>> Krzysztof Chmielewski <kr...@gmail.com> 于2021年12月15日周三
>>> 05:47写道：
>>>
>>>> Hi,
>>>> Is there a way to build an UDF [1] for FLink SQL that can be used with
>>>> Broadcast State Pattern [2]?
>>>>
>>>> I have a use case, where I would like to be able to use broadcast
>>>> control stream to change logic in UDF.
>>>>
>>>> Regards,
>>>> Krzysztof Chmielewski
>>>>
>>>> [1]
>>>> https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/functions/udfs/#user-defined-functions
>>>> [2]
>>>> https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/fault-tolerance/broadcast_state/#the-broadcast-state-pattern
>>>>
>>>
Re: UDF and Broadcast State Pattern
Posted by Seth Wiesman <sj...@gmail.com>.
Hi Krzysztof,
There is a difference in semantics here between yourself and Caizhi. SQL
UDFs can be used statefully - see AggregateFunction and
TableAggregateFunction for examples. You even have access to ListView and
MapView which are backed by ListState and MapState accordingly. These
functions contain aggregates which do participate in checkpointing and are
strongly consistent.
What is not supported are more low level process function type operations
(custom state registration, user access to timers, broadcast state as
you've discovered). There have been some discussions about how to add this
sort of functionality in a SQL compliant manner but nothing concrete.
In the meantime, Flink SQL has strong interop with the DataStream API. You
can always transform a Table into a DataStream, do some low level
processing, and then transform it back into a table to run further SQL.
Seth
On Wed, Dec 15, 2021 at 3:52 AM Krzysztof Chmielewski <
krzysiek.chmielewski@gmail.com> wrote:
> Thank you,
> yes I was thinking about simply running my own thread in UDF and consume
> some queue something like that.
> Having some background with DataStreamAPI i was hoping that I can reuse
> same mechanisms (like Broadcast State Pattern or CoProcessFunction) in
> Flink SQL.
> However it seems there is a quite noticeable gap between what you can do
> with SQL and what you can control comparing to DataStreamAPI.
>
> Regarding UDF being stateless. I assume you mean that UDF does not
> participate in checkpoint mechanism and I cannot initialize a Flink state
> in UDF right?
> I'm wondering why it is not possible. Seems like an "obligatory" feature
> for Statefull stream processing platform that supports SQL. In a time when
> there is a huge interest with Flink SQL and everyone is talking about it,
> thing like state support is not available in SQL UDF is suprissing.
>
> Are there any plans, maybe FLIP to change it?
>
> Regards,
> Krzysztof Chmielewski
>
> śr., 15 gru 2021 o 02:36 Caizhi Weng <ts...@gmail.com> napisał(a):
>
>> Hi!
>>
>> Currently you can't use broadcast state in Flink SQL UDF because UDFs are
>> all stateless.
>>
>> However you mentioned your use case that you want to control the logic in
>> UDF with some information. If that is the case, you can just run a thread
>> in your UDF to read that information and change the behavior of the eval
>> method accordingly.
>>
>> Krzysztof Chmielewski <kr...@gmail.com> 于2021年12月15日周三
>> 05:47写道：
>>
>>> Hi,
>>> Is there a way to build an UDF [1] for FLink SQL that can be used with
>>> Broadcast State Pattern [2]?
>>>
>>> I have a use case, where I would like to be able to use broadcast
>>> control stream to change logic in UDF.
>>>
>>> Regards,
>>> Krzysztof Chmielewski
>>>
>>> [1]
>>> https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/functions/udfs/#user-defined-functions
>>> [2]
>>> https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/fault-tolerance/broadcast_state/#the-broadcast-state-pattern
>>>
>>
Re: UDF and Broadcast State Pattern
Posted by Krzysztof Chmielewski <kr...@gmail.com>.
Thank you,
yes I was thinking about simply running my own thread in UDF and consume
some queue something like that.
Having some background with DataStreamAPI i was hoping that I can reuse
same mechanisms (like Broadcast State Pattern or CoProcessFunction) in
Flink SQL.
However it seems there is a quite noticeable gap between what you can do
with SQL and what you can control comparing to DataStreamAPI.
Regarding UDF being stateless. I assume you mean that UDF does not
participate in checkpoint mechanism and I cannot initialize a Flink state
in UDF right?
I'm wondering why it is not possible. Seems like an "obligatory" feature
for Statefull stream processing platform that supports SQL. In a time when
there is a huge interest with Flink SQL and everyone is talking about it,
thing like state support is not available in SQL UDF is suprissing.
Are there any plans, maybe FLIP to change it?
Regards,
Krzysztof Chmielewski
śr., 15 gru 2021 o 02:36 Caizhi Weng <ts...@gmail.com> napisał(a):
> Hi!
>
> Currently you can't use broadcast state in Flink SQL UDF because UDFs are
> all stateless.
>
> However you mentioned your use case that you want to control the logic in
> UDF with some information. If that is the case, you can just run a thread
> in your UDF to read that information and change the behavior of the eval
> method accordingly.
>
> Krzysztof Chmielewski <kr...@gmail.com> 于2021年12月15日周三
> 05:47写道：
>
>> Hi,
>> Is there a way to build an UDF [1] for FLink SQL that can be used with
>> Broadcast State Pattern [2]?
>>
>> I have a use case, where I would like to be able to use broadcast control
>> stream to change logic in UDF.
>>
>> Regards,
>> Krzysztof Chmielewski
>>
>> [1]
>> https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/functions/udfs/#user-defined-functions
>> [2]
>> https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/fault-tolerance/broadcast_state/#the-broadcast-state-pattern
>>
>
Re: UDF and Broadcast State Pattern
Posted by Caizhi Weng <ts...@gmail.com>.
Hi!
Currently you can't use broadcast state in Flink SQL UDF because UDFs are
all stateless.
However you mentioned your use case that you want to control the logic in
UDF with some information. If that is the case, you can just run a thread
in your UDF to read that information and change the behavior of the eval
method accordingly.
Krzysztof Chmielewski <kr...@gmail.com> 于2021年12月15日周三
05:47写道：
> Hi,
> Is there a way to build an UDF [1] for FLink SQL that can be used with
> Broadcast State Pattern [2]?
>
> I have a use case, where I would like to be able to use broadcast control
> stream to change logic in UDF.
>
> Regards,
> Krzysztof Chmielewski
>
> [1]
> https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/functions/udfs/#user-defined-functions
> [2]
> https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/fault-tolerance/broadcast_state/#the-broadcast-state-pattern
>