Apache Beam SQL and UDF

[Talat Uyarer <tu...@paloaltonetworks.com>]

Hi,

We plan to use UDF on our sql. We want to achieve some kind of
filtering based on internal states. We want to update that internal state
with a separate async thread in UDF. Before implementing that thing I want
to get your options. Is there any limitation for UDF to have multi-thread
implementation ?  Our UDF is a scalar function. It will get 1 or 2 input
and return boolean.

I will appreciate your comments in advance.

Thanks

Re: Apache Beam SQL and UDF

[Talat Uyarer <tu...@paloaltonetworks.com>]

Thanks Rui to remind me lifecycle of UDF. LOoks liek there is no any
lifecycle. I checked the code looks like we create UDF's instance for each
message:

org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.runtime.SqlFunctions.isTrue(new
> com.paloaltonetworks.cortex.streamcompute.functions.MyUDFFunction().apply(current.getString(2))


Do you think we should put the UDF instance in the setup function and call
in processlement ? Do you see anything besides this ? Or How can i achieve
my goal in a different way ? I thought I could use the Table Provider
approach. But it does not update data with any mechanisim.

Thanks


On Wed, Feb 10, 2021 at 12:41 PM Talat Uyarer <tu...@paloaltonetworks.com>
wrote:

> Does beam create UDF function for every bundle or in setup of pipeline ?
>
> I will keep internal state in memory. The Async thread will update that in
> memory state based on an interval such as every hour etc. If beam keeps UDF
> instance more than one bundle it is ok for me.
>
>
> On Wed, Feb 10, 2021, 12:37 PM Rui Wang <ru...@google.com> wrote:
>
>> The problem that I can think of is maybe before the async call is
>> completed, the UDF life cycle has reached to the end.
>>
>>
>> -Rui
>>
>> On Wed, Feb 10, 2021 at 12:34 PM Talat Uyarer <
>> tuyarer@paloaltonetworks.com> wrote:
>>
>>> Hi,
>>>
>>> We plan to use UDF on our sql. We want to achieve some kind of
>>> filtering based on internal states. We want to update that internal state
>>> with a separate async thread in UDF. Before implementing that thing I want
>>> to get your options. Is there any limitation for UDF to have multi-thread
>>> implementation ?  Our UDF is a scalar function. It will get 1 or 2 input
>>> and return boolean.
>>>
>>> I will appreciate your comments in advance.
>>>
>>> Thanks
>>>
>>

Re: Apache Beam SQL and UDF

[Talat Uyarer <tu...@paloaltonetworks.com>]

Does beam create UDF function for every bundle or in setup of pipeline ?

I will keep internal state in memory. The Async thread will update that in
memory state based on an interval such as every hour etc. If beam keeps UDF
instance more than one bundle it is ok for me.


On Wed, Feb 10, 2021, 12:37 PM Rui Wang <ru...@google.com> wrote:

> The problem that I can think of is maybe before the async call is
> completed, the UDF life cycle has reached to the end.
>
>
> -Rui
>
> On Wed, Feb 10, 2021 at 12:34 PM Talat Uyarer <
> tuyarer@paloaltonetworks.com> wrote:
>
>> Hi,
>>
>> We plan to use UDF on our sql. We want to achieve some kind of
>> filtering based on internal states. We want to update that internal state
>> with a separate async thread in UDF. Before implementing that thing I want
>> to get your options. Is there any limitation for UDF to have multi-thread
>> implementation ?  Our UDF is a scalar function. It will get 1 or 2 input
>> and return boolean.
>>
>> I will appreciate your comments in advance.
>>
>> Thanks
>>
>

Re: Apache Beam SQL and UDF

[Rui Wang <ru...@google.com>]

The problem that I can think of is maybe before the async call is
completed, the UDF life cycle has reached to the end.


-Rui

On Wed, Feb 10, 2021 at 12:34 PM Talat Uyarer <tu...@paloaltonetworks.com>
wrote:

> Hi,
>
> We plan to use UDF on our sql. We want to achieve some kind of
> filtering based on internal states. We want to update that internal state
> with a separate async thread in UDF. Before implementing that thing I want
> to get your options. Is there any limitation for UDF to have multi-thread
> implementation ?  Our UDF is a scalar function. It will get 1 or 2 input
> and return boolean.
>
> I will appreciate your comments in advance.
>
> Thanks
>