You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@flink.apache.org by Aitozi <gj...@gmail.com> on 2023/05/23 09:45:57 UTC

[DISCUSS] FLIP-313 Add support of User Defined AsyncTableFunction

Hi guys,
    I want to bring up a discussion about adding support of User
Defined AsyncTableFunction in Flink.
Currently, async table function are special functions for table source
to perform
async lookup. However, it's worth to support the user defined async
table function.
Because, in this way, the end SQL user can leverage it to perform the
async operation
which is useful to maximum the system throughput especially for IO
bottleneck case.

You can find some more detail in [1].

Looking forward to feedback


[1]: https://cwiki.apache.org/confluence/display/FLINK/%5BFLIP-313%5D+Add+support+of+User+Defined+AsyncTableFunction

Thanks,
Aitozi.

Re: [DISCUSS] FLIP-313 Add support of User Defined AsyncTableFunction

Posted by Aitozi <gj...@gmail.com>.

Hi guys,
    Since there are no further comments, Kindly ping for the vote thread
[1] :D

Thanks,
Aitozi.

[1]: https://lists.apache.org/thread/7g5n2vshosom2dj9bp7x4n01okrnx4xx

Aitozi <gj...@gmail.com> 于2023年6月26日周一 10:31写道：

> Hi Lincoln,
>     Thanks for your confirmation. I have updated the consensus to the FLIP
> doc.
> If there are no other comments, I'd like to restart the vote process in
> [1] today.
>
> https://lists.apache.org/thread/7g5n2vshosom2dj9bp7x4n01okrnx4xx
>
> Thanks,
> Aitozi.
>
> Lincoln Lee <li...@gmail.com> 于2023年6月21日周三 22:29写道：
>
>> Hi Aitozi,
>>
>> Thanks for your updates!
>>
>> By the design of hints, the hints after select clause belong to the query
>> hints category, and this new hint is also a kind of join hints[1].
>> Join table function is one of the join type defined by flink sql joins[2],
>> all existing join hints[1] omit the 'join' keyword,
>> so I would prefer the 'ASYNC_TABLE_FUNC' (which is actually the one for
>> 'ASYNC_TABLE_FUNC_JOIN').
>>
>> Since a short Chinese holiday is coming, I suggest waiting for other
>> people's responses before continuing to vote (next monday?)
>>
>> Btw, I discussed with @fudian offline about pyflink support, there should
>> be no known issues, so you can create a subtask with pyflink support after
>> the vote passed.
>>
>> [1]
>>
>> https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/sql/queries/hints/#join-hints
>> [2]
>>
>> https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/sql/queries/joins/
>>
>> Best,
>> Lincoln Lee
>>
>>
>> Aitozi <gj...@gmail.com> 于2023年6月18日周日 21:18写道：
>>
>> > Hi all,
>> >     Sorry for the late reply, I have a discussion with Lincoln offline,
>> > mainly about
>> > the naming of the hints option. Thanks Lincoln for the valuable
>> > suggestions.
>> >
>> > Let me answer the last email inline.
>> >
>> > >For `JavaAsyncTableFunc0` in flip, can you use a scenario like RPC
>> call as
>> > an example?
>> >
>> > Sure, will give an example when adding the doc of async udtf and will
>> > update the FLIP simultaneously
>> >
>> > >For the name of this query hint, 'LATERAL' (include its internal
>> options)
>> > don't show any relevance to async, but I haven't thought of a suitable
>> name
>> > at the moment,
>> >
>> > After some discussion with Lincoln, We prefer to choose one of the
>> > `ASYNC_TABLE_FUNC` and `ASYNC_LATERAL`.
>> > Besides, In my opinion the keyword `lateral`'s use scenario is wider
>> than
>> > the table function join, but in this case we only want to config
>> > the async table function, So I'm a bit more lean to the
>> `ASYNC_TABLE_FUNC`.
>> > Looking forward to some inputs if you guys have
>> > some better suggestion on the naming.
>> >
>> > For the usage of the hints config option, I have updated the section
>> > of ConfigOption, you can refer to the FLIP
>> > for more details.
>> >
>> > >Also, the terms 'correlate join' and 'lateral join' are not the same
>> as in
>> > the current joins page[1], so maybe it would be better if we unified
>> them
>> > into  'join table function'
>> >
>> > Yes, we should unified to the 'join table function', updated.
>> >
>> > Best,
>> > Aitozi
>> >
>> > Lincoln Lee <li...@gmail.com> 于2023年6月15日周四 09:15写道：
>> >
>> > > Hi Aitozi,
>> > >
>> > > Thanks for your reply!  Gives sql users more flexibility to get
>> > > asynchronous processing capabilities via lateral join table function
>> +1
>> > for
>> > > this
>> > >
>> > > For `JavaAsyncTableFunc0` in flip, can you use a scenario like RPC
>> call
>> > as
>> > > an example?
>> > >
>> > > For the name of this query hint, 'LATERAL' (include its internal
>> options)
>> > > don't show any relevance to async, but I haven't thought of a suitable
>> > name
>> > > at the moment,
>> > > maybe we need to highlight the async keyword directly, we can also
>> see if
>> > > others have better candidates
>> > >
>> > > For the hint option "timeout = '180s'" should be "'timeout' = '180s'",
>> > > seems a typo in the flip. And use upper case for all keywords in sql
>> > > examples.
>> > > Also, the terms 'correlate join' and 'lateral join' are not the same
>> as
>> > in
>> > > the current joins page[1], so maybe it would be better if we unified
>> them
>> > > into  'join table function'
>> > >
>> > > [1]
>> > >
>> > >
>> >
>> https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/sql/queries/joins/#table-function
>> > >
>> > > Best,
>> > > Lincoln Lee
>> > >
>> > >
>> > > Aitozi <gj...@gmail.com> 于2023年6月14日周三 16:11写道：
>> > >
>> > > > Hi Lincoln
>> > > >
>> > > >     Very thanks for your valuable question. I will try to answer
>> your
>> > > > questions inline.
>> > > >
>> > > > >Does the async udtf bring any additional benefits besides a
>> > > > lighter implementation？
>> > > >
>> > > > IMO, async udtf is more than a lighter implementation. It can act
>> as a
>> > > > general way for sql users to use the async operator. And they don't
>> > have
>> > > to
>> > > > bind the async function with a table (a LookupTable), and they are
>> not
>> > > > forced to join on an equality join condition, and they can use it
>> to do
>> > > > more than enrich data.
>> > > >
>> > > > The async lookup join is more like a subset/specific usage of async
>> > udtf.
>> > > > The specific version has more opportunity to be optimized (like push
>> > > down)
>> > > > is acceptable. Async table function should be categorized to
>> > used-defined
>> > > > function.
>> > > >
>> > > > >Should users
>> > > >
>> > > > migrate to the lookup source when they encounter similar
>> requirements
>> > or
>> > > >
>> > > > problems, or should we develop an additional set of similar
>> mechanisms？
>> > > >
>> > > > As I clarified above, the lookup join is a specific usage of async
>> > udtf.
>> > > So
>> > > > it deserves more refined optimization like caching / retryable. But
>> it
>> > > may
>> > > > not all
>> > > >
>> > > > suitable for the async udtf. As function, it can be deterministic/or
>> > > > non-deterministic. So caching is not suitable, and we also do not
>> have
>> > a
>> > > > common cache for the udf now. So I think optimization like
>> > caching/retry
>> > > > should be handed over to the function implementor.
>> > > >
>> > > > > the newly added query hint need a different name that
>> > > > can be easier related to the lateral operation as the current join
>> > > hints[5]
>> > > > do.
>> > > >
>> > > >
>> > > > What about using LATERAL?
>> > > >
>> > > > as below
>> > > >
>> > > > SELECT /*+ LATERAL('output-mode' = 'ordered', 'capacity' = '200',
>> > > timeout =
>> > > > '180s') */ a, c1, c2
>> > > >
>> > > > FROM T1
>> > > >
>> > > > LEFT JOIN lateral TABLE (async_split(b)) AS T(c1, c2) ON true
>> > > >
>> > > > >For the async func example, since the target scenario is an
>> external
>> > io
>> > > > operation, it's better to add the `close` method to actively release
>> > > > resources as a good example for users
>> > > >
>> > > >
>> > > > Make sense to me, will update the FLIP
>> > > >
>> > > > Best,
>> > > >
>> > > > Aitozi.
>> > > >
>> > > > Lincoln Lee <li...@gmail.com> 于2023年6月14日周三 14:24写道：
>> > > >
>> > > > > Hi Aitozi,
>> > > > >
>> > > > > Sorry for the lately reply here!  Supports async
>> > > > udtf(`AsyncTableFunction`)
>> > > > > directly in sql seems like an attractive feature, but there're two
>> > > issues
>> > > > > that need to be addressed before we can be sure to add it:
>> > > > > 1. As mentioned in the flip[1], the current lookup function can
>> > already
>> > > > > implement the requirements, but it requires implementing an extra
>> > > > > `LookupTableSource` and explicitly declaring the table schema
>> (which
>> > > can
>> > > > > help implementers the various push-down optimizations supported by
>> > the
>> > > > > planner). Does the async udtf bring any additional benefits
>> besides a
>> > > > > lighter implementation?
>> > > > > 2. FLIP-221[2] abstracts a reusable cache and metric
>> infrastructure
>> > for
>> > > > > lookup sources, which are important to improve performance and
>> > > > > observability for high overhead external io scenarios, how do we
>> > > > integrate
>> > > > > and reuse these capabilities after introducing async udtf? Should
>> > users
>> > > > > migrate to the lookup source when they encounter similar
>> requirements
>> > > or
>> > > > > problems, or should we develop an additional set of similar
>> > mechanisms?
>> > > > (a
>> > > > > similarly case:  FLIP-234[3] introduced the retryable capability
>> for
>> > > > lookup
>> > > > > join)
>> > > > >
>> > > > > For the flip itself,
>> > > > > 1. Considering the 'options' is already used as the dynamic table
>> > > > > options[4] in flink, the newly added query hint need a different
>> name
>> > > > that
>> > > > > can be easier related to the lateral operation as the current join
>> > > > hints[5]
>> > > > > do.
>> > > > > 2. For the async func example, since the target scenario is an
>> > external
>> > > > io
>> > > > > operation, it's better to add the `close` method to actively
>> release
>> > > > > resources as a good example for users. Also in terms of the
>> > determinism
>> > > > of
>> > > > > a function, it is important to remind users that unless the
>> behavior
>> > of
>> > > > the
>> > > > > function is deterministic, it needs to be explicitly declared as
>> > > > > non-deterministic.
>> > > > >
>> > > > > [1].
>> > > > >
>> > > > >
>> > > >
>> > >
>> >
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-313%3A+Add+support+of+User+Defined+AsyncTableFunction?src=contextnavpagetreemode
>> > > > > [2].
>> > > > >
>> > > > >
>> > > >
>> > >
>> >
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-221%3A+Abstraction+for+lookup+source+cache+and+metric?src=contextnavpagetreemode
>> > > > > [3].
>> > > > >
>> > > > >
>> > > >
>> > >
>> >
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-234%3A+Support+Retryable+Lookup+Join+To+Solve+Delayed+Updates+Issue+In+External+Systems?src=contextnavpagetreemode
>> > > > > [4].
>> > > > >
>> > > > >
>> > > >
>> > >
>> >
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-113%3A+Supports+Dynamic+Table+Options+for+Flink+SQL?src=contextnavpagetreemode
>> > > > > [5].
>> > > > >
>> > > > >
>> > > >
>> > >
>> >
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-229%3A+Introduces+Join+Hint+for+Flink+SQL+Batch+Job?src=contextnavpagetreemode
>> > > > >
>> > > > > Best,
>> > > > > Lincoln Lee
>> > > > >
>> > > > >
>> > > > > Aitozi <gj...@gmail.com> 于2023年6月13日周二 11:30写道：
>> > > > >
>> > > > > > Get your meaning now, thanks :)
>> > > > > >
>> > > > > > Best,
>> > > > > > Aitozi.
>> > > > > >
>> > > > > > Feng Jin <ji...@gmail.com> 于2023年6月13日周二 11:16写道：
>> > > > > >
>> > > > > > > Hi Aitozi,
>> > > > > > >
>> > > > > > > Sorry for the confusing description.
>> > > > > > >
>> > > > > > > What I meant was that if we need to remind users about tire
>> > safety
>> > > > > > issues,
>> > > > > > > we should introduce the new UDTF interface instead of
>> executing
>> > the
>> > > > > > > original UDTF asynchronously. Therefore, I agree with
>> introducing
>> > > the
>> > > > > > > AsyncTableFunction.
>> > > > > > >
>> > > > > > > Best,
>> > > > > > > Feng
>> > > > > > >
>> > > > > > > On Tue, Jun 13, 2023 at 10:42 AM Aitozi <gjying1314@gmail.com
>> >
>> > > > wrote:
>> > > > > > >
>> > > > > > > > Hi Feng,
>> > > > > > > >     Thanks for your question. We do not provide a way to
>> switch
>> > > the
>> > > > > > UDTF
>> > > > > > > > between sync and async way,
>> > > > > > > > So there should be no thread safety problem here.
>> > > > > > > >
>> > > > > > > > Best,
>> > > > > > > > Aitozi
>> > > > > > > >
>> > > > > > > > Feng Jin <ji...@gmail.com> 于2023年6月13日周二 10:31写道：
>> > > > > > > >
>> > > > > > > > > Hi Aitozi, We do need to remind users about thread safety
>> > > issues.
>> > > > > > Thank
>> > > > > > > > you
>> > > > > > > > > for your efforts on this FLIP. I have no further
>> questions.
>> > > > > > > > > Best, Feng
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > > On Tue, Jun 13, 2023 at 6:05 AM Jing Ge
>> > > > <jing@ververica.com.invalid
>> > > > > >
>> > > > > > > > > wrote:
>> > > > > > > > >
>> > > > > > > > > > Hi Aitozi,
>> > > > > > > > > >
>> > > > > > > > > > Thanks for taking care of that part. I have no other
>> > concern.
>> > > > > > > > > >
>> > > > > > > > > > Best regards,
>> > > > > > > > > > Jing
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > > On Mon, Jun 12, 2023 at 5:38 PM Aitozi <
>> > gjying1314@gmail.com
>> > > >
>> > > > > > wrote:
>> > > > > > > > > >
>> > > > > > > > > > > BTW, If there are no other more blocking issue /
>> > comments,
>> > > I
>> > > > > > would
>> > > > > > > > like
>> > > > > > > > > > to
>> > > > > > > > > > > start a VOTE in another thread this wednesday 6.14
>> > > > > > > > > > >
>> > > > > > > > > > > Thanks,
>> > > > > > > > > > > Aitozi.
>> > > > > > > > > > >
>> > > > > > > > > > > Aitozi <gj...@gmail.com> 于2023年6月12日周一 23:34写道：
>> > > > > > > > > > >
>> > > > > > > > > > > > Hi, Jing,
>> > > > > > > > > > > >     Thanks for your explanation. I get your point
>> now.
>> > > > > > > > > > > >
>> > > > > > > > > > > > For the performance part, I think it's a good idea
>> to
>> > run
>> > > > > with
>> > > > > > > > > > returning
>> > > > > > > > > > > a
>> > > > > > > > > > > > big table case, the memory consumption
>> > > > > > > > > > > > should be a point to be taken care about. Because in
>> > the
>> > > > > > ordered
>> > > > > > > > > mode,
>> > > > > > > > > > > the
>> > > > > > > > > > > > head element in buffer may affect the
>> > > > > > > > > > > > total memory consumption.
>> > > > > > > > > > > >
>> > > > > > > > > > > >
>> > > > > > > > > > > > Thanks,
>> > > > > > > > > > > > Aitozi.
>> > > > > > > > > > > >
>> > > > > > > > > > > >
>> > > > > > > > > > > >
>> > > > > > > > > > > > Jing Ge <ji...@ververica.com.invalid> 于2023年6月12日周一
>> > > > 20:28写道：
>> > > > > > > > > > > >
>> > > > > > > > > > > >> Hi Aitozi,
>> > > > > > > > > > > >>
>> > > > > > > > > > > >> Which key will be used for lookup is not an issue,
>> > only
>> > > > one
>> > > > > > row
>> > > > > > > > will
>> > > > > > > > > > be
>> > > > > > > > > > > >> required for each key in order to enrich it. True,
>> it
>> > > > > depends
>> > > > > > on
>> > > > > > > > the
>> > > > > > > > > > > >> implementation whether multiple rows or single row
>> for
>> > > > each
>> > > > > > key
>> > > > > > > > will
>> > > > > > > > > > be
>> > > > > > > > > > > >> returned. However, for the lookup & enrichment
>> > scenario,
>> > > > one
>> > > > > > > > row/key
>> > > > > > > > > > is
>> > > > > > > > > > > >> recommended, otherwise, like I mentioned
>> previously,
>> > > > > > enrichment
>> > > > > > > > > won't
>> > > > > > > > > > > >> work.
>> > > > > > > > > > > >>
>> > > > > > > > > > > >> I am a little bit concerned about returning a big
>> > table
>> > > > for
>> > > > > > each
>> > > > > > > > > key,
>> > > > > > > > > > > >> since
>> > > > > > > > > > > >> it will take the async call longer to return and
>> need
>> > > more
>> > > > > > > memory.
>> > > > > > > > > The
>> > > > > > > > > > > >> performance tests should cover this scenario. This
>> is
>> > > not
>> > > > a
>> > > > > > > > blocking
>> > > > > > > > > > > issue
>> > > > > > > > > > > >> for this FLIP.
>> > > > > > > > > > > >>
>> > > > > > > > > > > >> Best regards,
>> > > > > > > > > > > >> Jing
>> > > > > > > > > > > >>
>> > > > > > > > > > > >> On Sat, Jun 10, 2023 at 4:11 AM Aitozi <
>> > > > > gjying1314@gmail.com>
>> > > > > > > > > wrote:
>> > > > > > > > > > > >>
>> > > > > > > > > > > >> > Hi Jing,
>> > > > > > > > > > > >> >     I means the join key is not necessary to be
>> the
>> > > > > primary
>> > > > > > > key
>> > > > > > > > or
>> > > > > > > > > > > >> unique
>> > > > > > > > > > > >> > index of the database.
>> > > > > > > > > > > >> > In this situation, we may queried out multi rows
>> for
>> > > one
>> > > > > > join
>> > > > > > > > > key. I
>> > > > > > > > > > > >> think
>> > > > > > > > > > > >> > that's why the
>> > > > > > > > > > > >> > LookupFunction#lookup will return a collection of
>> > > > RowData.
>> > > > > > > > > > > >> >
>> > > > > > > > > > > >> > BTW, I think the behavior of lookup join will not
>> > > affect
>> > > > > the
>> > > > > > > > > > semantic
>> > > > > > > > > > > of
>> > > > > > > > > > > >> > the async udtf.
>> > > > > > > > > > > >> > We use the Async TableFunction here and the table
>> > > > function
>> > > > > > can
>> > > > > > > > > > collect
>> > > > > > > > > > > >> > multiple rows.
>> > > > > > > > > > > >> >
>> > > > > > > > > > > >> > Thanks,
>> > > > > > > > > > > >> > Atiozi.
>> > > > > > > > > > > >> >
>> > > > > > > > > > > >> >
>> > > > > > > > > > > >> >
>> > > > > > > > > > > >> > Jing Ge <ji...@ververica.com.invalid>
>> 于2023年6月10日周六
>> > > > > 00:15写道：
>> > > > > > > > > > > >> >
>> > > > > > > > > > > >> > > Hi Aitozi,
>> > > > > > > > > > > >> > >
>> > > > > > > > > > > >> > > The keyRow used in this case contains all
>> keys[1].
>> > > > > > > > > > > >> > >
>> > > > > > > > > > > >> > > Best regards,
>> > > > > > > > > > > >> > > Jing
>> > > > > > > > > > > >> > >
>> > > > > > > > > > > >> > > [1]
>> > > > > > > > > > > >> > >
>> > > > > > > > > > > >> > >
>> > > > > > > > > > > >> >
>> > > > > > > > > > > >>
>> > > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> https://github.com/apache/flink/blob/191ec6ca3943d7119f14837efe112e074d815c47/flink-table/flink-table-common/src/main/java/org/apache/flink/table/functions/LookupFunction.java#L49
>> > > > > > > > > > > >> > >
>> > > > > > > > > > > >> > >
>> > > > > > > > > > > >> > > On Fri, Jun 9, 2023 at 3:42 PM Aitozi <
>> > > > > > gjying1314@gmail.com
>> > > > > > > >
>> > > > > > > > > > wrote:
>> > > > > > > > > > > >> > >
>> > > > > > > > > > > >> > > > Hi Jing,
>> > > > > > > > > > > >> > > >
>> > > > > > > > > > > >> > > >      The performance test is added to the
>> FLIP.
>> > > > > > > > > > > >> > > >
>> > > > > > > > > > > >> > > >      As I know, The lookup join can return
>> multi
>> > > > rows,
>> > > > > > it
>> > > > > > > > > > depends
>> > > > > > > > > > > on
>> > > > > > > > > > > >> > > > whether  the join key
>> > > > > > > > > > > >> > > > is the primary key of the external database
>> or
>> > > not.
>> > > > > The
>> > > > > > > > > `lookup`
>> > > > > > > > > > > [1]
>> > > > > > > > > > > >> > will
>> > > > > > > > > > > >> > > > return a collection of
>> > > > > > > > > > > >> > > > joined result, and each of them will be
>> > collected
>> > > > > > > > > > > >> > > >
>> > > > > > > > > > > >> > > >
>> > > > > > > > > > > >> > > > [1]:
>> > > > > > > > > > > >> > > >
>> > > > > > > > > > > >> > > >
>> > > > > > > > > > > >> > >
>> > > > > > > > > > > >> >
>> > > > > > > > > > > >>
>> > > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> https://github.com/apache/flink/blob/191ec6ca3943d7119f14837efe112e074d815c47/flink-table/flink-table-common/src/main/java/org/apache/flink/table/functions/LookupFunction.java#L52
>> > > > > > > > > > > >> > > >
>> > > > > > > > > > > >> > > >
>> > > > > > > > > > > >> > > > Thanks,
>> > > > > > > > > > > >> > > > Aitozi.
>> > > > > > > > > > > >> > > >
>> > > > > > > > > > > >> > > > Jing Ge <ji...@ververica.com.invalid>
>> > 于2023年6月9日周五
>> > > > > > > 17:05写道：
>> > > > > > > > > > > >> > > >
>> > > > > > > > > > > >> > > > > Hi Aitozi,
>> > > > > > > > > > > >> > > > >
>> > > > > > > > > > > >> > > > > Thanks for the feedback. Looking forward to
>> > the
>> > > > > > > > performance
>> > > > > > > > > > > tests.
>> > > > > > > > > > > >> > > > >
>> > > > > > > > > > > >> > > > > Afaik, lookup returns one row for each key
>> [1]
>> > > > [2].
>> > > > > > > > > > > Conceptually,
>> > > > > > > > > > > >> the
>> > > > > > > > > > > >> > > > > lookup function is used to enrich column(s)
>> > from
>> > > > the
>> > > > > > > > > dimension
>> > > > > > > > > > > >> table.
>> > > > > > > > > > > >> > > If,
>> > > > > > > > > > > >> > > > > for the given key, there will be more than
>> one
>> > > > row,
>> > > > > > > there
>> > > > > > > > > will
>> > > > > > > > > > > be
>> > > > > > > > > > > >> no
>> > > > > > > > > > > >> > > way
>> > > > > > > > > > > >> > > > to
>> > > > > > > > > > > >> > > > > know which row will be used to enrich the
>> key.
>> > > > > > > > > > > >> > > > >
>> > > > > > > > > > > >> > > > > [1]
>> > > > > > > > > > > >> > > > >
>> > > > > > > > > > > >> > > > >
>> > > > > > > > > > > >> > > >
>> > > > > > > > > > > >> > >
>> > > > > > > > > > > >> >
>> > > > > > > > > > > >>
>> > > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> https://github.com/apache/flink/blob/191ec6ca3943d7119f14837efe112e074d815c47/flink-table/flink-table-common/src/main/java/org/apache/flink/table/functions/LookupFunction.java#L49
>> > > > > > > > > > > >> > > > > [2]
>> > > > > > > > > > > >> > > > >
>> > > > > > > > > > > >> > > > >
>> > > > > > > > > > > >> > > >
>> > > > > > > > > > > >> > >
>> > > > > > > > > > > >> >
>> > > > > > > > > > > >>
>> > > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> https://github.com/apache/flink/blob/191ec6ca3943d7119f14837efe112e074d815c47/flink-table/flink-table-common/src/main/java/org/apache/flink/table/functions/TableFunction.java#L196
>> > > > > > > > > > > >> > > > >
>> > > > > > > > > > > >> > > > > Best regards,
>> > > > > > > > > > > >> > > > > Jing
>> > > > > > > > > > > >> > > > >
>> > > > > > > > > > > >> > > > > On Fri, Jun 9, 2023 at 5:18 AM Aitozi <
>> > > > > > > > gjying1314@gmail.com
>> > > > > > > > > >
>> > > > > > > > > > > >> wrote:
>> > > > > > > > > > > >> > > > >
>> > > > > > > > > > > >> > > > > > Hi Jing
>> > > > > > > > > > > >> > > > > >     Thanks for your good questions. I
>> have
>> > > > updated
>> > > > > > the
>> > > > > > > > > > example
>> > > > > > > > > > > >> to
>> > > > > > > > > > > >> > the
>> > > > > > > > > > > >> > > > > FLIP.
>> > > > > > > > > > > >> > > > > >
>> > > > > > > > > > > >> > > > > > > Only one row for each lookup
>> > > > > > > > > > > >> > > > > > lookup can also return multi rows, based
>> on
>> > > the
>> > > > > > query
>> > > > > > > > > > result.
>> > > > > > > > > > > >> [1]
>> > > > > > > > > > > >> > > > > >
>> > > > > > > > > > > >> > > > > > [1]:
>> > > > > > > > > > > >> > > > > >
>> > > > > > > > > > > >> > > > > >
>> > > > > > > > > > > >> > > > >
>> > > > > > > > > > > >> > > >
>> > > > > > > > > > > >> > >
>> > > > > > > > > > > >> >
>> > > > > > > > > > > >>
>> > > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> https://github.com/apache/flink/blob/191ec6ca3943d7119f14837efe112e074d815c47/flink-table/flink-table-common/src/main/java/org/apache/flink/table/functions/LookupFunction.java#L56
>> > > > > > > > > > > >> > > > > >
>> > > > > > > > > > > >> > > > > > > If we use async calls with lateral
>> join,
>> > my
>> > > > gut
>> > > > > > > > feeling
>> > > > > > > > > is
>> > > > > > > > > > > >> > > > > > that we might have many more async calls
>> > than
>> > > > > lookup
>> > > > > > > > > join. I
>> > > > > > > > > > > am
>> > > > > > > > > > > >> not
>> > > > > > > > > > > >> > > > > really
>> > > > > > > > > > > >> > > > > > sure if we will be facing potential
>> issues
>> > in
>> > > > this
>> > > > > > > case
>> > > > > > > > or
>> > > > > > > > > > > not.
>> > > > > > > > > > > >> > > > > >
>> > > > > > > > > > > >> > > > > > IMO, the work pattern is similar to the
>> > lookup
>> > > > > > > function,
>> > > > > > > > > for
>> > > > > > > > > > > >> each
>> > > > > > > > > > > >> > row
>> > > > > > > > > > > >> > > > > from
>> > > > > > > > > > > >> > > > > > the left table,
>> > > > > > > > > > > >> > > > > > it will evaluate the eval method once, so
>> > the
>> > > > > async
>> > > > > > > call
>> > > > > > > > > > > numbers
>> > > > > > > > > > > >> > will
>> > > > > > > > > > > >> > > > not
>> > > > > > > > > > > >> > > > > > change.
>> > > > > > > > > > > >> > > > > > and the maximum calls in flight is
>> limited
>> > by
>> > > > the
>> > > > > > > Async
>> > > > > > > > > > > >> operators
>> > > > > > > > > > > >> > > > buffer
>> > > > > > > > > > > >> > > > > > capacity
>> > > > > > > > > > > >> > > > > > which will be controlled by the option.
>> > > > > > > > > > > >> > > > > >
>> > > > > > > > > > > >> > > > > > BTW, for the naming of these option, I
>> > updated
>> > > > the
>> > > > > > > FLIP
>> > > > > > > > > > about
>> > > > > > > > > > > >> this
>> > > > > > > > > > > >> > > you
>> > > > > > > > > > > >> > > > > can
>> > > > > > > > > > > >> > > > > > refer to
>> > > > > > > > > > > >> > > > > > the section of "ConfigOption" and
>> "Rejected
>> > > > > > > > Alternatives"
>> > > > > > > > > > > >> > > > > >
>> > > > > > > > > > > >> > > > > > In the end, for the performance
>> evaluation,
>> > > I'd
>> > > > > like
>> > > > > > > to
>> > > > > > > > do
>> > > > > > > > > > > some
>> > > > > > > > > > > >> > tests
>> > > > > > > > > > > >> > > > and
>> > > > > > > > > > > >> > > > > > will update it to the FLIP doc
>> > > > > > > > > > > >> > > > > >
>> > > > > > > > > > > >> > > > > > Thanks,
>> > > > > > > > > > > >> > > > > > Aitozi.
>> > > > > > > > > > > >> > > > > >
>> > > > > > > > > > > >> > > > > >
>> > > > > > > > > > > >> > > > > > Jing Ge <ji...@ververica.com.invalid>
>> > > > 于2023年6月9日周五
>> > > > > > > > > 07:23写道：
>> > > > > > > > > > > >> > > > > >
>> > > > > > > > > > > >> > > > > > > Hi Aitozi,
>> > > > > > > > > > > >> > > > > > >
>> > > > > > > > > > > >> > > > > > > Thanks for the clarification. The code
>> > > example
>> > > > > > looks
>> > > > > > > > > > > >> > interesting. I
>> > > > > > > > > > > >> > > > > would
>> > > > > > > > > > > >> > > > > > > suggest adding them into the FLIP. The
>> > > > > description
>> > > > > > > > with
>> > > > > > > > > > code
>> > > > > > > > > > > >> > > examples
>> > > > > > > > > > > >> > > > > > will
>> > > > > > > > > > > >> > > > > > > help readers understand the motivation
>> and
>> > > how
>> > > > > to
>> > > > > > > use
>> > > > > > > > > it.
>> > > > > > > > > > > >> Afaiac,
>> > > > > > > > > > > >> > > it
>> > > > > > > > > > > >> > > > > is a
>> > > > > > > > > > > >> > > > > > > valid feature for Flink users.
>> > > > > > > > > > > >> > > > > > >
>> > > > > > > > > > > >> > > > > > > As we knew, lookup join is based on
>> > temporal
>> > > > > join,
>> > > > > > > > i.e.
>> > > > > > > > > > FOR
>> > > > > > > > > > > >> > > > SYSTEM_TIME
>> > > > > > > > > > > >> > > > > > AS
>> > > > > > > > > > > >> > > > > > > OF which is also used in your code
>> > example.
>> > > > > > Temporal
>> > > > > > > > > join
>> > > > > > > > > > > >> > performs
>> > > > > > > > > > > >> > > > the
>> > > > > > > > > > > >> > > > > > > lookup based on the processing time
>> match.
>> > > > Only
>> > > > > > one
>> > > > > > > > row
>> > > > > > > > > > for
>> > > > > > > > > > > >> each
>> > > > > > > > > > > >> > > > > > > lookup(afaiu, I need to check the
>> source
>> > > code
>> > > > to
>> > > > > > > > double
>> > > > > > > > > > > >> confirm)
>> > > > > > > > > > > >> > > will
>> > > > > > > > > > > >> > > > > > > return for further enrichment. One the
>> > other
>> > > > > hand,
>> > > > > > > > > lateral
>> > > > > > > > > > > >> join
>> > > > > > > > > > > >> > > will
>> > > > > > > > > > > >> > > > > have
>> > > > > > > > > > > >> > > > > > > sub-queries correlated with every
>> > individual
>> > > > > value
>> > > > > > > of
>> > > > > > > > > the
>> > > > > > > > > > > >> > reference
>> > > > > > > > > > > >> > > > > table
>> > > > > > > > > > > >> > > > > > > from the preceding part of the query
>> and
>> > > each
>> > > > > sub
>> > > > > > > > query
>> > > > > > > > > > will
>> > > > > > > > > > > >> > return
>> > > > > > > > > > > >> > > > > > > multiple rows. If we use async calls
>> with
>> > > > > lateral
>> > > > > > > > join,
>> > > > > > > > > my
>> > > > > > > > > > > gut
>> > > > > > > > > > > >> > > > feeling
>> > > > > > > > > > > >> > > > > is
>> > > > > > > > > > > >> > > > > > > that we might have many more async
>> calls
>> > > than
>> > > > > > lookup
>> > > > > > > > > > join. I
>> > > > > > > > > > > >> am
>> > > > > > > > > > > >> > not
>> > > > > > > > > > > >> > > > > > really
>> > > > > > > > > > > >> > > > > > > sure if we will be facing potential
>> issues
>> > > in
>> > > > > this
>> > > > > > > > case
>> > > > > > > > > or
>> > > > > > > > > > > >> not.
>> > > > > > > > > > > >> > > > > Possible
>> > > > > > > > > > > >> > > > > > > issues I can think of now e.g. too many
>> > PRC
>> > > > > calls,
>> > > > > > > too
>> > > > > > > > > > many
>> > > > > > > > > > > >> async
>> > > > > > > > > > > >> > > > calls
>> > > > > > > > > > > >> > > > > > > processing, the sub query will return a
>> > > table
>> > > > > > which
>> > > > > > > > > might
>> > > > > > > > > > be
>> > > > > > > > > > > >> > (too)
>> > > > > > > > > > > >> > > > big,
>> > > > > > > > > > > >> > > > > > and
>> > > > > > > > > > > >> > > > > > > might cause performance issues. I would
>> > > > suggest
>> > > > > > > > > preparing
>> > > > > > > > > > > some
>> > > > > > > > > > > >> > use
>> > > > > > > > > > > >> > > > > cases
>> > > > > > > > > > > >> > > > > > > and running some performance tests to
>> > check
>> > > > it.
>> > > > > > > These
>> > > > > > > > > are
>> > > > > > > > > > my
>> > > > > > > > > > > >> > > concerns
>> > > > > > > > > > > >> > > > > > about
>> > > > > > > > > > > >> > > > > > > using async calls with lateral join and
>> > I'd
>> > > > like
>> > > > > > to
>> > > > > > > > > share
>> > > > > > > > > > > with
>> > > > > > > > > > > >> > you,
>> > > > > > > > > > > >> > > > > happy
>> > > > > > > > > > > >> > > > > > > to discuss with you and hear different
>> > > > opinions,
>> > > > > > > > > hopefully
>> > > > > > > > > > > the
>> > > > > > > > > > > >> > > > > > > discussion could help me understand it
>> > more
>> > > > > > deeply.
>> > > > > > > > > Please
>> > > > > > > > > > > >> > correct
>> > > > > > > > > > > >> > > me
>> > > > > > > > > > > >> > > > > if
>> > > > > > > > > > > >> > > > > > I
>> > > > > > > > > > > >> > > > > > > am wrong.
>> > > > > > > > > > > >> > > > > > >
>> > > > > > > > > > > >> > > > > > > Best regards,
>> > > > > > > > > > > >> > > > > > > Jing
>> > > > > > > > > > > >> > > > > > >
>> > > > > > > > > > > >> > > > > > >
>> > > > > > > > > > > >> > > > > > > On Thu, Jun 8, 2023 at 7:22 AM Aitozi <
>> > > > > > > > > > gjying1314@gmail.com
>> > > > > > > > > > > >
>> > > > > > > > > > > >> > > wrote:
>> > > > > > > > > > > >> > > > > > >
>> > > > > > > > > > > >> > > > > > > > Hi Mason,
>> > > > > > > > > > > >> > > > > > > >     Thanks for your input. I think
>> if we
>> > > > > support
>> > > > > > > the
>> > > > > > > > > > user
>> > > > > > > > > > > >> > defined
>> > > > > > > > > > > >> > > > > async
>> > > > > > > > > > > >> > > > > > > > table function,
>> > > > > > > > > > > >> > > > > > > > user will be able to use it to hold a
>> > > batch
>> > > > > data
>> > > > > > > > then
>> > > > > > > > > > > >> handle it
>> > > > > > > > > > > >> > > at
>> > > > > > > > > > > >> > > > > one
>> > > > > > > > > > > >> > > > > > > time
>> > > > > > > > > > > >> > > > > > > > in the customized function.
>> > > > > > > > > > > >> > > > > > > >
>> > > > > > > > > > > >> > > > > > > > AsyncSink is meant for the sink
>> > operator.
>> > > I
>> > > > > have
>> > > > > > > not
>> > > > > > > > > > > figure
>> > > > > > > > > > > >> out
>> > > > > > > > > > > >> > > how
>> > > > > > > > > > > >> > > > > to
>> > > > > > > > > > > >> > > > > > > > integrate in this case.
>> > > > > > > > > > > >> > > > > > > >
>> > > > > > > > > > > >> > > > > > > > Thanks,
>> > > > > > > > > > > >> > > > > > > > Atiozi.
>> > > > > > > > > > > >> > > > > > > >
>> > > > > > > > > > > >> > > > > > > >
>> > > > > > > > > > > >> > > > > > > > Mason Chen <ma...@gmail.com>
>> > > > > > 于2023年6月8日周四
>> > > > > > > > > > 12:40写道：
>> > > > > > > > > > > >> > > > > > > >
>> > > > > > > > > > > >> > > > > > > > > Hi Aitozi,
>> > > > > > > > > > > >> > > > > > > > >
>> > > > > > > > > > > >> > > > > > > > > I think it makes sense to make it
>> > easier
>> > > > for
>> > > > > > SQL
>> > > > > > > > > users
>> > > > > > > > > > > to
>> > > > > > > > > > > >> > make
>> > > > > > > > > > > >> > > > > RPCs.
>> > > > > > > > > > > >> > > > > > Do
>> > > > > > > > > > > >> > > > > > > > you
>> > > > > > > > > > > >> > > > > > > > > think your proposal can extend to
>> the
>> > > > > ability
>> > > > > > to
>> > > > > > > > > batch
>> > > > > > > > > > > >> data
>> > > > > > > > > > > >> > for
>> > > > > > > > > > > >> > > > the
>> > > > > > > > > > > >> > > > > > > RPC?
>> > > > > > > > > > > >> > > > > > > > > This is also another common
>> strategy
>> > to
>> > > > > > increase
>> > > > > > > > > > > >> throughput.
>> > > > > > > > > > > >> > > > Also,
>> > > > > > > > > > > >> > > > > > have
>> > > > > > > > > > > >> > > > > > > > you
>> > > > > > > > > > > >> > > > > > > > > considered solving this a bit
>> > > differently
>> > > > by
>> > > > > > > > > > leveraging
>> > > > > > > > > > > >> > Flink's
>> > > > > > > > > > > >> > > > > > > > AsyncSink?
>> > > > > > > > > > > >> > > > > > > > >
>> > > > > > > > > > > >> > > > > > > > > Best,
>> > > > > > > > > > > >> > > > > > > > > Mason
>> > > > > > > > > > > >> > > > > > > > >
>> > > > > > > > > > > >> > > > > > > > > On Mon, Jun 5, 2023 at 1:50 AM
>> Aitozi
>> > <
>> > > > > > > > > > > >> gjying1314@gmail.com>
>> > > > > > > > > > > >> > > > > wrote:
>> > > > > > > > > > > >> > > > > > > > >
>> > > > > > > > > > > >> > > > > > > > > > One more thing for discussion:
>> > > > > > > > > > > >> > > > > > > > > >
>> > > > > > > > > > > >> > > > > > > > > > In our internal implementation,
>> we
>> > > reuse
>> > > > > the
>> > > > > > > > > option
>> > > > > > > > > > > >> > > > > > > > > >
>> > > > `table.exec.async-lookup.buffer-capacity`
>> > > > > > and
>> > > > > > > > > > > >> > > > > > > > > >
>> `table.exec.async-lookup.timeout` to
>> > > > > config
>> > > > > > > > > > > >> > > > > > > > > > the async udtf. Do you think we
>> > should
>> > > > add
>> > > > > > two
>> > > > > > > > > extra
>> > > > > > > > > > > >> option
>> > > > > > > > > > > >> > > to
>> > > > > > > > > > > >> > > > > > > > > distinguish
>> > > > > > > > > > > >> > > > > > > > > > from the lookup option such as
>> > > > > > > > > > > >> > > > > > > > > >
>> > > > > > > > > > > >> > > > > > > > > >
>> > > `table.exec.async-udtf.buffer-capacity`
>> > > > > > > > > > > >> > > > > > > > > > `table.exec.async-udtf.timeout`
>> > > > > > > > > > > >> > > > > > > > > >
>> > > > > > > > > > > >> > > > > > > > > >
>> > > > > > > > > > > >> > > > > > > > > > Best,
>> > > > > > > > > > > >> > > > > > > > > > Aitozi.
>> > > > > > > > > > > >> > > > > > > > > >
>> > > > > > > > > > > >> > > > > > > > > >
>> > > > > > > > > > > >> > > > > > > > > >
>> > > > > > > > > > > >> > > > > > > > > > Aitozi <gj...@gmail.com>
>> > > > > 于2023年6月5日周一
>> > > > > > > > > 12:20写道：
>> > > > > > > > > > > >> > > > > > > > > >
>> > > > > > > > > > > >> > > > > > > > > > > Hi Jing,
>> > > > > > > > > > > >> > > > > > > > > > >
>> > > > > > > > > > > >> > > > > > > > > > >     > what is the difference
>> > between
>> > > > the
>> > > > > > RPC
>> > > > > > > > > call
>> > > > > > > > > > or
>> > > > > > > > > > > >> > query
>> > > > > > > > > > > >> > > > you
>> > > > > > > > > > > >> > > > > > > > > mentioned
>> > > > > > > > > > > >> > > > > > > > > > > and the lookup in a very
>> > > > > > > > > > > >> > > > > > > > > > > general way
>> > > > > > > > > > > >> > > > > > > > > > >
>> > > > > > > > > > > >> > > > > > > > > > > I think the RPC call or query
>> > > service
>> > > > is
>> > > > > > > quite
>> > > > > > > > > > > >> similar to
>> > > > > > > > > > > >> > > the
>> > > > > > > > > > > >> > > > > > > lookup
>> > > > > > > > > > > >> > > > > > > > > > join.
>> > > > > > > > > > > >> > > > > > > > > > > But lookup join should work
>> > > > > > > > > > > >> > > > > > > > > > > with `LookupTableSource`.
>> > > > > > > > > > > >> > > > > > > > > > >
>> > > > > > > > > > > >> > > > > > > > > > > Let's see how we can perform an
>> > > async
>> > > > > RPC
>> > > > > > > call
>> > > > > > > > > > with
>> > > > > > > > > > > >> > lookup
>> > > > > > > > > > > >> > > > > join:
>> > > > > > > > > > > >> > > > > > > > > > >
>> > > > > > > > > > > >> > > > > > > > > > > (1) Implement an
>> > AsyncTableFunction
>> > > > with
>> > > > > > RPC
>> > > > > > > > > call
>> > > > > > > > > > > >> logic.
>> > > > > > > > > > > >> > > > > > > > > > > (2) Implement a
>> > `LookupTableSource`
>> > > > > > > connector
>> > > > > > > > > run
>> > > > > > > > > > > with
>> > > > > > > > > > > >> > the
>> > > > > > > > > > > >> > > > > async
>> > > > > > > > > > > >> > > > > > > udtf
>> > > > > > > > > > > >> > > > > > > > > > > defined in (1).
>> > > > > > > > > > > >> > > > > > > > > > > (3) Then define a DDL of this
>> look
>> > > up
>> > > > > > table
>> > > > > > > in
>> > > > > > > > > SQL
>> > > > > > > > > > > >> > > > > > > > > > >
>> > > > > > > > > > > >> > > > > > > > > > > CREATE TEMPORARY TABLE
>> Customers (
>> > > > > > > > > > > >> > > > > > > > > > >   id INT,
>> > > > > > > > > > > >> > > > > > > > > > >   name STRING,
>> > > > > > > > > > > >> > > > > > > > > > >   country STRING,
>> > > > > > > > > > > >> > > > > > > > > > >   zip STRING
>> > > > > > > > > > > >> > > > > > > > > > > ) WITH (
>> > > > > > > > > > > >> > > > > > > > > > >   'connector' = 'custom'
>> > > > > > > > > > > >> > > > > > > > > > > );
>> > > > > > > > > > > >> > > > > > > > > > >
>> > > > > > > > > > > >> > > > > > > > > > > (4) Run with the query as
>> below:
>> > > > > > > > > > > >> > > > > > > > > > >
>> > > > > > > > > > > >> > > > > > > > > > > SELECT o.order_id, o.total,
>> > > c.country,
>> > > > > > c.zip
>> > > > > > > > > > > >> > > > > > > > > > > FROM Orders AS o
>> > > > > > > > > > > >> > > > > > > > > > >   JOIN Customers FOR
>> SYSTEM_TIME
>> > AS
>> > > OF
>> > > > > > > > > o.proc_time
>> > > > > > > > > > > AS
>> > > > > > > > > > > >> c
>> > > > > > > > > > > >> > > > > > > > > > >     ON o.customer_id = c.id;
>> > > > > > > > > > > >> > > > > > > > > > >
>> > > > > > > > > > > >> > > > > > > > > > > This example is from doc
>> > > > > > > > > > > >> > > > > > > > > > > <
>> > > > > > > > > > > >> > > > > > > > > >
>> > > > > > > > > > > >> > > > > > > > >
>> > > > > > > > > > > >> > > > > > > >
>> > > > > > > > > > > >> > > > > > >
>> > > > > > > > > > > >> > > > > >
>> > > > > > > > > > > >> > > > >
>> > > > > > > > > > > >> > > >
>> > > > > > > > > > > >> > >
>> > > > > > > > > > > >> >
>> > > > > > > > > > > >>
>> > > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/dev/table/sql/queries/joins/#lookup-join
>> > > > > > > > > > > >> > > > > > > > > > >.You
>> > > > > > > > > > > >> > > > > > > > > > > can image the look up process
>> as
>> > an
>> > > > > async
>> > > > > > > RPC
>> > > > > > > > > call
>> > > > > > > > > > > >> > process.
>> > > > > > > > > > > >> > > > > > > > > > >
>> > > > > > > > > > > >> > > > > > > > > > > Let's see how we can perform an
>> > > async
>> > > > > RPC
>> > > > > > > call
>> > > > > > > > > > with
>> > > > > > > > > > > >> > lateral
>> > > > > > > > > > > >> > > > > join:
>> > > > > > > > > > > >> > > > > > > > > > >
>> > > > > > > > > > > >> > > > > > > > > > > (1) Implement an
>> > AsyncTableFunction
>> > > > with
>> > > > > > RPC
>> > > > > > > > > call
>> > > > > > > > > > > >> logic.
>> > > > > > > > > > > >> > > > > > > > > > > (2) Run query with
>> > > > > > > > > > > >> > > > > > > > > > >
>> > > > > > > > > > > >> > > > > > > > > > > Create function f1 as '...' ;
>> > > > > > > > > > > >> > > > > > > > > > >
>> > > > > > > > > > > >> > > > > > > > > > > SELECT o.order_id, o.total,
>> > > c.country,
>> > > > > > c.zip
>> > > > > > > > > FROM
>> > > > > > > > > > > >> Orders
>> > > > > > > > > > > >> > > > > lateral
>> > > > > > > > > > > >> > > > > > > > table
>> > > > > > > > > > > >> > > > > > > > > > > (f1(order_id)) as T(...);
>> > > > > > > > > > > >> > > > > > > > > > >
>> > > > > > > > > > > >> > > > > > > > > > > As you can see, the lateral
>> join
>> > > > version
>> > > > > > is
>> > > > > > > > more
>> > > > > > > > > > > >> simple
>> > > > > > > > > > > >> > and
>> > > > > > > > > > > >> > > > > > > intuitive
>> > > > > > > > > > > >> > > > > > > > > to
>> > > > > > > > > > > >> > > > > > > > > > > users. Users do not have to
>> wrap a
>> > > > > > > > > > > >> > > > > > > > > > > LookupTableSource for the
>> purpose
>> > of
>> > > > > using
>> > > > > > > > async
>> > > > > > > > > > > udtf.
>> > > > > > > > > > > >> > > > > > > > > > >
>> > > > > > > > > > > >> > > > > > > > > > > In the end, We can also see the
>> > user
>> > > > > > defined
>> > > > > > > > > async
>> > > > > > > > > > > >> table
>> > > > > > > > > > > >> > > > > function
>> > > > > > > > > > > >> > > > > > > is
>> > > > > > > > > > > >> > > > > > > > an
>> > > > > > > > > > > >> > > > > > > > > > > enhancement of the current
>> lateral
>> > > > table
>> > > > > > > join
>> > > > > > > > > > > >> > > > > > > > > > > which only supports sync
>> lateral
>> > > join
>> > > > > now.
>> > > > > > > > > > > >> > > > > > > > > > >
>> > > > > > > > > > > >> > > > > > > > > > > Best,
>> > > > > > > > > > > >> > > > > > > > > > > Aitozi.
>> > > > > > > > > > > >> > > > > > > > > > >
>> > > > > > > > > > > >> > > > > > > > > > >
>> > > > > > > > > > > >> > > > > > > > > > > Jing Ge
>> > <jing@ververica.com.invalid
>> > > >
>> > > > > > > > > 于2023年6月2日周五
>> > > > > > > > > > > >> > 19:37写道：
>> > > > > > > > > > > >> > > > > > > > > > >
>> > > > > > > > > > > >> > > > > > > > > > >> Hi Aitozi,
>> > > > > > > > > > > >> > > > > > > > > > >>
>> > > > > > > > > > > >> > > > > > > > > > >> Thanks for the update. Just
>> out
>> > of
>> > > > > > > curiosity,
>> > > > > > > > > > what
>> > > > > > > > > > > is
>> > > > > > > > > > > >> > the
>> > > > > > > > > > > >> > > > > > > difference
>> > > > > > > > > > > >> > > > > > > > > > >> between the RPC call or query
>> you
>> > > > > > mentioned
>> > > > > > > > and
>> > > > > > > > > > the
>> > > > > > > > > > > >> > lookup
>> > > > > > > > > > > >> > > > in
>> > > > > > > > > > > >> > > > > a
>> > > > > > > > > > > >> > > > > > > very
>> > > > > > > > > > > >> > > > > > > > > > >> general way? Since Lateral
>> join
>> > is
>> > > > used
>> > > > > > in
>> > > > > > > > the
>> > > > > > > > > > > FLIP.
>> > > > > > > > > > > >> Is
>> > > > > > > > > > > >> > > > there
>> > > > > > > > > > > >> > > > > > any
>> > > > > > > > > > > >> > > > > > > > > > special
>> > > > > > > > > > > >> > > > > > > > > > >> thought for that? Sorry for
>> > asking
>> > > so
>> > > > > > many
>> > > > > > > > > > > questions.
>> > > > > > > > > > > >> > The
>> > > > > > > > > > > >> > > > FLIP
>> > > > > > > > > > > >> > > > > > > > > contains
>> > > > > > > > > > > >> > > > > > > > > > >> limited information to
>> understand
>> > > the
>> > > > > > > > > motivation.
>> > > > > > > > > > > >> > > > > > > > > > >>
>> > > > > > > > > > > >> > > > > > > > > > >> Best regards,
>> > > > > > > > > > > >> > > > > > > > > > >> Jing
>> > > > > > > > > > > >> > > > > > > > > > >>
>> > > > > > > > > > > >> > > > > > > > > > >> On Fri, Jun 2, 2023 at 3:48 AM
>> > > > Aitozi <
>> > > > > > > > > > > >> > > gjying1314@gmail.com
>> > > > > > > > > > > >> > > > >
>> > > > > > > > > > > >> > > > > > > wrote:
>> > > > > > > > > > > >> > > > > > > > > > >>
>> > > > > > > > > > > >> > > > > > > > > > >> > Hi Jing,
>> > > > > > > > > > > >> > > > > > > > > > >> >     I have updated the
>> proposed
>> > > > > changes
>> > > > > > > to
>> > > > > > > > > the
>> > > > > > > > > > > >> FLIP.
>> > > > > > > > > > > >> > > IMO,
>> > > > > > > > > > > >> > > > > > lookup
>> > > > > > > > > > > >> > > > > > > > has
>> > > > > > > > > > > >> > > > > > > > > > its
>> > > > > > > > > > > >> > > > > > > > > > >> > clear
>> > > > > > > > > > > >> > > > > > > > > > >> > async call requirement is
>> due
>> > to
>> > > > its
>> > > > > IO
>> > > > > > > > heavy
>> > > > > > > > > > > >> > operator.
>> > > > > > > > > > > >> > > In
>> > > > > > > > > > > >> > > > > our
>> > > > > > > > > > > >> > > > > > > > > usage,
>> > > > > > > > > > > >> > > > > > > > > > >> sql
>> > > > > > > > > > > >> > > > > > > > > > >> > users have
>> > > > > > > > > > > >> > > > > > > > > > >> > logic to do some RPC call or
>> > > query
>> > > > > the
>> > > > > > > > > > > third-party
>> > > > > > > > > > > >> > > service
>> > > > > > > > > > > >> > > > > > which
>> > > > > > > > > > > >> > > > > > > > is
>> > > > > > > > > > > >> > > > > > > > > > >> also IO
>> > > > > > > > > > > >> > > > > > > > > > >> > intensive.
>> > > > > > > > > > > >> > > > > > > > > > >> > In these case, we'd like to
>> > > > leverage
>> > > > > > the
>> > > > > > > > > async
>> > > > > > > > > > > >> > function
>> > > > > > > > > > > >> > > to
>> > > > > > > > > > > >> > > > > > > improve
>> > > > > > > > > > > >> > > > > > > > > the
>> > > > > > > > > > > >> > > > > > > > > > >> > throughput.
>> > > > > > > > > > > >> > > > > > > > > > >> >
>> > > > > > > > > > > >> > > > > > > > > > >> > Thanks,
>> > > > > > > > > > > >> > > > > > > > > > >> > Aitozi.
>> > > > > > > > > > > >> > > > > > > > > > >> >
>> > > > > > > > > > > >> > > > > > > > > > >> > Jing Ge
>> > > <jing@ververica.com.invalid
>> > > > >
>> > > > > > > > > > > 于2023年6月1日周四
>> > > > > > > > > > > >> > > > 22:55写道：
>> > > > > > > > > > > >> > > > > > > > > > >> >
>> > > > > > > > > > > >> > > > > > > > > > >> > > Hi Aitozi,
>> > > > > > > > > > > >> > > > > > > > > > >> > >
>> > > > > > > > > > > >> > > > > > > > > > >> > > Sorry for the late reply.
>> > Would
>> > > > you
>> > > > > > > like
>> > > > > > > > to
>> > > > > > > > > > > >> update
>> > > > > > > > > > > >> > the
>> > > > > > > > > > > >> > > > > > > proposed
>> > > > > > > > > > > >> > > > > > > > > > >> changes
>> > > > > > > > > > > >> > > > > > > > > > >> > > with more details into the
>> > FLIP
>> > > > > too?
>> > > > > > > > > > > >> > > > > > > > > > >> > > I got your point. It looks
>> > > like a
>> > > > > > > > rational
>> > > > > > > > > > > idea.
>> > > > > > > > > > > >> > > > However,
>> > > > > > > > > > > >> > > > > > > since
>> > > > > > > > > > > >> > > > > > > > > > lookup
>> > > > > > > > > > > >> > > > > > > > > > >> > has
>> > > > > > > > > > > >> > > > > > > > > > >> > > its clear async call
>> > > requirement,
>> > > > > are
>> > > > > > > > there
>> > > > > > > > > > any
>> > > > > > > > > > > >> real
>> > > > > > > > > > > >> > > use
>> > > > > > > > > > > >> > > > > > cases
>> > > > > > > > > > > >> > > > > > > > > that
>> > > > > > > > > > > >> > > > > > > > > > >> > > need this change? This
>> will
>> > > help
>> > > > us
>> > > > > > > > > > understand
>> > > > > > > > > > > >> the
>> > > > > > > > > > > >> > > > > > motivation.
>> > > > > > > > > > > >> > > > > > > > > After
>> > > > > > > > > > > >> > > > > > > > > > >> all,
>> > > > > > > > > > > >> > > > > > > > > > >> > > lateral join and temporal
>> > > lookup
>> > > > > > > join[1]
>> > > > > > > > > are
>> > > > > > > > > > > >> quite
>> > > > > > > > > > > >> > > > > > different.
>> > > > > > > > > > > >> > > > > > > > > > >> > >
>> > > > > > > > > > > >> > > > > > > > > > >> > > Best regards,
>> > > > > > > > > > > >> > > > > > > > > > >> > > Jing
>> > > > > > > > > > > >> > > > > > > > > > >> > >
>> > > > > > > > > > > >> > > > > > > > > > >> > >
>> > > > > > > > > > > >> > > > > > > > > > >> > > [1]
>> > > > > > > > > > > >> > > > > > > > > > >> > >
>> > > > > > > > > > > >> > > > > > > > > > >> > >
>> > > > > > > > > > > >> > > > > > > > > > >> >
>> > > > > > > > > > > >> > > > > > > > > > >>
>> > > > > > > > > > > >> > > > > > > > > >
>> > > > > > > > > > > >> > > > > > > > >
>> > > > > > > > > > > >> > > > > > > >
>> > > > > > > > > > > >> > > > > > >
>> > > > > > > > > > > >> > > > > >
>> > > > > > > > > > > >> > > > >
>> > > > > > > > > > > >> > > >
>> > > > > > > > > > > >> > >
>> > > > > > > > > > > >> >
>> > > > > > > > > > > >>
>> > > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> https://github.com/apache/flink/blob/d90a72da2fd601ca4e2a46700e91ec5b348de2ad/flink-table/flink-table-common/src/main/java/org/apache/flink/table/functions/AsyncTableFunction.java#L54
>> > > > > > > > > > > >> > > > > > > > > > >> > >
>> > > > > > > > > > > >> > > > > > > > > > >> > > On Wed, May 31, 2023 at
>> > 8:53 AM
>> > > > > > Aitozi
>> > > > > > > <
>> > > > > > > > > > > >> > > > > > gjying1314@gmail.com>
>> > > > > > > > > > > >> > > > > > > > > > wrote:
>> > > > > > > > > > > >> > > > > > > > > > >> > >
>> > > > > > > > > > > >> > > > > > > > > > >> > > > Hi Jing,
>> > > > > > > > > > > >> > > > > > > > > > >> > > >     What do you think
>> about
>> > > it?
>> > > > > Can
>> > > > > > > we
>> > > > > > > > > move
>> > > > > > > > > > > >> > forward
>> > > > > > > > > > > >> > > > this
>> > > > > > > > > > > >> > > > > > > > > feature?
>> > > > > > > > > > > >> > > > > > > > > > >> > > >
>> > > > > > > > > > > >> > > > > > > > > > >> > > > Thanks,
>> > > > > > > > > > > >> > > > > > > > > > >> > > > Aitozi.
>> > > > > > > > > > > >> > > > > > > > > > >> > > >
>> > > > > > > > > > > >> > > > > > > > > > >> > > > Aitozi <
>> > gjying1314@gmail.com
>> > > >
>> > > > > > > > > > 于2023年5月29日周一
>> > > > > > > > > > > >> > > 09:56写道：
>> > > > > > > > > > > >> > > > > > > > > > >> > > >
>> > > > > > > > > > > >> > > > > > > > > > >> > > > > Hi Jing,
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >     > "Do you mean to
>> > > support
>> > > > > the
>> > > > > > > > > > > >> > > AyncTableFunction
>> > > > > > > > > > > >> > > > > > beyond
>> > > > > > > > > > > >> > > > > > > > the
>> > > > > > > > > > > >> > > > > > > > > > >> > > > > LookupTableSource?"
>> > > > > > > > > > > >> > > > > > > > > > >> > > > > Yes, I mean to support
>> > the
>> > > > > > > > > > > AyncTableFunction
>> > > > > > > > > > > >> > > beyond
>> > > > > > > > > > > >> > > > > the
>> > > > > > > > > > > >> > > > > > > > > > >> > > > LookupTableSource.
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >
>> > > > > > > > > > > >> > > > > > > > > > >> > > > > The
>> "AsyncTableFunction"
>> > is
>> > > > the
>> > > > > > > > > function
>> > > > > > > > > > > with
>> > > > > > > > > > > >> > > > ability
>> > > > > > > > > > > >> > > > > to
>> > > > > > > > > > > >> > > > > > > be
>> > > > > > > > > > > >> > > > > > > > > > >> executed
>> > > > > > > > > > > >> > > > > > > > > > >> > > > async
>> > > > > > > > > > > >> > > > > > > > > > >> > > > > (with
>> AsyncWaitOperator).
>> > > > > > > > > > > >> > > > > > > > > > >> > > > > The async lookup join
>> is
>> > a
>> > > > one
>> > > > > of
>> > > > > > > > usage
>> > > > > > > > > > of
>> > > > > > > > > > > >> this.
>> > > > > > > > > > > >> > > So,
>> > > > > > > > > > > >> > > > > we
>> > > > > > > > > > > >> > > > > > > > don't
>> > > > > > > > > > > >> > > > > > > > > > >> have to
>> > > > > > > > > > > >> > > > > > > > > > >> > > > bind
>> > > > > > > > > > > >> > > > > > > > > > >> > > > > the AyncTableFunction
>> > with
>> > > > > > > > > > > LookupTableSource.
>> > > > > > > > > > > >> > > > > > > > > > >> > > > > If User-defined
>> > > > > > AsyncTableFunction
>> > > > > > > is
>> > > > > > > > > > > >> supported,
>> > > > > > > > > > > >> > > > user
>> > > > > > > > > > > >> > > > > > can
>> > > > > > > > > > > >> > > > > > > > > > directly
>> > > > > > > > > > > >> > > > > > > > > > >> > use
>> > > > > > > > > > > >> > > > > > > > > > >> > > > > lateral table syntax
>> to
>> > > > perform
>> > > > > > > async
>> > > > > > > > > > > >> operation.
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >
>> > > > > > > > > > > >> > > > > > > > > > >> > > > > > "It would be better
>> if
>> > > you
>> > > > > > could
>> > > > > > > > > > > elaborate
>> > > > > > > > > > > >> the
>> > > > > > > > > > > >> > > > > > proposed
>> > > > > > > > > > > >> > > > > > > > > > changes
>> > > > > > > > > > > >> > > > > > > > > > >> wrt
>> > > > > > > > > > > >> > > > > > > > > > >> > > the
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >
>> CorrelatedCodeGenerator
>> > > with
>> > > > > more
>> > > > > > > > > > details"
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >
>> > > > > > > > > > > >> > > > > > > > > > >> > > > > In the proposal, we
>> use
>> > > > lateral
>> > > > > > > table
>> > > > > > > > > > > syntax
>> > > > > > > > > > > >> to
>> > > > > > > > > > > >> > > > > support
>> > > > > > > > > > > >> > > > > > > the
>> > > > > > > > > > > >> > > > > > > > > > async
>> > > > > > > > > > > >> > > > > > > > > > >> > table
>> > > > > > > > > > > >> > > > > > > > > > >> > > > > function. So the
>> planner
>> > > will
>> > > > > > also
>> > > > > > > > > treat
>> > > > > > > > > > > this
>> > > > > > > > > > > >> > > > > statement
>> > > > > > > > > > > >> > > > > > > to a
>> > > > > > > > > > > >> > > > > > > > > > >> > > > > CommonExecCorrelate
>> node.
>> > > So
>> > > > > the
>> > > > > > > > > runtime
>> > > > > > > > > > > code
>> > > > > > > > > > > >> > > should
>> > > > > > > > > > > >> > > > > be
>> > > > > > > > > > > >> > > > > > > > > > generated
>> > > > > > > > > > > >> > > > > > > > > > >> in
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >
>> CorrelatedCodeGenerator.
>> > > > > > > > > > > >> > > > > > > > > > >> > > > > In
>> > CorrelatedCodeGenerator,
>> > > > we
>> > > > > > will
>> > > > > > > > > know
>> > > > > > > > > > > the
>> > > > > > > > > > > >> > > > > > > TableFunction's
>> > > > > > > > > > > >> > > > > > > > > > Kind
>> > > > > > > > > > > >> > > > > > > > > > >> of
>> > > > > > > > > > > >> > > > > > > > > > >> > > > > `FunctionKind.Table`
>> or
>> > > > > > > > > > > >> > `FunctionKind.ASYNC_TABLE`
>> > > > > > > > > > > >> > > > > > > > > > >> > > > > For
>> > > > `FunctionKind.ASYNC_TABLE`
>> > > > > > we
>> > > > > > > > can
>> > > > > > > > > > > >> generate
>> > > > > > > > > > > >> > a
>> > > > > > > > > > > >> > > > > > > > > > >> AsyncWaitOperator
>> > > > > > > > > > > >> > > > > > > > > > >> > to
>> > > > > > > > > > > >> > > > > > > > > > >> > > > > execute the async
>> table
>> > > > > function.
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >
>> > > > > > > > > > > >> > > > > > > > > > >> > > > > Thanks,
>> > > > > > > > > > > >> > > > > > > > > > >> > > > > Aitozi.
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >
>> > > > > > > > > > > >> > > > > > > > > > >> > > > > Jing Ge
>> > > > > > <jing@ververica.com.invalid
>> > > > > > > >
>> > > > > > > > > > > >> > > 于2023年5月29日周一
>> > > > > > > > > > > >> > > > > > > 03:22写道：
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >> Hi Aitozi,
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >>
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >> Thanks for the
>> > > > clarification.
>> > > > > > The
>> > > > > > > > > naming
>> > > > > > > > > > > >> > "Lookup"
>> > > > > > > > > > > >> > > > > might
>> > > > > > > > > > > >> > > > > > > > > suggest
>> > > > > > > > > > > >> > > > > > > > > > >> > using
>> > > > > > > > > > > >> > > > > > > > > > >> > > it
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >> for table look up.
>> But
>> > > > > > > conceptually
>> > > > > > > > > what
>> > > > > > > > > > > the
>> > > > > > > > > > > >> > > eval()
>> > > > > > > > > > > >> > > > > > > method
>> > > > > > > > > > > >> > > > > > > > > will
>> > > > > > > > > > > >> > > > > > > > > > >> do
>> > > > > > > > > > > >> > > > > > > > > > >> > is
>> > > > > > > > > > > >> > > > > > > > > > >> > > to
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >> get a collection of
>> > > > > results(Row,
>> > > > > > > > > > RowData)
>> > > > > > > > > > > >> from
>> > > > > > > > > > > >> > > the
>> > > > > > > > > > > >> > > > > > given
>> > > > > > > > > > > >> > > > > > > > > keys.
>> > > > > > > > > > > >> > > > > > > > > > >> How
>> > > > > > > > > > > >> > > > > > > > > > >> > it
>> > > > > > > > > > > >> > > > > > > > > > >> > > > will
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >> be done depends on
>> the
>> > > > > > > > implementation,
>> > > > > > > > > > > i.e.
>> > > > > > > > > > > >> you
>> > > > > > > > > > > >> > > can
>> > > > > > > > > > > >> > > > > > > > implement
>> > > > > > > > > > > >> > > > > > > > > > >> your
>> > > > > > > > > > > >> > > > > > > > > > >> > own
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >> Source[1][2]. The
>> > example
>> > > in
>> > > > > the
>> > > > > > > > FLIP
>> > > > > > > > > > > >> should be
>> > > > > > > > > > > >> > > > able
>> > > > > > > > > > > >> > > > > to
>> > > > > > > > > > > >> > > > > > > be
>> > > > > > > > > > > >> > > > > > > > > > >> handled
>> > > > > > > > > > > >> > > > > > > > > > >> > in
>> > > > > > > > > > > >> > > > > > > > > > >> > > > this
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >> way.
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >>
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >> Do you mean to
>> support
>> > the
>> > > > > > > > > > > AyncTableFunction
>> > > > > > > > > > > >> > > beyond
>> > > > > > > > > > > >> > > > > the
>> > > > > > > > > > > >> > > > > > > > > > >> > > > LookupTableSource?
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >> It would be better if
>> > you
>> > > > > could
>> > > > > > > > > > elaborate
>> > > > > > > > > > > >> the
>> > > > > > > > > > > >> > > > > proposed
>> > > > > > > > > > > >> > > > > > > > > changes
>> > > > > > > > > > > >> > > > > > > > > > >> wrt
>> > > > > > > > > > > >> > > > > > > > > > >> > the
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >>
>> CorrelatedCodeGenerator
>> > > with
>> > > > > > more
>> > > > > > > > > > details.
>> > > > > > > > > > > >> > > Thanks!
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >>
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >> Best regards,
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >> Jing
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >>
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >> [1]
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >>
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >>
>> > > > > > > > > > > >> > > > > > > > > > >> > > >
>> > > > > > > > > > > >> > > > > > > > > > >> > >
>> > > > > > > > > > > >> > > > > > > > > > >> >
>> > > > > > > > > > > >> > > > > > > > > > >>
>> > > > > > > > > > > >> > > > > > > > > >
>> > > > > > > > > > > >> > > > > > > > >
>> > > > > > > > > > > >> > > > > > > >
>> > > > > > > > > > > >> > > > > > >
>> > > > > > > > > > > >> > > > > >
>> > > > > > > > > > > >> > > > >
>> > > > > > > > > > > >> > > >
>> > > > > > > > > > > >> > >
>> > > > > > > > > > > >> >
>> > > > > > > > > > > >>
>> > > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> https://github.com/apache/flink/blob/678370b18e1b6c4a23e5ce08f8efd05675a0cc17/flink-table/flink-table-common/src/main/java/org/apache/flink/table/connector/source/LookupTableSource.java#L64
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >> [2]
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >>
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >>
>> > > > > > > > > > > >> > > > > > > > > > >> > > >
>> > > > > > > > > > > >> > > > > > > > > > >> > >
>> > > > > > > > > > > >> > > > > > > > > > >> >
>> > > > > > > > > > > >> > > > > > > > > > >>
>> > > > > > > > > > > >> > > > > > > > > >
>> > > > > > > > > > > >> > > > > > > > >
>> > > > > > > > > > > >> > > > > > > >
>> > > > > > > > > > > >> > > > > > >
>> > > > > > > > > > > >> > > > > >
>> > > > > > > > > > > >> > > > >
>> > > > > > > > > > > >> > > >
>> > > > > > > > > > > >> > >
>> > > > > > > > > > > >> >
>> > > > > > > > > > > >>
>> > > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> https://github.com/apache/flink/blob/678370b18e1b6c4a23e5ce08f8efd05675a0cc17/flink-table/flink-table-common/src/main/java/org/apache/flink/table/connector/source/AsyncTableFunctionProvider.java#L49
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >>
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >> On Sat, May 27, 2023
>> at
>> > > > > 9:48 AM
>> > > > > > > > > Aitozi <
>> > > > > > > > > > > >> > > > > > > > gjying1314@gmail.com
>> > > > > > > > > > > >> > > > > > > > > >
>> > > > > > > > > > > >> > > > > > > > > > >> > wrote:
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >>
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >> > Hi Jing,
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >> >     Thanks for your
>> > > > > response.
>> > > > > > As
>> > > > > > > > > > stated
>> > > > > > > > > > > in
>> > > > > > > > > > > >> > the
>> > > > > > > > > > > >> > > > > FLIP,
>> > > > > > > > > > > >> > > > > > > the
>> > > > > > > > > > > >> > > > > > > > > > >> purpose
>> > > > > > > > > > > >> > > > > > > > > > >> > of
>> > > > > > > > > > > >> > > > > > > > > > >> > > > this
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >> > FLIP is meant to
>> > support
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >> > user-defined async
>> > table
>> > > > > > > function.
>> > > > > > > > > As
>> > > > > > > > > > > >> > described
>> > > > > > > > > > > >> > > > in
>> > > > > > > > > > > >> > > > > > > flink
>> > > > > > > > > > > >> > > > > > > > > > >> document
>> > > > > > > > > > > >> > > > > > > > > > >> > > [1]
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >> >
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >> > Async table
>> functions
>> > > are
>> > > > > > > special
>> > > > > > > > > > > >> functions
>> > > > > > > > > > > >> > for
>> > > > > > > > > > > >> > > > > table
>> > > > > > > > > > > >> > > > > > > > > sources
>> > > > > > > > > > > >> > > > > > > > > > >> that
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >> perform
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >> > > a lookup.
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >> > >
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >> >
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >> > So end user can not
>> > > > directly
>> > > > > > > > define
>> > > > > > > > > > and
>> > > > > > > > > > > >> use
>> > > > > > > > > > > >> > > async
>> > > > > > > > > > > >> > > > > > table
>> > > > > > > > > > > >> > > > > > > > > > >> function
>> > > > > > > > > > > >> > > > > > > > > > >> > > now.
>> > > > > > > > > > > >> > > > > > > > > > >> > > > An
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >> > user case is
>> reported
>> > in
>> > > > [2]
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >> >
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >> > So, in conclusion,
>> no
>> > > new
>> > > > > > > > interface
>> > > > > > > > > is
>> > > > > > > > > > > >> > > > introduced,
>> > > > > > > > > > > >> > > > > > but
>> > > > > > > > > > > >> > > > > > > we
>> > > > > > > > > > > >> > > > > > > > > > >> extend
>> > > > > > > > > > > >> > > > > > > > > > >> > the
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >> > ability to support
>> > > > > > user-defined
>> > > > > > > > > async
>> > > > > > > > > > > >> table
>> > > > > > > > > > > >> > > > > function.
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >> >
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >> > [1]:
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >> >
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >> >
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >>
>> > > > > > > > > > > >> > > > > > > > > > >> > > >
>> > > > > > > > > > > >> > > > > > > > > > >> > >
>> > > > > > > > > > > >> > > > > > > > > > >> >
>> > > > > > > > > > > >> > > > > > > > > > >>
>> > > > > > > > > > > >> > > > > > > > > >
>> > > > > > > > > > > >> > > > > > > > >
>> > > > > > > > > > > >> > > > > > > >
>> > > > > > > > > > > >> > > > > > >
>> > > > > > > > > > > >> > > > > >
>> > > > > > > > > > > >> > > > >
>> > > > > > > > > > > >> > > >
>> > > > > > > > > > > >> > >
>> > > > > > > > > > > >> >
>> > > > > > > > > > > >>
>> > > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/dev/table/functions/udfs/
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >> > [2]:
>> > > > > > > > > > > >> > > > > > > > > > >> > >
>> > > > > > > > > > > >> > > > > > > >
>> > > > > > > > > > > >> >
>> > > > > > > >
>> > https://lists.apache.org/thread/qljwd40v5ntz6733cwcdr8s4z97b343b
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >> >
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >> > Thanks.
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >> > Aitozi.
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >> >
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >> >
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >> > Jing Ge
>> > > > > > > > <jing@ververica.com.invalid
>> > > > > > > > > >
>> > > > > > > > > > > >> > > > 于2023年5月27日周六
>> > > > > > > > > > > >> > > > > > > > > 06:40写道：
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >> >
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >> > > Hi Aitozi,
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >> > >
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >> > > Thanks for your
>> > > > proposal.
>> > > > > I
>> > > > > > am
>> > > > > > > > not
>> > > > > > > > > > > quite
>> > > > > > > > > > > >> > sure
>> > > > > > > > > > > >> > > > if
>> > > > > > > > > > > >> > > > > I
>> > > > > > > > > > > >> > > > > > > > > > understood
>> > > > > > > > > > > >> > > > > > > > > > >> > your
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >> > thoughts
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >> > > correctly. You
>> > > > described a
>> > > > > > > > special
>> > > > > > > > > > > case
>> > > > > > > > > > > >> > > > > > > implementation
>> > > > > > > > > > > >> > > > > > > > of
>> > > > > > > > > > > >> > > > > > > > > > the
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >> > >
>> AsyncTableFunction
>> > > with
>> > > > on
>> > > > > > > > public
>> > > > > > > > > > API
>> > > > > > > > > > > >> > > changes.
>> > > > > > > > > > > >> > > > > > Would
>> > > > > > > > > > > >> > > > > > > > you
>> > > > > > > > > > > >> > > > > > > > > > >> please
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >> elaborate
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >> > > your purpose of
>> > > writing
>> > > > a
>> > > > > > FLIP
>> > > > > > > > > > > >> according to
>> > > > > > > > > > > >> > > the
>> > > > > > > > > > > >> > > > > > FLIP
>> > > > > > > > > > > >> > > > > > > > > > >> > > > documentation[1]?
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >> > > Thanks!
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >> > >
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >> > > [1]
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >> > >
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >> > >
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >> >
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >>
>> > > > > > > > > > > >> > > > > > > > > > >> > > >
>> > > > > > > > > > > >> > > > > > > > > > >> > >
>> > > > > > > > > > > >> > > > > > > > > > >> >
>> > > > > > > > > > > >> > > > > > > > > > >>
>> > > > > > > > > > > >> > > > > > > > > >
>> > > > > > > > > > > >> > > > > > > > >
>> > > > > > > > > > > >> > > > > > > >
>> > > > > > > > > > > >> > > > > > >
>> > > > > > > > > > > >> > > > > >
>> > > > > > > > > > > >> > > > >
>> > > > > > > > > > > >> > > >
>> > > > > > > > > > > >> > >
>> > > > > > > > > > > >> >
>> > > > > > > > > > > >>
>> > > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> https://cwiki.apache.org/confluence/display/FLINK/Flink+Improvement+Proposals
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >> > >
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >> > > Best regards,
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >> > > Jing
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >> > >
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >> > > On Wed, May 24,
>> 2023
>> > > at
>> > > > > > > 1:07 PM
>> > > > > > > > > > > Aitozi <
>> > > > > > > > > > > >> > > > > > > > > > gjying1314@gmail.com
>> > > > > > > > > > > >> > > > > > > > > > >> >
>> > > > > > > > > > > >> > > > > > > > > > >> > > > wrote:
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >> > >
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >> > > > May I ask for
>> some
>> > > > > > feedback
>> > > > > > > > :D
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >> > > >
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >> > > > Thanks,
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >> > > > Aitozi
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >> > > >
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >> > > > Aitozi <
>> > > > > > > gjying1314@gmail.com>
>> > > > > > > > > > > >> > > 于2023年5月23日周二
>> > > > > > > > > > > >> > > > > > > 19:14写道：
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >> > > > >
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >> > > > > Just catch an
>> > user
>> > > > > case
>> > > > > > > > report
>> > > > > > > > > > > from
>> > > > > > > > > > > >> > > Giannis
>> > > > > > > > > > > >> > > > > > > Polyzos
>> > > > > > > > > > > >> > > > > > > > > for
>> > > > > > > > > > > >> > > > > > > > > > >> this
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >> usage:
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >> > > > >
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >> > > > >
>> > > > > > > > > > > >> > > > > > > > > > >> > > >
>> > > > > > > > > > > >> > > > > > > > >
>> > > > > > > > > > > >> > >
>> > > > > > > > >
>> > > https://lists.apache.org/thread/qljwd40v5ntz6733cwcdr8s4z97b343b
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >> > > > >
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >> > > > > Aitozi <
>> > > > > > > > gjying1314@gmail.com>
>> > > > > > > > > > > >> > > > 于2023年5月23日周二
>> > > > > > > > > > > >> > > > > > > > 17:45写道：
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >> > > > > >
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >> > > > > > Hi guys,
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >> > > > > >     I want
>> to
>> > > > bring
>> > > > > > up a
>> > > > > > > > > > > >> discussion
>> > > > > > > > > > > >> > > about
>> > > > > > > > > > > >> > > > > > > adding
>> > > > > > > > > > > >> > > > > > > > > > >> support
>> > > > > > > > > > > >> > > > > > > > > > >> > of
>> > > > > > > > > > > >> > > > > > > > > > >> > > > User
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >> > > > > > Defined
>> > > > > > > AsyncTableFunction
>> > > > > > > > > in
>> > > > > > > > > > > >> Flink.
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >> > > > > > Currently,
>> > async
>> > > > > table
>> > > > > > > > > > function
>> > > > > > > > > > > >> are
>> > > > > > > > > > > >> > > > special
>> > > > > > > > > > > >> > > > > > > > > functions
>> > > > > > > > > > > >> > > > > > > > > > >> for
>> > > > > > > > > > > >> > > > > > > > > > >> > > > table
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >> > > source
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >> > > > > > to perform
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >> > > > > > async
>> lookup.
>> > > > > However,
>> > > > > > > > it's
>> > > > > > > > > > > worth
>> > > > > > > > > > > >> to
>> > > > > > > > > > > >> > > > > support
>> > > > > > > > > > > >> > > > > > > the
>> > > > > > > > > > > >> > > > > > > > > user
>> > > > > > > > > > > >> > > > > > > > > > >> > > defined
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >> async
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >> > > > > > table
>> > function.
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >> > > > > > Because, in
>> > this
>> > > > > way,
>> > > > > > > the
>> > > > > > > > > end
>> > > > > > > > > > > SQL
>> > > > > > > > > > > >> > user
>> > > > > > > > > > > >> > > > can
>> > > > > > > > > > > >> > > > > > > > leverage
>> > > > > > > > > > > >> > > > > > > > > > it
>> > > > > > > > > > > >> > > > > > > > > > >> to
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >> perform
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >> > the
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >> > > > > > async
>> > operation
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >> > > > > > which is
>> > useful
>> > > to
>> > > > > > > maximum
>> > > > > > > > > the
>> > > > > > > > > > > >> system
>> > > > > > > > > > > >> > > > > > > throughput
>> > > > > > > > > > > >> > > > > > > > > > >> > especially
>> > > > > > > > > > > >> > > > > > > > > > >> > > > for
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >> IO
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >> > > > > > bottleneck
>> > case.
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >> > > > > >
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >> > > > > > You can
>> find
>> > > some
>> > > > > more
>> > > > > > > > > detail
>> > > > > > > > > > in
>> > > > > > > > > > > >> [1].
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >> > > > > >
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >> > > > > > Looking
>> > forward
>> > > to
>> > > > > > > > feedback
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >> > > > > >
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >> > > > > >
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >> > > > > > [1]:
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >> > > >
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >> > >
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >> >
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >>
>> > > > > > > > > > > >> > > > > > > > > > >> > > >
>> > > > > > > > > > > >> > > > > > > > > > >> > >
>> > > > > > > > > > > >> > > > > > > > > > >> >
>> > > > > > > > > > > >> > > > > > > > > > >>
>> > > > > > > > > > > >> > > > > > > > > >
>> > > > > > > > > > > >> > > > > > > > >
>> > > > > > > > > > > >> > > > > > > >
>> > > > > > > > > > > >> > > > > > >
>> > > > > > > > > > > >> > > > > >
>> > > > > > > > > > > >> > > > >
>> > > > > > > > > > > >> > > >
>> > > > > > > > > > > >> > >
>> > > > > > > > > > > >> >
>> > > > > > > > > > > >>
>> > > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> https://cwiki.apache.org/confluence/display/FLINK/%5BFLIP-313%5D+Add+support+of+User+Defined+AsyncTableFunction
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >> > > > > >
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >> > > > > > Thanks,
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >> > > > > > Aitozi.
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >> > > >
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >> > >
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >> >
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >>
>> > > > > > > > > > > >> > > > > > > > > > >> > > > >
>> > > > > > > > > > > >> > > > > > > > > > >> > > >
>> > > > > > > > > > > >> > > > > > > > > > >> > >
>> > > > > > > > > > > >> > > > > > > > > > >> >
>> > > > > > > > > > > >> > > > > > > > > > >>
>> > > > > > > > > > > >> > > > > > > > > > >
>> > > > > > > > > > > >> > > > > > > > > >
>> > > > > > > > > > > >> > > > > > > > >
>> > > > > > > > > > > >> > > > > > > >
>> > > > > > > > > > > >> > > > > > >
>> > > > > > > > > > > >> > > > > >
>> > > > > > > > > > > >> > > > >
>> > > > > > > > > > > >> > > >
>> > > > > > > > > > > >> > >
>> > > > > > > > > > > >> >
>> > > > > > > > > > > >>
>> > > > > > > > > > > >
>> > > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>>
>

Re: [DISCUSS] FLIP-313 Add support of User Defined AsyncTableFunction

Posted by Aitozi <gj...@gmail.com>.

Hi Lincoln,
    Thanks for your confirmation. I have updated the consensus to the FLIP
doc.
If there are no other comments, I'd like to restart the vote process in [1]
today.

https://lists.apache.org/thread/7g5n2vshosom2dj9bp7x4n01okrnx4xx

Thanks,
Aitozi.

Lincoln Lee <li...@gmail.com> 于2023年6月21日周三 22:29写道：

> Hi Aitozi,
>
> Thanks for your updates!
>
> By the design of hints, the hints after select clause belong to the query
> hints category, and this new hint is also a kind of join hints[1].
> Join table function is one of the join type defined by flink sql joins[2],
> all existing join hints[1] omit the 'join' keyword,
> so I would prefer the 'ASYNC_TABLE_FUNC' (which is actually the one for
> 'ASYNC_TABLE_FUNC_JOIN').
>
> Since a short Chinese holiday is coming, I suggest waiting for other
> people's responses before continuing to vote (next monday?)
>
> Btw, I discussed with @fudian offline about pyflink support, there should
> be no known issues, so you can create a subtask with pyflink support after
> the vote passed.
>
> [1]
>
> https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/sql/queries/hints/#join-hints
> [2]
>
> https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/sql/queries/joins/
>
> Best,
> Lincoln Lee
>
>
> Aitozi <gj...@gmail.com> 于2023年6月18日周日 21:18写道：
>
> > Hi all,
> >     Sorry for the late reply, I have a discussion with Lincoln offline,
> > mainly about
> > the naming of the hints option. Thanks Lincoln for the valuable
> > suggestions.
> >
> > Let me answer the last email inline.
> >
> > >For `JavaAsyncTableFunc0` in flip, can you use a scenario like RPC call
> as
> > an example?
> >
> > Sure, will give an example when adding the doc of async udtf and will
> > update the FLIP simultaneously
> >
> > >For the name of this query hint, 'LATERAL' (include its internal
> options)
> > don't show any relevance to async, but I haven't thought of a suitable
> name
> > at the moment,
> >
> > After some discussion with Lincoln, We prefer to choose one of the
> > `ASYNC_TABLE_FUNC` and `ASYNC_LATERAL`.
> > Besides, In my opinion the keyword `lateral`'s use scenario is wider than
> > the table function join, but in this case we only want to config
> > the async table function, So I'm a bit more lean to the
> `ASYNC_TABLE_FUNC`.
> > Looking forward to some inputs if you guys have
> > some better suggestion on the naming.
> >
> > For the usage of the hints config option, I have updated the section
> > of ConfigOption, you can refer to the FLIP
> > for more details.
> >
> > >Also, the terms 'correlate join' and 'lateral join' are not the same as
> in
> > the current joins page[1], so maybe it would be better if we unified them
> > into  'join table function'
> >
> > Yes, we should unified to the 'join table function', updated.
> >
> > Best,
> > Aitozi
> >
> > Lincoln Lee <li...@gmail.com> 于2023年6月15日周四 09:15写道：
> >
> > > Hi Aitozi,
> > >
> > > Thanks for your reply!  Gives sql users more flexibility to get
> > > asynchronous processing capabilities via lateral join table function +1
> > for
> > > this
> > >
> > > For `JavaAsyncTableFunc0` in flip, can you use a scenario like RPC call
> > as
> > > an example?
> > >
> > > For the name of this query hint, 'LATERAL' (include its internal
> options)
> > > don't show any relevance to async, but I haven't thought of a suitable
> > name
> > > at the moment,
> > > maybe we need to highlight the async keyword directly, we can also see
> if
> > > others have better candidates
> > >
> > > For the hint option "timeout = '180s'" should be "'timeout' = '180s'",
> > > seems a typo in the flip. And use upper case for all keywords in sql
> > > examples.
> > > Also, the terms 'correlate join' and 'lateral join' are not the same as
> > in
> > > the current joins page[1], so maybe it would be better if we unified
> them
> > > into  'join table function'
> > >
> > > [1]
> > >
> > >
> >
> https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/sql/queries/joins/#table-function
> > >
> > > Best,
> > > Lincoln Lee
> > >
> > >
> > > Aitozi <gj...@gmail.com> 于2023年6月14日周三 16:11写道：
> > >
> > > > Hi Lincoln
> > > >
> > > >     Very thanks for your valuable question. I will try to answer your
> > > > questions inline.
> > > >
> > > > >Does the async udtf bring any additional benefits besides a
> > > > lighter implementation？
> > > >
> > > > IMO, async udtf is more than a lighter implementation. It can act as
> a
> > > > general way for sql users to use the async operator. And they don't
> > have
> > > to
> > > > bind the async function with a table (a LookupTable), and they are
> not
> > > > forced to join on an equality join condition, and they can use it to
> do
> > > > more than enrich data.
> > > >
> > > > The async lookup join is more like a subset/specific usage of async
> > udtf.
> > > > The specific version has more opportunity to be optimized (like push
> > > down)
> > > > is acceptable. Async table function should be categorized to
> > used-defined
> > > > function.
> > > >
> > > > >Should users
> > > >
> > > > migrate to the lookup source when they encounter similar requirements
> > or
> > > >
> > > > problems, or should we develop an additional set of similar
> mechanisms？
> > > >
> > > > As I clarified above, the lookup join is a specific usage of async
> > udtf.
> > > So
> > > > it deserves more refined optimization like caching / retryable. But
> it
> > > may
> > > > not all
> > > >
> > > > suitable for the async udtf. As function, it can be deterministic/or
> > > > non-deterministic. So caching is not suitable, and we also do not
> have
> > a
> > > > common cache for the udf now. So I think optimization like
> > caching/retry
> > > > should be handed over to the function implementor.
> > > >
> > > > > the newly added query hint need a different name that
> > > > can be easier related to the lateral operation as the current join
> > > hints[5]
> > > > do.
> > > >
> > > >
> > > > What about using LATERAL?
> > > >
> > > > as below
> > > >
> > > > SELECT /*+ LATERAL('output-mode' = 'ordered', 'capacity' = '200',
> > > timeout =
> > > > '180s') */ a, c1, c2
> > > >
> > > > FROM T1
> > > >
> > > > LEFT JOIN lateral TABLE (async_split(b)) AS T(c1, c2) ON true
> > > >
> > > > >For the async func example, since the target scenario is an external
> > io
> > > > operation, it's better to add the `close` method to actively release
> > > > resources as a good example for users
> > > >
> > > >
> > > > Make sense to me, will update the FLIP
> > > >
> > > > Best,
> > > >
> > > > Aitozi.
> > > >
> > > > Lincoln Lee <li...@gmail.com> 于2023年6月14日周三 14:24写道：
> > > >
> > > > > Hi Aitozi,
> > > > >
> > > > > Sorry for the lately reply here!  Supports async
> > > > udtf(`AsyncTableFunction`)
> > > > > directly in sql seems like an attractive feature, but there're two
> > > issues
> > > > > that need to be addressed before we can be sure to add it:
> > > > > 1. As mentioned in the flip[1], the current lookup function can
> > already
> > > > > implement the requirements, but it requires implementing an extra
> > > > > `LookupTableSource` and explicitly declaring the table schema
> (which
> > > can
> > > > > help implementers the various push-down optimizations supported by
> > the
> > > > > planner). Does the async udtf bring any additional benefits
> besides a
> > > > > lighter implementation?
> > > > > 2. FLIP-221[2] abstracts a reusable cache and metric infrastructure
> > for
> > > > > lookup sources, which are important to improve performance and
> > > > > observability for high overhead external io scenarios, how do we
> > > > integrate
> > > > > and reuse these capabilities after introducing async udtf? Should
> > users
> > > > > migrate to the lookup source when they encounter similar
> requirements
> > > or
> > > > > problems, or should we develop an additional set of similar
> > mechanisms?
> > > > (a
> > > > > similarly case:  FLIP-234[3] introduced the retryable capability
> for
> > > > lookup
> > > > > join)
> > > > >
> > > > > For the flip itself,
> > > > > 1. Considering the 'options' is already used as the dynamic table
> > > > > options[4] in flink, the newly added query hint need a different
> name
> > > > that
> > > > > can be easier related to the lateral operation as the current join
> > > > hints[5]
> > > > > do.
> > > > > 2. For the async func example, since the target scenario is an
> > external
> > > > io
> > > > > operation, it's better to add the `close` method to actively
> release
> > > > > resources as a good example for users. Also in terms of the
> > determinism
> > > > of
> > > > > a function, it is important to remind users that unless the
> behavior
> > of
> > > > the
> > > > > function is deterministic, it needs to be explicitly declared as
> > > > > non-deterministic.
> > > > >
> > > > > [1].
> > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-313%3A+Add+support+of+User+Defined+AsyncTableFunction?src=contextnavpagetreemode
> > > > > [2].
> > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-221%3A+Abstraction+for+lookup+source+cache+and+metric?src=contextnavpagetreemode
> > > > > [3].
> > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-234%3A+Support+Retryable+Lookup+Join+To+Solve+Delayed+Updates+Issue+In+External+Systems?src=contextnavpagetreemode
> > > > > [4].
> > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-113%3A+Supports+Dynamic+Table+Options+for+Flink+SQL?src=contextnavpagetreemode
> > > > > [5].
> > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-229%3A+Introduces+Join+Hint+for+Flink+SQL+Batch+Job?src=contextnavpagetreemode
> > > > >
> > > > > Best,
> > > > > Lincoln Lee
> > > > >
> > > > >
> > > > > Aitozi <gj...@gmail.com> 于2023年6月13日周二 11:30写道：
> > > > >
> > > > > > Get your meaning now, thanks :)
> > > > > >
> > > > > > Best,
> > > > > > Aitozi.
> > > > > >
> > > > > > Feng Jin <ji...@gmail.com> 于2023年6月13日周二 11:16写道：
> > > > > >
> > > > > > > Hi Aitozi,
> > > > > > >
> > > > > > > Sorry for the confusing description.
> > > > > > >
> > > > > > > What I meant was that if we need to remind users about tire
> > safety
> > > > > > issues,
> > > > > > > we should introduce the new UDTF interface instead of executing
> > the
> > > > > > > original UDTF asynchronously. Therefore, I agree with
> introducing
> > > the
> > > > > > > AsyncTableFunction.
> > > > > > >
> > > > > > > Best,
> > > > > > > Feng
> > > > > > >
> > > > > > > On Tue, Jun 13, 2023 at 10:42 AM Aitozi <gj...@gmail.com>
> > > > wrote:
> > > > > > >
> > > > > > > > Hi Feng,
> > > > > > > >     Thanks for your question. We do not provide a way to
> switch
> > > the
> > > > > > UDTF
> > > > > > > > between sync and async way,
> > > > > > > > So there should be no thread safety problem here.
> > > > > > > >
> > > > > > > > Best,
> > > > > > > > Aitozi
> > > > > > > >
> > > > > > > > Feng Jin <ji...@gmail.com> 于2023年6月13日周二 10:31写道：
> > > > > > > >
> > > > > > > > > Hi Aitozi, We do need to remind users about thread safety
> > > issues.
> > > > > > Thank
> > > > > > > > you
> > > > > > > > > for your efforts on this FLIP. I have no further questions.
> > > > > > > > > Best, Feng
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Tue, Jun 13, 2023 at 6:05 AM Jing Ge
> > > > <jing@ververica.com.invalid
> > > > > >
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hi Aitozi,
> > > > > > > > > >
> > > > > > > > > > Thanks for taking care of that part. I have no other
> > concern.
> > > > > > > > > >
> > > > > > > > > > Best regards,
> > > > > > > > > > Jing
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > On Mon, Jun 12, 2023 at 5:38 PM Aitozi <
> > gjying1314@gmail.com
> > > >
> > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > BTW, If there are no other more blocking issue /
> > comments,
> > > I
> > > > > > would
> > > > > > > > like
> > > > > > > > > > to
> > > > > > > > > > > start a VOTE in another thread this wednesday 6.14
> > > > > > > > > > >
> > > > > > > > > > > Thanks,
> > > > > > > > > > > Aitozi.
> > > > > > > > > > >
> > > > > > > > > > > Aitozi <gj...@gmail.com> 于2023年6月12日周一 23:34写道：
> > > > > > > > > > >
> > > > > > > > > > > > Hi, Jing,
> > > > > > > > > > > >     Thanks for your explanation. I get your point
> now.
> > > > > > > > > > > >
> > > > > > > > > > > > For the performance part, I think it's a good idea to
> > run
> > > > > with
> > > > > > > > > > returning
> > > > > > > > > > > a
> > > > > > > > > > > > big table case, the memory consumption
> > > > > > > > > > > > should be a point to be taken care about. Because in
> > the
> > > > > > ordered
> > > > > > > > > mode,
> > > > > > > > > > > the
> > > > > > > > > > > > head element in buffer may affect the
> > > > > > > > > > > > total memory consumption.
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks,
> > > > > > > > > > > > Aitozi.
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > Jing Ge <ji...@ververica.com.invalid> 于2023年6月12日周一
> > > > 20:28写道：
> > > > > > > > > > > >
> > > > > > > > > > > >> Hi Aitozi,
> > > > > > > > > > > >>
> > > > > > > > > > > >> Which key will be used for lookup is not an issue,
> > only
> > > > one
> > > > > > row
> > > > > > > > will
> > > > > > > > > > be
> > > > > > > > > > > >> required for each key in order to enrich it. True,
> it
> > > > > depends
> > > > > > on
> > > > > > > > the
> > > > > > > > > > > >> implementation whether multiple rows or single row
> for
> > > > each
> > > > > > key
> > > > > > > > will
> > > > > > > > > > be
> > > > > > > > > > > >> returned. However, for the lookup & enrichment
> > scenario,
> > > > one
> > > > > > > > row/key
> > > > > > > > > > is
> > > > > > > > > > > >> recommended, otherwise, like I mentioned previously,
> > > > > > enrichment
> > > > > > > > > won't
> > > > > > > > > > > >> work.
> > > > > > > > > > > >>
> > > > > > > > > > > >> I am a little bit concerned about returning a big
> > table
> > > > for
> > > > > > each
> > > > > > > > > key,
> > > > > > > > > > > >> since
> > > > > > > > > > > >> it will take the async call longer to return and
> need
> > > more
> > > > > > > memory.
> > > > > > > > > The
> > > > > > > > > > > >> performance tests should cover this scenario. This
> is
> > > not
> > > > a
> > > > > > > > blocking
> > > > > > > > > > > issue
> > > > > > > > > > > >> for this FLIP.
> > > > > > > > > > > >>
> > > > > > > > > > > >> Best regards,
> > > > > > > > > > > >> Jing
> > > > > > > > > > > >>
> > > > > > > > > > > >> On Sat, Jun 10, 2023 at 4:11 AM Aitozi <
> > > > > gjying1314@gmail.com>
> > > > > > > > > wrote:
> > > > > > > > > > > >>
> > > > > > > > > > > >> > Hi Jing,
> > > > > > > > > > > >> >     I means the join key is not necessary to be
> the
> > > > > primary
> > > > > > > key
> > > > > > > > or
> > > > > > > > > > > >> unique
> > > > > > > > > > > >> > index of the database.
> > > > > > > > > > > >> > In this situation, we may queried out multi rows
> for
> > > one
> > > > > > join
> > > > > > > > > key. I
> > > > > > > > > > > >> think
> > > > > > > > > > > >> > that's why the
> > > > > > > > > > > >> > LookupFunction#lookup will return a collection of
> > > > RowData.
> > > > > > > > > > > >> >
> > > > > > > > > > > >> > BTW, I think the behavior of lookup join will not
> > > affect
> > > > > the
> > > > > > > > > > semantic
> > > > > > > > > > > of
> > > > > > > > > > > >> > the async udtf.
> > > > > > > > > > > >> > We use the Async TableFunction here and the table
> > > > function
> > > > > > can
> > > > > > > > > > collect
> > > > > > > > > > > >> > multiple rows.
> > > > > > > > > > > >> >
> > > > > > > > > > > >> > Thanks,
> > > > > > > > > > > >> > Atiozi.
> > > > > > > > > > > >> >
> > > > > > > > > > > >> >
> > > > > > > > > > > >> >
> > > > > > > > > > > >> > Jing Ge <ji...@ververica.com.invalid>
> 于2023年6月10日周六
> > > > > 00:15写道：
> > > > > > > > > > > >> >
> > > > > > > > > > > >> > > Hi Aitozi,
> > > > > > > > > > > >> > >
> > > > > > > > > > > >> > > The keyRow used in this case contains all
> keys[1].
> > > > > > > > > > > >> > >
> > > > > > > > > > > >> > > Best regards,
> > > > > > > > > > > >> > > Jing
> > > > > > > > > > > >> > >
> > > > > > > > > > > >> > > [1]
> > > > > > > > > > > >> > >
> > > > > > > > > > > >> > >
> > > > > > > > > > > >> >
> > > > > > > > > > > >>
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/flink/blob/191ec6ca3943d7119f14837efe112e074d815c47/flink-table/flink-table-common/src/main/java/org/apache/flink/table/functions/LookupFunction.java#L49
> > > > > > > > > > > >> > >
> > > > > > > > > > > >> > >
> > > > > > > > > > > >> > > On Fri, Jun 9, 2023 at 3:42 PM Aitozi <
> > > > > > gjying1314@gmail.com
> > > > > > > >
> > > > > > > > > > wrote:
> > > > > > > > > > > >> > >
> > > > > > > > > > > >> > > > Hi Jing,
> > > > > > > > > > > >> > > >
> > > > > > > > > > > >> > > >      The performance test is added to the
> FLIP.
> > > > > > > > > > > >> > > >
> > > > > > > > > > > >> > > >      As I know, The lookup join can return
> multi
> > > > rows,
> > > > > > it
> > > > > > > > > > depends
> > > > > > > > > > > on
> > > > > > > > > > > >> > > > whether  the join key
> > > > > > > > > > > >> > > > is the primary key of the external database or
> > > not.
> > > > > The
> > > > > > > > > `lookup`
> > > > > > > > > > > [1]
> > > > > > > > > > > >> > will
> > > > > > > > > > > >> > > > return a collection of
> > > > > > > > > > > >> > > > joined result, and each of them will be
> > collected
> > > > > > > > > > > >> > > >
> > > > > > > > > > > >> > > >
> > > > > > > > > > > >> > > > [1]:
> > > > > > > > > > > >> > > >
> > > > > > > > > > > >> > > >
> > > > > > > > > > > >> > >
> > > > > > > > > > > >> >
> > > > > > > > > > > >>
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/flink/blob/191ec6ca3943d7119f14837efe112e074d815c47/flink-table/flink-table-common/src/main/java/org/apache/flink/table/functions/LookupFunction.java#L52
> > > > > > > > > > > >> > > >
> > > > > > > > > > > >> > > >
> > > > > > > > > > > >> > > > Thanks,
> > > > > > > > > > > >> > > > Aitozi.
> > > > > > > > > > > >> > > >
> > > > > > > > > > > >> > > > Jing Ge <ji...@ververica.com.invalid>
> > 于2023年6月9日周五
> > > > > > > 17:05写道：
> > > > > > > > > > > >> > > >
> > > > > > > > > > > >> > > > > Hi Aitozi,
> > > > > > > > > > > >> > > > >
> > > > > > > > > > > >> > > > > Thanks for the feedback. Looking forward to
> > the
> > > > > > > > performance
> > > > > > > > > > > tests.
> > > > > > > > > > > >> > > > >
> > > > > > > > > > > >> > > > > Afaik, lookup returns one row for each key
> [1]
> > > > [2].
> > > > > > > > > > > Conceptually,
> > > > > > > > > > > >> the
> > > > > > > > > > > >> > > > > lookup function is used to enrich column(s)
> > from
> > > > the
> > > > > > > > > dimension
> > > > > > > > > > > >> table.
> > > > > > > > > > > >> > > If,
> > > > > > > > > > > >> > > > > for the given key, there will be more than
> one
> > > > row,
> > > > > > > there
> > > > > > > > > will
> > > > > > > > > > > be
> > > > > > > > > > > >> no
> > > > > > > > > > > >> > > way
> > > > > > > > > > > >> > > > to
> > > > > > > > > > > >> > > > > know which row will be used to enrich the
> key.
> > > > > > > > > > > >> > > > >
> > > > > > > > > > > >> > > > > [1]
> > > > > > > > > > > >> > > > >
> > > > > > > > > > > >> > > > >
> > > > > > > > > > > >> > > >
> > > > > > > > > > > >> > >
> > > > > > > > > > > >> >
> > > > > > > > > > > >>
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/flink/blob/191ec6ca3943d7119f14837efe112e074d815c47/flink-table/flink-table-common/src/main/java/org/apache/flink/table/functions/LookupFunction.java#L49
> > > > > > > > > > > >> > > > > [2]
> > > > > > > > > > > >> > > > >
> > > > > > > > > > > >> > > > >
> > > > > > > > > > > >> > > >
> > > > > > > > > > > >> > >
> > > > > > > > > > > >> >
> > > > > > > > > > > >>
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/flink/blob/191ec6ca3943d7119f14837efe112e074d815c47/flink-table/flink-table-common/src/main/java/org/apache/flink/table/functions/TableFunction.java#L196
> > > > > > > > > > > >> > > > >
> > > > > > > > > > > >> > > > > Best regards,
> > > > > > > > > > > >> > > > > Jing
> > > > > > > > > > > >> > > > >
> > > > > > > > > > > >> > > > > On Fri, Jun 9, 2023 at 5:18 AM Aitozi <
> > > > > > > > gjying1314@gmail.com
> > > > > > > > > >
> > > > > > > > > > > >> wrote:
> > > > > > > > > > > >> > > > >
> > > > > > > > > > > >> > > > > > Hi Jing
> > > > > > > > > > > >> > > > > >     Thanks for your good questions. I have
> > > > updated
> > > > > > the
> > > > > > > > > > example
> > > > > > > > > > > >> to
> > > > > > > > > > > >> > the
> > > > > > > > > > > >> > > > > FLIP.
> > > > > > > > > > > >> > > > > >
> > > > > > > > > > > >> > > > > > > Only one row for each lookup
> > > > > > > > > > > >> > > > > > lookup can also return multi rows, based
> on
> > > the
> > > > > > query
> > > > > > > > > > result.
> > > > > > > > > > > >> [1]
> > > > > > > > > > > >> > > > > >
> > > > > > > > > > > >> > > > > > [1]:
> > > > > > > > > > > >> > > > > >
> > > > > > > > > > > >> > > > > >
> > > > > > > > > > > >> > > > >
> > > > > > > > > > > >> > > >
> > > > > > > > > > > >> > >
> > > > > > > > > > > >> >
> > > > > > > > > > > >>
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/flink/blob/191ec6ca3943d7119f14837efe112e074d815c47/flink-table/flink-table-common/src/main/java/org/apache/flink/table/functions/LookupFunction.java#L56
> > > > > > > > > > > >> > > > > >
> > > > > > > > > > > >> > > > > > > If we use async calls with lateral join,
> > my
> > > > gut
> > > > > > > > feeling
> > > > > > > > > is
> > > > > > > > > > > >> > > > > > that we might have many more async calls
> > than
> > > > > lookup
> > > > > > > > > join. I
> > > > > > > > > > > am
> > > > > > > > > > > >> not
> > > > > > > > > > > >> > > > > really
> > > > > > > > > > > >> > > > > > sure if we will be facing potential issues
> > in
> > > > this
> > > > > > > case
> > > > > > > > or
> > > > > > > > > > > not.
> > > > > > > > > > > >> > > > > >
> > > > > > > > > > > >> > > > > > IMO, the work pattern is similar to the
> > lookup
> > > > > > > function,
> > > > > > > > > for
> > > > > > > > > > > >> each
> > > > > > > > > > > >> > row
> > > > > > > > > > > >> > > > > from
> > > > > > > > > > > >> > > > > > the left table,
> > > > > > > > > > > >> > > > > > it will evaluate the eval method once, so
> > the
> > > > > async
> > > > > > > call
> > > > > > > > > > > numbers
> > > > > > > > > > > >> > will
> > > > > > > > > > > >> > > > not
> > > > > > > > > > > >> > > > > > change.
> > > > > > > > > > > >> > > > > > and the maximum calls in flight is limited
> > by
> > > > the
> > > > > > > Async
> > > > > > > > > > > >> operators
> > > > > > > > > > > >> > > > buffer
> > > > > > > > > > > >> > > > > > capacity
> > > > > > > > > > > >> > > > > > which will be controlled by the option.
> > > > > > > > > > > >> > > > > >
> > > > > > > > > > > >> > > > > > BTW, for the naming of these option, I
> > updated
> > > > the
> > > > > > > FLIP
> > > > > > > > > > about
> > > > > > > > > > > >> this
> > > > > > > > > > > >> > > you
> > > > > > > > > > > >> > > > > can
> > > > > > > > > > > >> > > > > > refer to
> > > > > > > > > > > >> > > > > > the section of "ConfigOption" and
> "Rejected
> > > > > > > > Alternatives"
> > > > > > > > > > > >> > > > > >
> > > > > > > > > > > >> > > > > > In the end, for the performance
> evaluation,
> > > I'd
> > > > > like
> > > > > > > to
> > > > > > > > do
> > > > > > > > > > > some
> > > > > > > > > > > >> > tests
> > > > > > > > > > > >> > > > and
> > > > > > > > > > > >> > > > > > will update it to the FLIP doc
> > > > > > > > > > > >> > > > > >
> > > > > > > > > > > >> > > > > > Thanks,
> > > > > > > > > > > >> > > > > > Aitozi.
> > > > > > > > > > > >> > > > > >
> > > > > > > > > > > >> > > > > >
> > > > > > > > > > > >> > > > > > Jing Ge <ji...@ververica.com.invalid>
> > > > 于2023年6月9日周五
> > > > > > > > > 07:23写道：
> > > > > > > > > > > >> > > > > >
> > > > > > > > > > > >> > > > > > > Hi Aitozi,
> > > > > > > > > > > >> > > > > > >
> > > > > > > > > > > >> > > > > > > Thanks for the clarification. The code
> > > example
> > > > > > looks
> > > > > > > > > > > >> > interesting. I
> > > > > > > > > > > >> > > > > would
> > > > > > > > > > > >> > > > > > > suggest adding them into the FLIP. The
> > > > > description
> > > > > > > > with
> > > > > > > > > > code
> > > > > > > > > > > >> > > examples
> > > > > > > > > > > >> > > > > > will
> > > > > > > > > > > >> > > > > > > help readers understand the motivation
> and
> > > how
> > > > > to
> > > > > > > use
> > > > > > > > > it.
> > > > > > > > > > > >> Afaiac,
> > > > > > > > > > > >> > > it
> > > > > > > > > > > >> > > > > is a
> > > > > > > > > > > >> > > > > > > valid feature for Flink users.
> > > > > > > > > > > >> > > > > > >
> > > > > > > > > > > >> > > > > > > As we knew, lookup join is based on
> > temporal
> > > > > join,
> > > > > > > > i.e.
> > > > > > > > > > FOR
> > > > > > > > > > > >> > > > SYSTEM_TIME
> > > > > > > > > > > >> > > > > > AS
> > > > > > > > > > > >> > > > > > > OF which is also used in your code
> > example.
> > > > > > Temporal
> > > > > > > > > join
> > > > > > > > > > > >> > performs
> > > > > > > > > > > >> > > > the
> > > > > > > > > > > >> > > > > > > lookup based on the processing time
> match.
> > > > Only
> > > > > > one
> > > > > > > > row
> > > > > > > > > > for
> > > > > > > > > > > >> each
> > > > > > > > > > > >> > > > > > > lookup(afaiu, I need to check the source
> > > code
> > > > to
> > > > > > > > double
> > > > > > > > > > > >> confirm)
> > > > > > > > > > > >> > > will
> > > > > > > > > > > >> > > > > > > return for further enrichment. One the
> > other
> > > > > hand,
> > > > > > > > > lateral
> > > > > > > > > > > >> join
> > > > > > > > > > > >> > > will
> > > > > > > > > > > >> > > > > have
> > > > > > > > > > > >> > > > > > > sub-queries correlated with every
> > individual
> > > > > value
> > > > > > > of
> > > > > > > > > the
> > > > > > > > > > > >> > reference
> > > > > > > > > > > >> > > > > table
> > > > > > > > > > > >> > > > > > > from the preceding part of the query and
> > > each
> > > > > sub
> > > > > > > > query
> > > > > > > > > > will
> > > > > > > > > > > >> > return
> > > > > > > > > > > >> > > > > > > multiple rows. If we use async calls
> with
> > > > > lateral
> > > > > > > > join,
> > > > > > > > > my
> > > > > > > > > > > gut
> > > > > > > > > > > >> > > > feeling
> > > > > > > > > > > >> > > > > is
> > > > > > > > > > > >> > > > > > > that we might have many more async calls
> > > than
> > > > > > lookup
> > > > > > > > > > join. I
> > > > > > > > > > > >> am
> > > > > > > > > > > >> > not
> > > > > > > > > > > >> > > > > > really
> > > > > > > > > > > >> > > > > > > sure if we will be facing potential
> issues
> > > in
> > > > > this
> > > > > > > > case
> > > > > > > > > or
> > > > > > > > > > > >> not.
> > > > > > > > > > > >> > > > > Possible
> > > > > > > > > > > >> > > > > > > issues I can think of now e.g. too many
> > PRC
> > > > > calls,
> > > > > > > too
> > > > > > > > > > many
> > > > > > > > > > > >> async
> > > > > > > > > > > >> > > > calls
> > > > > > > > > > > >> > > > > > > processing, the sub query will return a
> > > table
> > > > > > which
> > > > > > > > > might
> > > > > > > > > > be
> > > > > > > > > > > >> > (too)
> > > > > > > > > > > >> > > > big,
> > > > > > > > > > > >> > > > > > and
> > > > > > > > > > > >> > > > > > > might cause performance issues. I would
> > > > suggest
> > > > > > > > > preparing
> > > > > > > > > > > some
> > > > > > > > > > > >> > use
> > > > > > > > > > > >> > > > > cases
> > > > > > > > > > > >> > > > > > > and running some performance tests to
> > check
> > > > it.
> > > > > > > These
> > > > > > > > > are
> > > > > > > > > > my
> > > > > > > > > > > >> > > concerns
> > > > > > > > > > > >> > > > > > about
> > > > > > > > > > > >> > > > > > > using async calls with lateral join and
> > I'd
> > > > like
> > > > > > to
> > > > > > > > > share
> > > > > > > > > > > with
> > > > > > > > > > > >> > you,
> > > > > > > > > > > >> > > > > happy
> > > > > > > > > > > >> > > > > > > to discuss with you and hear different
> > > > opinions,
> > > > > > > > > hopefully
> > > > > > > > > > > the
> > > > > > > > > > > >> > > > > > > discussion could help me understand it
> > more
> > > > > > deeply.
> > > > > > > > > Please
> > > > > > > > > > > >> > correct
> > > > > > > > > > > >> > > me
> > > > > > > > > > > >> > > > > if
> > > > > > > > > > > >> > > > > > I
> > > > > > > > > > > >> > > > > > > am wrong.
> > > > > > > > > > > >> > > > > > >
> > > > > > > > > > > >> > > > > > > Best regards,
> > > > > > > > > > > >> > > > > > > Jing
> > > > > > > > > > > >> > > > > > >
> > > > > > > > > > > >> > > > > > >
> > > > > > > > > > > >> > > > > > > On Thu, Jun 8, 2023 at 7:22 AM Aitozi <
> > > > > > > > > > gjying1314@gmail.com
> > > > > > > > > > > >
> > > > > > > > > > > >> > > wrote:
> > > > > > > > > > > >> > > > > > >
> > > > > > > > > > > >> > > > > > > > Hi Mason,
> > > > > > > > > > > >> > > > > > > >     Thanks for your input. I think if
> we
> > > > > support
> > > > > > > the
> > > > > > > > > > user
> > > > > > > > > > > >> > defined
> > > > > > > > > > > >> > > > > async
> > > > > > > > > > > >> > > > > > > > table function,
> > > > > > > > > > > >> > > > > > > > user will be able to use it to hold a
> > > batch
> > > > > data
> > > > > > > > then
> > > > > > > > > > > >> handle it
> > > > > > > > > > > >> > > at
> > > > > > > > > > > >> > > > > one
> > > > > > > > > > > >> > > > > > > time
> > > > > > > > > > > >> > > > > > > > in the customized function.
> > > > > > > > > > > >> > > > > > > >
> > > > > > > > > > > >> > > > > > > > AsyncSink is meant for the sink
> > operator.
> > > I
> > > > > have
> > > > > > > not
> > > > > > > > > > > figure
> > > > > > > > > > > >> out
> > > > > > > > > > > >> > > how
> > > > > > > > > > > >> > > > > to
> > > > > > > > > > > >> > > > > > > > integrate in this case.
> > > > > > > > > > > >> > > > > > > >
> > > > > > > > > > > >> > > > > > > > Thanks,
> > > > > > > > > > > >> > > > > > > > Atiozi.
> > > > > > > > > > > >> > > > > > > >
> > > > > > > > > > > >> > > > > > > >
> > > > > > > > > > > >> > > > > > > > Mason Chen <ma...@gmail.com>
> > > > > > 于2023年6月8日周四
> > > > > > > > > > 12:40写道：
> > > > > > > > > > > >> > > > > > > >
> > > > > > > > > > > >> > > > > > > > > Hi Aitozi,
> > > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > > >> > > > > > > > > I think it makes sense to make it
> > easier
> > > > for
> > > > > > SQL
> > > > > > > > > users
> > > > > > > > > > > to
> > > > > > > > > > > >> > make
> > > > > > > > > > > >> > > > > RPCs.
> > > > > > > > > > > >> > > > > > Do
> > > > > > > > > > > >> > > > > > > > you
> > > > > > > > > > > >> > > > > > > > > think your proposal can extend to
> the
> > > > > ability
> > > > > > to
> > > > > > > > > batch
> > > > > > > > > > > >> data
> > > > > > > > > > > >> > for
> > > > > > > > > > > >> > > > the
> > > > > > > > > > > >> > > > > > > RPC?
> > > > > > > > > > > >> > > > > > > > > This is also another common strategy
> > to
> > > > > > increase
> > > > > > > > > > > >> throughput.
> > > > > > > > > > > >> > > > Also,
> > > > > > > > > > > >> > > > > > have
> > > > > > > > > > > >> > > > > > > > you
> > > > > > > > > > > >> > > > > > > > > considered solving this a bit
> > > differently
> > > > by
> > > > > > > > > > leveraging
> > > > > > > > > > > >> > Flink's
> > > > > > > > > > > >> > > > > > > > AsyncSink?
> > > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > > >> > > > > > > > > Best,
> > > > > > > > > > > >> > > > > > > > > Mason
> > > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > > >> > > > > > > > > On Mon, Jun 5, 2023 at 1:50 AM
> Aitozi
> > <
> > > > > > > > > > > >> gjying1314@gmail.com>
> > > > > > > > > > > >> > > > > wrote:
> > > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > > >> > > > > > > > > > One more thing for discussion:
> > > > > > > > > > > >> > > > > > > > > >
> > > > > > > > > > > >> > > > > > > > > > In our internal implementation, we
> > > reuse
> > > > > the
> > > > > > > > > option
> > > > > > > > > > > >> > > > > > > > > >
> > > > `table.exec.async-lookup.buffer-capacity`
> > > > > > and
> > > > > > > > > > > >> > > > > > > > > > `table.exec.async-lookup.timeout`
> to
> > > > > config
> > > > > > > > > > > >> > > > > > > > > > the async udtf. Do you think we
> > should
> > > > add
> > > > > > two
> > > > > > > > > extra
> > > > > > > > > > > >> option
> > > > > > > > > > > >> > > to
> > > > > > > > > > > >> > > > > > > > > distinguish
> > > > > > > > > > > >> > > > > > > > > > from the lookup option such as
> > > > > > > > > > > >> > > > > > > > > >
> > > > > > > > > > > >> > > > > > > > > >
> > > `table.exec.async-udtf.buffer-capacity`
> > > > > > > > > > > >> > > > > > > > > > `table.exec.async-udtf.timeout`
> > > > > > > > > > > >> > > > > > > > > >
> > > > > > > > > > > >> > > > > > > > > >
> > > > > > > > > > > >> > > > > > > > > > Best,
> > > > > > > > > > > >> > > > > > > > > > Aitozi.
> > > > > > > > > > > >> > > > > > > > > >
> > > > > > > > > > > >> > > > > > > > > >
> > > > > > > > > > > >> > > > > > > > > >
> > > > > > > > > > > >> > > > > > > > > > Aitozi <gj...@gmail.com>
> > > > > 于2023年6月5日周一
> > > > > > > > > 12:20写道：
> > > > > > > > > > > >> > > > > > > > > >
> > > > > > > > > > > >> > > > > > > > > > > Hi Jing,
> > > > > > > > > > > >> > > > > > > > > > >
> > > > > > > > > > > >> > > > > > > > > > >     > what is the difference
> > between
> > > > the
> > > > > > RPC
> > > > > > > > > call
> > > > > > > > > > or
> > > > > > > > > > > >> > query
> > > > > > > > > > > >> > > > you
> > > > > > > > > > > >> > > > > > > > > mentioned
> > > > > > > > > > > >> > > > > > > > > > > and the lookup in a very
> > > > > > > > > > > >> > > > > > > > > > > general way
> > > > > > > > > > > >> > > > > > > > > > >
> > > > > > > > > > > >> > > > > > > > > > > I think the RPC call or query
> > > service
> > > > is
> > > > > > > quite
> > > > > > > > > > > >> similar to
> > > > > > > > > > > >> > > the
> > > > > > > > > > > >> > > > > > > lookup
> > > > > > > > > > > >> > > > > > > > > > join.
> > > > > > > > > > > >> > > > > > > > > > > But lookup join should work
> > > > > > > > > > > >> > > > > > > > > > > with `LookupTableSource`.
> > > > > > > > > > > >> > > > > > > > > > >
> > > > > > > > > > > >> > > > > > > > > > > Let's see how we can perform an
> > > async
> > > > > RPC
> > > > > > > call
> > > > > > > > > > with
> > > > > > > > > > > >> > lookup
> > > > > > > > > > > >> > > > > join:
> > > > > > > > > > > >> > > > > > > > > > >
> > > > > > > > > > > >> > > > > > > > > > > (1) Implement an
> > AsyncTableFunction
> > > > with
> > > > > > RPC
> > > > > > > > > call
> > > > > > > > > > > >> logic.
> > > > > > > > > > > >> > > > > > > > > > > (2) Implement a
> > `LookupTableSource`
> > > > > > > connector
> > > > > > > > > run
> > > > > > > > > > > with
> > > > > > > > > > > >> > the
> > > > > > > > > > > >> > > > > async
> > > > > > > > > > > >> > > > > > > udtf
> > > > > > > > > > > >> > > > > > > > > > > defined in (1).
> > > > > > > > > > > >> > > > > > > > > > > (3) Then define a DDL of this
> look
> > > up
> > > > > > table
> > > > > > > in
> > > > > > > > > SQL
> > > > > > > > > > > >> > > > > > > > > > >
> > > > > > > > > > > >> > > > > > > > > > > CREATE TEMPORARY TABLE
> Customers (
> > > > > > > > > > > >> > > > > > > > > > >   id INT,
> > > > > > > > > > > >> > > > > > > > > > >   name STRING,
> > > > > > > > > > > >> > > > > > > > > > >   country STRING,
> > > > > > > > > > > >> > > > > > > > > > >   zip STRING
> > > > > > > > > > > >> > > > > > > > > > > ) WITH (
> > > > > > > > > > > >> > > > > > > > > > >   'connector' = 'custom'
> > > > > > > > > > > >> > > > > > > > > > > );
> > > > > > > > > > > >> > > > > > > > > > >
> > > > > > > > > > > >> > > > > > > > > > > (4) Run with the query as below:
> > > > > > > > > > > >> > > > > > > > > > >
> > > > > > > > > > > >> > > > > > > > > > > SELECT o.order_id, o.total,
> > > c.country,
> > > > > > c.zip
> > > > > > > > > > > >> > > > > > > > > > > FROM Orders AS o
> > > > > > > > > > > >> > > > > > > > > > >   JOIN Customers FOR SYSTEM_TIME
> > AS
> > > OF
> > > > > > > > > o.proc_time
> > > > > > > > > > > AS
> > > > > > > > > > > >> c
> > > > > > > > > > > >> > > > > > > > > > >     ON o.customer_id = c.id;
> > > > > > > > > > > >> > > > > > > > > > >
> > > > > > > > > > > >> > > > > > > > > > > This example is from doc
> > > > > > > > > > > >> > > > > > > > > > > <
> > > > > > > > > > > >> > > > > > > > > >
> > > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > > >> > > > > > > >
> > > > > > > > > > > >> > > > > > >
> > > > > > > > > > > >> > > > > >
> > > > > > > > > > > >> > > > >
> > > > > > > > > > > >> > > >
> > > > > > > > > > > >> > >
> > > > > > > > > > > >> >
> > > > > > > > > > > >>
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/dev/table/sql/queries/joins/#lookup-join
> > > > > > > > > > > >> > > > > > > > > > >.You
> > > > > > > > > > > >> > > > > > > > > > > can image the look up process as
> > an
> > > > > async
> > > > > > > RPC
> > > > > > > > > call
> > > > > > > > > > > >> > process.
> > > > > > > > > > > >> > > > > > > > > > >
> > > > > > > > > > > >> > > > > > > > > > > Let's see how we can perform an
> > > async
> > > > > RPC
> > > > > > > call
> > > > > > > > > > with
> > > > > > > > > > > >> > lateral
> > > > > > > > > > > >> > > > > join:
> > > > > > > > > > > >> > > > > > > > > > >
> > > > > > > > > > > >> > > > > > > > > > > (1) Implement an
> > AsyncTableFunction
> > > > with
> > > > > > RPC
> > > > > > > > > call
> > > > > > > > > > > >> logic.
> > > > > > > > > > > >> > > > > > > > > > > (2) Run query with
> > > > > > > > > > > >> > > > > > > > > > >
> > > > > > > > > > > >> > > > > > > > > > > Create function f1 as '...' ;
> > > > > > > > > > > >> > > > > > > > > > >
> > > > > > > > > > > >> > > > > > > > > > > SELECT o.order_id, o.total,
> > > c.country,
> > > > > > c.zip
> > > > > > > > > FROM
> > > > > > > > > > > >> Orders
> > > > > > > > > > > >> > > > > lateral
> > > > > > > > > > > >> > > > > > > > table
> > > > > > > > > > > >> > > > > > > > > > > (f1(order_id)) as T(...);
> > > > > > > > > > > >> > > > > > > > > > >
> > > > > > > > > > > >> > > > > > > > > > > As you can see, the lateral join
> > > > version
> > > > > > is
> > > > > > > > more
> > > > > > > > > > > >> simple
> > > > > > > > > > > >> > and
> > > > > > > > > > > >> > > > > > > intuitive
> > > > > > > > > > > >> > > > > > > > > to
> > > > > > > > > > > >> > > > > > > > > > > users. Users do not have to
> wrap a
> > > > > > > > > > > >> > > > > > > > > > > LookupTableSource for the
> purpose
> > of
> > > > > using
> > > > > > > > async
> > > > > > > > > > > udtf.
> > > > > > > > > > > >> > > > > > > > > > >
> > > > > > > > > > > >> > > > > > > > > > > In the end, We can also see the
> > user
> > > > > > defined
> > > > > > > > > async
> > > > > > > > > > > >> table
> > > > > > > > > > > >> > > > > function
> > > > > > > > > > > >> > > > > > > is
> > > > > > > > > > > >> > > > > > > > an
> > > > > > > > > > > >> > > > > > > > > > > enhancement of the current
> lateral
> > > > table
> > > > > > > join
> > > > > > > > > > > >> > > > > > > > > > > which only supports sync lateral
> > > join
> > > > > now.
> > > > > > > > > > > >> > > > > > > > > > >
> > > > > > > > > > > >> > > > > > > > > > > Best,
> > > > > > > > > > > >> > > > > > > > > > > Aitozi.
> > > > > > > > > > > >> > > > > > > > > > >
> > > > > > > > > > > >> > > > > > > > > > >
> > > > > > > > > > > >> > > > > > > > > > > Jing Ge
> > <jing@ververica.com.invalid
> > > >
> > > > > > > > > 于2023年6月2日周五
> > > > > > > > > > > >> > 19:37写道：
> > > > > > > > > > > >> > > > > > > > > > >
> > > > > > > > > > > >> > > > > > > > > > >> Hi Aitozi,
> > > > > > > > > > > >> > > > > > > > > > >>
> > > > > > > > > > > >> > > > > > > > > > >> Thanks for the update. Just out
> > of
> > > > > > > curiosity,
> > > > > > > > > > what
> > > > > > > > > > > is
> > > > > > > > > > > >> > the
> > > > > > > > > > > >> > > > > > > difference
> > > > > > > > > > > >> > > > > > > > > > >> between the RPC call or query
> you
> > > > > > mentioned
> > > > > > > > and
> > > > > > > > > > the
> > > > > > > > > > > >> > lookup
> > > > > > > > > > > >> > > > in
> > > > > > > > > > > >> > > > > a
> > > > > > > > > > > >> > > > > > > very
> > > > > > > > > > > >> > > > > > > > > > >> general way? Since Lateral join
> > is
> > > > used
> > > > > > in
> > > > > > > > the
> > > > > > > > > > > FLIP.
> > > > > > > > > > > >> Is
> > > > > > > > > > > >> > > > there
> > > > > > > > > > > >> > > > > > any
> > > > > > > > > > > >> > > > > > > > > > special
> > > > > > > > > > > >> > > > > > > > > > >> thought for that? Sorry for
> > asking
> > > so
> > > > > > many
> > > > > > > > > > > questions.
> > > > > > > > > > > >> > The
> > > > > > > > > > > >> > > > FLIP
> > > > > > > > > > > >> > > > > > > > > contains
> > > > > > > > > > > >> > > > > > > > > > >> limited information to
> understand
> > > the
> > > > > > > > > motivation.
> > > > > > > > > > > >> > > > > > > > > > >>
> > > > > > > > > > > >> > > > > > > > > > >> Best regards,
> > > > > > > > > > > >> > > > > > > > > > >> Jing
> > > > > > > > > > > >> > > > > > > > > > >>
> > > > > > > > > > > >> > > > > > > > > > >> On Fri, Jun 2, 2023 at 3:48 AM
> > > > Aitozi <
> > > > > > > > > > > >> > > gjying1314@gmail.com
> > > > > > > > > > > >> > > > >
> > > > > > > > > > > >> > > > > > > wrote:
> > > > > > > > > > > >> > > > > > > > > > >>
> > > > > > > > > > > >> > > > > > > > > > >> > Hi Jing,
> > > > > > > > > > > >> > > > > > > > > > >> >     I have updated the
> proposed
> > > > > changes
> > > > > > > to
> > > > > > > > > the
> > > > > > > > > > > >> FLIP.
> > > > > > > > > > > >> > > IMO,
> > > > > > > > > > > >> > > > > > lookup
> > > > > > > > > > > >> > > > > > > > has
> > > > > > > > > > > >> > > > > > > > > > its
> > > > > > > > > > > >> > > > > > > > > > >> > clear
> > > > > > > > > > > >> > > > > > > > > > >> > async call requirement is due
> > to
> > > > its
> > > > > IO
> > > > > > > > heavy
> > > > > > > > > > > >> > operator.
> > > > > > > > > > > >> > > In
> > > > > > > > > > > >> > > > > our
> > > > > > > > > > > >> > > > > > > > > usage,
> > > > > > > > > > > >> > > > > > > > > > >> sql
> > > > > > > > > > > >> > > > > > > > > > >> > users have
> > > > > > > > > > > >> > > > > > > > > > >> > logic to do some RPC call or
> > > query
> > > > > the
> > > > > > > > > > > third-party
> > > > > > > > > > > >> > > service
> > > > > > > > > > > >> > > > > > which
> > > > > > > > > > > >> > > > > > > > is
> > > > > > > > > > > >> > > > > > > > > > >> also IO
> > > > > > > > > > > >> > > > > > > > > > >> > intensive.
> > > > > > > > > > > >> > > > > > > > > > >> > In these case, we'd like to
> > > > leverage
> > > > > > the
> > > > > > > > > async
> > > > > > > > > > > >> > function
> > > > > > > > > > > >> > > to
> > > > > > > > > > > >> > > > > > > improve
> > > > > > > > > > > >> > > > > > > > > the
> > > > > > > > > > > >> > > > > > > > > > >> > throughput.
> > > > > > > > > > > >> > > > > > > > > > >> >
> > > > > > > > > > > >> > > > > > > > > > >> > Thanks,
> > > > > > > > > > > >> > > > > > > > > > >> > Aitozi.
> > > > > > > > > > > >> > > > > > > > > > >> >
> > > > > > > > > > > >> > > > > > > > > > >> > Jing Ge
> > > <jing@ververica.com.invalid
> > > > >
> > > > > > > > > > > 于2023年6月1日周四
> > > > > > > > > > > >> > > > 22:55写道：
> > > > > > > > > > > >> > > > > > > > > > >> >
> > > > > > > > > > > >> > > > > > > > > > >> > > Hi Aitozi,
> > > > > > > > > > > >> > > > > > > > > > >> > >
> > > > > > > > > > > >> > > > > > > > > > >> > > Sorry for the late reply.
> > Would
> > > > you
> > > > > > > like
> > > > > > > > to
> > > > > > > > > > > >> update
> > > > > > > > > > > >> > the
> > > > > > > > > > > >> > > > > > > proposed
> > > > > > > > > > > >> > > > > > > > > > >> changes
> > > > > > > > > > > >> > > > > > > > > > >> > > with more details into the
> > FLIP
> > > > > too?
> > > > > > > > > > > >> > > > > > > > > > >> > > I got your point. It looks
> > > like a
> > > > > > > > rational
> > > > > > > > > > > idea.
> > > > > > > > > > > >> > > > However,
> > > > > > > > > > > >> > > > > > > since
> > > > > > > > > > > >> > > > > > > > > > lookup
> > > > > > > > > > > >> > > > > > > > > > >> > has
> > > > > > > > > > > >> > > > > > > > > > >> > > its clear async call
> > > requirement,
> > > > > are
> > > > > > > > there
> > > > > > > > > > any
> > > > > > > > > > > >> real
> > > > > > > > > > > >> > > use
> > > > > > > > > > > >> > > > > > cases
> > > > > > > > > > > >> > > > > > > > > that
> > > > > > > > > > > >> > > > > > > > > > >> > > need this change? This will
> > > help
> > > > us
> > > > > > > > > > understand
> > > > > > > > > > > >> the
> > > > > > > > > > > >> > > > > > motivation.
> > > > > > > > > > > >> > > > > > > > > After
> > > > > > > > > > > >> > > > > > > > > > >> all,
> > > > > > > > > > > >> > > > > > > > > > >> > > lateral join and temporal
> > > lookup
> > > > > > > join[1]
> > > > > > > > > are
> > > > > > > > > > > >> quite
> > > > > > > > > > > >> > > > > > different.
> > > > > > > > > > > >> > > > > > > > > > >> > >
> > > > > > > > > > > >> > > > > > > > > > >> > > Best regards,
> > > > > > > > > > > >> > > > > > > > > > >> > > Jing
> > > > > > > > > > > >> > > > > > > > > > >> > >
> > > > > > > > > > > >> > > > > > > > > > >> > >
> > > > > > > > > > > >> > > > > > > > > > >> > > [1]
> > > > > > > > > > > >> > > > > > > > > > >> > >
> > > > > > > > > > > >> > > > > > > > > > >> > >
> > > > > > > > > > > >> > > > > > > > > > >> >
> > > > > > > > > > > >> > > > > > > > > > >>
> > > > > > > > > > > >> > > > > > > > > >
> > > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > > >> > > > > > > >
> > > > > > > > > > > >> > > > > > >
> > > > > > > > > > > >> > > > > >
> > > > > > > > > > > >> > > > >
> > > > > > > > > > > >> > > >
> > > > > > > > > > > >> > >
> > > > > > > > > > > >> >
> > > > > > > > > > > >>
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/flink/blob/d90a72da2fd601ca4e2a46700e91ec5b348de2ad/flink-table/flink-table-common/src/main/java/org/apache/flink/table/functions/AsyncTableFunction.java#L54
> > > > > > > > > > > >> > > > > > > > > > >> > >
> > > > > > > > > > > >> > > > > > > > > > >> > > On Wed, May 31, 2023 at
> > 8:53 AM
> > > > > > Aitozi
> > > > > > > <
> > > > > > > > > > > >> > > > > > gjying1314@gmail.com>
> > > > > > > > > > > >> > > > > > > > > > wrote:
> > > > > > > > > > > >> > > > > > > > > > >> > >
> > > > > > > > > > > >> > > > > > > > > > >> > > > Hi Jing,
> > > > > > > > > > > >> > > > > > > > > > >> > > >     What do you think
> about
> > > it?
> > > > > Can
> > > > > > > we
> > > > > > > > > move
> > > > > > > > > > > >> > forward
> > > > > > > > > > > >> > > > this
> > > > > > > > > > > >> > > > > > > > > feature?
> > > > > > > > > > > >> > > > > > > > > > >> > > >
> > > > > > > > > > > >> > > > > > > > > > >> > > > Thanks,
> > > > > > > > > > > >> > > > > > > > > > >> > > > Aitozi.
> > > > > > > > > > > >> > > > > > > > > > >> > > >
> > > > > > > > > > > >> > > > > > > > > > >> > > > Aitozi <
> > gjying1314@gmail.com
> > > >
> > > > > > > > > > 于2023年5月29日周一
> > > > > > > > > > > >> > > 09:56写道：
> > > > > > > > > > > >> > > > > > > > > > >> > > >
> > > > > > > > > > > >> > > > > > > > > > >> > > > > Hi Jing,
> > > > > > > > > > > >> > > > > > > > > > >> > > > >     > "Do you mean to
> > > support
> > > > > the
> > > > > > > > > > > >> > > AyncTableFunction
> > > > > > > > > > > >> > > > > > beyond
> > > > > > > > > > > >> > > > > > > > the
> > > > > > > > > > > >> > > > > > > > > > >> > > > > LookupTableSource?"
> > > > > > > > > > > >> > > > > > > > > > >> > > > > Yes, I mean to support
> > the
> > > > > > > > > > > AyncTableFunction
> > > > > > > > > > > >> > > beyond
> > > > > > > > > > > >> > > > > the
> > > > > > > > > > > >> > > > > > > > > > >> > > > LookupTableSource.
> > > > > > > > > > > >> > > > > > > > > > >> > > > >
> > > > > > > > > > > >> > > > > > > > > > >> > > > > The
> "AsyncTableFunction"
> > is
> > > > the
> > > > > > > > > function
> > > > > > > > > > > with
> > > > > > > > > > > >> > > > ability
> > > > > > > > > > > >> > > > > to
> > > > > > > > > > > >> > > > > > > be
> > > > > > > > > > > >> > > > > > > > > > >> executed
> > > > > > > > > > > >> > > > > > > > > > >> > > > async
> > > > > > > > > > > >> > > > > > > > > > >> > > > > (with
> AsyncWaitOperator).
> > > > > > > > > > > >> > > > > > > > > > >> > > > > The async lookup join
> is
> > a
> > > > one
> > > > > of
> > > > > > > > usage
> > > > > > > > > > of
> > > > > > > > > > > >> this.
> > > > > > > > > > > >> > > So,
> > > > > > > > > > > >> > > > > we
> > > > > > > > > > > >> > > > > > > > don't
> > > > > > > > > > > >> > > > > > > > > > >> have to
> > > > > > > > > > > >> > > > > > > > > > >> > > > bind
> > > > > > > > > > > >> > > > > > > > > > >> > > > > the AyncTableFunction
> > with
> > > > > > > > > > > LookupTableSource.
> > > > > > > > > > > >> > > > > > > > > > >> > > > > If User-defined
> > > > > > AsyncTableFunction
> > > > > > > is
> > > > > > > > > > > >> supported,
> > > > > > > > > > > >> > > > user
> > > > > > > > > > > >> > > > > > can
> > > > > > > > > > > >> > > > > > > > > > directly
> > > > > > > > > > > >> > > > > > > > > > >> > use
> > > > > > > > > > > >> > > > > > > > > > >> > > > > lateral table syntax to
> > > > perform
> > > > > > > async
> > > > > > > > > > > >> operation.
> > > > > > > > > > > >> > > > > > > > > > >> > > > >
> > > > > > > > > > > >> > > > > > > > > > >> > > > > > "It would be better
> if
> > > you
> > > > > > could
> > > > > > > > > > > elaborate
> > > > > > > > > > > >> the
> > > > > > > > > > > >> > > > > > proposed
> > > > > > > > > > > >> > > > > > > > > > changes
> > > > > > > > > > > >> > > > > > > > > > >> wrt
> > > > > > > > > > > >> > > > > > > > > > >> > > the
> > > > > > > > > > > >> > > > > > > > > > >> > > > > CorrelatedCodeGenerator
> > > with
> > > > > more
> > > > > > > > > > details"
> > > > > > > > > > > >> > > > > > > > > > >> > > > >
> > > > > > > > > > > >> > > > > > > > > > >> > > > > In the proposal, we use
> > > > lateral
> > > > > > > table
> > > > > > > > > > > syntax
> > > > > > > > > > > >> to
> > > > > > > > > > > >> > > > > support
> > > > > > > > > > > >> > > > > > > the
> > > > > > > > > > > >> > > > > > > > > > async
> > > > > > > > > > > >> > > > > > > > > > >> > table
> > > > > > > > > > > >> > > > > > > > > > >> > > > > function. So the
> planner
> > > will
> > > > > > also
> > > > > > > > > treat
> > > > > > > > > > > this
> > > > > > > > > > > >> > > > > statement
> > > > > > > > > > > >> > > > > > > to a
> > > > > > > > > > > >> > > > > > > > > > >> > > > > CommonExecCorrelate
> node.
> > > So
> > > > > the
> > > > > > > > > runtime
> > > > > > > > > > > code
> > > > > > > > > > > >> > > should
> > > > > > > > > > > >> > > > > be
> > > > > > > > > > > >> > > > > > > > > > generated
> > > > > > > > > > > >> > > > > > > > > > >> in
> > > > > > > > > > > >> > > > > > > > > > >> > > > >
> CorrelatedCodeGenerator.
> > > > > > > > > > > >> > > > > > > > > > >> > > > > In
> > CorrelatedCodeGenerator,
> > > > we
> > > > > > will
> > > > > > > > > know
> > > > > > > > > > > the
> > > > > > > > > > > >> > > > > > > TableFunction's
> > > > > > > > > > > >> > > > > > > > > > Kind
> > > > > > > > > > > >> > > > > > > > > > >> of
> > > > > > > > > > > >> > > > > > > > > > >> > > > > `FunctionKind.Table` or
> > > > > > > > > > > >> > `FunctionKind.ASYNC_TABLE`
> > > > > > > > > > > >> > > > > > > > > > >> > > > > For
> > > > `FunctionKind.ASYNC_TABLE`
> > > > > > we
> > > > > > > > can
> > > > > > > > > > > >> generate
> > > > > > > > > > > >> > a
> > > > > > > > > > > >> > > > > > > > > > >> AsyncWaitOperator
> > > > > > > > > > > >> > > > > > > > > > >> > to
> > > > > > > > > > > >> > > > > > > > > > >> > > > > execute the async table
> > > > > function.
> > > > > > > > > > > >> > > > > > > > > > >> > > > >
> > > > > > > > > > > >> > > > > > > > > > >> > > > >
> > > > > > > > > > > >> > > > > > > > > > >> > > > > Thanks,
> > > > > > > > > > > >> > > > > > > > > > >> > > > > Aitozi.
> > > > > > > > > > > >> > > > > > > > > > >> > > > >
> > > > > > > > > > > >> > > > > > > > > > >> > > > >
> > > > > > > > > > > >> > > > > > > > > > >> > > > > Jing Ge
> > > > > > <jing@ververica.com.invalid
> > > > > > > >
> > > > > > > > > > > >> > > 于2023年5月29日周一
> > > > > > > > > > > >> > > > > > > 03:22写道：
> > > > > > > > > > > >> > > > > > > > > > >> > > > >
> > > > > > > > > > > >> > > > > > > > > > >> > > > >> Hi Aitozi,
> > > > > > > > > > > >> > > > > > > > > > >> > > > >>
> > > > > > > > > > > >> > > > > > > > > > >> > > > >> Thanks for the
> > > > clarification.
> > > > > > The
> > > > > > > > > naming
> > > > > > > > > > > >> > "Lookup"
> > > > > > > > > > > >> > > > > might
> > > > > > > > > > > >> > > > > > > > > suggest
> > > > > > > > > > > >> > > > > > > > > > >> > using
> > > > > > > > > > > >> > > > > > > > > > >> > > it
> > > > > > > > > > > >> > > > > > > > > > >> > > > >> for table look up. But
> > > > > > > conceptually
> > > > > > > > > what
> > > > > > > > > > > the
> > > > > > > > > > > >> > > eval()
> > > > > > > > > > > >> > > > > > > method
> > > > > > > > > > > >> > > > > > > > > will
> > > > > > > > > > > >> > > > > > > > > > >> do
> > > > > > > > > > > >> > > > > > > > > > >> > is
> > > > > > > > > > > >> > > > > > > > > > >> > > to
> > > > > > > > > > > >> > > > > > > > > > >> > > > >> get a collection of
> > > > > results(Row,
> > > > > > > > > > RowData)
> > > > > > > > > > > >> from
> > > > > > > > > > > >> > > the
> > > > > > > > > > > >> > > > > > given
> > > > > > > > > > > >> > > > > > > > > keys.
> > > > > > > > > > > >> > > > > > > > > > >> How
> > > > > > > > > > > >> > > > > > > > > > >> > it
> > > > > > > > > > > >> > > > > > > > > > >> > > > will
> > > > > > > > > > > >> > > > > > > > > > >> > > > >> be done depends on the
> > > > > > > > implementation,
> > > > > > > > > > > i.e.
> > > > > > > > > > > >> you
> > > > > > > > > > > >> > > can
> > > > > > > > > > > >> > > > > > > > implement
> > > > > > > > > > > >> > > > > > > > > > >> your
> > > > > > > > > > > >> > > > > > > > > > >> > own
> > > > > > > > > > > >> > > > > > > > > > >> > > > >> Source[1][2]. The
> > example
> > > in
> > > > > the
> > > > > > > > FLIP
> > > > > > > > > > > >> should be
> > > > > > > > > > > >> > > > able
> > > > > > > > > > > >> > > > > to
> > > > > > > > > > > >> > > > > > > be
> > > > > > > > > > > >> > > > > > > > > > >> handled
> > > > > > > > > > > >> > > > > > > > > > >> > in
> > > > > > > > > > > >> > > > > > > > > > >> > > > this
> > > > > > > > > > > >> > > > > > > > > > >> > > > >> way.
> > > > > > > > > > > >> > > > > > > > > > >> > > > >>
> > > > > > > > > > > >> > > > > > > > > > >> > > > >> Do you mean to support
> > the
> > > > > > > > > > > AyncTableFunction
> > > > > > > > > > > >> > > beyond
> > > > > > > > > > > >> > > > > the
> > > > > > > > > > > >> > > > > > > > > > >> > > > LookupTableSource?
> > > > > > > > > > > >> > > > > > > > > > >> > > > >> It would be better if
> > you
> > > > > could
> > > > > > > > > > elaborate
> > > > > > > > > > > >> the
> > > > > > > > > > > >> > > > > proposed
> > > > > > > > > > > >> > > > > > > > > changes
> > > > > > > > > > > >> > > > > > > > > > >> wrt
> > > > > > > > > > > >> > > > > > > > > > >> > the
> > > > > > > > > > > >> > > > > > > > > > >> > > > >>
> CorrelatedCodeGenerator
> > > with
> > > > > > more
> > > > > > > > > > details.
> > > > > > > > > > > >> > > Thanks!
> > > > > > > > > > > >> > > > > > > > > > >> > > > >>
> > > > > > > > > > > >> > > > > > > > > > >> > > > >> Best regards,
> > > > > > > > > > > >> > > > > > > > > > >> > > > >> Jing
> > > > > > > > > > > >> > > > > > > > > > >> > > > >>
> > > > > > > > > > > >> > > > > > > > > > >> > > > >> [1]
> > > > > > > > > > > >> > > > > > > > > > >> > > > >>
> > > > > > > > > > > >> > > > > > > > > > >> > > > >>
> > > > > > > > > > > >> > > > > > > > > > >> > > >
> > > > > > > > > > > >> > > > > > > > > > >> > >
> > > > > > > > > > > >> > > > > > > > > > >> >
> > > > > > > > > > > >> > > > > > > > > > >>
> > > > > > > > > > > >> > > > > > > > > >
> > > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > > >> > > > > > > >
> > > > > > > > > > > >> > > > > > >
> > > > > > > > > > > >> > > > > >
> > > > > > > > > > > >> > > > >
> > > > > > > > > > > >> > > >
> > > > > > > > > > > >> > >
> > > > > > > > > > > >> >
> > > > > > > > > > > >>
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/flink/blob/678370b18e1b6c4a23e5ce08f8efd05675a0cc17/flink-table/flink-table-common/src/main/java/org/apache/flink/table/connector/source/LookupTableSource.java#L64
> > > > > > > > > > > >> > > > > > > > > > >> > > > >> [2]
> > > > > > > > > > > >> > > > > > > > > > >> > > > >>
> > > > > > > > > > > >> > > > > > > > > > >> > > > >>
> > > > > > > > > > > >> > > > > > > > > > >> > > >
> > > > > > > > > > > >> > > > > > > > > > >> > >
> > > > > > > > > > > >> > > > > > > > > > >> >
> > > > > > > > > > > >> > > > > > > > > > >>
> > > > > > > > > > > >> > > > > > > > > >
> > > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > > >> > > > > > > >
> > > > > > > > > > > >> > > > > > >
> > > > > > > > > > > >> > > > > >
> > > > > > > > > > > >> > > > >
> > > > > > > > > > > >> > > >
> > > > > > > > > > > >> > >
> > > > > > > > > > > >> >
> > > > > > > > > > > >>
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/flink/blob/678370b18e1b6c4a23e5ce08f8efd05675a0cc17/flink-table/flink-table-common/src/main/java/org/apache/flink/table/connector/source/AsyncTableFunctionProvider.java#L49
> > > > > > > > > > > >> > > > > > > > > > >> > > > >>
> > > > > > > > > > > >> > > > > > > > > > >> > > > >> On Sat, May 27, 2023
> at
> > > > > 9:48 AM
> > > > > > > > > Aitozi <
> > > > > > > > > > > >> > > > > > > > gjying1314@gmail.com
> > > > > > > > > > > >> > > > > > > > > >
> > > > > > > > > > > >> > > > > > > > > > >> > wrote:
> > > > > > > > > > > >> > > > > > > > > > >> > > > >>
> > > > > > > > > > > >> > > > > > > > > > >> > > > >> > Hi Jing,
> > > > > > > > > > > >> > > > > > > > > > >> > > > >> >     Thanks for your
> > > > > response.
> > > > > > As
> > > > > > > > > > stated
> > > > > > > > > > > in
> > > > > > > > > > > >> > the
> > > > > > > > > > > >> > > > > FLIP,
> > > > > > > > > > > >> > > > > > > the
> > > > > > > > > > > >> > > > > > > > > > >> purpose
> > > > > > > > > > > >> > > > > > > > > > >> > of
> > > > > > > > > > > >> > > > > > > > > > >> > > > this
> > > > > > > > > > > >> > > > > > > > > > >> > > > >> > FLIP is meant to
> > support
> > > > > > > > > > > >> > > > > > > > > > >> > > > >> > user-defined async
> > table
> > > > > > > function.
> > > > > > > > > As
> > > > > > > > > > > >> > described
> > > > > > > > > > > >> > > > in
> > > > > > > > > > > >> > > > > > > flink
> > > > > > > > > > > >> > > > > > > > > > >> document
> > > > > > > > > > > >> > > > > > > > > > >> > > [1]
> > > > > > > > > > > >> > > > > > > > > > >> > > > >> >
> > > > > > > > > > > >> > > > > > > > > > >> > > > >> > Async table
> functions
> > > are
> > > > > > > special
> > > > > > > > > > > >> functions
> > > > > > > > > > > >> > for
> > > > > > > > > > > >> > > > > table
> > > > > > > > > > > >> > > > > > > > > sources
> > > > > > > > > > > >> > > > > > > > > > >> that
> > > > > > > > > > > >> > > > > > > > > > >> > > > >> perform
> > > > > > > > > > > >> > > > > > > > > > >> > > > >> > > a lookup.
> > > > > > > > > > > >> > > > > > > > > > >> > > > >> > >
> > > > > > > > > > > >> > > > > > > > > > >> > > > >> >
> > > > > > > > > > > >> > > > > > > > > > >> > > > >> > So end user can not
> > > > directly
> > > > > > > > define
> > > > > > > > > > and
> > > > > > > > > > > >> use
> > > > > > > > > > > >> > > async
> > > > > > > > > > > >> > > > > > table
> > > > > > > > > > > >> > > > > > > > > > >> function
> > > > > > > > > > > >> > > > > > > > > > >> > > now.
> > > > > > > > > > > >> > > > > > > > > > >> > > > An
> > > > > > > > > > > >> > > > > > > > > > >> > > > >> > user case is
> reported
> > in
> > > > [2]
> > > > > > > > > > > >> > > > > > > > > > >> > > > >> >
> > > > > > > > > > > >> > > > > > > > > > >> > > > >> > So, in conclusion,
> no
> > > new
> > > > > > > > interface
> > > > > > > > > is
> > > > > > > > > > > >> > > > introduced,
> > > > > > > > > > > >> > > > > > but
> > > > > > > > > > > >> > > > > > > we
> > > > > > > > > > > >> > > > > > > > > > >> extend
> > > > > > > > > > > >> > > > > > > > > > >> > the
> > > > > > > > > > > >> > > > > > > > > > >> > > > >> > ability to support
> > > > > > user-defined
> > > > > > > > > async
> > > > > > > > > > > >> table
> > > > > > > > > > > >> > > > > function.
> > > > > > > > > > > >> > > > > > > > > > >> > > > >> >
> > > > > > > > > > > >> > > > > > > > > > >> > > > >> > [1]:
> > > > > > > > > > > >> > > > > > > > > > >> > > > >> >
> > > > > > > > > > > >> > > > > > > > > > >> > > > >> >
> > > > > > > > > > > >> > > > > > > > > > >> > > > >>
> > > > > > > > > > > >> > > > > > > > > > >> > > >
> > > > > > > > > > > >> > > > > > > > > > >> > >
> > > > > > > > > > > >> > > > > > > > > > >> >
> > > > > > > > > > > >> > > > > > > > > > >>
> > > > > > > > > > > >> > > > > > > > > >
> > > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > > >> > > > > > > >
> > > > > > > > > > > >> > > > > > >
> > > > > > > > > > > >> > > > > >
> > > > > > > > > > > >> > > > >
> > > > > > > > > > > >> > > >
> > > > > > > > > > > >> > >
> > > > > > > > > > > >> >
> > > > > > > > > > > >>
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/dev/table/functions/udfs/
> > > > > > > > > > > >> > > > > > > > > > >> > > > >> > [2]:
> > > > > > > > > > > >> > > > > > > > > > >> > >
> > > > > > > > > > > >> > > > > > > >
> > > > > > > > > > > >> >
> > > > > > > >
> > https://lists.apache.org/thread/qljwd40v5ntz6733cwcdr8s4z97b343b
> > > > > > > > > > > >> > > > > > > > > > >> > > > >> >
> > > > > > > > > > > >> > > > > > > > > > >> > > > >> > Thanks.
> > > > > > > > > > > >> > > > > > > > > > >> > > > >> > Aitozi.
> > > > > > > > > > > >> > > > > > > > > > >> > > > >> >
> > > > > > > > > > > >> > > > > > > > > > >> > > > >> >
> > > > > > > > > > > >> > > > > > > > > > >> > > > >> > Jing Ge
> > > > > > > > <jing@ververica.com.invalid
> > > > > > > > > >
> > > > > > > > > > > >> > > > 于2023年5月27日周六
> > > > > > > > > > > >> > > > > > > > > 06:40写道：
> > > > > > > > > > > >> > > > > > > > > > >> > > > >> >
> > > > > > > > > > > >> > > > > > > > > > >> > > > >> > > Hi Aitozi,
> > > > > > > > > > > >> > > > > > > > > > >> > > > >> > >
> > > > > > > > > > > >> > > > > > > > > > >> > > > >> > > Thanks for your
> > > > proposal.
> > > > > I
> > > > > > am
> > > > > > > > not
> > > > > > > > > > > quite
> > > > > > > > > > > >> > sure
> > > > > > > > > > > >> > > > if
> > > > > > > > > > > >> > > > > I
> > > > > > > > > > > >> > > > > > > > > > understood
> > > > > > > > > > > >> > > > > > > > > > >> > your
> > > > > > > > > > > >> > > > > > > > > > >> > > > >> > thoughts
> > > > > > > > > > > >> > > > > > > > > > >> > > > >> > > correctly. You
> > > > described a
> > > > > > > > special
> > > > > > > > > > > case
> > > > > > > > > > > >> > > > > > > implementation
> > > > > > > > > > > >> > > > > > > > of
> > > > > > > > > > > >> > > > > > > > > > the
> > > > > > > > > > > >> > > > > > > > > > >> > > > >> > > AsyncTableFunction
> > > with
> > > > on
> > > > > > > > public
> > > > > > > > > > API
> > > > > > > > > > > >> > > changes.
> > > > > > > > > > > >> > > > > > Would
> > > > > > > > > > > >> > > > > > > > you
> > > > > > > > > > > >> > > > > > > > > > >> please
> > > > > > > > > > > >> > > > > > > > > > >> > > > >> elaborate
> > > > > > > > > > > >> > > > > > > > > > >> > > > >> > > your purpose of
> > > writing
> > > > a
> > > > > > FLIP
> > > > > > > > > > > >> according to
> > > > > > > > > > > >> > > the
> > > > > > > > > > > >> > > > > > FLIP
> > > > > > > > > > > >> > > > > > > > > > >> > > > documentation[1]?
> > > > > > > > > > > >> > > > > > > > > > >> > > > >> > > Thanks!
> > > > > > > > > > > >> > > > > > > > > > >> > > > >> > >
> > > > > > > > > > > >> > > > > > > > > > >> > > > >> > > [1]
> > > > > > > > > > > >> > > > > > > > > > >> > > > >> > >
> > > > > > > > > > > >> > > > > > > > > > >> > > > >> > >
> > > > > > > > > > > >> > > > > > > > > > >> > > > >> >
> > > > > > > > > > > >> > > > > > > > > > >> > > > >>
> > > > > > > > > > > >> > > > > > > > > > >> > > >
> > > > > > > > > > > >> > > > > > > > > > >> > >
> > > > > > > > > > > >> > > > > > > > > > >> >
> > > > > > > > > > > >> > > > > > > > > > >>
> > > > > > > > > > > >> > > > > > > > > >
> > > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > > >> > > > > > > >
> > > > > > > > > > > >> > > > > > >
> > > > > > > > > > > >> > > > > >
> > > > > > > > > > > >> > > > >
> > > > > > > > > > > >> > > >
> > > > > > > > > > > >> > >
> > > > > > > > > > > >> >
> > > > > > > > > > > >>
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/Flink+Improvement+Proposals
> > > > > > > > > > > >> > > > > > > > > > >> > > > >> > >
> > > > > > > > > > > >> > > > > > > > > > >> > > > >> > > Best regards,
> > > > > > > > > > > >> > > > > > > > > > >> > > > >> > > Jing
> > > > > > > > > > > >> > > > > > > > > > >> > > > >> > >
> > > > > > > > > > > >> > > > > > > > > > >> > > > >> > > On Wed, May 24,
> 2023
> > > at
> > > > > > > 1:07 PM
> > > > > > > > > > > Aitozi <
> > > > > > > > > > > >> > > > > > > > > > gjying1314@gmail.com
> > > > > > > > > > > >> > > > > > > > > > >> >
> > > > > > > > > > > >> > > > > > > > > > >> > > > wrote:
> > > > > > > > > > > >> > > > > > > > > > >> > > > >> > >
> > > > > > > > > > > >> > > > > > > > > > >> > > > >> > > > May I ask for
> some
> > > > > > feedback
> > > > > > > > :D
> > > > > > > > > > > >> > > > > > > > > > >> > > > >> > > >
> > > > > > > > > > > >> > > > > > > > > > >> > > > >> > > > Thanks,
> > > > > > > > > > > >> > > > > > > > > > >> > > > >> > > > Aitozi
> > > > > > > > > > > >> > > > > > > > > > >> > > > >> > > >
> > > > > > > > > > > >> > > > > > > > > > >> > > > >> > > > Aitozi <
> > > > > > > gjying1314@gmail.com>
> > > > > > > > > > > >> > > 于2023年5月23日周二
> > > > > > > > > > > >> > > > > > > 19:14写道：
> > > > > > > > > > > >> > > > > > > > > > >> > > > >> > > > >
> > > > > > > > > > > >> > > > > > > > > > >> > > > >> > > > > Just catch an
> > user
> > > > > case
> > > > > > > > report
> > > > > > > > > > > from
> > > > > > > > > > > >> > > Giannis
> > > > > > > > > > > >> > > > > > > Polyzos
> > > > > > > > > > > >> > > > > > > > > for
> > > > > > > > > > > >> > > > > > > > > > >> this
> > > > > > > > > > > >> > > > > > > > > > >> > > > >> usage:
> > > > > > > > > > > >> > > > > > > > > > >> > > > >> > > > >
> > > > > > > > > > > >> > > > > > > > > > >> > > > >> > > > >
> > > > > > > > > > > >> > > > > > > > > > >> > > >
> > > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > > >> > >
> > > > > > > > >
> > > https://lists.apache.org/thread/qljwd40v5ntz6733cwcdr8s4z97b343b
> > > > > > > > > > > >> > > > > > > > > > >> > > > >> > > > >
> > > > > > > > > > > >> > > > > > > > > > >> > > > >> > > > > Aitozi <
> > > > > > > > gjying1314@gmail.com>
> > > > > > > > > > > >> > > > 于2023年5月23日周二
> > > > > > > > > > > >> > > > > > > > 17:45写道：
> > > > > > > > > > > >> > > > > > > > > > >> > > > >> > > > > >
> > > > > > > > > > > >> > > > > > > > > > >> > > > >> > > > > > Hi guys,
> > > > > > > > > > > >> > > > > > > > > > >> > > > >> > > > > >     I want
> to
> > > > bring
> > > > > > up a
> > > > > > > > > > > >> discussion
> > > > > > > > > > > >> > > about
> > > > > > > > > > > >> > > > > > > adding
> > > > > > > > > > > >> > > > > > > > > > >> support
> > > > > > > > > > > >> > > > > > > > > > >> > of
> > > > > > > > > > > >> > > > > > > > > > >> > > > User
> > > > > > > > > > > >> > > > > > > > > > >> > > > >> > > > > > Defined
> > > > > > > AsyncTableFunction
> > > > > > > > > in
> > > > > > > > > > > >> Flink.
> > > > > > > > > > > >> > > > > > > > > > >> > > > >> > > > > > Currently,
> > async
> > > > > table
> > > > > > > > > > function
> > > > > > > > > > > >> are
> > > > > > > > > > > >> > > > special
> > > > > > > > > > > >> > > > > > > > > functions
> > > > > > > > > > > >> > > > > > > > > > >> for
> > > > > > > > > > > >> > > > > > > > > > >> > > > table
> > > > > > > > > > > >> > > > > > > > > > >> > > > >> > > source
> > > > > > > > > > > >> > > > > > > > > > >> > > > >> > > > > > to perform
> > > > > > > > > > > >> > > > > > > > > > >> > > > >> > > > > > async
> lookup.
> > > > > However,
> > > > > > > > it's
> > > > > > > > > > > worth
> > > > > > > > > > > >> to
> > > > > > > > > > > >> > > > > support
> > > > > > > > > > > >> > > > > > > the
> > > > > > > > > > > >> > > > > > > > > user
> > > > > > > > > > > >> > > > > > > > > > >> > > defined
> > > > > > > > > > > >> > > > > > > > > > >> > > > >> async
> > > > > > > > > > > >> > > > > > > > > > >> > > > >> > > > > > table
> > function.
> > > > > > > > > > > >> > > > > > > > > > >> > > > >> > > > > > Because, in
> > this
> > > > > way,
> > > > > > > the
> > > > > > > > > end
> > > > > > > > > > > SQL
> > > > > > > > > > > >> > user
> > > > > > > > > > > >> > > > can
> > > > > > > > > > > >> > > > > > > > leverage
> > > > > > > > > > > >> > > > > > > > > > it
> > > > > > > > > > > >> > > > > > > > > > >> to
> > > > > > > > > > > >> > > > > > > > > > >> > > > >> perform
> > > > > > > > > > > >> > > > > > > > > > >> > > > >> > the
> > > > > > > > > > > >> > > > > > > > > > >> > > > >> > > > > > async
> > operation
> > > > > > > > > > > >> > > > > > > > > > >> > > > >> > > > > > which is
> > useful
> > > to
> > > > > > > maximum
> > > > > > > > > the
> > > > > > > > > > > >> system
> > > > > > > > > > > >> > > > > > > throughput
> > > > > > > > > > > >> > > > > > > > > > >> > especially
> > > > > > > > > > > >> > > > > > > > > > >> > > > for
> > > > > > > > > > > >> > > > > > > > > > >> > > > >> IO
> > > > > > > > > > > >> > > > > > > > > > >> > > > >> > > > > > bottleneck
> > case.
> > > > > > > > > > > >> > > > > > > > > > >> > > > >> > > > > >
> > > > > > > > > > > >> > > > > > > > > > >> > > > >> > > > > > You can find
> > > some
> > > > > more
> > > > > > > > > detail
> > > > > > > > > > in
> > > > > > > > > > > >> [1].
> > > > > > > > > > > >> > > > > > > > > > >> > > > >> > > > > >
> > > > > > > > > > > >> > > > > > > > > > >> > > > >> > > > > > Looking
> > forward
> > > to
> > > > > > > > feedback
> > > > > > > > > > > >> > > > > > > > > > >> > > > >> > > > > >
> > > > > > > > > > > >> > > > > > > > > > >> > > > >> > > > > >
> > > > > > > > > > > >> > > > > > > > > > >> > > > >> > > > > > [1]:
> > > > > > > > > > > >> > > > > > > > > > >> > > > >> > > >
> > > > > > > > > > > >> > > > > > > > > > >> > > > >> > >
> > > > > > > > > > > >> > > > > > > > > > >> > > > >> >
> > > > > > > > > > > >> > > > > > > > > > >> > > > >>
> > > > > > > > > > > >> > > > > > > > > > >> > > >
> > > > > > > > > > > >> > > > > > > > > > >> > >
> > > > > > > > > > > >> > > > > > > > > > >> >
> > > > > > > > > > > >> > > > > > > > > > >>
> > > > > > > > > > > >> > > > > > > > > >
> > > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > > >> > > > > > > >
> > > > > > > > > > > >> > > > > > >
> > > > > > > > > > > >> > > > > >
> > > > > > > > > > > >> > > > >
> > > > > > > > > > > >> > > >
> > > > > > > > > > > >> > >
> > > > > > > > > > > >> >
> > > > > > > > > > > >>
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/%5BFLIP-313%5D+Add+support+of+User+Defined+AsyncTableFunction
> > > > > > > > > > > >> > > > > > > > > > >> > > > >> > > > > >
> > > > > > > > > > > >> > > > > > > > > > >> > > > >> > > > > > Thanks,
> > > > > > > > > > > >> > > > > > > > > > >> > > > >> > > > > > Aitozi.
> > > > > > > > > > > >> > > > > > > > > > >> > > > >> > > >
> > > > > > > > > > > >> > > > > > > > > > >> > > > >> > >
> > > > > > > > > > > >> > > > > > > > > > >> > > > >> >
> > > > > > > > > > > >> > > > > > > > > > >> > > > >>
> > > > > > > > > > > >> > > > > > > > > > >> > > > >
> > > > > > > > > > > >> > > > > > > > > > >> > > >
> > > > > > > > > > > >> > > > > > > > > > >> > >
> > > > > > > > > > > >> > > > > > > > > > >> >
> > > > > > > > > > > >> > > > > > > > > > >>
> > > > > > > > > > > >> > > > > > > > > > >
> > > > > > > > > > > >> > > > > > > > > >
> > > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > > >> > > > > > > >
> > > > > > > > > > > >> > > > > > >
> > > > > > > > > > > >> > > > > >
> > > > > > > > > > > >> > > > >
> > > > > > > > > > > >> > > >
> > > > > > > > > > > >> > >
> > > > > > > > > > > >> >
> > > > > > > > > > > >>
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] FLIP-313 Add support of User Defined AsyncTableFunction

Posted by Lincoln Lee <li...@gmail.com>.

Hi Aitozi,

Thanks for your updates!

By the design of hints, the hints after select clause belong to the query
hints category, and this new hint is also a kind of join hints[1].
Join table function is one of the join type defined by flink sql joins[2],
all existing join hints[1] omit the 'join' keyword,
so I would prefer the 'ASYNC_TABLE_FUNC' (which is actually the one for
'ASYNC_TABLE_FUNC_JOIN').

Since a short Chinese holiday is coming, I suggest waiting for other
people's responses before continuing to vote (next monday?)

Btw, I discussed with @fudian offline about pyflink support, there should
be no known issues, so you can create a subtask with pyflink support after
the vote passed.

[1]
https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/sql/queries/hints/#join-hints
[2]
https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/sql/queries/joins/

Best,
Lincoln Lee


Aitozi <gj...@gmail.com> 于2023年6月18日周日 21:18写道：

> Hi all,
>     Sorry for the late reply, I have a discussion with Lincoln offline,
> mainly about
> the naming of the hints option. Thanks Lincoln for the valuable
> suggestions.
>
> Let me answer the last email inline.
>
> >For `JavaAsyncTableFunc0` in flip, can you use a scenario like RPC call as
> an example?
>
> Sure, will give an example when adding the doc of async udtf and will
> update the FLIP simultaneously
>
> >For the name of this query hint, 'LATERAL' (include its internal options)
> don't show any relevance to async, but I haven't thought of a suitable name
> at the moment,
>
> After some discussion with Lincoln, We prefer to choose one of the
> `ASYNC_TABLE_FUNC` and `ASYNC_LATERAL`.
> Besides, In my opinion the keyword `lateral`'s use scenario is wider than
> the table function join, but in this case we only want to config
> the async table function, So I'm a bit more lean to the `ASYNC_TABLE_FUNC`.
> Looking forward to some inputs if you guys have
> some better suggestion on the naming.
>
> For the usage of the hints config option, I have updated the section
> of ConfigOption, you can refer to the FLIP
> for more details.
>
> >Also, the terms 'correlate join' and 'lateral join' are not the same as in
> the current joins page[1], so maybe it would be better if we unified them
> into  'join table function'
>
> Yes, we should unified to the 'join table function', updated.
>
> Best,
> Aitozi
>
> Lincoln Lee <li...@gmail.com> 于2023年6月15日周四 09:15写道：
>
> > Hi Aitozi,
> >
> > Thanks for your reply!  Gives sql users more flexibility to get
> > asynchronous processing capabilities via lateral join table function +1
> for
> > this
> >
> > For `JavaAsyncTableFunc0` in flip, can you use a scenario like RPC call
> as
> > an example?
> >
> > For the name of this query hint, 'LATERAL' (include its internal options)
> > don't show any relevance to async, but I haven't thought of a suitable
> name
> > at the moment,
> > maybe we need to highlight the async keyword directly, we can also see if
> > others have better candidates
> >
> > For the hint option "timeout = '180s'" should be "'timeout' = '180s'",
> > seems a typo in the flip. And use upper case for all keywords in sql
> > examples.
> > Also, the terms 'correlate join' and 'lateral join' are not the same as
> in
> > the current joins page[1], so maybe it would be better if we unified them
> > into  'join table function'
> >
> > [1]
> >
> >
> https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/sql/queries/joins/#table-function
> >
> > Best,
> > Lincoln Lee
> >
> >
> > Aitozi <gj...@gmail.com> 于2023年6月14日周三 16:11写道：
> >
> > > Hi Lincoln
> > >
> > >     Very thanks for your valuable question. I will try to answer your
> > > questions inline.
> > >
> > > >Does the async udtf bring any additional benefits besides a
> > > lighter implementation？
> > >
> > > IMO, async udtf is more than a lighter implementation. It can act as a
> > > general way for sql users to use the async operator. And they don't
> have
> > to
> > > bind the async function with a table (a LookupTable), and they are not
> > > forced to join on an equality join condition, and they can use it to do
> > > more than enrich data.
> > >
> > > The async lookup join is more like a subset/specific usage of async
> udtf.
> > > The specific version has more opportunity to be optimized (like push
> > down)
> > > is acceptable. Async table function should be categorized to
> used-defined
> > > function.
> > >
> > > >Should users
> > >
> > > migrate to the lookup source when they encounter similar requirements
> or
> > >
> > > problems, or should we develop an additional set of similar mechanisms？
> > >
> > > As I clarified above, the lookup join is a specific usage of async
> udtf.
> > So
> > > it deserves more refined optimization like caching / retryable. But it
> > may
> > > not all
> > >
> > > suitable for the async udtf. As function, it can be deterministic/or
> > > non-deterministic. So caching is not suitable, and we also do not have
> a
> > > common cache for the udf now. So I think optimization like
> caching/retry
> > > should be handed over to the function implementor.
> > >
> > > > the newly added query hint need a different name that
> > > can be easier related to the lateral operation as the current join
> > hints[5]
> > > do.
> > >
> > >
> > > What about using LATERAL?
> > >
> > > as below
> > >
> > > SELECT /*+ LATERAL('output-mode' = 'ordered', 'capacity' = '200',
> > timeout =
> > > '180s') */ a, c1, c2
> > >
> > > FROM T1
> > >
> > > LEFT JOIN lateral TABLE (async_split(b)) AS T(c1, c2) ON true
> > >
> > > >For the async func example, since the target scenario is an external
> io
> > > operation, it's better to add the `close` method to actively release
> > > resources as a good example for users
> > >
> > >
> > > Make sense to me, will update the FLIP
> > >
> > > Best,
> > >
> > > Aitozi.
> > >
> > > Lincoln Lee <li...@gmail.com> 于2023年6月14日周三 14:24写道：
> > >
> > > > Hi Aitozi,
> > > >
> > > > Sorry for the lately reply here!  Supports async
> > > udtf(`AsyncTableFunction`)
> > > > directly in sql seems like an attractive feature, but there're two
> > issues
> > > > that need to be addressed before we can be sure to add it:
> > > > 1. As mentioned in the flip[1], the current lookup function can
> already
> > > > implement the requirements, but it requires implementing an extra
> > > > `LookupTableSource` and explicitly declaring the table schema (which
> > can
> > > > help implementers the various push-down optimizations supported by
> the
> > > > planner). Does the async udtf bring any additional benefits besides a
> > > > lighter implementation?
> > > > 2. FLIP-221[2] abstracts a reusable cache and metric infrastructure
> for
> > > > lookup sources, which are important to improve performance and
> > > > observability for high overhead external io scenarios, how do we
> > > integrate
> > > > and reuse these capabilities after introducing async udtf? Should
> users
> > > > migrate to the lookup source when they encounter similar requirements
> > or
> > > > problems, or should we develop an additional set of similar
> mechanisms?
> > > (a
> > > > similarly case:  FLIP-234[3] introduced the retryable capability for
> > > lookup
> > > > join)
> > > >
> > > > For the flip itself,
> > > > 1. Considering the 'options' is already used as the dynamic table
> > > > options[4] in flink, the newly added query hint need a different name
> > > that
> > > > can be easier related to the lateral operation as the current join
> > > hints[5]
> > > > do.
> > > > 2. For the async func example, since the target scenario is an
> external
> > > io
> > > > operation, it's better to add the `close` method to actively release
> > > > resources as a good example for users. Also in terms of the
> determinism
> > > of
> > > > a function, it is important to remind users that unless the behavior
> of
> > > the
> > > > function is deterministic, it needs to be explicitly declared as
> > > > non-deterministic.
> > > >
> > > > [1].
> > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-313%3A+Add+support+of+User+Defined+AsyncTableFunction?src=contextnavpagetreemode
> > > > [2].
> > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-221%3A+Abstraction+for+lookup+source+cache+and+metric?src=contextnavpagetreemode
> > > > [3].
> > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-234%3A+Support+Retryable+Lookup+Join+To+Solve+Delayed+Updates+Issue+In+External+Systems?src=contextnavpagetreemode
> > > > [4].
> > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-113%3A+Supports+Dynamic+Table+Options+for+Flink+SQL?src=contextnavpagetreemode
> > > > [5].
> > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-229%3A+Introduces+Join+Hint+for+Flink+SQL+Batch+Job?src=contextnavpagetreemode
> > > >
> > > > Best,
> > > > Lincoln Lee
> > > >
> > > >
> > > > Aitozi <gj...@gmail.com> 于2023年6月13日周二 11:30写道：
> > > >
> > > > > Get your meaning now, thanks :)
> > > > >
> > > > > Best,
> > > > > Aitozi.
> > > > >
> > > > > Feng Jin <ji...@gmail.com> 于2023年6月13日周二 11:16写道：
> > > > >
> > > > > > Hi Aitozi,
> > > > > >
> > > > > > Sorry for the confusing description.
> > > > > >
> > > > > > What I meant was that if we need to remind users about tire
> safety
> > > > > issues,
> > > > > > we should introduce the new UDTF interface instead of executing
> the
> > > > > > original UDTF asynchronously. Therefore, I agree with introducing
> > the
> > > > > > AsyncTableFunction.
> > > > > >
> > > > > > Best,
> > > > > > Feng
> > > > > >
> > > > > > On Tue, Jun 13, 2023 at 10:42 AM Aitozi <gj...@gmail.com>
> > > wrote:
> > > > > >
> > > > > > > Hi Feng,
> > > > > > >     Thanks for your question. We do not provide a way to switch
> > the
> > > > > UDTF
> > > > > > > between sync and async way,
> > > > > > > So there should be no thread safety problem here.
> > > > > > >
> > > > > > > Best,
> > > > > > > Aitozi
> > > > > > >
> > > > > > > Feng Jin <ji...@gmail.com> 于2023年6月13日周二 10:31写道：
> > > > > > >
> > > > > > > > Hi Aitozi, We do need to remind users about thread safety
> > issues.
> > > > > Thank
> > > > > > > you
> > > > > > > > for your efforts on this FLIP. I have no further questions.
> > > > > > > > Best, Feng
> > > > > > > >
> > > > > > > >
> > > > > > > > On Tue, Jun 13, 2023 at 6:05 AM Jing Ge
> > > <jing@ververica.com.invalid
> > > > >
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi Aitozi,
> > > > > > > > >
> > > > > > > > > Thanks for taking care of that part. I have no other
> concern.
> > > > > > > > >
> > > > > > > > > Best regards,
> > > > > > > > > Jing
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Mon, Jun 12, 2023 at 5:38 PM Aitozi <
> gjying1314@gmail.com
> > >
> > > > > wrote:
> > > > > > > > >
> > > > > > > > > > BTW, If there are no other more blocking issue /
> comments,
> > I
> > > > > would
> > > > > > > like
> > > > > > > > > to
> > > > > > > > > > start a VOTE in another thread this wednesday 6.14
> > > > > > > > > >
> > > > > > > > > > Thanks,
> > > > > > > > > > Aitozi.
> > > > > > > > > >
> > > > > > > > > > Aitozi <gj...@gmail.com> 于2023年6月12日周一 23:34写道：
> > > > > > > > > >
> > > > > > > > > > > Hi, Jing,
> > > > > > > > > > >     Thanks for your explanation. I get your point now.
> > > > > > > > > > >
> > > > > > > > > > > For the performance part, I think it's a good idea to
> run
> > > > with
> > > > > > > > > returning
> > > > > > > > > > a
> > > > > > > > > > > big table case, the memory consumption
> > > > > > > > > > > should be a point to be taken care about. Because in
> the
> > > > > ordered
> > > > > > > > mode,
> > > > > > > > > > the
> > > > > > > > > > > head element in buffer may affect the
> > > > > > > > > > > total memory consumption.
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > Thanks,
> > > > > > > > > > > Aitozi.
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > Jing Ge <ji...@ververica.com.invalid> 于2023年6月12日周一
> > > 20:28写道：
> > > > > > > > > > >
> > > > > > > > > > >> Hi Aitozi,
> > > > > > > > > > >>
> > > > > > > > > > >> Which key will be used for lookup is not an issue,
> only
> > > one
> > > > > row
> > > > > > > will
> > > > > > > > > be
> > > > > > > > > > >> required for each key in order to enrich it. True, it
> > > > depends
> > > > > on
> > > > > > > the
> > > > > > > > > > >> implementation whether multiple rows or single row for
> > > each
> > > > > key
> > > > > > > will
> > > > > > > > > be
> > > > > > > > > > >> returned. However, for the lookup & enrichment
> scenario,
> > > one
> > > > > > > row/key
> > > > > > > > > is
> > > > > > > > > > >> recommended, otherwise, like I mentioned previously,
> > > > > enrichment
> > > > > > > > won't
> > > > > > > > > > >> work.
> > > > > > > > > > >>
> > > > > > > > > > >> I am a little bit concerned about returning a big
> table
> > > for
> > > > > each
> > > > > > > > key,
> > > > > > > > > > >> since
> > > > > > > > > > >> it will take the async call longer to return and need
> > more
> > > > > > memory.
> > > > > > > > The
> > > > > > > > > > >> performance tests should cover this scenario. This is
> > not
> > > a
> > > > > > > blocking
> > > > > > > > > > issue
> > > > > > > > > > >> for this FLIP.
> > > > > > > > > > >>
> > > > > > > > > > >> Best regards,
> > > > > > > > > > >> Jing
> > > > > > > > > > >>
> > > > > > > > > > >> On Sat, Jun 10, 2023 at 4:11 AM Aitozi <
> > > > gjying1314@gmail.com>
> > > > > > > > wrote:
> > > > > > > > > > >>
> > > > > > > > > > >> > Hi Jing,
> > > > > > > > > > >> >     I means the join key is not necessary to be the
> > > > primary
> > > > > > key
> > > > > > > or
> > > > > > > > > > >> unique
> > > > > > > > > > >> > index of the database.
> > > > > > > > > > >> > In this situation, we may queried out multi rows for
> > one
> > > > > join
> > > > > > > > key. I
> > > > > > > > > > >> think
> > > > > > > > > > >> > that's why the
> > > > > > > > > > >> > LookupFunction#lookup will return a collection of
> > > RowData.
> > > > > > > > > > >> >
> > > > > > > > > > >> > BTW, I think the behavior of lookup join will not
> > affect
> > > > the
> > > > > > > > > semantic
> > > > > > > > > > of
> > > > > > > > > > >> > the async udtf.
> > > > > > > > > > >> > We use the Async TableFunction here and the table
> > > function
> > > > > can
> > > > > > > > > collect
> > > > > > > > > > >> > multiple rows.
> > > > > > > > > > >> >
> > > > > > > > > > >> > Thanks,
> > > > > > > > > > >> > Atiozi.
> > > > > > > > > > >> >
> > > > > > > > > > >> >
> > > > > > > > > > >> >
> > > > > > > > > > >> > Jing Ge <ji...@ververica.com.invalid> 于2023年6月10日周六
> > > > 00:15写道：
> > > > > > > > > > >> >
> > > > > > > > > > >> > > Hi Aitozi,
> > > > > > > > > > >> > >
> > > > > > > > > > >> > > The keyRow used in this case contains all keys[1].
> > > > > > > > > > >> > >
> > > > > > > > > > >> > > Best regards,
> > > > > > > > > > >> > > Jing
> > > > > > > > > > >> > >
> > > > > > > > > > >> > > [1]
> > > > > > > > > > >> > >
> > > > > > > > > > >> > >
> > > > > > > > > > >> >
> > > > > > > > > > >>
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/flink/blob/191ec6ca3943d7119f14837efe112e074d815c47/flink-table/flink-table-common/src/main/java/org/apache/flink/table/functions/LookupFunction.java#L49
> > > > > > > > > > >> > >
> > > > > > > > > > >> > >
> > > > > > > > > > >> > > On Fri, Jun 9, 2023 at 3:42 PM Aitozi <
> > > > > gjying1314@gmail.com
> > > > > > >
> > > > > > > > > wrote:
> > > > > > > > > > >> > >
> > > > > > > > > > >> > > > Hi Jing,
> > > > > > > > > > >> > > >
> > > > > > > > > > >> > > >      The performance test is added to the FLIP.
> > > > > > > > > > >> > > >
> > > > > > > > > > >> > > >      As I know, The lookup join can return multi
> > > rows,
> > > > > it
> > > > > > > > > depends
> > > > > > > > > > on
> > > > > > > > > > >> > > > whether  the join key
> > > > > > > > > > >> > > > is the primary key of the external database or
> > not.
> > > > The
> > > > > > > > `lookup`
> > > > > > > > > > [1]
> > > > > > > > > > >> > will
> > > > > > > > > > >> > > > return a collection of
> > > > > > > > > > >> > > > joined result, and each of them will be
> collected
> > > > > > > > > > >> > > >
> > > > > > > > > > >> > > >
> > > > > > > > > > >> > > > [1]:
> > > > > > > > > > >> > > >
> > > > > > > > > > >> > > >
> > > > > > > > > > >> > >
> > > > > > > > > > >> >
> > > > > > > > > > >>
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/flink/blob/191ec6ca3943d7119f14837efe112e074d815c47/flink-table/flink-table-common/src/main/java/org/apache/flink/table/functions/LookupFunction.java#L52
> > > > > > > > > > >> > > >
> > > > > > > > > > >> > > >
> > > > > > > > > > >> > > > Thanks,
> > > > > > > > > > >> > > > Aitozi.
> > > > > > > > > > >> > > >
> > > > > > > > > > >> > > > Jing Ge <ji...@ververica.com.invalid>
> 于2023年6月9日周五
> > > > > > 17:05写道：
> > > > > > > > > > >> > > >
> > > > > > > > > > >> > > > > Hi Aitozi,
> > > > > > > > > > >> > > > >
> > > > > > > > > > >> > > > > Thanks for the feedback. Looking forward to
> the
> > > > > > > performance
> > > > > > > > > > tests.
> > > > > > > > > > >> > > > >
> > > > > > > > > > >> > > > > Afaik, lookup returns one row for each key [1]
> > > [2].
> > > > > > > > > > Conceptually,
> > > > > > > > > > >> the
> > > > > > > > > > >> > > > > lookup function is used to enrich column(s)
> from
> > > the
> > > > > > > > dimension
> > > > > > > > > > >> table.
> > > > > > > > > > >> > > If,
> > > > > > > > > > >> > > > > for the given key, there will be more than one
> > > row,
> > > > > > there
> > > > > > > > will
> > > > > > > > > > be
> > > > > > > > > > >> no
> > > > > > > > > > >> > > way
> > > > > > > > > > >> > > > to
> > > > > > > > > > >> > > > > know which row will be used to enrich the key.
> > > > > > > > > > >> > > > >
> > > > > > > > > > >> > > > > [1]
> > > > > > > > > > >> > > > >
> > > > > > > > > > >> > > > >
> > > > > > > > > > >> > > >
> > > > > > > > > > >> > >
> > > > > > > > > > >> >
> > > > > > > > > > >>
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/flink/blob/191ec6ca3943d7119f14837efe112e074d815c47/flink-table/flink-table-common/src/main/java/org/apache/flink/table/functions/LookupFunction.java#L49
> > > > > > > > > > >> > > > > [2]
> > > > > > > > > > >> > > > >
> > > > > > > > > > >> > > > >
> > > > > > > > > > >> > > >
> > > > > > > > > > >> > >
> > > > > > > > > > >> >
> > > > > > > > > > >>
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/flink/blob/191ec6ca3943d7119f14837efe112e074d815c47/flink-table/flink-table-common/src/main/java/org/apache/flink/table/functions/TableFunction.java#L196
> > > > > > > > > > >> > > > >
> > > > > > > > > > >> > > > > Best regards,
> > > > > > > > > > >> > > > > Jing
> > > > > > > > > > >> > > > >
> > > > > > > > > > >> > > > > On Fri, Jun 9, 2023 at 5:18 AM Aitozi <
> > > > > > > gjying1314@gmail.com
> > > > > > > > >
> > > > > > > > > > >> wrote:
> > > > > > > > > > >> > > > >
> > > > > > > > > > >> > > > > > Hi Jing
> > > > > > > > > > >> > > > > >     Thanks for your good questions. I have
> > > updated
> > > > > the
> > > > > > > > > example
> > > > > > > > > > >> to
> > > > > > > > > > >> > the
> > > > > > > > > > >> > > > > FLIP.
> > > > > > > > > > >> > > > > >
> > > > > > > > > > >> > > > > > > Only one row for each lookup
> > > > > > > > > > >> > > > > > lookup can also return multi rows, based on
> > the
> > > > > query
> > > > > > > > > result.
> > > > > > > > > > >> [1]
> > > > > > > > > > >> > > > > >
> > > > > > > > > > >> > > > > > [1]:
> > > > > > > > > > >> > > > > >
> > > > > > > > > > >> > > > > >
> > > > > > > > > > >> > > > >
> > > > > > > > > > >> > > >
> > > > > > > > > > >> > >
> > > > > > > > > > >> >
> > > > > > > > > > >>
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/flink/blob/191ec6ca3943d7119f14837efe112e074d815c47/flink-table/flink-table-common/src/main/java/org/apache/flink/table/functions/LookupFunction.java#L56
> > > > > > > > > > >> > > > > >
> > > > > > > > > > >> > > > > > > If we use async calls with lateral join,
> my
> > > gut
> > > > > > > feeling
> > > > > > > > is
> > > > > > > > > > >> > > > > > that we might have many more async calls
> than
> > > > lookup
> > > > > > > > join. I
> > > > > > > > > > am
> > > > > > > > > > >> not
> > > > > > > > > > >> > > > > really
> > > > > > > > > > >> > > > > > sure if we will be facing potential issues
> in
> > > this
> > > > > > case
> > > > > > > or
> > > > > > > > > > not.
> > > > > > > > > > >> > > > > >
> > > > > > > > > > >> > > > > > IMO, the work pattern is similar to the
> lookup
> > > > > > function,
> > > > > > > > for
> > > > > > > > > > >> each
> > > > > > > > > > >> > row
> > > > > > > > > > >> > > > > from
> > > > > > > > > > >> > > > > > the left table,
> > > > > > > > > > >> > > > > > it will evaluate the eval method once, so
> the
> > > > async
> > > > > > call
> > > > > > > > > > numbers
> > > > > > > > > > >> > will
> > > > > > > > > > >> > > > not
> > > > > > > > > > >> > > > > > change.
> > > > > > > > > > >> > > > > > and the maximum calls in flight is limited
> by
> > > the
> > > > > > Async
> > > > > > > > > > >> operators
> > > > > > > > > > >> > > > buffer
> > > > > > > > > > >> > > > > > capacity
> > > > > > > > > > >> > > > > > which will be controlled by the option.
> > > > > > > > > > >> > > > > >
> > > > > > > > > > >> > > > > > BTW, for the naming of these option, I
> updated
> > > the
> > > > > > FLIP
> > > > > > > > > about
> > > > > > > > > > >> this
> > > > > > > > > > >> > > you
> > > > > > > > > > >> > > > > can
> > > > > > > > > > >> > > > > > refer to
> > > > > > > > > > >> > > > > > the section of "ConfigOption" and "Rejected
> > > > > > > Alternatives"
> > > > > > > > > > >> > > > > >
> > > > > > > > > > >> > > > > > In the end, for the performance evaluation,
> > I'd
> > > > like
> > > > > > to
> > > > > > > do
> > > > > > > > > > some
> > > > > > > > > > >> > tests
> > > > > > > > > > >> > > > and
> > > > > > > > > > >> > > > > > will update it to the FLIP doc
> > > > > > > > > > >> > > > > >
> > > > > > > > > > >> > > > > > Thanks,
> > > > > > > > > > >> > > > > > Aitozi.
> > > > > > > > > > >> > > > > >
> > > > > > > > > > >> > > > > >
> > > > > > > > > > >> > > > > > Jing Ge <ji...@ververica.com.invalid>
> > > 于2023年6月9日周五
> > > > > > > > 07:23写道：
> > > > > > > > > > >> > > > > >
> > > > > > > > > > >> > > > > > > Hi Aitozi,
> > > > > > > > > > >> > > > > > >
> > > > > > > > > > >> > > > > > > Thanks for the clarification. The code
> > example
> > > > > looks
> > > > > > > > > > >> > interesting. I
> > > > > > > > > > >> > > > > would
> > > > > > > > > > >> > > > > > > suggest adding them into the FLIP. The
> > > > description
> > > > > > > with
> > > > > > > > > code
> > > > > > > > > > >> > > examples
> > > > > > > > > > >> > > > > > will
> > > > > > > > > > >> > > > > > > help readers understand the motivation and
> > how
> > > > to
> > > > > > use
> > > > > > > > it.
> > > > > > > > > > >> Afaiac,
> > > > > > > > > > >> > > it
> > > > > > > > > > >> > > > > is a
> > > > > > > > > > >> > > > > > > valid feature for Flink users.
> > > > > > > > > > >> > > > > > >
> > > > > > > > > > >> > > > > > > As we knew, lookup join is based on
> temporal
> > > > join,
> > > > > > > i.e.
> > > > > > > > > FOR
> > > > > > > > > > >> > > > SYSTEM_TIME
> > > > > > > > > > >> > > > > > AS
> > > > > > > > > > >> > > > > > > OF which is also used in your code
> example.
> > > > > Temporal
> > > > > > > > join
> > > > > > > > > > >> > performs
> > > > > > > > > > >> > > > the
> > > > > > > > > > >> > > > > > > lookup based on the processing time match.
> > > Only
> > > > > one
> > > > > > > row
> > > > > > > > > for
> > > > > > > > > > >> each
> > > > > > > > > > >> > > > > > > lookup(afaiu, I need to check the source
> > code
> > > to
> > > > > > > double
> > > > > > > > > > >> confirm)
> > > > > > > > > > >> > > will
> > > > > > > > > > >> > > > > > > return for further enrichment. One the
> other
> > > > hand,
> > > > > > > > lateral
> > > > > > > > > > >> join
> > > > > > > > > > >> > > will
> > > > > > > > > > >> > > > > have
> > > > > > > > > > >> > > > > > > sub-queries correlated with every
> individual
> > > > value
> > > > > > of
> > > > > > > > the
> > > > > > > > > > >> > reference
> > > > > > > > > > >> > > > > table
> > > > > > > > > > >> > > > > > > from the preceding part of the query and
> > each
> > > > sub
> > > > > > > query
> > > > > > > > > will
> > > > > > > > > > >> > return
> > > > > > > > > > >> > > > > > > multiple rows. If we use async calls with
> > > > lateral
> > > > > > > join,
> > > > > > > > my
> > > > > > > > > > gut
> > > > > > > > > > >> > > > feeling
> > > > > > > > > > >> > > > > is
> > > > > > > > > > >> > > > > > > that we might have many more async calls
> > than
> > > > > lookup
> > > > > > > > > join. I
> > > > > > > > > > >> am
> > > > > > > > > > >> > not
> > > > > > > > > > >> > > > > > really
> > > > > > > > > > >> > > > > > > sure if we will be facing potential issues
> > in
> > > > this
> > > > > > > case
> > > > > > > > or
> > > > > > > > > > >> not.
> > > > > > > > > > >> > > > > Possible
> > > > > > > > > > >> > > > > > > issues I can think of now e.g. too many
> PRC
> > > > calls,
> > > > > > too
> > > > > > > > > many
> > > > > > > > > > >> async
> > > > > > > > > > >> > > > calls
> > > > > > > > > > >> > > > > > > processing, the sub query will return a
> > table
> > > > > which
> > > > > > > > might
> > > > > > > > > be
> > > > > > > > > > >> > (too)
> > > > > > > > > > >> > > > big,
> > > > > > > > > > >> > > > > > and
> > > > > > > > > > >> > > > > > > might cause performance issues. I would
> > > suggest
> > > > > > > > preparing
> > > > > > > > > > some
> > > > > > > > > > >> > use
> > > > > > > > > > >> > > > > cases
> > > > > > > > > > >> > > > > > > and running some performance tests to
> check
> > > it.
> > > > > > These
> > > > > > > > are
> > > > > > > > > my
> > > > > > > > > > >> > > concerns
> > > > > > > > > > >> > > > > > about
> > > > > > > > > > >> > > > > > > using async calls with lateral join and
> I'd
> > > like
> > > > > to
> > > > > > > > share
> > > > > > > > > > with
> > > > > > > > > > >> > you,
> > > > > > > > > > >> > > > > happy
> > > > > > > > > > >> > > > > > > to discuss with you and hear different
> > > opinions,
> > > > > > > > hopefully
> > > > > > > > > > the
> > > > > > > > > > >> > > > > > > discussion could help me understand it
> more
> > > > > deeply.
> > > > > > > > Please
> > > > > > > > > > >> > correct
> > > > > > > > > > >> > > me
> > > > > > > > > > >> > > > > if
> > > > > > > > > > >> > > > > > I
> > > > > > > > > > >> > > > > > > am wrong.
> > > > > > > > > > >> > > > > > >
> > > > > > > > > > >> > > > > > > Best regards,
> > > > > > > > > > >> > > > > > > Jing
> > > > > > > > > > >> > > > > > >
> > > > > > > > > > >> > > > > > >
> > > > > > > > > > >> > > > > > > On Thu, Jun 8, 2023 at 7:22 AM Aitozi <
> > > > > > > > > gjying1314@gmail.com
> > > > > > > > > > >
> > > > > > > > > > >> > > wrote:
> > > > > > > > > > >> > > > > > >
> > > > > > > > > > >> > > > > > > > Hi Mason,
> > > > > > > > > > >> > > > > > > >     Thanks for your input. I think if we
> > > > support
> > > > > > the
> > > > > > > > > user
> > > > > > > > > > >> > defined
> > > > > > > > > > >> > > > > async
> > > > > > > > > > >> > > > > > > > table function,
> > > > > > > > > > >> > > > > > > > user will be able to use it to hold a
> > batch
> > > > data
> > > > > > > then
> > > > > > > > > > >> handle it
> > > > > > > > > > >> > > at
> > > > > > > > > > >> > > > > one
> > > > > > > > > > >> > > > > > > time
> > > > > > > > > > >> > > > > > > > in the customized function.
> > > > > > > > > > >> > > > > > > >
> > > > > > > > > > >> > > > > > > > AsyncSink is meant for the sink
> operator.
> > I
> > > > have
> > > > > > not
> > > > > > > > > > figure
> > > > > > > > > > >> out
> > > > > > > > > > >> > > how
> > > > > > > > > > >> > > > > to
> > > > > > > > > > >> > > > > > > > integrate in this case.
> > > > > > > > > > >> > > > > > > >
> > > > > > > > > > >> > > > > > > > Thanks,
> > > > > > > > > > >> > > > > > > > Atiozi.
> > > > > > > > > > >> > > > > > > >
> > > > > > > > > > >> > > > > > > >
> > > > > > > > > > >> > > > > > > > Mason Chen <ma...@gmail.com>
> > > > > 于2023年6月8日周四
> > > > > > > > > 12:40写道：
> > > > > > > > > > >> > > > > > > >
> > > > > > > > > > >> > > > > > > > > Hi Aitozi,
> > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > >> > > > > > > > > I think it makes sense to make it
> easier
> > > for
> > > > > SQL
> > > > > > > > users
> > > > > > > > > > to
> > > > > > > > > > >> > make
> > > > > > > > > > >> > > > > RPCs.
> > > > > > > > > > >> > > > > > Do
> > > > > > > > > > >> > > > > > > > you
> > > > > > > > > > >> > > > > > > > > think your proposal can extend to the
> > > > ability
> > > > > to
> > > > > > > > batch
> > > > > > > > > > >> data
> > > > > > > > > > >> > for
> > > > > > > > > > >> > > > the
> > > > > > > > > > >> > > > > > > RPC?
> > > > > > > > > > >> > > > > > > > > This is also another common strategy
> to
> > > > > increase
> > > > > > > > > > >> throughput.
> > > > > > > > > > >> > > > Also,
> > > > > > > > > > >> > > > > > have
> > > > > > > > > > >> > > > > > > > you
> > > > > > > > > > >> > > > > > > > > considered solving this a bit
> > differently
> > > by
> > > > > > > > > leveraging
> > > > > > > > > > >> > Flink's
> > > > > > > > > > >> > > > > > > > AsyncSink?
> > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > >> > > > > > > > > Best,
> > > > > > > > > > >> > > > > > > > > Mason
> > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > >> > > > > > > > > On Mon, Jun 5, 2023 at 1:50 AM Aitozi
> <
> > > > > > > > > > >> gjying1314@gmail.com>
> > > > > > > > > > >> > > > > wrote:
> > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > >> > > > > > > > > > One more thing for discussion:
> > > > > > > > > > >> > > > > > > > > >
> > > > > > > > > > >> > > > > > > > > > In our internal implementation, we
> > reuse
> > > > the
> > > > > > > > option
> > > > > > > > > > >> > > > > > > > > >
> > > `table.exec.async-lookup.buffer-capacity`
> > > > > and
> > > > > > > > > > >> > > > > > > > > > `table.exec.async-lookup.timeout` to
> > > > config
> > > > > > > > > > >> > > > > > > > > > the async udtf. Do you think we
> should
> > > add
> > > > > two
> > > > > > > > extra
> > > > > > > > > > >> option
> > > > > > > > > > >> > > to
> > > > > > > > > > >> > > > > > > > > distinguish
> > > > > > > > > > >> > > > > > > > > > from the lookup option such as
> > > > > > > > > > >> > > > > > > > > >
> > > > > > > > > > >> > > > > > > > > >
> > `table.exec.async-udtf.buffer-capacity`
> > > > > > > > > > >> > > > > > > > > > `table.exec.async-udtf.timeout`
> > > > > > > > > > >> > > > > > > > > >
> > > > > > > > > > >> > > > > > > > > >
> > > > > > > > > > >> > > > > > > > > > Best,
> > > > > > > > > > >> > > > > > > > > > Aitozi.
> > > > > > > > > > >> > > > > > > > > >
> > > > > > > > > > >> > > > > > > > > >
> > > > > > > > > > >> > > > > > > > > >
> > > > > > > > > > >> > > > > > > > > > Aitozi <gj...@gmail.com>
> > > > 于2023年6月5日周一
> > > > > > > > 12:20写道：
> > > > > > > > > > >> > > > > > > > > >
> > > > > > > > > > >> > > > > > > > > > > Hi Jing,
> > > > > > > > > > >> > > > > > > > > > >
> > > > > > > > > > >> > > > > > > > > > >     > what is the difference
> between
> > > the
> > > > > RPC
> > > > > > > > call
> > > > > > > > > or
> > > > > > > > > > >> > query
> > > > > > > > > > >> > > > you
> > > > > > > > > > >> > > > > > > > > mentioned
> > > > > > > > > > >> > > > > > > > > > > and the lookup in a very
> > > > > > > > > > >> > > > > > > > > > > general way
> > > > > > > > > > >> > > > > > > > > > >
> > > > > > > > > > >> > > > > > > > > > > I think the RPC call or query
> > service
> > > is
> > > > > > quite
> > > > > > > > > > >> similar to
> > > > > > > > > > >> > > the
> > > > > > > > > > >> > > > > > > lookup
> > > > > > > > > > >> > > > > > > > > > join.
> > > > > > > > > > >> > > > > > > > > > > But lookup join should work
> > > > > > > > > > >> > > > > > > > > > > with `LookupTableSource`.
> > > > > > > > > > >> > > > > > > > > > >
> > > > > > > > > > >> > > > > > > > > > > Let's see how we can perform an
> > async
> > > > RPC
> > > > > > call
> > > > > > > > > with
> > > > > > > > > > >> > lookup
> > > > > > > > > > >> > > > > join:
> > > > > > > > > > >> > > > > > > > > > >
> > > > > > > > > > >> > > > > > > > > > > (1) Implement an
> AsyncTableFunction
> > > with
> > > > > RPC
> > > > > > > > call
> > > > > > > > > > >> logic.
> > > > > > > > > > >> > > > > > > > > > > (2) Implement a
> `LookupTableSource`
> > > > > > connector
> > > > > > > > run
> > > > > > > > > > with
> > > > > > > > > > >> > the
> > > > > > > > > > >> > > > > async
> > > > > > > > > > >> > > > > > > udtf
> > > > > > > > > > >> > > > > > > > > > > defined in (1).
> > > > > > > > > > >> > > > > > > > > > > (3) Then define a DDL of this look
> > up
> > > > > table
> > > > > > in
> > > > > > > > SQL
> > > > > > > > > > >> > > > > > > > > > >
> > > > > > > > > > >> > > > > > > > > > > CREATE TEMPORARY TABLE Customers (
> > > > > > > > > > >> > > > > > > > > > >   id INT,
> > > > > > > > > > >> > > > > > > > > > >   name STRING,
> > > > > > > > > > >> > > > > > > > > > >   country STRING,
> > > > > > > > > > >> > > > > > > > > > >   zip STRING
> > > > > > > > > > >> > > > > > > > > > > ) WITH (
> > > > > > > > > > >> > > > > > > > > > >   'connector' = 'custom'
> > > > > > > > > > >> > > > > > > > > > > );
> > > > > > > > > > >> > > > > > > > > > >
> > > > > > > > > > >> > > > > > > > > > > (4) Run with the query as below:
> > > > > > > > > > >> > > > > > > > > > >
> > > > > > > > > > >> > > > > > > > > > > SELECT o.order_id, o.total,
> > c.country,
> > > > > c.zip
> > > > > > > > > > >> > > > > > > > > > > FROM Orders AS o
> > > > > > > > > > >> > > > > > > > > > >   JOIN Customers FOR SYSTEM_TIME
> AS
> > OF
> > > > > > > > o.proc_time
> > > > > > > > > > AS
> > > > > > > > > > >> c
> > > > > > > > > > >> > > > > > > > > > >     ON o.customer_id = c.id;
> > > > > > > > > > >> > > > > > > > > > >
> > > > > > > > > > >> > > > > > > > > > > This example is from doc
> > > > > > > > > > >> > > > > > > > > > > <
> > > > > > > > > > >> > > > > > > > > >
> > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > >> > > > > > > >
> > > > > > > > > > >> > > > > > >
> > > > > > > > > > >> > > > > >
> > > > > > > > > > >> > > > >
> > > > > > > > > > >> > > >
> > > > > > > > > > >> > >
> > > > > > > > > > >> >
> > > > > > > > > > >>
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/dev/table/sql/queries/joins/#lookup-join
> > > > > > > > > > >> > > > > > > > > > >.You
> > > > > > > > > > >> > > > > > > > > > > can image the look up process as
> an
> > > > async
> > > > > > RPC
> > > > > > > > call
> > > > > > > > > > >> > process.
> > > > > > > > > > >> > > > > > > > > > >
> > > > > > > > > > >> > > > > > > > > > > Let's see how we can perform an
> > async
> > > > RPC
> > > > > > call
> > > > > > > > > with
> > > > > > > > > > >> > lateral
> > > > > > > > > > >> > > > > join:
> > > > > > > > > > >> > > > > > > > > > >
> > > > > > > > > > >> > > > > > > > > > > (1) Implement an
> AsyncTableFunction
> > > with
> > > > > RPC
> > > > > > > > call
> > > > > > > > > > >> logic.
> > > > > > > > > > >> > > > > > > > > > > (2) Run query with
> > > > > > > > > > >> > > > > > > > > > >
> > > > > > > > > > >> > > > > > > > > > > Create function f1 as '...' ;
> > > > > > > > > > >> > > > > > > > > > >
> > > > > > > > > > >> > > > > > > > > > > SELECT o.order_id, o.total,
> > c.country,
> > > > > c.zip
> > > > > > > > FROM
> > > > > > > > > > >> Orders
> > > > > > > > > > >> > > > > lateral
> > > > > > > > > > >> > > > > > > > table
> > > > > > > > > > >> > > > > > > > > > > (f1(order_id)) as T(...);
> > > > > > > > > > >> > > > > > > > > > >
> > > > > > > > > > >> > > > > > > > > > > As you can see, the lateral join
> > > version
> > > > > is
> > > > > > > more
> > > > > > > > > > >> simple
> > > > > > > > > > >> > and
> > > > > > > > > > >> > > > > > > intuitive
> > > > > > > > > > >> > > > > > > > > to
> > > > > > > > > > >> > > > > > > > > > > users. Users do not have to wrap a
> > > > > > > > > > >> > > > > > > > > > > LookupTableSource for the purpose
> of
> > > > using
> > > > > > > async
> > > > > > > > > > udtf.
> > > > > > > > > > >> > > > > > > > > > >
> > > > > > > > > > >> > > > > > > > > > > In the end, We can also see the
> user
> > > > > defined
> > > > > > > > async
> > > > > > > > > > >> table
> > > > > > > > > > >> > > > > function
> > > > > > > > > > >> > > > > > > is
> > > > > > > > > > >> > > > > > > > an
> > > > > > > > > > >> > > > > > > > > > > enhancement of the current lateral
> > > table
> > > > > > join
> > > > > > > > > > >> > > > > > > > > > > which only supports sync lateral
> > join
> > > > now.
> > > > > > > > > > >> > > > > > > > > > >
> > > > > > > > > > >> > > > > > > > > > > Best,
> > > > > > > > > > >> > > > > > > > > > > Aitozi.
> > > > > > > > > > >> > > > > > > > > > >
> > > > > > > > > > >> > > > > > > > > > >
> > > > > > > > > > >> > > > > > > > > > > Jing Ge
> <jing@ververica.com.invalid
> > >
> > > > > > > > 于2023年6月2日周五
> > > > > > > > > > >> > 19:37写道：
> > > > > > > > > > >> > > > > > > > > > >
> > > > > > > > > > >> > > > > > > > > > >> Hi Aitozi,
> > > > > > > > > > >> > > > > > > > > > >>
> > > > > > > > > > >> > > > > > > > > > >> Thanks for the update. Just out
> of
> > > > > > curiosity,
> > > > > > > > > what
> > > > > > > > > > is
> > > > > > > > > > >> > the
> > > > > > > > > > >> > > > > > > difference
> > > > > > > > > > >> > > > > > > > > > >> between the RPC call or query you
> > > > > mentioned
> > > > > > > and
> > > > > > > > > the
> > > > > > > > > > >> > lookup
> > > > > > > > > > >> > > > in
> > > > > > > > > > >> > > > > a
> > > > > > > > > > >> > > > > > > very
> > > > > > > > > > >> > > > > > > > > > >> general way? Since Lateral join
> is
> > > used
> > > > > in
> > > > > > > the
> > > > > > > > > > FLIP.
> > > > > > > > > > >> Is
> > > > > > > > > > >> > > > there
> > > > > > > > > > >> > > > > > any
> > > > > > > > > > >> > > > > > > > > > special
> > > > > > > > > > >> > > > > > > > > > >> thought for that? Sorry for
> asking
> > so
> > > > > many
> > > > > > > > > > questions.
> > > > > > > > > > >> > The
> > > > > > > > > > >> > > > FLIP
> > > > > > > > > > >> > > > > > > > > contains
> > > > > > > > > > >> > > > > > > > > > >> limited information to understand
> > the
> > > > > > > > motivation.
> > > > > > > > > > >> > > > > > > > > > >>
> > > > > > > > > > >> > > > > > > > > > >> Best regards,
> > > > > > > > > > >> > > > > > > > > > >> Jing
> > > > > > > > > > >> > > > > > > > > > >>
> > > > > > > > > > >> > > > > > > > > > >> On Fri, Jun 2, 2023 at 3:48 AM
> > > Aitozi <
> > > > > > > > > > >> > > gjying1314@gmail.com
> > > > > > > > > > >> > > > >
> > > > > > > > > > >> > > > > > > wrote:
> > > > > > > > > > >> > > > > > > > > > >>
> > > > > > > > > > >> > > > > > > > > > >> > Hi Jing,
> > > > > > > > > > >> > > > > > > > > > >> >     I have updated the proposed
> > > > changes
> > > > > > to
> > > > > > > > the
> > > > > > > > > > >> FLIP.
> > > > > > > > > > >> > > IMO,
> > > > > > > > > > >> > > > > > lookup
> > > > > > > > > > >> > > > > > > > has
> > > > > > > > > > >> > > > > > > > > > its
> > > > > > > > > > >> > > > > > > > > > >> > clear
> > > > > > > > > > >> > > > > > > > > > >> > async call requirement is due
> to
> > > its
> > > > IO
> > > > > > > heavy
> > > > > > > > > > >> > operator.
> > > > > > > > > > >> > > In
> > > > > > > > > > >> > > > > our
> > > > > > > > > > >> > > > > > > > > usage,
> > > > > > > > > > >> > > > > > > > > > >> sql
> > > > > > > > > > >> > > > > > > > > > >> > users have
> > > > > > > > > > >> > > > > > > > > > >> > logic to do some RPC call or
> > query
> > > > the
> > > > > > > > > > third-party
> > > > > > > > > > >> > > service
> > > > > > > > > > >> > > > > > which
> > > > > > > > > > >> > > > > > > > is
> > > > > > > > > > >> > > > > > > > > > >> also IO
> > > > > > > > > > >> > > > > > > > > > >> > intensive.
> > > > > > > > > > >> > > > > > > > > > >> > In these case, we'd like to
> > > leverage
> > > > > the
> > > > > > > > async
> > > > > > > > > > >> > function
> > > > > > > > > > >> > > to
> > > > > > > > > > >> > > > > > > improve
> > > > > > > > > > >> > > > > > > > > the
> > > > > > > > > > >> > > > > > > > > > >> > throughput.
> > > > > > > > > > >> > > > > > > > > > >> >
> > > > > > > > > > >> > > > > > > > > > >> > Thanks,
> > > > > > > > > > >> > > > > > > > > > >> > Aitozi.
> > > > > > > > > > >> > > > > > > > > > >> >
> > > > > > > > > > >> > > > > > > > > > >> > Jing Ge
> > <jing@ververica.com.invalid
> > > >
> > > > > > > > > > 于2023年6月1日周四
> > > > > > > > > > >> > > > 22:55写道：
> > > > > > > > > > >> > > > > > > > > > >> >
> > > > > > > > > > >> > > > > > > > > > >> > > Hi Aitozi,
> > > > > > > > > > >> > > > > > > > > > >> > >
> > > > > > > > > > >> > > > > > > > > > >> > > Sorry for the late reply.
> Would
> > > you
> > > > > > like
> > > > > > > to
> > > > > > > > > > >> update
> > > > > > > > > > >> > the
> > > > > > > > > > >> > > > > > > proposed
> > > > > > > > > > >> > > > > > > > > > >> changes
> > > > > > > > > > >> > > > > > > > > > >> > > with more details into the
> FLIP
> > > > too?
> > > > > > > > > > >> > > > > > > > > > >> > > I got your point. It looks
> > like a
> > > > > > > rational
> > > > > > > > > > idea.
> > > > > > > > > > >> > > > However,
> > > > > > > > > > >> > > > > > > since
> > > > > > > > > > >> > > > > > > > > > lookup
> > > > > > > > > > >> > > > > > > > > > >> > has
> > > > > > > > > > >> > > > > > > > > > >> > > its clear async call
> > requirement,
> > > > are
> > > > > > > there
> > > > > > > > > any
> > > > > > > > > > >> real
> > > > > > > > > > >> > > use
> > > > > > > > > > >> > > > > > cases
> > > > > > > > > > >> > > > > > > > > that
> > > > > > > > > > >> > > > > > > > > > >> > > need this change? This will
> > help
> > > us
> > > > > > > > > understand
> > > > > > > > > > >> the
> > > > > > > > > > >> > > > > > motivation.
> > > > > > > > > > >> > > > > > > > > After
> > > > > > > > > > >> > > > > > > > > > >> all,
> > > > > > > > > > >> > > > > > > > > > >> > > lateral join and temporal
> > lookup
> > > > > > join[1]
> > > > > > > > are
> > > > > > > > > > >> quite
> > > > > > > > > > >> > > > > > different.
> > > > > > > > > > >> > > > > > > > > > >> > >
> > > > > > > > > > >> > > > > > > > > > >> > > Best regards,
> > > > > > > > > > >> > > > > > > > > > >> > > Jing
> > > > > > > > > > >> > > > > > > > > > >> > >
> > > > > > > > > > >> > > > > > > > > > >> > >
> > > > > > > > > > >> > > > > > > > > > >> > > [1]
> > > > > > > > > > >> > > > > > > > > > >> > >
> > > > > > > > > > >> > > > > > > > > > >> > >
> > > > > > > > > > >> > > > > > > > > > >> >
> > > > > > > > > > >> > > > > > > > > > >>
> > > > > > > > > > >> > > > > > > > > >
> > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > >> > > > > > > >
> > > > > > > > > > >> > > > > > >
> > > > > > > > > > >> > > > > >
> > > > > > > > > > >> > > > >
> > > > > > > > > > >> > > >
> > > > > > > > > > >> > >
> > > > > > > > > > >> >
> > > > > > > > > > >>
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/flink/blob/d90a72da2fd601ca4e2a46700e91ec5b348de2ad/flink-table/flink-table-common/src/main/java/org/apache/flink/table/functions/AsyncTableFunction.java#L54
> > > > > > > > > > >> > > > > > > > > > >> > >
> > > > > > > > > > >> > > > > > > > > > >> > > On Wed, May 31, 2023 at
> 8:53 AM
> > > > > Aitozi
> > > > > > <
> > > > > > > > > > >> > > > > > gjying1314@gmail.com>
> > > > > > > > > > >> > > > > > > > > > wrote:
> > > > > > > > > > >> > > > > > > > > > >> > >
> > > > > > > > > > >> > > > > > > > > > >> > > > Hi Jing,
> > > > > > > > > > >> > > > > > > > > > >> > > >     What do you think about
> > it?
> > > > Can
> > > > > > we
> > > > > > > > move
> > > > > > > > > > >> > forward
> > > > > > > > > > >> > > > this
> > > > > > > > > > >> > > > > > > > > feature?
> > > > > > > > > > >> > > > > > > > > > >> > > >
> > > > > > > > > > >> > > > > > > > > > >> > > > Thanks,
> > > > > > > > > > >> > > > > > > > > > >> > > > Aitozi.
> > > > > > > > > > >> > > > > > > > > > >> > > >
> > > > > > > > > > >> > > > > > > > > > >> > > > Aitozi <
> gjying1314@gmail.com
> > >
> > > > > > > > > 于2023年5月29日周一
> > > > > > > > > > >> > > 09:56写道：
> > > > > > > > > > >> > > > > > > > > > >> > > >
> > > > > > > > > > >> > > > > > > > > > >> > > > > Hi Jing,
> > > > > > > > > > >> > > > > > > > > > >> > > > >     > "Do you mean to
> > support
> > > > the
> > > > > > > > > > >> > > AyncTableFunction
> > > > > > > > > > >> > > > > > beyond
> > > > > > > > > > >> > > > > > > > the
> > > > > > > > > > >> > > > > > > > > > >> > > > > LookupTableSource?"
> > > > > > > > > > >> > > > > > > > > > >> > > > > Yes, I mean to support
> the
> > > > > > > > > > AyncTableFunction
> > > > > > > > > > >> > > beyond
> > > > > > > > > > >> > > > > the
> > > > > > > > > > >> > > > > > > > > > >> > > > LookupTableSource.
> > > > > > > > > > >> > > > > > > > > > >> > > > >
> > > > > > > > > > >> > > > > > > > > > >> > > > > The "AsyncTableFunction"
> is
> > > the
> > > > > > > > function
> > > > > > > > > > with
> > > > > > > > > > >> > > > ability
> > > > > > > > > > >> > > > > to
> > > > > > > > > > >> > > > > > > be
> > > > > > > > > > >> > > > > > > > > > >> executed
> > > > > > > > > > >> > > > > > > > > > >> > > > async
> > > > > > > > > > >> > > > > > > > > > >> > > > > (with AsyncWaitOperator).
> > > > > > > > > > >> > > > > > > > > > >> > > > > The async lookup join is
> a
> > > one
> > > > of
> > > > > > > usage
> > > > > > > > > of
> > > > > > > > > > >> this.
> > > > > > > > > > >> > > So,
> > > > > > > > > > >> > > > > we
> > > > > > > > > > >> > > > > > > > don't
> > > > > > > > > > >> > > > > > > > > > >> have to
> > > > > > > > > > >> > > > > > > > > > >> > > > bind
> > > > > > > > > > >> > > > > > > > > > >> > > > > the AyncTableFunction
> with
> > > > > > > > > > LookupTableSource.
> > > > > > > > > > >> > > > > > > > > > >> > > > > If User-defined
> > > > > AsyncTableFunction
> > > > > > is
> > > > > > > > > > >> supported,
> > > > > > > > > > >> > > > user
> > > > > > > > > > >> > > > > > can
> > > > > > > > > > >> > > > > > > > > > directly
> > > > > > > > > > >> > > > > > > > > > >> > use
> > > > > > > > > > >> > > > > > > > > > >> > > > > lateral table syntax to
> > > perform
> > > > > > async
> > > > > > > > > > >> operation.
> > > > > > > > > > >> > > > > > > > > > >> > > > >
> > > > > > > > > > >> > > > > > > > > > >> > > > > > "It would be better if
> > you
> > > > > could
> > > > > > > > > > elaborate
> > > > > > > > > > >> the
> > > > > > > > > > >> > > > > > proposed
> > > > > > > > > > >> > > > > > > > > > changes
> > > > > > > > > > >> > > > > > > > > > >> wrt
> > > > > > > > > > >> > > > > > > > > > >> > > the
> > > > > > > > > > >> > > > > > > > > > >> > > > > CorrelatedCodeGenerator
> > with
> > > > more
> > > > > > > > > details"
> > > > > > > > > > >> > > > > > > > > > >> > > > >
> > > > > > > > > > >> > > > > > > > > > >> > > > > In the proposal, we use
> > > lateral
> > > > > > table
> > > > > > > > > > syntax
> > > > > > > > > > >> to
> > > > > > > > > > >> > > > > support
> > > > > > > > > > >> > > > > > > the
> > > > > > > > > > >> > > > > > > > > > async
> > > > > > > > > > >> > > > > > > > > > >> > table
> > > > > > > > > > >> > > > > > > > > > >> > > > > function. So the planner
> > will
> > > > > also
> > > > > > > > treat
> > > > > > > > > > this
> > > > > > > > > > >> > > > > statement
> > > > > > > > > > >> > > > > > > to a
> > > > > > > > > > >> > > > > > > > > > >> > > > > CommonExecCorrelate node.
> > So
> > > > the
> > > > > > > > runtime
> > > > > > > > > > code
> > > > > > > > > > >> > > should
> > > > > > > > > > >> > > > > be
> > > > > > > > > > >> > > > > > > > > > generated
> > > > > > > > > > >> > > > > > > > > > >> in
> > > > > > > > > > >> > > > > > > > > > >> > > > > CorrelatedCodeGenerator.
> > > > > > > > > > >> > > > > > > > > > >> > > > > In
> CorrelatedCodeGenerator,
> > > we
> > > > > will
> > > > > > > > know
> > > > > > > > > > the
> > > > > > > > > > >> > > > > > > TableFunction's
> > > > > > > > > > >> > > > > > > > > > Kind
> > > > > > > > > > >> > > > > > > > > > >> of
> > > > > > > > > > >> > > > > > > > > > >> > > > > `FunctionKind.Table` or
> > > > > > > > > > >> > `FunctionKind.ASYNC_TABLE`
> > > > > > > > > > >> > > > > > > > > > >> > > > > For
> > > `FunctionKind.ASYNC_TABLE`
> > > > > we
> > > > > > > can
> > > > > > > > > > >> generate
> > > > > > > > > > >> > a
> > > > > > > > > > >> > > > > > > > > > >> AsyncWaitOperator
> > > > > > > > > > >> > > > > > > > > > >> > to
> > > > > > > > > > >> > > > > > > > > > >> > > > > execute the async table
> > > > function.
> > > > > > > > > > >> > > > > > > > > > >> > > > >
> > > > > > > > > > >> > > > > > > > > > >> > > > >
> > > > > > > > > > >> > > > > > > > > > >> > > > > Thanks,
> > > > > > > > > > >> > > > > > > > > > >> > > > > Aitozi.
> > > > > > > > > > >> > > > > > > > > > >> > > > >
> > > > > > > > > > >> > > > > > > > > > >> > > > >
> > > > > > > > > > >> > > > > > > > > > >> > > > > Jing Ge
> > > > > <jing@ververica.com.invalid
> > > > > > >
> > > > > > > > > > >> > > 于2023年5月29日周一
> > > > > > > > > > >> > > > > > > 03:22写道：
> > > > > > > > > > >> > > > > > > > > > >> > > > >
> > > > > > > > > > >> > > > > > > > > > >> > > > >> Hi Aitozi,
> > > > > > > > > > >> > > > > > > > > > >> > > > >>
> > > > > > > > > > >> > > > > > > > > > >> > > > >> Thanks for the
> > > clarification.
> > > > > The
> > > > > > > > naming
> > > > > > > > > > >> > "Lookup"
> > > > > > > > > > >> > > > > might
> > > > > > > > > > >> > > > > > > > > suggest
> > > > > > > > > > >> > > > > > > > > > >> > using
> > > > > > > > > > >> > > > > > > > > > >> > > it
> > > > > > > > > > >> > > > > > > > > > >> > > > >> for table look up. But
> > > > > > conceptually
> > > > > > > > what
> > > > > > > > > > the
> > > > > > > > > > >> > > eval()
> > > > > > > > > > >> > > > > > > method
> > > > > > > > > > >> > > > > > > > > will
> > > > > > > > > > >> > > > > > > > > > >> do
> > > > > > > > > > >> > > > > > > > > > >> > is
> > > > > > > > > > >> > > > > > > > > > >> > > to
> > > > > > > > > > >> > > > > > > > > > >> > > > >> get a collection of
> > > > results(Row,
> > > > > > > > > RowData)
> > > > > > > > > > >> from
> > > > > > > > > > >> > > the
> > > > > > > > > > >> > > > > > given
> > > > > > > > > > >> > > > > > > > > keys.
> > > > > > > > > > >> > > > > > > > > > >> How
> > > > > > > > > > >> > > > > > > > > > >> > it
> > > > > > > > > > >> > > > > > > > > > >> > > > will
> > > > > > > > > > >> > > > > > > > > > >> > > > >> be done depends on the
> > > > > > > implementation,
> > > > > > > > > > i.e.
> > > > > > > > > > >> you
> > > > > > > > > > >> > > can
> > > > > > > > > > >> > > > > > > > implement
> > > > > > > > > > >> > > > > > > > > > >> your
> > > > > > > > > > >> > > > > > > > > > >> > own
> > > > > > > > > > >> > > > > > > > > > >> > > > >> Source[1][2]. The
> example
> > in
> > > > the
> > > > > > > FLIP
> > > > > > > > > > >> should be
> > > > > > > > > > >> > > > able
> > > > > > > > > > >> > > > > to
> > > > > > > > > > >> > > > > > > be
> > > > > > > > > > >> > > > > > > > > > >> handled
> > > > > > > > > > >> > > > > > > > > > >> > in
> > > > > > > > > > >> > > > > > > > > > >> > > > this
> > > > > > > > > > >> > > > > > > > > > >> > > > >> way.
> > > > > > > > > > >> > > > > > > > > > >> > > > >>
> > > > > > > > > > >> > > > > > > > > > >> > > > >> Do you mean to support
> the
> > > > > > > > > > AyncTableFunction
> > > > > > > > > > >> > > beyond
> > > > > > > > > > >> > > > > the
> > > > > > > > > > >> > > > > > > > > > >> > > > LookupTableSource?
> > > > > > > > > > >> > > > > > > > > > >> > > > >> It would be better if
> you
> > > > could
> > > > > > > > > elaborate
> > > > > > > > > > >> the
> > > > > > > > > > >> > > > > proposed
> > > > > > > > > > >> > > > > > > > > changes
> > > > > > > > > > >> > > > > > > > > > >> wrt
> > > > > > > > > > >> > > > > > > > > > >> > the
> > > > > > > > > > >> > > > > > > > > > >> > > > >> CorrelatedCodeGenerator
> > with
> > > > > more
> > > > > > > > > details.
> > > > > > > > > > >> > > Thanks!
> > > > > > > > > > >> > > > > > > > > > >> > > > >>
> > > > > > > > > > >> > > > > > > > > > >> > > > >> Best regards,
> > > > > > > > > > >> > > > > > > > > > >> > > > >> Jing
> > > > > > > > > > >> > > > > > > > > > >> > > > >>
> > > > > > > > > > >> > > > > > > > > > >> > > > >> [1]
> > > > > > > > > > >> > > > > > > > > > >> > > > >>
> > > > > > > > > > >> > > > > > > > > > >> > > > >>
> > > > > > > > > > >> > > > > > > > > > >> > > >
> > > > > > > > > > >> > > > > > > > > > >> > >
> > > > > > > > > > >> > > > > > > > > > >> >
> > > > > > > > > > >> > > > > > > > > > >>
> > > > > > > > > > >> > > > > > > > > >
> > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > >> > > > > > > >
> > > > > > > > > > >> > > > > > >
> > > > > > > > > > >> > > > > >
> > > > > > > > > > >> > > > >
> > > > > > > > > > >> > > >
> > > > > > > > > > >> > >
> > > > > > > > > > >> >
> > > > > > > > > > >>
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/flink/blob/678370b18e1b6c4a23e5ce08f8efd05675a0cc17/flink-table/flink-table-common/src/main/java/org/apache/flink/table/connector/source/LookupTableSource.java#L64
> > > > > > > > > > >> > > > > > > > > > >> > > > >> [2]
> > > > > > > > > > >> > > > > > > > > > >> > > > >>
> > > > > > > > > > >> > > > > > > > > > >> > > > >>
> > > > > > > > > > >> > > > > > > > > > >> > > >
> > > > > > > > > > >> > > > > > > > > > >> > >
> > > > > > > > > > >> > > > > > > > > > >> >
> > > > > > > > > > >> > > > > > > > > > >>
> > > > > > > > > > >> > > > > > > > > >
> > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > >> > > > > > > >
> > > > > > > > > > >> > > > > > >
> > > > > > > > > > >> > > > > >
> > > > > > > > > > >> > > > >
> > > > > > > > > > >> > > >
> > > > > > > > > > >> > >
> > > > > > > > > > >> >
> > > > > > > > > > >>
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/flink/blob/678370b18e1b6c4a23e5ce08f8efd05675a0cc17/flink-table/flink-table-common/src/main/java/org/apache/flink/table/connector/source/AsyncTableFunctionProvider.java#L49
> > > > > > > > > > >> > > > > > > > > > >> > > > >>
> > > > > > > > > > >> > > > > > > > > > >> > > > >> On Sat, May 27, 2023 at
> > > > 9:48 AM
> > > > > > > > Aitozi <
> > > > > > > > > > >> > > > > > > > gjying1314@gmail.com
> > > > > > > > > > >> > > > > > > > > >
> > > > > > > > > > >> > > > > > > > > > >> > wrote:
> > > > > > > > > > >> > > > > > > > > > >> > > > >>
> > > > > > > > > > >> > > > > > > > > > >> > > > >> > Hi Jing,
> > > > > > > > > > >> > > > > > > > > > >> > > > >> >     Thanks for your
> > > > response.
> > > > > As
> > > > > > > > > stated
> > > > > > > > > > in
> > > > > > > > > > >> > the
> > > > > > > > > > >> > > > > FLIP,
> > > > > > > > > > >> > > > > > > the
> > > > > > > > > > >> > > > > > > > > > >> purpose
> > > > > > > > > > >> > > > > > > > > > >> > of
> > > > > > > > > > >> > > > > > > > > > >> > > > this
> > > > > > > > > > >> > > > > > > > > > >> > > > >> > FLIP is meant to
> support
> > > > > > > > > > >> > > > > > > > > > >> > > > >> > user-defined async
> table
> > > > > > function.
> > > > > > > > As
> > > > > > > > > > >> > described
> > > > > > > > > > >> > > > in
> > > > > > > > > > >> > > > > > > flink
> > > > > > > > > > >> > > > > > > > > > >> document
> > > > > > > > > > >> > > > > > > > > > >> > > [1]
> > > > > > > > > > >> > > > > > > > > > >> > > > >> >
> > > > > > > > > > >> > > > > > > > > > >> > > > >> > Async table functions
> > are
> > > > > > special
> > > > > > > > > > >> functions
> > > > > > > > > > >> > for
> > > > > > > > > > >> > > > > table
> > > > > > > > > > >> > > > > > > > > sources
> > > > > > > > > > >> > > > > > > > > > >> that
> > > > > > > > > > >> > > > > > > > > > >> > > > >> perform
> > > > > > > > > > >> > > > > > > > > > >> > > > >> > > a lookup.
> > > > > > > > > > >> > > > > > > > > > >> > > > >> > >
> > > > > > > > > > >> > > > > > > > > > >> > > > >> >
> > > > > > > > > > >> > > > > > > > > > >> > > > >> > So end user can not
> > > directly
> > > > > > > define
> > > > > > > > > and
> > > > > > > > > > >> use
> > > > > > > > > > >> > > async
> > > > > > > > > > >> > > > > > table
> > > > > > > > > > >> > > > > > > > > > >> function
> > > > > > > > > > >> > > > > > > > > > >> > > now.
> > > > > > > > > > >> > > > > > > > > > >> > > > An
> > > > > > > > > > >> > > > > > > > > > >> > > > >> > user case is reported
> in
> > > [2]
> > > > > > > > > > >> > > > > > > > > > >> > > > >> >
> > > > > > > > > > >> > > > > > > > > > >> > > > >> > So, in conclusion, no
> > new
> > > > > > > interface
> > > > > > > > is
> > > > > > > > > > >> > > > introduced,
> > > > > > > > > > >> > > > > > but
> > > > > > > > > > >> > > > > > > we
> > > > > > > > > > >> > > > > > > > > > >> extend
> > > > > > > > > > >> > > > > > > > > > >> > the
> > > > > > > > > > >> > > > > > > > > > >> > > > >> > ability to support
> > > > > user-defined
> > > > > > > > async
> > > > > > > > > > >> table
> > > > > > > > > > >> > > > > function.
> > > > > > > > > > >> > > > > > > > > > >> > > > >> >
> > > > > > > > > > >> > > > > > > > > > >> > > > >> > [1]:
> > > > > > > > > > >> > > > > > > > > > >> > > > >> >
> > > > > > > > > > >> > > > > > > > > > >> > > > >> >
> > > > > > > > > > >> > > > > > > > > > >> > > > >>
> > > > > > > > > > >> > > > > > > > > > >> > > >
> > > > > > > > > > >> > > > > > > > > > >> > >
> > > > > > > > > > >> > > > > > > > > > >> >
> > > > > > > > > > >> > > > > > > > > > >>
> > > > > > > > > > >> > > > > > > > > >
> > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > >> > > > > > > >
> > > > > > > > > > >> > > > > > >
> > > > > > > > > > >> > > > > >
> > > > > > > > > > >> > > > >
> > > > > > > > > > >> > > >
> > > > > > > > > > >> > >
> > > > > > > > > > >> >
> > > > > > > > > > >>
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/dev/table/functions/udfs/
> > > > > > > > > > >> > > > > > > > > > >> > > > >> > [2]:
> > > > > > > > > > >> > > > > > > > > > >> > >
> > > > > > > > > > >> > > > > > > >
> > > > > > > > > > >> >
> > > > > > >
> https://lists.apache.org/thread/qljwd40v5ntz6733cwcdr8s4z97b343b
> > > > > > > > > > >> > > > > > > > > > >> > > > >> >
> > > > > > > > > > >> > > > > > > > > > >> > > > >> > Thanks.
> > > > > > > > > > >> > > > > > > > > > >> > > > >> > Aitozi.
> > > > > > > > > > >> > > > > > > > > > >> > > > >> >
> > > > > > > > > > >> > > > > > > > > > >> > > > >> >
> > > > > > > > > > >> > > > > > > > > > >> > > > >> > Jing Ge
> > > > > > > <jing@ververica.com.invalid
> > > > > > > > >
> > > > > > > > > > >> > > > 于2023年5月27日周六
> > > > > > > > > > >> > > > > > > > > 06:40写道：
> > > > > > > > > > >> > > > > > > > > > >> > > > >> >
> > > > > > > > > > >> > > > > > > > > > >> > > > >> > > Hi Aitozi,
> > > > > > > > > > >> > > > > > > > > > >> > > > >> > >
> > > > > > > > > > >> > > > > > > > > > >> > > > >> > > Thanks for your
> > > proposal.
> > > > I
> > > > > am
> > > > > > > not
> > > > > > > > > > quite
> > > > > > > > > > >> > sure
> > > > > > > > > > >> > > > if
> > > > > > > > > > >> > > > > I
> > > > > > > > > > >> > > > > > > > > > understood
> > > > > > > > > > >> > > > > > > > > > >> > your
> > > > > > > > > > >> > > > > > > > > > >> > > > >> > thoughts
> > > > > > > > > > >> > > > > > > > > > >> > > > >> > > correctly. You
> > > described a
> > > > > > > special
> > > > > > > > > > case
> > > > > > > > > > >> > > > > > > implementation
> > > > > > > > > > >> > > > > > > > of
> > > > > > > > > > >> > > > > > > > > > the
> > > > > > > > > > >> > > > > > > > > > >> > > > >> > > AsyncTableFunction
> > with
> > > on
> > > > > > > public
> > > > > > > > > API
> > > > > > > > > > >> > > changes.
> > > > > > > > > > >> > > > > > Would
> > > > > > > > > > >> > > > > > > > you
> > > > > > > > > > >> > > > > > > > > > >> please
> > > > > > > > > > >> > > > > > > > > > >> > > > >> elaborate
> > > > > > > > > > >> > > > > > > > > > >> > > > >> > > your purpose of
> > writing
> > > a
> > > > > FLIP
> > > > > > > > > > >> according to
> > > > > > > > > > >> > > the
> > > > > > > > > > >> > > > > > FLIP
> > > > > > > > > > >> > > > > > > > > > >> > > > documentation[1]?
> > > > > > > > > > >> > > > > > > > > > >> > > > >> > > Thanks!
> > > > > > > > > > >> > > > > > > > > > >> > > > >> > >
> > > > > > > > > > >> > > > > > > > > > >> > > > >> > > [1]
> > > > > > > > > > >> > > > > > > > > > >> > > > >> > >
> > > > > > > > > > >> > > > > > > > > > >> > > > >> > >
> > > > > > > > > > >> > > > > > > > > > >> > > > >> >
> > > > > > > > > > >> > > > > > > > > > >> > > > >>
> > > > > > > > > > >> > > > > > > > > > >> > > >
> > > > > > > > > > >> > > > > > > > > > >> > >
> > > > > > > > > > >> > > > > > > > > > >> >
> > > > > > > > > > >> > > > > > > > > > >>
> > > > > > > > > > >> > > > > > > > > >
> > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > >> > > > > > > >
> > > > > > > > > > >> > > > > > >
> > > > > > > > > > >> > > > > >
> > > > > > > > > > >> > > > >
> > > > > > > > > > >> > > >
> > > > > > > > > > >> > >
> > > > > > > > > > >> >
> > > > > > > > > > >>
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/Flink+Improvement+Proposals
> > > > > > > > > > >> > > > > > > > > > >> > > > >> > >
> > > > > > > > > > >> > > > > > > > > > >> > > > >> > > Best regards,
> > > > > > > > > > >> > > > > > > > > > >> > > > >> > > Jing
> > > > > > > > > > >> > > > > > > > > > >> > > > >> > >
> > > > > > > > > > >> > > > > > > > > > >> > > > >> > > On Wed, May 24, 2023
> > at
> > > > > > 1:07 PM
> > > > > > > > > > Aitozi <
> > > > > > > > > > >> > > > > > > > > > gjying1314@gmail.com
> > > > > > > > > > >> > > > > > > > > > >> >
> > > > > > > > > > >> > > > > > > > > > >> > > > wrote:
> > > > > > > > > > >> > > > > > > > > > >> > > > >> > >
> > > > > > > > > > >> > > > > > > > > > >> > > > >> > > > May I ask for some
> > > > > feedback
> > > > > > > :D
> > > > > > > > > > >> > > > > > > > > > >> > > > >> > > >
> > > > > > > > > > >> > > > > > > > > > >> > > > >> > > > Thanks,
> > > > > > > > > > >> > > > > > > > > > >> > > > >> > > > Aitozi
> > > > > > > > > > >> > > > > > > > > > >> > > > >> > > >
> > > > > > > > > > >> > > > > > > > > > >> > > > >> > > > Aitozi <
> > > > > > gjying1314@gmail.com>
> > > > > > > > > > >> > > 于2023年5月23日周二
> > > > > > > > > > >> > > > > > > 19:14写道：
> > > > > > > > > > >> > > > > > > > > > >> > > > >> > > > >
> > > > > > > > > > >> > > > > > > > > > >> > > > >> > > > > Just catch an
> user
> > > > case
> > > > > > > report
> > > > > > > > > > from
> > > > > > > > > > >> > > Giannis
> > > > > > > > > > >> > > > > > > Polyzos
> > > > > > > > > > >> > > > > > > > > for
> > > > > > > > > > >> > > > > > > > > > >> this
> > > > > > > > > > >> > > > > > > > > > >> > > > >> usage:
> > > > > > > > > > >> > > > > > > > > > >> > > > >> > > > >
> > > > > > > > > > >> > > > > > > > > > >> > > > >> > > > >
> > > > > > > > > > >> > > > > > > > > > >> > > >
> > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > >> > >
> > > > > > > >
> > https://lists.apache.org/thread/qljwd40v5ntz6733cwcdr8s4z97b343b
> > > > > > > > > > >> > > > > > > > > > >> > > > >> > > > >
> > > > > > > > > > >> > > > > > > > > > >> > > > >> > > > > Aitozi <
> > > > > > > gjying1314@gmail.com>
> > > > > > > > > > >> > > > 于2023年5月23日周二
> > > > > > > > > > >> > > > > > > > 17:45写道：
> > > > > > > > > > >> > > > > > > > > > >> > > > >> > > > > >
> > > > > > > > > > >> > > > > > > > > > >> > > > >> > > > > > Hi guys,
> > > > > > > > > > >> > > > > > > > > > >> > > > >> > > > > >     I want to
> > > bring
> > > > > up a
> > > > > > > > > > >> discussion
> > > > > > > > > > >> > > about
> > > > > > > > > > >> > > > > > > adding
> > > > > > > > > > >> > > > > > > > > > >> support
> > > > > > > > > > >> > > > > > > > > > >> > of
> > > > > > > > > > >> > > > > > > > > > >> > > > User
> > > > > > > > > > >> > > > > > > > > > >> > > > >> > > > > > Defined
> > > > > > AsyncTableFunction
> > > > > > > > in
> > > > > > > > > > >> Flink.
> > > > > > > > > > >> > > > > > > > > > >> > > > >> > > > > > Currently,
> async
> > > > table
> > > > > > > > > function
> > > > > > > > > > >> are
> > > > > > > > > > >> > > > special
> > > > > > > > > > >> > > > > > > > > functions
> > > > > > > > > > >> > > > > > > > > > >> for
> > > > > > > > > > >> > > > > > > > > > >> > > > table
> > > > > > > > > > >> > > > > > > > > > >> > > > >> > > source
> > > > > > > > > > >> > > > > > > > > > >> > > > >> > > > > > to perform
> > > > > > > > > > >> > > > > > > > > > >> > > > >> > > > > > async lookup.
> > > > However,
> > > > > > > it's
> > > > > > > > > > worth
> > > > > > > > > > >> to
> > > > > > > > > > >> > > > > support
> > > > > > > > > > >> > > > > > > the
> > > > > > > > > > >> > > > > > > > > user
> > > > > > > > > > >> > > > > > > > > > >> > > defined
> > > > > > > > > > >> > > > > > > > > > >> > > > >> async
> > > > > > > > > > >> > > > > > > > > > >> > > > >> > > > > > table
> function.
> > > > > > > > > > >> > > > > > > > > > >> > > > >> > > > > > Because, in
> this
> > > > way,
> > > > > > the
> > > > > > > > end
> > > > > > > > > > SQL
> > > > > > > > > > >> > user
> > > > > > > > > > >> > > > can
> > > > > > > > > > >> > > > > > > > leverage
> > > > > > > > > > >> > > > > > > > > > it
> > > > > > > > > > >> > > > > > > > > > >> to
> > > > > > > > > > >> > > > > > > > > > >> > > > >> perform
> > > > > > > > > > >> > > > > > > > > > >> > > > >> > the
> > > > > > > > > > >> > > > > > > > > > >> > > > >> > > > > > async
> operation
> > > > > > > > > > >> > > > > > > > > > >> > > > >> > > > > > which is
> useful
> > to
> > > > > > maximum
> > > > > > > > the
> > > > > > > > > > >> system
> > > > > > > > > > >> > > > > > > throughput
> > > > > > > > > > >> > > > > > > > > > >> > especially
> > > > > > > > > > >> > > > > > > > > > >> > > > for
> > > > > > > > > > >> > > > > > > > > > >> > > > >> IO
> > > > > > > > > > >> > > > > > > > > > >> > > > >> > > > > > bottleneck
> case.
> > > > > > > > > > >> > > > > > > > > > >> > > > >> > > > > >
> > > > > > > > > > >> > > > > > > > > > >> > > > >> > > > > > You can find
> > some
> > > > more
> > > > > > > > detail
> > > > > > > > > in
> > > > > > > > > > >> [1].
> > > > > > > > > > >> > > > > > > > > > >> > > > >> > > > > >
> > > > > > > > > > >> > > > > > > > > > >> > > > >> > > > > > Looking
> forward
> > to
> > > > > > > feedback
> > > > > > > > > > >> > > > > > > > > > >> > > > >> > > > > >
> > > > > > > > > > >> > > > > > > > > > >> > > > >> > > > > >
> > > > > > > > > > >> > > > > > > > > > >> > > > >> > > > > > [1]:
> > > > > > > > > > >> > > > > > > > > > >> > > > >> > > >
> > > > > > > > > > >> > > > > > > > > > >> > > > >> > >
> > > > > > > > > > >> > > > > > > > > > >> > > > >> >
> > > > > > > > > > >> > > > > > > > > > >> > > > >>
> > > > > > > > > > >> > > > > > > > > > >> > > >
> > > > > > > > > > >> > > > > > > > > > >> > >
> > > > > > > > > > >> > > > > > > > > > >> >
> > > > > > > > > > >> > > > > > > > > > >>
> > > > > > > > > > >> > > > > > > > > >
> > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > >> > > > > > > >
> > > > > > > > > > >> > > > > > >
> > > > > > > > > > >> > > > > >
> > > > > > > > > > >> > > > >
> > > > > > > > > > >> > > >
> > > > > > > > > > >> > >
> > > > > > > > > > >> >
> > > > > > > > > > >>
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/%5BFLIP-313%5D+Add+support+of+User+Defined+AsyncTableFunction
> > > > > > > > > > >> > > > > > > > > > >> > > > >> > > > > >
> > > > > > > > > > >> > > > > > > > > > >> > > > >> > > > > > Thanks,
> > > > > > > > > > >> > > > > > > > > > >> > > > >> > > > > > Aitozi.
> > > > > > > > > > >> > > > > > > > > > >> > > > >> > > >
> > > > > > > > > > >> > > > > > > > > > >> > > > >> > >
> > > > > > > > > > >> > > > > > > > > > >> > > > >> >
> > > > > > > > > > >> > > > > > > > > > >> > > > >>
> > > > > > > > > > >> > > > > > > > > > >> > > > >
> > > > > > > > > > >> > > > > > > > > > >> > > >
> > > > > > > > > > >> > > > > > > > > > >> > >
> > > > > > > > > > >> > > > > > > > > > >> >
> > > > > > > > > > >> > > > > > > > > > >>
> > > > > > > > > > >> > > > > > > > > > >
> > > > > > > > > > >> > > > > > > > > >
> > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > >> > > > > > > >
> > > > > > > > > > >> > > > > > >
> > > > > > > > > > >> > > > > >
> > > > > > > > > > >> > > > >
> > > > > > > > > > >> > > >
> > > > > > > > > > >> > >
> > > > > > > > > > >> >
> > > > > > > > > > >>
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] FLIP-313 Add support of User Defined AsyncTableFunction

Posted by Aitozi <gj...@gmail.com>.

Hi all,
    Sorry for the late reply, I have a discussion with Lincoln offline,
mainly about
the naming of the hints option. Thanks Lincoln for the valuable suggestions.

Let me answer the last email inline.

>For `JavaAsyncTableFunc0` in flip, can you use a scenario like RPC call as
an example?

Sure, will give an example when adding the doc of async udtf and will
update the FLIP simultaneously

>For the name of this query hint, 'LATERAL' (include its internal options)
don't show any relevance to async, but I haven't thought of a suitable name
at the moment,

After some discussion with Lincoln, We prefer to choose one of the
`ASYNC_TABLE_FUNC` and `ASYNC_LATERAL`.
Besides, In my opinion the keyword `lateral`'s use scenario is wider than
the table function join, but in this case we only want to config
the async table function, So I'm a bit more lean to the `ASYNC_TABLE_FUNC`.
Looking forward to some inputs if you guys have
some better suggestion on the naming.

For the usage of the hints config option, I have updated the section
of ConfigOption, you can refer to the FLIP
for more details.

>Also, the terms 'correlate join' and 'lateral join' are not the same as in
the current joins page[1], so maybe it would be better if we unified them
into  'join table function'

Yes, we should unified to the 'join table function', updated.

Best,
Aitozi

Lincoln Lee <li...@gmail.com> 于2023年6月15日周四 09:15写道：

> Hi Aitozi,
>
> Thanks for your reply!  Gives sql users more flexibility to get
> asynchronous processing capabilities via lateral join table function +1 for
> this
>
> For `JavaAsyncTableFunc0` in flip, can you use a scenario like RPC call as
> an example?
>
> For the name of this query hint, 'LATERAL' (include its internal options)
> don't show any relevance to async, but I haven't thought of a suitable name
> at the moment,
> maybe we need to highlight the async keyword directly, we can also see if
> others have better candidates
>
> For the hint option "timeout = '180s'" should be "'timeout' = '180s'",
> seems a typo in the flip. And use upper case for all keywords in sql
> examples.
> Also, the terms 'correlate join' and 'lateral join' are not the same as in
> the current joins page[1], so maybe it would be better if we unified them
> into  'join table function'
>
> [1]
>
> https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/sql/queries/joins/#table-function
>
> Best,
> Lincoln Lee
>
>
> Aitozi <gj...@gmail.com> 于2023年6月14日周三 16:11写道：
>
> > Hi Lincoln
> >
> >     Very thanks for your valuable question. I will try to answer your
> > questions inline.
> >
> > >Does the async udtf bring any additional benefits besides a
> > lighter implementation？
> >
> > IMO, async udtf is more than a lighter implementation. It can act as a
> > general way for sql users to use the async operator. And they don't have
> to
> > bind the async function with a table (a LookupTable), and they are not
> > forced to join on an equality join condition, and they can use it to do
> > more than enrich data.
> >
> > The async lookup join is more like a subset/specific usage of async udtf.
> > The specific version has more opportunity to be optimized (like push
> down)
> > is acceptable. Async table function should be categorized to used-defined
> > function.
> >
> > >Should users
> >
> > migrate to the lookup source when they encounter similar requirements or
> >
> > problems, or should we develop an additional set of similar mechanisms？
> >
> > As I clarified above, the lookup join is a specific usage of async udtf.
> So
> > it deserves more refined optimization like caching / retryable. But it
> may
> > not all
> >
> > suitable for the async udtf. As function, it can be deterministic/or
> > non-deterministic. So caching is not suitable, and we also do not have a
> > common cache for the udf now. So I think optimization like caching/retry
> > should be handed over to the function implementor.
> >
> > > the newly added query hint need a different name that
> > can be easier related to the lateral operation as the current join
> hints[5]
> > do.
> >
> >
> > What about using LATERAL?
> >
> > as below
> >
> > SELECT /*+ LATERAL('output-mode' = 'ordered', 'capacity' = '200',
> timeout =
> > '180s') */ a, c1, c2
> >
> > FROM T1
> >
> > LEFT JOIN lateral TABLE (async_split(b)) AS T(c1, c2) ON true
> >
> > >For the async func example, since the target scenario is an external io
> > operation, it's better to add the `close` method to actively release
> > resources as a good example for users
> >
> >
> > Make sense to me, will update the FLIP
> >
> > Best,
> >
> > Aitozi.
> >
> > Lincoln Lee <li...@gmail.com> 于2023年6月14日周三 14:24写道：
> >
> > > Hi Aitozi,
> > >
> > > Sorry for the lately reply here!  Supports async
> > udtf(`AsyncTableFunction`)
> > > directly in sql seems like an attractive feature, but there're two
> issues
> > > that need to be addressed before we can be sure to add it:
> > > 1. As mentioned in the flip[1], the current lookup function can already
> > > implement the requirements, but it requires implementing an extra
> > > `LookupTableSource` and explicitly declaring the table schema (which
> can
> > > help implementers the various push-down optimizations supported by the
> > > planner). Does the async udtf bring any additional benefits besides a
> > > lighter implementation?
> > > 2. FLIP-221[2] abstracts a reusable cache and metric infrastructure for
> > > lookup sources, which are important to improve performance and
> > > observability for high overhead external io scenarios, how do we
> > integrate
> > > and reuse these capabilities after introducing async udtf? Should users
> > > migrate to the lookup source when they encounter similar requirements
> or
> > > problems, or should we develop an additional set of similar mechanisms?
> > (a
> > > similarly case:  FLIP-234[3] introduced the retryable capability for
> > lookup
> > > join)
> > >
> > > For the flip itself,
> > > 1. Considering the 'options' is already used as the dynamic table
> > > options[4] in flink, the newly added query hint need a different name
> > that
> > > can be easier related to the lateral operation as the current join
> > hints[5]
> > > do.
> > > 2. For the async func example, since the target scenario is an external
> > io
> > > operation, it's better to add the `close` method to actively release
> > > resources as a good example for users. Also in terms of the determinism
> > of
> > > a function, it is important to remind users that unless the behavior of
> > the
> > > function is deterministic, it needs to be explicitly declared as
> > > non-deterministic.
> > >
> > > [1].
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-313%3A+Add+support+of+User+Defined+AsyncTableFunction?src=contextnavpagetreemode
> > > [2].
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-221%3A+Abstraction+for+lookup+source+cache+and+metric?src=contextnavpagetreemode
> > > [3].
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-234%3A+Support+Retryable+Lookup+Join+To+Solve+Delayed+Updates+Issue+In+External+Systems?src=contextnavpagetreemode
> > > [4].
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-113%3A+Supports+Dynamic+Table+Options+for+Flink+SQL?src=contextnavpagetreemode
> > > [5].
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-229%3A+Introduces+Join+Hint+for+Flink+SQL+Batch+Job?src=contextnavpagetreemode
> > >
> > > Best,
> > > Lincoln Lee
> > >
> > >
> > > Aitozi <gj...@gmail.com> 于2023年6月13日周二 11:30写道：
> > >
> > > > Get your meaning now, thanks :)
> > > >
> > > > Best,
> > > > Aitozi.
> > > >
> > > > Feng Jin <ji...@gmail.com> 于2023年6月13日周二 11:16写道：
> > > >
> > > > > Hi Aitozi,
> > > > >
> > > > > Sorry for the confusing description.
> > > > >
> > > > > What I meant was that if we need to remind users about tire safety
> > > > issues,
> > > > > we should introduce the new UDTF interface instead of executing the
> > > > > original UDTF asynchronously. Therefore, I agree with introducing
> the
> > > > > AsyncTableFunction.
> > > > >
> > > > > Best,
> > > > > Feng
> > > > >
> > > > > On Tue, Jun 13, 2023 at 10:42 AM Aitozi <gj...@gmail.com>
> > wrote:
> > > > >
> > > > > > Hi Feng,
> > > > > >     Thanks for your question. We do not provide a way to switch
> the
> > > > UDTF
> > > > > > between sync and async way,
> > > > > > So there should be no thread safety problem here.
> > > > > >
> > > > > > Best,
> > > > > > Aitozi
> > > > > >
> > > > > > Feng Jin <ji...@gmail.com> 于2023年6月13日周二 10:31写道：
> > > > > >
> > > > > > > Hi Aitozi, We do need to remind users about thread safety
> issues.
> > > > Thank
> > > > > > you
> > > > > > > for your efforts on this FLIP. I have no further questions.
> > > > > > > Best, Feng
> > > > > > >
> > > > > > >
> > > > > > > On Tue, Jun 13, 2023 at 6:05 AM Jing Ge
> > <jing@ververica.com.invalid
> > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hi Aitozi,
> > > > > > > >
> > > > > > > > Thanks for taking care of that part. I have no other concern.
> > > > > > > >
> > > > > > > > Best regards,
> > > > > > > > Jing
> > > > > > > >
> > > > > > > >
> > > > > > > > On Mon, Jun 12, 2023 at 5:38 PM Aitozi <gjying1314@gmail.com
> >
> > > > wrote:
> > > > > > > >
> > > > > > > > > BTW, If there are no other more blocking issue / comments,
> I
> > > > would
> > > > > > like
> > > > > > > > to
> > > > > > > > > start a VOTE in another thread this wednesday 6.14
> > > > > > > > >
> > > > > > > > > Thanks,
> > > > > > > > > Aitozi.
> > > > > > > > >
> > > > > > > > > Aitozi <gj...@gmail.com> 于2023年6月12日周一 23:34写道：
> > > > > > > > >
> > > > > > > > > > Hi, Jing,
> > > > > > > > > >     Thanks for your explanation. I get your point now.
> > > > > > > > > >
> > > > > > > > > > For the performance part, I think it's a good idea to run
> > > with
> > > > > > > > returning
> > > > > > > > > a
> > > > > > > > > > big table case, the memory consumption
> > > > > > > > > > should be a point to be taken care about. Because in the
> > > > ordered
> > > > > > > mode,
> > > > > > > > > the
> > > > > > > > > > head element in buffer may affect the
> > > > > > > > > > total memory consumption.
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Thanks,
> > > > > > > > > > Aitozi.
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Jing Ge <ji...@ververica.com.invalid> 于2023年6月12日周一
> > 20:28写道：
> > > > > > > > > >
> > > > > > > > > >> Hi Aitozi,
> > > > > > > > > >>
> > > > > > > > > >> Which key will be used for lookup is not an issue, only
> > one
> > > > row
> > > > > > will
> > > > > > > > be
> > > > > > > > > >> required for each key in order to enrich it. True, it
> > > depends
> > > > on
> > > > > > the
> > > > > > > > > >> implementation whether multiple rows or single row for
> > each
> > > > key
> > > > > > will
> > > > > > > > be
> > > > > > > > > >> returned. However, for the lookup & enrichment scenario,
> > one
> > > > > > row/key
> > > > > > > > is
> > > > > > > > > >> recommended, otherwise, like I mentioned previously,
> > > > enrichment
> > > > > > > won't
> > > > > > > > > >> work.
> > > > > > > > > >>
> > > > > > > > > >> I am a little bit concerned about returning a big table
> > for
> > > > each
> > > > > > > key,
> > > > > > > > > >> since
> > > > > > > > > >> it will take the async call longer to return and need
> more
> > > > > memory.
> > > > > > > The
> > > > > > > > > >> performance tests should cover this scenario. This is
> not
> > a
> > > > > > blocking
> > > > > > > > > issue
> > > > > > > > > >> for this FLIP.
> > > > > > > > > >>
> > > > > > > > > >> Best regards,
> > > > > > > > > >> Jing
> > > > > > > > > >>
> > > > > > > > > >> On Sat, Jun 10, 2023 at 4:11 AM Aitozi <
> > > gjying1314@gmail.com>
> > > > > > > wrote:
> > > > > > > > > >>
> > > > > > > > > >> > Hi Jing,
> > > > > > > > > >> >     I means the join key is not necessary to be the
> > > primary
> > > > > key
> > > > > > or
> > > > > > > > > >> unique
> > > > > > > > > >> > index of the database.
> > > > > > > > > >> > In this situation, we may queried out multi rows for
> one
> > > > join
> > > > > > > key. I
> > > > > > > > > >> think
> > > > > > > > > >> > that's why the
> > > > > > > > > >> > LookupFunction#lookup will return a collection of
> > RowData.
> > > > > > > > > >> >
> > > > > > > > > >> > BTW, I think the behavior of lookup join will not
> affect
> > > the
> > > > > > > > semantic
> > > > > > > > > of
> > > > > > > > > >> > the async udtf.
> > > > > > > > > >> > We use the Async TableFunction here and the table
> > function
> > > > can
> > > > > > > > collect
> > > > > > > > > >> > multiple rows.
> > > > > > > > > >> >
> > > > > > > > > >> > Thanks,
> > > > > > > > > >> > Atiozi.
> > > > > > > > > >> >
> > > > > > > > > >> >
> > > > > > > > > >> >
> > > > > > > > > >> > Jing Ge <ji...@ververica.com.invalid> 于2023年6月10日周六
> > > 00:15写道：
> > > > > > > > > >> >
> > > > > > > > > >> > > Hi Aitozi,
> > > > > > > > > >> > >
> > > > > > > > > >> > > The keyRow used in this case contains all keys[1].
> > > > > > > > > >> > >
> > > > > > > > > >> > > Best regards,
> > > > > > > > > >> > > Jing
> > > > > > > > > >> > >
> > > > > > > > > >> > > [1]
> > > > > > > > > >> > >
> > > > > > > > > >> > >
> > > > > > > > > >> >
> > > > > > > > > >>
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/flink/blob/191ec6ca3943d7119f14837efe112e074d815c47/flink-table/flink-table-common/src/main/java/org/apache/flink/table/functions/LookupFunction.java#L49
> > > > > > > > > >> > >
> > > > > > > > > >> > >
> > > > > > > > > >> > > On Fri, Jun 9, 2023 at 3:42 PM Aitozi <
> > > > gjying1314@gmail.com
> > > > > >
> > > > > > > > wrote:
> > > > > > > > > >> > >
> > > > > > > > > >> > > > Hi Jing,
> > > > > > > > > >> > > >
> > > > > > > > > >> > > >      The performance test is added to the FLIP.
> > > > > > > > > >> > > >
> > > > > > > > > >> > > >      As I know, The lookup join can return multi
> > rows,
> > > > it
> > > > > > > > depends
> > > > > > > > > on
> > > > > > > > > >> > > > whether  the join key
> > > > > > > > > >> > > > is the primary key of the external database or
> not.
> > > The
> > > > > > > `lookup`
> > > > > > > > > [1]
> > > > > > > > > >> > will
> > > > > > > > > >> > > > return a collection of
> > > > > > > > > >> > > > joined result, and each of them will be collected
> > > > > > > > > >> > > >
> > > > > > > > > >> > > >
> > > > > > > > > >> > > > [1]:
> > > > > > > > > >> > > >
> > > > > > > > > >> > > >
> > > > > > > > > >> > >
> > > > > > > > > >> >
> > > > > > > > > >>
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/flink/blob/191ec6ca3943d7119f14837efe112e074d815c47/flink-table/flink-table-common/src/main/java/org/apache/flink/table/functions/LookupFunction.java#L52
> > > > > > > > > >> > > >
> > > > > > > > > >> > > >
> > > > > > > > > >> > > > Thanks,
> > > > > > > > > >> > > > Aitozi.
> > > > > > > > > >> > > >
> > > > > > > > > >> > > > Jing Ge <ji...@ververica.com.invalid> 于2023年6月9日周五
> > > > > 17:05写道：
> > > > > > > > > >> > > >
> > > > > > > > > >> > > > > Hi Aitozi,
> > > > > > > > > >> > > > >
> > > > > > > > > >> > > > > Thanks for the feedback. Looking forward to the
> > > > > > performance
> > > > > > > > > tests.
> > > > > > > > > >> > > > >
> > > > > > > > > >> > > > > Afaik, lookup returns one row for each key [1]
> > [2].
> > > > > > > > > Conceptually,
> > > > > > > > > >> the
> > > > > > > > > >> > > > > lookup function is used to enrich column(s) from
> > the
> > > > > > > dimension
> > > > > > > > > >> table.
> > > > > > > > > >> > > If,
> > > > > > > > > >> > > > > for the given key, there will be more than one
> > row,
> > > > > there
> > > > > > > will
> > > > > > > > > be
> > > > > > > > > >> no
> > > > > > > > > >> > > way
> > > > > > > > > >> > > > to
> > > > > > > > > >> > > > > know which row will be used to enrich the key.
> > > > > > > > > >> > > > >
> > > > > > > > > >> > > > > [1]
> > > > > > > > > >> > > > >
> > > > > > > > > >> > > > >
> > > > > > > > > >> > > >
> > > > > > > > > >> > >
> > > > > > > > > >> >
> > > > > > > > > >>
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/flink/blob/191ec6ca3943d7119f14837efe112e074d815c47/flink-table/flink-table-common/src/main/java/org/apache/flink/table/functions/LookupFunction.java#L49
> > > > > > > > > >> > > > > [2]
> > > > > > > > > >> > > > >
> > > > > > > > > >> > > > >
> > > > > > > > > >> > > >
> > > > > > > > > >> > >
> > > > > > > > > >> >
> > > > > > > > > >>
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/flink/blob/191ec6ca3943d7119f14837efe112e074d815c47/flink-table/flink-table-common/src/main/java/org/apache/flink/table/functions/TableFunction.java#L196
> > > > > > > > > >> > > > >
> > > > > > > > > >> > > > > Best regards,
> > > > > > > > > >> > > > > Jing
> > > > > > > > > >> > > > >
> > > > > > > > > >> > > > > On Fri, Jun 9, 2023 at 5:18 AM Aitozi <
> > > > > > gjying1314@gmail.com
> > > > > > > >
> > > > > > > > > >> wrote:
> > > > > > > > > >> > > > >
> > > > > > > > > >> > > > > > Hi Jing
> > > > > > > > > >> > > > > >     Thanks for your good questions. I have
> > updated
> > > > the
> > > > > > > > example
> > > > > > > > > >> to
> > > > > > > > > >> > the
> > > > > > > > > >> > > > > FLIP.
> > > > > > > > > >> > > > > >
> > > > > > > > > >> > > > > > > Only one row for each lookup
> > > > > > > > > >> > > > > > lookup can also return multi rows, based on
> the
> > > > query
> > > > > > > > result.
> > > > > > > > > >> [1]
> > > > > > > > > >> > > > > >
> > > > > > > > > >> > > > > > [1]:
> > > > > > > > > >> > > > > >
> > > > > > > > > >> > > > > >
> > > > > > > > > >> > > > >
> > > > > > > > > >> > > >
> > > > > > > > > >> > >
> > > > > > > > > >> >
> > > > > > > > > >>
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/flink/blob/191ec6ca3943d7119f14837efe112e074d815c47/flink-table/flink-table-common/src/main/java/org/apache/flink/table/functions/LookupFunction.java#L56
> > > > > > > > > >> > > > > >
> > > > > > > > > >> > > > > > > If we use async calls with lateral join, my
> > gut
> > > > > > feeling
> > > > > > > is
> > > > > > > > > >> > > > > > that we might have many more async calls than
> > > lookup
> > > > > > > join. I
> > > > > > > > > am
> > > > > > > > > >> not
> > > > > > > > > >> > > > > really
> > > > > > > > > >> > > > > > sure if we will be facing potential issues in
> > this
> > > > > case
> > > > > > or
> > > > > > > > > not.
> > > > > > > > > >> > > > > >
> > > > > > > > > >> > > > > > IMO, the work pattern is similar to the lookup
> > > > > function,
> > > > > > > for
> > > > > > > > > >> each
> > > > > > > > > >> > row
> > > > > > > > > >> > > > > from
> > > > > > > > > >> > > > > > the left table,
> > > > > > > > > >> > > > > > it will evaluate the eval method once, so the
> > > async
> > > > > call
> > > > > > > > > numbers
> > > > > > > > > >> > will
> > > > > > > > > >> > > > not
> > > > > > > > > >> > > > > > change.
> > > > > > > > > >> > > > > > and the maximum calls in flight is limited by
> > the
> > > > > Async
> > > > > > > > > >> operators
> > > > > > > > > >> > > > buffer
> > > > > > > > > >> > > > > > capacity
> > > > > > > > > >> > > > > > which will be controlled by the option.
> > > > > > > > > >> > > > > >
> > > > > > > > > >> > > > > > BTW, for the naming of these option, I updated
> > the
> > > > > FLIP
> > > > > > > > about
> > > > > > > > > >> this
> > > > > > > > > >> > > you
> > > > > > > > > >> > > > > can
> > > > > > > > > >> > > > > > refer to
> > > > > > > > > >> > > > > > the section of "ConfigOption" and "Rejected
> > > > > > Alternatives"
> > > > > > > > > >> > > > > >
> > > > > > > > > >> > > > > > In the end, for the performance evaluation,
> I'd
> > > like
> > > > > to
> > > > > > do
> > > > > > > > > some
> > > > > > > > > >> > tests
> > > > > > > > > >> > > > and
> > > > > > > > > >> > > > > > will update it to the FLIP doc
> > > > > > > > > >> > > > > >
> > > > > > > > > >> > > > > > Thanks,
> > > > > > > > > >> > > > > > Aitozi.
> > > > > > > > > >> > > > > >
> > > > > > > > > >> > > > > >
> > > > > > > > > >> > > > > > Jing Ge <ji...@ververica.com.invalid>
> > 于2023年6月9日周五
> > > > > > > 07:23写道：
> > > > > > > > > >> > > > > >
> > > > > > > > > >> > > > > > > Hi Aitozi,
> > > > > > > > > >> > > > > > >
> > > > > > > > > >> > > > > > > Thanks for the clarification. The code
> example
> > > > looks
> > > > > > > > > >> > interesting. I
> > > > > > > > > >> > > > > would
> > > > > > > > > >> > > > > > > suggest adding them into the FLIP. The
> > > description
> > > > > > with
> > > > > > > > code
> > > > > > > > > >> > > examples
> > > > > > > > > >> > > > > > will
> > > > > > > > > >> > > > > > > help readers understand the motivation and
> how
> > > to
> > > > > use
> > > > > > > it.
> > > > > > > > > >> Afaiac,
> > > > > > > > > >> > > it
> > > > > > > > > >> > > > > is a
> > > > > > > > > >> > > > > > > valid feature for Flink users.
> > > > > > > > > >> > > > > > >
> > > > > > > > > >> > > > > > > As we knew, lookup join is based on temporal
> > > join,
> > > > > > i.e.
> > > > > > > > FOR
> > > > > > > > > >> > > > SYSTEM_TIME
> > > > > > > > > >> > > > > > AS
> > > > > > > > > >> > > > > > > OF which is also used in your code example.
> > > > Temporal
> > > > > > > join
> > > > > > > > > >> > performs
> > > > > > > > > >> > > > the
> > > > > > > > > >> > > > > > > lookup based on the processing time match.
> > Only
> > > > one
> > > > > > row
> > > > > > > > for
> > > > > > > > > >> each
> > > > > > > > > >> > > > > > > lookup(afaiu, I need to check the source
> code
> > to
> > > > > > double
> > > > > > > > > >> confirm)
> > > > > > > > > >> > > will
> > > > > > > > > >> > > > > > > return for further enrichment. One the other
> > > hand,
> > > > > > > lateral
> > > > > > > > > >> join
> > > > > > > > > >> > > will
> > > > > > > > > >> > > > > have
> > > > > > > > > >> > > > > > > sub-queries correlated with every individual
> > > value
> > > > > of
> > > > > > > the
> > > > > > > > > >> > reference
> > > > > > > > > >> > > > > table
> > > > > > > > > >> > > > > > > from the preceding part of the query and
> each
> > > sub
> > > > > > query
> > > > > > > > will
> > > > > > > > > >> > return
> > > > > > > > > >> > > > > > > multiple rows. If we use async calls with
> > > lateral
> > > > > > join,
> > > > > > > my
> > > > > > > > > gut
> > > > > > > > > >> > > > feeling
> > > > > > > > > >> > > > > is
> > > > > > > > > >> > > > > > > that we might have many more async calls
> than
> > > > lookup
> > > > > > > > join. I
> > > > > > > > > >> am
> > > > > > > > > >> > not
> > > > > > > > > >> > > > > > really
> > > > > > > > > >> > > > > > > sure if we will be facing potential issues
> in
> > > this
> > > > > > case
> > > > > > > or
> > > > > > > > > >> not.
> > > > > > > > > >> > > > > Possible
> > > > > > > > > >> > > > > > > issues I can think of now e.g. too many PRC
> > > calls,
> > > > > too
> > > > > > > > many
> > > > > > > > > >> async
> > > > > > > > > >> > > > calls
> > > > > > > > > >> > > > > > > processing, the sub query will return a
> table
> > > > which
> > > > > > > might
> > > > > > > > be
> > > > > > > > > >> > (too)
> > > > > > > > > >> > > > big,
> > > > > > > > > >> > > > > > and
> > > > > > > > > >> > > > > > > might cause performance issues. I would
> > suggest
> > > > > > > preparing
> > > > > > > > > some
> > > > > > > > > >> > use
> > > > > > > > > >> > > > > cases
> > > > > > > > > >> > > > > > > and running some performance tests to check
> > it.
> > > > > These
> > > > > > > are
> > > > > > > > my
> > > > > > > > > >> > > concerns
> > > > > > > > > >> > > > > > about
> > > > > > > > > >> > > > > > > using async calls with lateral join and I'd
> > like
> > > > to
> > > > > > > share
> > > > > > > > > with
> > > > > > > > > >> > you,
> > > > > > > > > >> > > > > happy
> > > > > > > > > >> > > > > > > to discuss with you and hear different
> > opinions,
> > > > > > > hopefully
> > > > > > > > > the
> > > > > > > > > >> > > > > > > discussion could help me understand it more
> > > > deeply.
> > > > > > > Please
> > > > > > > > > >> > correct
> > > > > > > > > >> > > me
> > > > > > > > > >> > > > > if
> > > > > > > > > >> > > > > > I
> > > > > > > > > >> > > > > > > am wrong.
> > > > > > > > > >> > > > > > >
> > > > > > > > > >> > > > > > > Best regards,
> > > > > > > > > >> > > > > > > Jing
> > > > > > > > > >> > > > > > >
> > > > > > > > > >> > > > > > >
> > > > > > > > > >> > > > > > > On Thu, Jun 8, 2023 at 7:22 AM Aitozi <
> > > > > > > > gjying1314@gmail.com
> > > > > > > > > >
> > > > > > > > > >> > > wrote:
> > > > > > > > > >> > > > > > >
> > > > > > > > > >> > > > > > > > Hi Mason,
> > > > > > > > > >> > > > > > > >     Thanks for your input. I think if we
> > > support
> > > > > the
> > > > > > > > user
> > > > > > > > > >> > defined
> > > > > > > > > >> > > > > async
> > > > > > > > > >> > > > > > > > table function,
> > > > > > > > > >> > > > > > > > user will be able to use it to hold a
> batch
> > > data
> > > > > > then
> > > > > > > > > >> handle it
> > > > > > > > > >> > > at
> > > > > > > > > >> > > > > one
> > > > > > > > > >> > > > > > > time
> > > > > > > > > >> > > > > > > > in the customized function.
> > > > > > > > > >> > > > > > > >
> > > > > > > > > >> > > > > > > > AsyncSink is meant for the sink operator.
> I
> > > have
> > > > > not
> > > > > > > > > figure
> > > > > > > > > >> out
> > > > > > > > > >> > > how
> > > > > > > > > >> > > > > to
> > > > > > > > > >> > > > > > > > integrate in this case.
> > > > > > > > > >> > > > > > > >
> > > > > > > > > >> > > > > > > > Thanks,
> > > > > > > > > >> > > > > > > > Atiozi.
> > > > > > > > > >> > > > > > > >
> > > > > > > > > >> > > > > > > >
> > > > > > > > > >> > > > > > > > Mason Chen <ma...@gmail.com>
> > > > 于2023年6月8日周四
> > > > > > > > 12:40写道：
> > > > > > > > > >> > > > > > > >
> > > > > > > > > >> > > > > > > > > Hi Aitozi,
> > > > > > > > > >> > > > > > > > >
> > > > > > > > > >> > > > > > > > > I think it makes sense to make it easier
> > for
> > > > SQL
> > > > > > > users
> > > > > > > > > to
> > > > > > > > > >> > make
> > > > > > > > > >> > > > > RPCs.
> > > > > > > > > >> > > > > > Do
> > > > > > > > > >> > > > > > > > you
> > > > > > > > > >> > > > > > > > > think your proposal can extend to the
> > > ability
> > > > to
> > > > > > > batch
> > > > > > > > > >> data
> > > > > > > > > >> > for
> > > > > > > > > >> > > > the
> > > > > > > > > >> > > > > > > RPC?
> > > > > > > > > >> > > > > > > > > This is also another common strategy to
> > > > increase
> > > > > > > > > >> throughput.
> > > > > > > > > >> > > > Also,
> > > > > > > > > >> > > > > > have
> > > > > > > > > >> > > > > > > > you
> > > > > > > > > >> > > > > > > > > considered solving this a bit
> differently
> > by
> > > > > > > > leveraging
> > > > > > > > > >> > Flink's
> > > > > > > > > >> > > > > > > > AsyncSink?
> > > > > > > > > >> > > > > > > > >
> > > > > > > > > >> > > > > > > > > Best,
> > > > > > > > > >> > > > > > > > > Mason
> > > > > > > > > >> > > > > > > > >
> > > > > > > > > >> > > > > > > > > On Mon, Jun 5, 2023 at 1:50 AM Aitozi <
> > > > > > > > > >> gjying1314@gmail.com>
> > > > > > > > > >> > > > > wrote:
> > > > > > > > > >> > > > > > > > >
> > > > > > > > > >> > > > > > > > > > One more thing for discussion:
> > > > > > > > > >> > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > In our internal implementation, we
> reuse
> > > the
> > > > > > > option
> > > > > > > > > >> > > > > > > > > >
> > `table.exec.async-lookup.buffer-capacity`
> > > > and
> > > > > > > > > >> > > > > > > > > > `table.exec.async-lookup.timeout` to
> > > config
> > > > > > > > > >> > > > > > > > > > the async udtf. Do you think we should
> > add
> > > > two
> > > > > > > extra
> > > > > > > > > >> option
> > > > > > > > > >> > > to
> > > > > > > > > >> > > > > > > > > distinguish
> > > > > > > > > >> > > > > > > > > > from the lookup option such as
> > > > > > > > > >> > > > > > > > > >
> > > > > > > > > >> > > > > > > > > >
> `table.exec.async-udtf.buffer-capacity`
> > > > > > > > > >> > > > > > > > > > `table.exec.async-udtf.timeout`
> > > > > > > > > >> > > > > > > > > >
> > > > > > > > > >> > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > Best,
> > > > > > > > > >> > > > > > > > > > Aitozi.
> > > > > > > > > >> > > > > > > > > >
> > > > > > > > > >> > > > > > > > > >
> > > > > > > > > >> > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > Aitozi <gj...@gmail.com>
> > > 于2023年6月5日周一
> > > > > > > 12:20写道：
> > > > > > > > > >> > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > Hi Jing,
> > > > > > > > > >> > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > >     > what is the difference between
> > the
> > > > RPC
> > > > > > > call
> > > > > > > > or
> > > > > > > > > >> > query
> > > > > > > > > >> > > > you
> > > > > > > > > >> > > > > > > > > mentioned
> > > > > > > > > >> > > > > > > > > > > and the lookup in a very
> > > > > > > > > >> > > > > > > > > > > general way
> > > > > > > > > >> > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > I think the RPC call or query
> service
> > is
> > > > > quite
> > > > > > > > > >> similar to
> > > > > > > > > >> > > the
> > > > > > > > > >> > > > > > > lookup
> > > > > > > > > >> > > > > > > > > > join.
> > > > > > > > > >> > > > > > > > > > > But lookup join should work
> > > > > > > > > >> > > > > > > > > > > with `LookupTableSource`.
> > > > > > > > > >> > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > Let's see how we can perform an
> async
> > > RPC
> > > > > call
> > > > > > > > with
> > > > > > > > > >> > lookup
> > > > > > > > > >> > > > > join:
> > > > > > > > > >> > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > (1) Implement an AsyncTableFunction
> > with
> > > > RPC
> > > > > > > call
> > > > > > > > > >> logic.
> > > > > > > > > >> > > > > > > > > > > (2) Implement a `LookupTableSource`
> > > > > connector
> > > > > > > run
> > > > > > > > > with
> > > > > > > > > >> > the
> > > > > > > > > >> > > > > async
> > > > > > > > > >> > > > > > > udtf
> > > > > > > > > >> > > > > > > > > > > defined in (1).
> > > > > > > > > >> > > > > > > > > > > (3) Then define a DDL of this look
> up
> > > > table
> > > > > in
> > > > > > > SQL
> > > > > > > > > >> > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > CREATE TEMPORARY TABLE Customers (
> > > > > > > > > >> > > > > > > > > > >   id INT,
> > > > > > > > > >> > > > > > > > > > >   name STRING,
> > > > > > > > > >> > > > > > > > > > >   country STRING,
> > > > > > > > > >> > > > > > > > > > >   zip STRING
> > > > > > > > > >> > > > > > > > > > > ) WITH (
> > > > > > > > > >> > > > > > > > > > >   'connector' = 'custom'
> > > > > > > > > >> > > > > > > > > > > );
> > > > > > > > > >> > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > (4) Run with the query as below:
> > > > > > > > > >> > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > SELECT o.order_id, o.total,
> c.country,
> > > > c.zip
> > > > > > > > > >> > > > > > > > > > > FROM Orders AS o
> > > > > > > > > >> > > > > > > > > > >   JOIN Customers FOR SYSTEM_TIME AS
> OF
> > > > > > > o.proc_time
> > > > > > > > > AS
> > > > > > > > > >> c
> > > > > > > > > >> > > > > > > > > > >     ON o.customer_id = c.id;
> > > > > > > > > >> > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > This example is from doc
> > > > > > > > > >> > > > > > > > > > > <
> > > > > > > > > >> > > > > > > > > >
> > > > > > > > > >> > > > > > > > >
> > > > > > > > > >> > > > > > > >
> > > > > > > > > >> > > > > > >
> > > > > > > > > >> > > > > >
> > > > > > > > > >> > > > >
> > > > > > > > > >> > > >
> > > > > > > > > >> > >
> > > > > > > > > >> >
> > > > > > > > > >>
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/dev/table/sql/queries/joins/#lookup-join
> > > > > > > > > >> > > > > > > > > > >.You
> > > > > > > > > >> > > > > > > > > > > can image the look up process as an
> > > async
> > > > > RPC
> > > > > > > call
> > > > > > > > > >> > process.
> > > > > > > > > >> > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > Let's see how we can perform an
> async
> > > RPC
> > > > > call
> > > > > > > > with
> > > > > > > > > >> > lateral
> > > > > > > > > >> > > > > join:
> > > > > > > > > >> > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > (1) Implement an AsyncTableFunction
> > with
> > > > RPC
> > > > > > > call
> > > > > > > > > >> logic.
> > > > > > > > > >> > > > > > > > > > > (2) Run query with
> > > > > > > > > >> > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > Create function f1 as '...' ;
> > > > > > > > > >> > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > SELECT o.order_id, o.total,
> c.country,
> > > > c.zip
> > > > > > > FROM
> > > > > > > > > >> Orders
> > > > > > > > > >> > > > > lateral
> > > > > > > > > >> > > > > > > > table
> > > > > > > > > >> > > > > > > > > > > (f1(order_id)) as T(...);
> > > > > > > > > >> > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > As you can see, the lateral join
> > version
> > > > is
> > > > > > more
> > > > > > > > > >> simple
> > > > > > > > > >> > and
> > > > > > > > > >> > > > > > > intuitive
> > > > > > > > > >> > > > > > > > > to
> > > > > > > > > >> > > > > > > > > > > users. Users do not have to wrap a
> > > > > > > > > >> > > > > > > > > > > LookupTableSource for the purpose of
> > > using
> > > > > > async
> > > > > > > > > udtf.
> > > > > > > > > >> > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > In the end, We can also see the user
> > > > defined
> > > > > > > async
> > > > > > > > > >> table
> > > > > > > > > >> > > > > function
> > > > > > > > > >> > > > > > > is
> > > > > > > > > >> > > > > > > > an
> > > > > > > > > >> > > > > > > > > > > enhancement of the current lateral
> > table
> > > > > join
> > > > > > > > > >> > > > > > > > > > > which only supports sync lateral
> join
> > > now.
> > > > > > > > > >> > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > Best,
> > > > > > > > > >> > > > > > > > > > > Aitozi.
> > > > > > > > > >> > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > Jing Ge <jing@ververica.com.invalid
> >
> > > > > > > 于2023年6月2日周五
> > > > > > > > > >> > 19:37写道：
> > > > > > > > > >> > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > >> Hi Aitozi,
> > > > > > > > > >> > > > > > > > > > >>
> > > > > > > > > >> > > > > > > > > > >> Thanks for the update. Just out of
> > > > > curiosity,
> > > > > > > > what
> > > > > > > > > is
> > > > > > > > > >> > the
> > > > > > > > > >> > > > > > > difference
> > > > > > > > > >> > > > > > > > > > >> between the RPC call or query you
> > > > mentioned
> > > > > > and
> > > > > > > > the
> > > > > > > > > >> > lookup
> > > > > > > > > >> > > > in
> > > > > > > > > >> > > > > a
> > > > > > > > > >> > > > > > > very
> > > > > > > > > >> > > > > > > > > > >> general way? Since Lateral join is
> > used
> > > > in
> > > > > > the
> > > > > > > > > FLIP.
> > > > > > > > > >> Is
> > > > > > > > > >> > > > there
> > > > > > > > > >> > > > > > any
> > > > > > > > > >> > > > > > > > > > special
> > > > > > > > > >> > > > > > > > > > >> thought for that? Sorry for asking
> so
> > > > many
> > > > > > > > > questions.
> > > > > > > > > >> > The
> > > > > > > > > >> > > > FLIP
> > > > > > > > > >> > > > > > > > > contains
> > > > > > > > > >> > > > > > > > > > >> limited information to understand
> the
> > > > > > > motivation.
> > > > > > > > > >> > > > > > > > > > >>
> > > > > > > > > >> > > > > > > > > > >> Best regards,
> > > > > > > > > >> > > > > > > > > > >> Jing
> > > > > > > > > >> > > > > > > > > > >>
> > > > > > > > > >> > > > > > > > > > >> On Fri, Jun 2, 2023 at 3:48 AM
> > Aitozi <
> > > > > > > > > >> > > gjying1314@gmail.com
> > > > > > > > > >> > > > >
> > > > > > > > > >> > > > > > > wrote:
> > > > > > > > > >> > > > > > > > > > >>
> > > > > > > > > >> > > > > > > > > > >> > Hi Jing,
> > > > > > > > > >> > > > > > > > > > >> >     I have updated the proposed
> > > changes
> > > > > to
> > > > > > > the
> > > > > > > > > >> FLIP.
> > > > > > > > > >> > > IMO,
> > > > > > > > > >> > > > > > lookup
> > > > > > > > > >> > > > > > > > has
> > > > > > > > > >> > > > > > > > > > its
> > > > > > > > > >> > > > > > > > > > >> > clear
> > > > > > > > > >> > > > > > > > > > >> > async call requirement is due to
> > its
> > > IO
> > > > > > heavy
> > > > > > > > > >> > operator.
> > > > > > > > > >> > > In
> > > > > > > > > >> > > > > our
> > > > > > > > > >> > > > > > > > > usage,
> > > > > > > > > >> > > > > > > > > > >> sql
> > > > > > > > > >> > > > > > > > > > >> > users have
> > > > > > > > > >> > > > > > > > > > >> > logic to do some RPC call or
> query
> > > the
> > > > > > > > > third-party
> > > > > > > > > >> > > service
> > > > > > > > > >> > > > > > which
> > > > > > > > > >> > > > > > > > is
> > > > > > > > > >> > > > > > > > > > >> also IO
> > > > > > > > > >> > > > > > > > > > >> > intensive.
> > > > > > > > > >> > > > > > > > > > >> > In these case, we'd like to
> > leverage
> > > > the
> > > > > > > async
> > > > > > > > > >> > function
> > > > > > > > > >> > > to
> > > > > > > > > >> > > > > > > improve
> > > > > > > > > >> > > > > > > > > the
> > > > > > > > > >> > > > > > > > > > >> > throughput.
> > > > > > > > > >> > > > > > > > > > >> >
> > > > > > > > > >> > > > > > > > > > >> > Thanks,
> > > > > > > > > >> > > > > > > > > > >> > Aitozi.
> > > > > > > > > >> > > > > > > > > > >> >
> > > > > > > > > >> > > > > > > > > > >> > Jing Ge
> <jing@ververica.com.invalid
> > >
> > > > > > > > > 于2023年6月1日周四
> > > > > > > > > >> > > > 22:55写道：
> > > > > > > > > >> > > > > > > > > > >> >
> > > > > > > > > >> > > > > > > > > > >> > > Hi Aitozi,
> > > > > > > > > >> > > > > > > > > > >> > >
> > > > > > > > > >> > > > > > > > > > >> > > Sorry for the late reply. Would
> > you
> > > > > like
> > > > > > to
> > > > > > > > > >> update
> > > > > > > > > >> > the
> > > > > > > > > >> > > > > > > proposed
> > > > > > > > > >> > > > > > > > > > >> changes
> > > > > > > > > >> > > > > > > > > > >> > > with more details into the FLIP
> > > too?
> > > > > > > > > >> > > > > > > > > > >> > > I got your point. It looks
> like a
> > > > > > rational
> > > > > > > > > idea.
> > > > > > > > > >> > > > However,
> > > > > > > > > >> > > > > > > since
> > > > > > > > > >> > > > > > > > > > lookup
> > > > > > > > > >> > > > > > > > > > >> > has
> > > > > > > > > >> > > > > > > > > > >> > > its clear async call
> requirement,
> > > are
> > > > > > there
> > > > > > > > any
> > > > > > > > > >> real
> > > > > > > > > >> > > use
> > > > > > > > > >> > > > > > cases
> > > > > > > > > >> > > > > > > > > that
> > > > > > > > > >> > > > > > > > > > >> > > need this change? This will
> help
> > us
> > > > > > > > understand
> > > > > > > > > >> the
> > > > > > > > > >> > > > > > motivation.
> > > > > > > > > >> > > > > > > > > After
> > > > > > > > > >> > > > > > > > > > >> all,
> > > > > > > > > >> > > > > > > > > > >> > > lateral join and temporal
> lookup
> > > > > join[1]
> > > > > > > are
> > > > > > > > > >> quite
> > > > > > > > > >> > > > > > different.
> > > > > > > > > >> > > > > > > > > > >> > >
> > > > > > > > > >> > > > > > > > > > >> > > Best regards,
> > > > > > > > > >> > > > > > > > > > >> > > Jing
> > > > > > > > > >> > > > > > > > > > >> > >
> > > > > > > > > >> > > > > > > > > > >> > >
> > > > > > > > > >> > > > > > > > > > >> > > [1]
> > > > > > > > > >> > > > > > > > > > >> > >
> > > > > > > > > >> > > > > > > > > > >> > >
> > > > > > > > > >> > > > > > > > > > >> >
> > > > > > > > > >> > > > > > > > > > >>
> > > > > > > > > >> > > > > > > > > >
> > > > > > > > > >> > > > > > > > >
> > > > > > > > > >> > > > > > > >
> > > > > > > > > >> > > > > > >
> > > > > > > > > >> > > > > >
> > > > > > > > > >> > > > >
> > > > > > > > > >> > > >
> > > > > > > > > >> > >
> > > > > > > > > >> >
> > > > > > > > > >>
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/flink/blob/d90a72da2fd601ca4e2a46700e91ec5b348de2ad/flink-table/flink-table-common/src/main/java/org/apache/flink/table/functions/AsyncTableFunction.java#L54
> > > > > > > > > >> > > > > > > > > > >> > >
> > > > > > > > > >> > > > > > > > > > >> > > On Wed, May 31, 2023 at 8:53 AM
> > > > Aitozi
> > > > > <
> > > > > > > > > >> > > > > > gjying1314@gmail.com>
> > > > > > > > > >> > > > > > > > > > wrote:
> > > > > > > > > >> > > > > > > > > > >> > >
> > > > > > > > > >> > > > > > > > > > >> > > > Hi Jing,
> > > > > > > > > >> > > > > > > > > > >> > > >     What do you think about
> it?
> > > Can
> > > > > we
> > > > > > > move
> > > > > > > > > >> > forward
> > > > > > > > > >> > > > this
> > > > > > > > > >> > > > > > > > > feature?
> > > > > > > > > >> > > > > > > > > > >> > > >
> > > > > > > > > >> > > > > > > > > > >> > > > Thanks,
> > > > > > > > > >> > > > > > > > > > >> > > > Aitozi.
> > > > > > > > > >> > > > > > > > > > >> > > >
> > > > > > > > > >> > > > > > > > > > >> > > > Aitozi <gjying1314@gmail.com
> >
> > > > > > > > 于2023年5月29日周一
> > > > > > > > > >> > > 09:56写道：
> > > > > > > > > >> > > > > > > > > > >> > > >
> > > > > > > > > >> > > > > > > > > > >> > > > > Hi Jing,
> > > > > > > > > >> > > > > > > > > > >> > > > >     > "Do you mean to
> support
> > > the
> > > > > > > > > >> > > AyncTableFunction
> > > > > > > > > >> > > > > > beyond
> > > > > > > > > >> > > > > > > > the
> > > > > > > > > >> > > > > > > > > > >> > > > > LookupTableSource?"
> > > > > > > > > >> > > > > > > > > > >> > > > > Yes, I mean to support the
> > > > > > > > > AyncTableFunction
> > > > > > > > > >> > > beyond
> > > > > > > > > >> > > > > the
> > > > > > > > > >> > > > > > > > > > >> > > > LookupTableSource.
> > > > > > > > > >> > > > > > > > > > >> > > > >
> > > > > > > > > >> > > > > > > > > > >> > > > > The "AsyncTableFunction" is
> > the
> > > > > > > function
> > > > > > > > > with
> > > > > > > > > >> > > > ability
> > > > > > > > > >> > > > > to
> > > > > > > > > >> > > > > > > be
> > > > > > > > > >> > > > > > > > > > >> executed
> > > > > > > > > >> > > > > > > > > > >> > > > async
> > > > > > > > > >> > > > > > > > > > >> > > > > (with AsyncWaitOperator).
> > > > > > > > > >> > > > > > > > > > >> > > > > The async lookup join is a
> > one
> > > of
> > > > > > usage
> > > > > > > > of
> > > > > > > > > >> this.
> > > > > > > > > >> > > So,
> > > > > > > > > >> > > > > we
> > > > > > > > > >> > > > > > > > don't
> > > > > > > > > >> > > > > > > > > > >> have to
> > > > > > > > > >> > > > > > > > > > >> > > > bind
> > > > > > > > > >> > > > > > > > > > >> > > > > the AyncTableFunction with
> > > > > > > > > LookupTableSource.
> > > > > > > > > >> > > > > > > > > > >> > > > > If User-defined
> > > > AsyncTableFunction
> > > > > is
> > > > > > > > > >> supported,
> > > > > > > > > >> > > > user
> > > > > > > > > >> > > > > > can
> > > > > > > > > >> > > > > > > > > > directly
> > > > > > > > > >> > > > > > > > > > >> > use
> > > > > > > > > >> > > > > > > > > > >> > > > > lateral table syntax to
> > perform
> > > > > async
> > > > > > > > > >> operation.
> > > > > > > > > >> > > > > > > > > > >> > > > >
> > > > > > > > > >> > > > > > > > > > >> > > > > > "It would be better if
> you
> > > > could
> > > > > > > > > elaborate
> > > > > > > > > >> the
> > > > > > > > > >> > > > > > proposed
> > > > > > > > > >> > > > > > > > > > changes
> > > > > > > > > >> > > > > > > > > > >> wrt
> > > > > > > > > >> > > > > > > > > > >> > > the
> > > > > > > > > >> > > > > > > > > > >> > > > > CorrelatedCodeGenerator
> with
> > > more
> > > > > > > > details"
> > > > > > > > > >> > > > > > > > > > >> > > > >
> > > > > > > > > >> > > > > > > > > > >> > > > > In the proposal, we use
> > lateral
> > > > > table
> > > > > > > > > syntax
> > > > > > > > > >> to
> > > > > > > > > >> > > > > support
> > > > > > > > > >> > > > > > > the
> > > > > > > > > >> > > > > > > > > > async
> > > > > > > > > >> > > > > > > > > > >> > table
> > > > > > > > > >> > > > > > > > > > >> > > > > function. So the planner
> will
> > > > also
> > > > > > > treat
> > > > > > > > > this
> > > > > > > > > >> > > > > statement
> > > > > > > > > >> > > > > > > to a
> > > > > > > > > >> > > > > > > > > > >> > > > > CommonExecCorrelate node.
> So
> > > the
> > > > > > > runtime
> > > > > > > > > code
> > > > > > > > > >> > > should
> > > > > > > > > >> > > > > be
> > > > > > > > > >> > > > > > > > > > generated
> > > > > > > > > >> > > > > > > > > > >> in
> > > > > > > > > >> > > > > > > > > > >> > > > > CorrelatedCodeGenerator.
> > > > > > > > > >> > > > > > > > > > >> > > > > In CorrelatedCodeGenerator,
> > we
> > > > will
> > > > > > > know
> > > > > > > > > the
> > > > > > > > > >> > > > > > > TableFunction's
> > > > > > > > > >> > > > > > > > > > Kind
> > > > > > > > > >> > > > > > > > > > >> of
> > > > > > > > > >> > > > > > > > > > >> > > > > `FunctionKind.Table` or
> > > > > > > > > >> > `FunctionKind.ASYNC_TABLE`
> > > > > > > > > >> > > > > > > > > > >> > > > > For
> > `FunctionKind.ASYNC_TABLE`
> > > > we
> > > > > > can
> > > > > > > > > >> generate
> > > > > > > > > >> > a
> > > > > > > > > >> > > > > > > > > > >> AsyncWaitOperator
> > > > > > > > > >> > > > > > > > > > >> > to
> > > > > > > > > >> > > > > > > > > > >> > > > > execute the async table
> > > function.
> > > > > > > > > >> > > > > > > > > > >> > > > >
> > > > > > > > > >> > > > > > > > > > >> > > > >
> > > > > > > > > >> > > > > > > > > > >> > > > > Thanks,
> > > > > > > > > >> > > > > > > > > > >> > > > > Aitozi.
> > > > > > > > > >> > > > > > > > > > >> > > > >
> > > > > > > > > >> > > > > > > > > > >> > > > >
> > > > > > > > > >> > > > > > > > > > >> > > > > Jing Ge
> > > > <jing@ververica.com.invalid
> > > > > >
> > > > > > > > > >> > > 于2023年5月29日周一
> > > > > > > > > >> > > > > > > 03:22写道：
> > > > > > > > > >> > > > > > > > > > >> > > > >
> > > > > > > > > >> > > > > > > > > > >> > > > >> Hi Aitozi,
> > > > > > > > > >> > > > > > > > > > >> > > > >>
> > > > > > > > > >> > > > > > > > > > >> > > > >> Thanks for the
> > clarification.
> > > > The
> > > > > > > naming
> > > > > > > > > >> > "Lookup"
> > > > > > > > > >> > > > > might
> > > > > > > > > >> > > > > > > > > suggest
> > > > > > > > > >> > > > > > > > > > >> > using
> > > > > > > > > >> > > > > > > > > > >> > > it
> > > > > > > > > >> > > > > > > > > > >> > > > >> for table look up. But
> > > > > conceptually
> > > > > > > what
> > > > > > > > > the
> > > > > > > > > >> > > eval()
> > > > > > > > > >> > > > > > > method
> > > > > > > > > >> > > > > > > > > will
> > > > > > > > > >> > > > > > > > > > >> do
> > > > > > > > > >> > > > > > > > > > >> > is
> > > > > > > > > >> > > > > > > > > > >> > > to
> > > > > > > > > >> > > > > > > > > > >> > > > >> get a collection of
> > > results(Row,
> > > > > > > > RowData)
> > > > > > > > > >> from
> > > > > > > > > >> > > the
> > > > > > > > > >> > > > > > given
> > > > > > > > > >> > > > > > > > > keys.
> > > > > > > > > >> > > > > > > > > > >> How
> > > > > > > > > >> > > > > > > > > > >> > it
> > > > > > > > > >> > > > > > > > > > >> > > > will
> > > > > > > > > >> > > > > > > > > > >> > > > >> be done depends on the
> > > > > > implementation,
> > > > > > > > > i.e.
> > > > > > > > > >> you
> > > > > > > > > >> > > can
> > > > > > > > > >> > > > > > > > implement
> > > > > > > > > >> > > > > > > > > > >> your
> > > > > > > > > >> > > > > > > > > > >> > own
> > > > > > > > > >> > > > > > > > > > >> > > > >> Source[1][2]. The example
> in
> > > the
> > > > > > FLIP
> > > > > > > > > >> should be
> > > > > > > > > >> > > > able
> > > > > > > > > >> > > > > to
> > > > > > > > > >> > > > > > > be
> > > > > > > > > >> > > > > > > > > > >> handled
> > > > > > > > > >> > > > > > > > > > >> > in
> > > > > > > > > >> > > > > > > > > > >> > > > this
> > > > > > > > > >> > > > > > > > > > >> > > > >> way.
> > > > > > > > > >> > > > > > > > > > >> > > > >>
> > > > > > > > > >> > > > > > > > > > >> > > > >> Do you mean to support the
> > > > > > > > > AyncTableFunction
> > > > > > > > > >> > > beyond
> > > > > > > > > >> > > > > the
> > > > > > > > > >> > > > > > > > > > >> > > > LookupTableSource?
> > > > > > > > > >> > > > > > > > > > >> > > > >> It would be better if you
> > > could
> > > > > > > > elaborate
> > > > > > > > > >> the
> > > > > > > > > >> > > > > proposed
> > > > > > > > > >> > > > > > > > > changes
> > > > > > > > > >> > > > > > > > > > >> wrt
> > > > > > > > > >> > > > > > > > > > >> > the
> > > > > > > > > >> > > > > > > > > > >> > > > >> CorrelatedCodeGenerator
> with
> > > > more
> > > > > > > > details.
> > > > > > > > > >> > > Thanks!
> > > > > > > > > >> > > > > > > > > > >> > > > >>
> > > > > > > > > >> > > > > > > > > > >> > > > >> Best regards,
> > > > > > > > > >> > > > > > > > > > >> > > > >> Jing
> > > > > > > > > >> > > > > > > > > > >> > > > >>
> > > > > > > > > >> > > > > > > > > > >> > > > >> [1]
> > > > > > > > > >> > > > > > > > > > >> > > > >>
> > > > > > > > > >> > > > > > > > > > >> > > > >>
> > > > > > > > > >> > > > > > > > > > >> > > >
> > > > > > > > > >> > > > > > > > > > >> > >
> > > > > > > > > >> > > > > > > > > > >> >
> > > > > > > > > >> > > > > > > > > > >>
> > > > > > > > > >> > > > > > > > > >
> > > > > > > > > >> > > > > > > > >
> > > > > > > > > >> > > > > > > >
> > > > > > > > > >> > > > > > >
> > > > > > > > > >> > > > > >
> > > > > > > > > >> > > > >
> > > > > > > > > >> > > >
> > > > > > > > > >> > >
> > > > > > > > > >> >
> > > > > > > > > >>
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/flink/blob/678370b18e1b6c4a23e5ce08f8efd05675a0cc17/flink-table/flink-table-common/src/main/java/org/apache/flink/table/connector/source/LookupTableSource.java#L64
> > > > > > > > > >> > > > > > > > > > >> > > > >> [2]
> > > > > > > > > >> > > > > > > > > > >> > > > >>
> > > > > > > > > >> > > > > > > > > > >> > > > >>
> > > > > > > > > >> > > > > > > > > > >> > > >
> > > > > > > > > >> > > > > > > > > > >> > >
> > > > > > > > > >> > > > > > > > > > >> >
> > > > > > > > > >> > > > > > > > > > >>
> > > > > > > > > >> > > > > > > > > >
> > > > > > > > > >> > > > > > > > >
> > > > > > > > > >> > > > > > > >
> > > > > > > > > >> > > > > > >
> > > > > > > > > >> > > > > >
> > > > > > > > > >> > > > >
> > > > > > > > > >> > > >
> > > > > > > > > >> > >
> > > > > > > > > >> >
> > > > > > > > > >>
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/flink/blob/678370b18e1b6c4a23e5ce08f8efd05675a0cc17/flink-table/flink-table-common/src/main/java/org/apache/flink/table/connector/source/AsyncTableFunctionProvider.java#L49
> > > > > > > > > >> > > > > > > > > > >> > > > >>
> > > > > > > > > >> > > > > > > > > > >> > > > >> On Sat, May 27, 2023 at
> > > 9:48 AM
> > > > > > > Aitozi <
> > > > > > > > > >> > > > > > > > gjying1314@gmail.com
> > > > > > > > > >> > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > >> > wrote:
> > > > > > > > > >> > > > > > > > > > >> > > > >>
> > > > > > > > > >> > > > > > > > > > >> > > > >> > Hi Jing,
> > > > > > > > > >> > > > > > > > > > >> > > > >> >     Thanks for your
> > > response.
> > > > As
> > > > > > > > stated
> > > > > > > > > in
> > > > > > > > > >> > the
> > > > > > > > > >> > > > > FLIP,
> > > > > > > > > >> > > > > > > the
> > > > > > > > > >> > > > > > > > > > >> purpose
> > > > > > > > > >> > > > > > > > > > >> > of
> > > > > > > > > >> > > > > > > > > > >> > > > this
> > > > > > > > > >> > > > > > > > > > >> > > > >> > FLIP is meant to support
> > > > > > > > > >> > > > > > > > > > >> > > > >> > user-defined async table
> > > > > function.
> > > > > > > As
> > > > > > > > > >> > described
> > > > > > > > > >> > > > in
> > > > > > > > > >> > > > > > > flink
> > > > > > > > > >> > > > > > > > > > >> document
> > > > > > > > > >> > > > > > > > > > >> > > [1]
> > > > > > > > > >> > > > > > > > > > >> > > > >> >
> > > > > > > > > >> > > > > > > > > > >> > > > >> > Async table functions
> are
> > > > > special
> > > > > > > > > >> functions
> > > > > > > > > >> > for
> > > > > > > > > >> > > > > table
> > > > > > > > > >> > > > > > > > > sources
> > > > > > > > > >> > > > > > > > > > >> that
> > > > > > > > > >> > > > > > > > > > >> > > > >> perform
> > > > > > > > > >> > > > > > > > > > >> > > > >> > > a lookup.
> > > > > > > > > >> > > > > > > > > > >> > > > >> > >
> > > > > > > > > >> > > > > > > > > > >> > > > >> >
> > > > > > > > > >> > > > > > > > > > >> > > > >> > So end user can not
> > directly
> > > > > > define
> > > > > > > > and
> > > > > > > > > >> use
> > > > > > > > > >> > > async
> > > > > > > > > >> > > > > > table
> > > > > > > > > >> > > > > > > > > > >> function
> > > > > > > > > >> > > > > > > > > > >> > > now.
> > > > > > > > > >> > > > > > > > > > >> > > > An
> > > > > > > > > >> > > > > > > > > > >> > > > >> > user case is reported in
> > [2]
> > > > > > > > > >> > > > > > > > > > >> > > > >> >
> > > > > > > > > >> > > > > > > > > > >> > > > >> > So, in conclusion, no
> new
> > > > > > interface
> > > > > > > is
> > > > > > > > > >> > > > introduced,
> > > > > > > > > >> > > > > > but
> > > > > > > > > >> > > > > > > we
> > > > > > > > > >> > > > > > > > > > >> extend
> > > > > > > > > >> > > > > > > > > > >> > the
> > > > > > > > > >> > > > > > > > > > >> > > > >> > ability to support
> > > > user-defined
> > > > > > > async
> > > > > > > > > >> table
> > > > > > > > > >> > > > > function.
> > > > > > > > > >> > > > > > > > > > >> > > > >> >
> > > > > > > > > >> > > > > > > > > > >> > > > >> > [1]:
> > > > > > > > > >> > > > > > > > > > >> > > > >> >
> > > > > > > > > >> > > > > > > > > > >> > > > >> >
> > > > > > > > > >> > > > > > > > > > >> > > > >>
> > > > > > > > > >> > > > > > > > > > >> > > >
> > > > > > > > > >> > > > > > > > > > >> > >
> > > > > > > > > >> > > > > > > > > > >> >
> > > > > > > > > >> > > > > > > > > > >>
> > > > > > > > > >> > > > > > > > > >
> > > > > > > > > >> > > > > > > > >
> > > > > > > > > >> > > > > > > >
> > > > > > > > > >> > > > > > >
> > > > > > > > > >> > > > > >
> > > > > > > > > >> > > > >
> > > > > > > > > >> > > >
> > > > > > > > > >> > >
> > > > > > > > > >> >
> > > > > > > > > >>
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/dev/table/functions/udfs/
> > > > > > > > > >> > > > > > > > > > >> > > > >> > [2]:
> > > > > > > > > >> > > > > > > > > > >> > >
> > > > > > > > > >> > > > > > > >
> > > > > > > > > >> >
> > > > > > https://lists.apache.org/thread/qljwd40v5ntz6733cwcdr8s4z97b343b
> > > > > > > > > >> > > > > > > > > > >> > > > >> >
> > > > > > > > > >> > > > > > > > > > >> > > > >> > Thanks.
> > > > > > > > > >> > > > > > > > > > >> > > > >> > Aitozi.
> > > > > > > > > >> > > > > > > > > > >> > > > >> >
> > > > > > > > > >> > > > > > > > > > >> > > > >> >
> > > > > > > > > >> > > > > > > > > > >> > > > >> > Jing Ge
> > > > > > <jing@ververica.com.invalid
> > > > > > > >
> > > > > > > > > >> > > > 于2023年5月27日周六
> > > > > > > > > >> > > > > > > > > 06:40写道：
> > > > > > > > > >> > > > > > > > > > >> > > > >> >
> > > > > > > > > >> > > > > > > > > > >> > > > >> > > Hi Aitozi,
> > > > > > > > > >> > > > > > > > > > >> > > > >> > >
> > > > > > > > > >> > > > > > > > > > >> > > > >> > > Thanks for your
> > proposal.
> > > I
> > > > am
> > > > > > not
> > > > > > > > > quite
> > > > > > > > > >> > sure
> > > > > > > > > >> > > > if
> > > > > > > > > >> > > > > I
> > > > > > > > > >> > > > > > > > > > understood
> > > > > > > > > >> > > > > > > > > > >> > your
> > > > > > > > > >> > > > > > > > > > >> > > > >> > thoughts
> > > > > > > > > >> > > > > > > > > > >> > > > >> > > correctly. You
> > described a
> > > > > > special
> > > > > > > > > case
> > > > > > > > > >> > > > > > > implementation
> > > > > > > > > >> > > > > > > > of
> > > > > > > > > >> > > > > > > > > > the
> > > > > > > > > >> > > > > > > > > > >> > > > >> > > AsyncTableFunction
> with
> > on
> > > > > > public
> > > > > > > > API
> > > > > > > > > >> > > changes.
> > > > > > > > > >> > > > > > Would
> > > > > > > > > >> > > > > > > > you
> > > > > > > > > >> > > > > > > > > > >> please
> > > > > > > > > >> > > > > > > > > > >> > > > >> elaborate
> > > > > > > > > >> > > > > > > > > > >> > > > >> > > your purpose of
> writing
> > a
> > > > FLIP
> > > > > > > > > >> according to
> > > > > > > > > >> > > the
> > > > > > > > > >> > > > > > FLIP
> > > > > > > > > >> > > > > > > > > > >> > > > documentation[1]?
> > > > > > > > > >> > > > > > > > > > >> > > > >> > > Thanks!
> > > > > > > > > >> > > > > > > > > > >> > > > >> > >
> > > > > > > > > >> > > > > > > > > > >> > > > >> > > [1]
> > > > > > > > > >> > > > > > > > > > >> > > > >> > >
> > > > > > > > > >> > > > > > > > > > >> > > > >> > >
> > > > > > > > > >> > > > > > > > > > >> > > > >> >
> > > > > > > > > >> > > > > > > > > > >> > > > >>
> > > > > > > > > >> > > > > > > > > > >> > > >
> > > > > > > > > >> > > > > > > > > > >> > >
> > > > > > > > > >> > > > > > > > > > >> >
> > > > > > > > > >> > > > > > > > > > >>
> > > > > > > > > >> > > > > > > > > >
> > > > > > > > > >> > > > > > > > >
> > > > > > > > > >> > > > > > > >
> > > > > > > > > >> > > > > > >
> > > > > > > > > >> > > > > >
> > > > > > > > > >> > > > >
> > > > > > > > > >> > > >
> > > > > > > > > >> > >
> > > > > > > > > >> >
> > > > > > > > > >>
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/Flink+Improvement+Proposals
> > > > > > > > > >> > > > > > > > > > >> > > > >> > >
> > > > > > > > > >> > > > > > > > > > >> > > > >> > > Best regards,
> > > > > > > > > >> > > > > > > > > > >> > > > >> > > Jing
> > > > > > > > > >> > > > > > > > > > >> > > > >> > >
> > > > > > > > > >> > > > > > > > > > >> > > > >> > > On Wed, May 24, 2023
> at
> > > > > 1:07 PM
> > > > > > > > > Aitozi <
> > > > > > > > > >> > > > > > > > > > gjying1314@gmail.com
> > > > > > > > > >> > > > > > > > > > >> >
> > > > > > > > > >> > > > > > > > > > >> > > > wrote:
> > > > > > > > > >> > > > > > > > > > >> > > > >> > >
> > > > > > > > > >> > > > > > > > > > >> > > > >> > > > May I ask for some
> > > > feedback
> > > > > > :D
> > > > > > > > > >> > > > > > > > > > >> > > > >> > > >
> > > > > > > > > >> > > > > > > > > > >> > > > >> > > > Thanks,
> > > > > > > > > >> > > > > > > > > > >> > > > >> > > > Aitozi
> > > > > > > > > >> > > > > > > > > > >> > > > >> > > >
> > > > > > > > > >> > > > > > > > > > >> > > > >> > > > Aitozi <
> > > > > gjying1314@gmail.com>
> > > > > > > > > >> > > 于2023年5月23日周二
> > > > > > > > > >> > > > > > > 19:14写道：
> > > > > > > > > >> > > > > > > > > > >> > > > >> > > > >
> > > > > > > > > >> > > > > > > > > > >> > > > >> > > > > Just catch an user
> > > case
> > > > > > report
> > > > > > > > > from
> > > > > > > > > >> > > Giannis
> > > > > > > > > >> > > > > > > Polyzos
> > > > > > > > > >> > > > > > > > > for
> > > > > > > > > >> > > > > > > > > > >> this
> > > > > > > > > >> > > > > > > > > > >> > > > >> usage:
> > > > > > > > > >> > > > > > > > > > >> > > > >> > > > >
> > > > > > > > > >> > > > > > > > > > >> > > > >> > > > >
> > > > > > > > > >> > > > > > > > > > >> > > >
> > > > > > > > > >> > > > > > > > >
> > > > > > > > > >> > >
> > > > > > >
> https://lists.apache.org/thread/qljwd40v5ntz6733cwcdr8s4z97b343b
> > > > > > > > > >> > > > > > > > > > >> > > > >> > > > >
> > > > > > > > > >> > > > > > > > > > >> > > > >> > > > > Aitozi <
> > > > > > gjying1314@gmail.com>
> > > > > > > > > >> > > > 于2023年5月23日周二
> > > > > > > > > >> > > > > > > > 17:45写道：
> > > > > > > > > >> > > > > > > > > > >> > > > >> > > > > >
> > > > > > > > > >> > > > > > > > > > >> > > > >> > > > > > Hi guys,
> > > > > > > > > >> > > > > > > > > > >> > > > >> > > > > >     I want to
> > bring
> > > > up a
> > > > > > > > > >> discussion
> > > > > > > > > >> > > about
> > > > > > > > > >> > > > > > > adding
> > > > > > > > > >> > > > > > > > > > >> support
> > > > > > > > > >> > > > > > > > > > >> > of
> > > > > > > > > >> > > > > > > > > > >> > > > User
> > > > > > > > > >> > > > > > > > > > >> > > > >> > > > > > Defined
> > > > > AsyncTableFunction
> > > > > > > in
> > > > > > > > > >> Flink.
> > > > > > > > > >> > > > > > > > > > >> > > > >> > > > > > Currently, async
> > > table
> > > > > > > > function
> > > > > > > > > >> are
> > > > > > > > > >> > > > special
> > > > > > > > > >> > > > > > > > > functions
> > > > > > > > > >> > > > > > > > > > >> for
> > > > > > > > > >> > > > > > > > > > >> > > > table
> > > > > > > > > >> > > > > > > > > > >> > > > >> > > source
> > > > > > > > > >> > > > > > > > > > >> > > > >> > > > > > to perform
> > > > > > > > > >> > > > > > > > > > >> > > > >> > > > > > async lookup.
> > > However,
> > > > > > it's
> > > > > > > > > worth
> > > > > > > > > >> to
> > > > > > > > > >> > > > > support
> > > > > > > > > >> > > > > > > the
> > > > > > > > > >> > > > > > > > > user
> > > > > > > > > >> > > > > > > > > > >> > > defined
> > > > > > > > > >> > > > > > > > > > >> > > > >> async
> > > > > > > > > >> > > > > > > > > > >> > > > >> > > > > > table function.
> > > > > > > > > >> > > > > > > > > > >> > > > >> > > > > > Because, in this
> > > way,
> > > > > the
> > > > > > > end
> > > > > > > > > SQL
> > > > > > > > > >> > user
> > > > > > > > > >> > > > can
> > > > > > > > > >> > > > > > > > leverage
> > > > > > > > > >> > > > > > > > > > it
> > > > > > > > > >> > > > > > > > > > >> to
> > > > > > > > > >> > > > > > > > > > >> > > > >> perform
> > > > > > > > > >> > > > > > > > > > >> > > > >> > the
> > > > > > > > > >> > > > > > > > > > >> > > > >> > > > > > async operation
> > > > > > > > > >> > > > > > > > > > >> > > > >> > > > > > which is useful
> to
> > > > > maximum
> > > > > > > the
> > > > > > > > > >> system
> > > > > > > > > >> > > > > > > throughput
> > > > > > > > > >> > > > > > > > > > >> > especially
> > > > > > > > > >> > > > > > > > > > >> > > > for
> > > > > > > > > >> > > > > > > > > > >> > > > >> IO
> > > > > > > > > >> > > > > > > > > > >> > > > >> > > > > > bottleneck case.
> > > > > > > > > >> > > > > > > > > > >> > > > >> > > > > >
> > > > > > > > > >> > > > > > > > > > >> > > > >> > > > > > You can find
> some
> > > more
> > > > > > > detail
> > > > > > > > in
> > > > > > > > > >> [1].
> > > > > > > > > >> > > > > > > > > > >> > > > >> > > > > >
> > > > > > > > > >> > > > > > > > > > >> > > > >> > > > > > Looking forward
> to
> > > > > > feedback
> > > > > > > > > >> > > > > > > > > > >> > > > >> > > > > >
> > > > > > > > > >> > > > > > > > > > >> > > > >> > > > > >
> > > > > > > > > >> > > > > > > > > > >> > > > >> > > > > > [1]:
> > > > > > > > > >> > > > > > > > > > >> > > > >> > > >
> > > > > > > > > >> > > > > > > > > > >> > > > >> > >
> > > > > > > > > >> > > > > > > > > > >> > > > >> >
> > > > > > > > > >> > > > > > > > > > >> > > > >>
> > > > > > > > > >> > > > > > > > > > >> > > >
> > > > > > > > > >> > > > > > > > > > >> > >
> > > > > > > > > >> > > > > > > > > > >> >
> > > > > > > > > >> > > > > > > > > > >>
> > > > > > > > > >> > > > > > > > > >
> > > > > > > > > >> > > > > > > > >
> > > > > > > > > >> > > > > > > >
> > > > > > > > > >> > > > > > >
> > > > > > > > > >> > > > > >
> > > > > > > > > >> > > > >
> > > > > > > > > >> > > >
> > > > > > > > > >> > >
> > > > > > > > > >> >
> > > > > > > > > >>
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/%5BFLIP-313%5D+Add+support+of+User+Defined+AsyncTableFunction
> > > > > > > > > >> > > > > > > > > > >> > > > >> > > > > >
> > > > > > > > > >> > > > > > > > > > >> > > > >> > > > > > Thanks,
> > > > > > > > > >> > > > > > > > > > >> > > > >> > > > > > Aitozi.
> > > > > > > > > >> > > > > > > > > > >> > > > >> > > >
> > > > > > > > > >> > > > > > > > > > >> > > > >> > >
> > > > > > > > > >> > > > > > > > > > >> > > > >> >
> > > > > > > > > >> > > > > > > > > > >> > > > >>
> > > > > > > > > >> > > > > > > > > > >> > > > >
> > > > > > > > > >> > > > > > > > > > >> > > >
> > > > > > > > > >> > > > > > > > > > >> > >
> > > > > > > > > >> > > > > > > > > > >> >
> > > > > > > > > >> > > > > > > > > > >>
> > > > > > > > > >> > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > >
> > > > > > > > > >> > > > > > > > >
> > > > > > > > > >> > > > > > > >
> > > > > > > > > >> > > > > > >
> > > > > > > > > >> > > > > >
> > > > > > > > > >> > > > >
> > > > > > > > > >> > > >
> > > > > > > > > >> > >
> > > > > > > > > >> >
> > > > > > > > > >>
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] FLIP-313 Add support of User Defined AsyncTableFunction

Posted by Lincoln Lee <li...@gmail.com>.

Hi Aitozi,

Thanks for your reply!  Gives sql users more flexibility to get
asynchronous processing capabilities via lateral join table function +1 for
this

For `JavaAsyncTableFunc0` in flip, can you use a scenario like RPC call as
an example?

For the name of this query hint, 'LATERAL' (include its internal options)
don't show any relevance to async, but I haven't thought of a suitable name
at the moment,
maybe we need to highlight the async keyword directly, we can also see if
others have better candidates

For the hint option "timeout = '180s'" should be "'timeout' = '180s'",
seems a typo in the flip. And use upper case for all keywords in sql
examples.
Also, the terms 'correlate join' and 'lateral join' are not the same as in
the current joins page[1], so maybe it would be better if we unified them
into  'join table function'

[1]
https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/sql/queries/joins/#table-function

Best,
Lincoln Lee


Aitozi <gj...@gmail.com> 于2023年6月14日周三 16:11写道：

> Hi Lincoln
>
>     Very thanks for your valuable question. I will try to answer your
> questions inline.
>
> >Does the async udtf bring any additional benefits besides a
> lighter implementation？
>
> IMO, async udtf is more than a lighter implementation. It can act as a
> general way for sql users to use the async operator. And they don't have to
> bind the async function with a table (a LookupTable), and they are not
> forced to join on an equality join condition, and they can use it to do
> more than enrich data.
>
> The async lookup join is more like a subset/specific usage of async udtf.
> The specific version has more opportunity to be optimized (like push down)
> is acceptable. Async table function should be categorized to used-defined
> function.
>
> >Should users
>
> migrate to the lookup source when they encounter similar requirements or
>
> problems, or should we develop an additional set of similar mechanisms？
>
> As I clarified above, the lookup join is a specific usage of async udtf. So
> it deserves more refined optimization like caching / retryable. But it may
> not all
>
> suitable for the async udtf. As function, it can be deterministic/or
> non-deterministic. So caching is not suitable, and we also do not have a
> common cache for the udf now. So I think optimization like caching/retry
> should be handed over to the function implementor.
>
> > the newly added query hint need a different name that
> can be easier related to the lateral operation as the current join hints[5]
> do.
>
>
> What about using LATERAL?
>
> as below
>
> SELECT /*+ LATERAL('output-mode' = 'ordered', 'capacity' = '200', timeout =
> '180s') */ a, c1, c2
>
> FROM T1
>
> LEFT JOIN lateral TABLE (async_split(b)) AS T(c1, c2) ON true
>
> >For the async func example, since the target scenario is an external io
> operation, it's better to add the `close` method to actively release
> resources as a good example for users
>
>
> Make sense to me, will update the FLIP
>
> Best,
>
> Aitozi.
>
> Lincoln Lee <li...@gmail.com> 于2023年6月14日周三 14:24写道：
>
> > Hi Aitozi,
> >
> > Sorry for the lately reply here!  Supports async
> udtf(`AsyncTableFunction`)
> > directly in sql seems like an attractive feature, but there're two issues
> > that need to be addressed before we can be sure to add it:
> > 1. As mentioned in the flip[1], the current lookup function can already
> > implement the requirements, but it requires implementing an extra
> > `LookupTableSource` and explicitly declaring the table schema (which can
> > help implementers the various push-down optimizations supported by the
> > planner). Does the async udtf bring any additional benefits besides a
> > lighter implementation?
> > 2. FLIP-221[2] abstracts a reusable cache and metric infrastructure for
> > lookup sources, which are important to improve performance and
> > observability for high overhead external io scenarios, how do we
> integrate
> > and reuse these capabilities after introducing async udtf? Should users
> > migrate to the lookup source when they encounter similar requirements or
> > problems, or should we develop an additional set of similar mechanisms?
> (a
> > similarly case:  FLIP-234[3] introduced the retryable capability for
> lookup
> > join)
> >
> > For the flip itself,
> > 1. Considering the 'options' is already used as the dynamic table
> > options[4] in flink, the newly added query hint need a different name
> that
> > can be easier related to the lateral operation as the current join
> hints[5]
> > do.
> > 2. For the async func example, since the target scenario is an external
> io
> > operation, it's better to add the `close` method to actively release
> > resources as a good example for users. Also in terms of the determinism
> of
> > a function, it is important to remind users that unless the behavior of
> the
> > function is deterministic, it needs to be explicitly declared as
> > non-deterministic.
> >
> > [1].
> >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-313%3A+Add+support+of+User+Defined+AsyncTableFunction?src=contextnavpagetreemode
> > [2].
> >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-221%3A+Abstraction+for+lookup+source+cache+and+metric?src=contextnavpagetreemode
> > [3].
> >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-234%3A+Support+Retryable+Lookup+Join+To+Solve+Delayed+Updates+Issue+In+External+Systems?src=contextnavpagetreemode
> > [4].
> >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-113%3A+Supports+Dynamic+Table+Options+for+Flink+SQL?src=contextnavpagetreemode
> > [5].
> >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-229%3A+Introduces+Join+Hint+for+Flink+SQL+Batch+Job?src=contextnavpagetreemode
> >
> > Best,
> > Lincoln Lee
> >
> >
> > Aitozi <gj...@gmail.com> 于2023年6月13日周二 11:30写道：
> >
> > > Get your meaning now, thanks :)
> > >
> > > Best,
> > > Aitozi.
> > >
> > > Feng Jin <ji...@gmail.com> 于2023年6月13日周二 11:16写道：
> > >
> > > > Hi Aitozi,
> > > >
> > > > Sorry for the confusing description.
> > > >
> > > > What I meant was that if we need to remind users about tire safety
> > > issues,
> > > > we should introduce the new UDTF interface instead of executing the
> > > > original UDTF asynchronously. Therefore, I agree with introducing the
> > > > AsyncTableFunction.
> > > >
> > > > Best,
> > > > Feng
> > > >
> > > > On Tue, Jun 13, 2023 at 10:42 AM Aitozi <gj...@gmail.com>
> wrote:
> > > >
> > > > > Hi Feng,
> > > > >     Thanks for your question. We do not provide a way to switch the
> > > UDTF
> > > > > between sync and async way,
> > > > > So there should be no thread safety problem here.
> > > > >
> > > > > Best,
> > > > > Aitozi
> > > > >
> > > > > Feng Jin <ji...@gmail.com> 于2023年6月13日周二 10:31写道：
> > > > >
> > > > > > Hi Aitozi, We do need to remind users about thread safety issues.
> > > Thank
> > > > > you
> > > > > > for your efforts on this FLIP. I have no further questions.
> > > > > > Best, Feng
> > > > > >
> > > > > >
> > > > > > On Tue, Jun 13, 2023 at 6:05 AM Jing Ge
> <jing@ververica.com.invalid
> > >
> > > > > > wrote:
> > > > > >
> > > > > > > Hi Aitozi,
> > > > > > >
> > > > > > > Thanks for taking care of that part. I have no other concern.
> > > > > > >
> > > > > > > Best regards,
> > > > > > > Jing
> > > > > > >
> > > > > > >
> > > > > > > On Mon, Jun 12, 2023 at 5:38 PM Aitozi <gj...@gmail.com>
> > > wrote:
> > > > > > >
> > > > > > > > BTW, If there are no other more blocking issue / comments, I
> > > would
> > > > > like
> > > > > > > to
> > > > > > > > start a VOTE in another thread this wednesday 6.14
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > Aitozi.
> > > > > > > >
> > > > > > > > Aitozi <gj...@gmail.com> 于2023年6月12日周一 23:34写道：
> > > > > > > >
> > > > > > > > > Hi, Jing,
> > > > > > > > >     Thanks for your explanation. I get your point now.
> > > > > > > > >
> > > > > > > > > For the performance part, I think it's a good idea to run
> > with
> > > > > > > returning
> > > > > > > > a
> > > > > > > > > big table case, the memory consumption
> > > > > > > > > should be a point to be taken care about. Because in the
> > > ordered
> > > > > > mode,
> > > > > > > > the
> > > > > > > > > head element in buffer may affect the
> > > > > > > > > total memory consumption.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Thanks,
> > > > > > > > > Aitozi.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Jing Ge <ji...@ververica.com.invalid> 于2023年6月12日周一
> 20:28写道：
> > > > > > > > >
> > > > > > > > >> Hi Aitozi,
> > > > > > > > >>
> > > > > > > > >> Which key will be used for lookup is not an issue, only
> one
> > > row
> > > > > will
> > > > > > > be
> > > > > > > > >> required for each key in order to enrich it. True, it
> > depends
> > > on
> > > > > the
> > > > > > > > >> implementation whether multiple rows or single row for
> each
> > > key
> > > > > will
> > > > > > > be
> > > > > > > > >> returned. However, for the lookup & enrichment scenario,
> one
> > > > > row/key
> > > > > > > is
> > > > > > > > >> recommended, otherwise, like I mentioned previously,
> > > enrichment
> > > > > > won't
> > > > > > > > >> work.
> > > > > > > > >>
> > > > > > > > >> I am a little bit concerned about returning a big table
> for
> > > each
> > > > > > key,
> > > > > > > > >> since
> > > > > > > > >> it will take the async call longer to return and need more
> > > > memory.
> > > > > > The
> > > > > > > > >> performance tests should cover this scenario. This is not
> a
> > > > > blocking
> > > > > > > > issue
> > > > > > > > >> for this FLIP.
> > > > > > > > >>
> > > > > > > > >> Best regards,
> > > > > > > > >> Jing
> > > > > > > > >>
> > > > > > > > >> On Sat, Jun 10, 2023 at 4:11 AM Aitozi <
> > gjying1314@gmail.com>
> > > > > > wrote:
> > > > > > > > >>
> > > > > > > > >> > Hi Jing,
> > > > > > > > >> >     I means the join key is not necessary to be the
> > primary
> > > > key
> > > > > or
> > > > > > > > >> unique
> > > > > > > > >> > index of the database.
> > > > > > > > >> > In this situation, we may queried out multi rows for one
> > > join
> > > > > > key. I
> > > > > > > > >> think
> > > > > > > > >> > that's why the
> > > > > > > > >> > LookupFunction#lookup will return a collection of
> RowData.
> > > > > > > > >> >
> > > > > > > > >> > BTW, I think the behavior of lookup join will not affect
> > the
> > > > > > > semantic
> > > > > > > > of
> > > > > > > > >> > the async udtf.
> > > > > > > > >> > We use the Async TableFunction here and the table
> function
> > > can
> > > > > > > collect
> > > > > > > > >> > multiple rows.
> > > > > > > > >> >
> > > > > > > > >> > Thanks,
> > > > > > > > >> > Atiozi.
> > > > > > > > >> >
> > > > > > > > >> >
> > > > > > > > >> >
> > > > > > > > >> > Jing Ge <ji...@ververica.com.invalid> 于2023年6月10日周六
> > 00:15写道：
> > > > > > > > >> >
> > > > > > > > >> > > Hi Aitozi,
> > > > > > > > >> > >
> > > > > > > > >> > > The keyRow used in this case contains all keys[1].
> > > > > > > > >> > >
> > > > > > > > >> > > Best regards,
> > > > > > > > >> > > Jing
> > > > > > > > >> > >
> > > > > > > > >> > > [1]
> > > > > > > > >> > >
> > > > > > > > >> > >
> > > > > > > > >> >
> > > > > > > > >>
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/flink/blob/191ec6ca3943d7119f14837efe112e074d815c47/flink-table/flink-table-common/src/main/java/org/apache/flink/table/functions/LookupFunction.java#L49
> > > > > > > > >> > >
> > > > > > > > >> > >
> > > > > > > > >> > > On Fri, Jun 9, 2023 at 3:42 PM Aitozi <
> > > gjying1314@gmail.com
> > > > >
> > > > > > > wrote:
> > > > > > > > >> > >
> > > > > > > > >> > > > Hi Jing,
> > > > > > > > >> > > >
> > > > > > > > >> > > >      The performance test is added to the FLIP.
> > > > > > > > >> > > >
> > > > > > > > >> > > >      As I know, The lookup join can return multi
> rows,
> > > it
> > > > > > > depends
> > > > > > > > on
> > > > > > > > >> > > > whether  the join key
> > > > > > > > >> > > > is the primary key of the external database or not.
> > The
> > > > > > `lookup`
> > > > > > > > [1]
> > > > > > > > >> > will
> > > > > > > > >> > > > return a collection of
> > > > > > > > >> > > > joined result, and each of them will be collected
> > > > > > > > >> > > >
> > > > > > > > >> > > >
> > > > > > > > >> > > > [1]:
> > > > > > > > >> > > >
> > > > > > > > >> > > >
> > > > > > > > >> > >
> > > > > > > > >> >
> > > > > > > > >>
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/flink/blob/191ec6ca3943d7119f14837efe112e074d815c47/flink-table/flink-table-common/src/main/java/org/apache/flink/table/functions/LookupFunction.java#L52
> > > > > > > > >> > > >
> > > > > > > > >> > > >
> > > > > > > > >> > > > Thanks,
> > > > > > > > >> > > > Aitozi.
> > > > > > > > >> > > >
> > > > > > > > >> > > > Jing Ge <ji...@ververica.com.invalid> 于2023年6月9日周五
> > > > 17:05写道：
> > > > > > > > >> > > >
> > > > > > > > >> > > > > Hi Aitozi,
> > > > > > > > >> > > > >
> > > > > > > > >> > > > > Thanks for the feedback. Looking forward to the
> > > > > performance
> > > > > > > > tests.
> > > > > > > > >> > > > >
> > > > > > > > >> > > > > Afaik, lookup returns one row for each key [1]
> [2].
> > > > > > > > Conceptually,
> > > > > > > > >> the
> > > > > > > > >> > > > > lookup function is used to enrich column(s) from
> the
> > > > > > dimension
> > > > > > > > >> table.
> > > > > > > > >> > > If,
> > > > > > > > >> > > > > for the given key, there will be more than one
> row,
> > > > there
> > > > > > will
> > > > > > > > be
> > > > > > > > >> no
> > > > > > > > >> > > way
> > > > > > > > >> > > > to
> > > > > > > > >> > > > > know which row will be used to enrich the key.
> > > > > > > > >> > > > >
> > > > > > > > >> > > > > [1]
> > > > > > > > >> > > > >
> > > > > > > > >> > > > >
> > > > > > > > >> > > >
> > > > > > > > >> > >
> > > > > > > > >> >
> > > > > > > > >>
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/flink/blob/191ec6ca3943d7119f14837efe112e074d815c47/flink-table/flink-table-common/src/main/java/org/apache/flink/table/functions/LookupFunction.java#L49
> > > > > > > > >> > > > > [2]
> > > > > > > > >> > > > >
> > > > > > > > >> > > > >
> > > > > > > > >> > > >
> > > > > > > > >> > >
> > > > > > > > >> >
> > > > > > > > >>
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/flink/blob/191ec6ca3943d7119f14837efe112e074d815c47/flink-table/flink-table-common/src/main/java/org/apache/flink/table/functions/TableFunction.java#L196
> > > > > > > > >> > > > >
> > > > > > > > >> > > > > Best regards,
> > > > > > > > >> > > > > Jing
> > > > > > > > >> > > > >
> > > > > > > > >> > > > > On Fri, Jun 9, 2023 at 5:18 AM Aitozi <
> > > > > gjying1314@gmail.com
> > > > > > >
> > > > > > > > >> wrote:
> > > > > > > > >> > > > >
> > > > > > > > >> > > > > > Hi Jing
> > > > > > > > >> > > > > >     Thanks for your good questions. I have
> updated
> > > the
> > > > > > > example
> > > > > > > > >> to
> > > > > > > > >> > the
> > > > > > > > >> > > > > FLIP.
> > > > > > > > >> > > > > >
> > > > > > > > >> > > > > > > Only one row for each lookup
> > > > > > > > >> > > > > > lookup can also return multi rows, based on the
> > > query
> > > > > > > result.
> > > > > > > > >> [1]
> > > > > > > > >> > > > > >
> > > > > > > > >> > > > > > [1]:
> > > > > > > > >> > > > > >
> > > > > > > > >> > > > > >
> > > > > > > > >> > > > >
> > > > > > > > >> > > >
> > > > > > > > >> > >
> > > > > > > > >> >
> > > > > > > > >>
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/flink/blob/191ec6ca3943d7119f14837efe112e074d815c47/flink-table/flink-table-common/src/main/java/org/apache/flink/table/functions/LookupFunction.java#L56
> > > > > > > > >> > > > > >
> > > > > > > > >> > > > > > > If we use async calls with lateral join, my
> gut
> > > > > feeling
> > > > > > is
> > > > > > > > >> > > > > > that we might have many more async calls than
> > lookup
> > > > > > join. I
> > > > > > > > am
> > > > > > > > >> not
> > > > > > > > >> > > > > really
> > > > > > > > >> > > > > > sure if we will be facing potential issues in
> this
> > > > case
> > > > > or
> > > > > > > > not.
> > > > > > > > >> > > > > >
> > > > > > > > >> > > > > > IMO, the work pattern is similar to the lookup
> > > > function,
> > > > > > for
> > > > > > > > >> each
> > > > > > > > >> > row
> > > > > > > > >> > > > > from
> > > > > > > > >> > > > > > the left table,
> > > > > > > > >> > > > > > it will evaluate the eval method once, so the
> > async
> > > > call
> > > > > > > > numbers
> > > > > > > > >> > will
> > > > > > > > >> > > > not
> > > > > > > > >> > > > > > change.
> > > > > > > > >> > > > > > and the maximum calls in flight is limited by
> the
> > > > Async
> > > > > > > > >> operators
> > > > > > > > >> > > > buffer
> > > > > > > > >> > > > > > capacity
> > > > > > > > >> > > > > > which will be controlled by the option.
> > > > > > > > >> > > > > >
> > > > > > > > >> > > > > > BTW, for the naming of these option, I updated
> the
> > > > FLIP
> > > > > > > about
> > > > > > > > >> this
> > > > > > > > >> > > you
> > > > > > > > >> > > > > can
> > > > > > > > >> > > > > > refer to
> > > > > > > > >> > > > > > the section of "ConfigOption" and "Rejected
> > > > > Alternatives"
> > > > > > > > >> > > > > >
> > > > > > > > >> > > > > > In the end, for the performance evaluation, I'd
> > like
> > > > to
> > > > > do
> > > > > > > > some
> > > > > > > > >> > tests
> > > > > > > > >> > > > and
> > > > > > > > >> > > > > > will update it to the FLIP doc
> > > > > > > > >> > > > > >
> > > > > > > > >> > > > > > Thanks,
> > > > > > > > >> > > > > > Aitozi.
> > > > > > > > >> > > > > >
> > > > > > > > >> > > > > >
> > > > > > > > >> > > > > > Jing Ge <ji...@ververica.com.invalid>
> 于2023年6月9日周五
> > > > > > 07:23写道：
> > > > > > > > >> > > > > >
> > > > > > > > >> > > > > > > Hi Aitozi,
> > > > > > > > >> > > > > > >
> > > > > > > > >> > > > > > > Thanks for the clarification. The code example
> > > looks
> > > > > > > > >> > interesting. I
> > > > > > > > >> > > > > would
> > > > > > > > >> > > > > > > suggest adding them into the FLIP. The
> > description
> > > > > with
> > > > > > > code
> > > > > > > > >> > > examples
> > > > > > > > >> > > > > > will
> > > > > > > > >> > > > > > > help readers understand the motivation and how
> > to
> > > > use
> > > > > > it.
> > > > > > > > >> Afaiac,
> > > > > > > > >> > > it
> > > > > > > > >> > > > > is a
> > > > > > > > >> > > > > > > valid feature for Flink users.
> > > > > > > > >> > > > > > >
> > > > > > > > >> > > > > > > As we knew, lookup join is based on temporal
> > join,
> > > > > i.e.
> > > > > > > FOR
> > > > > > > > >> > > > SYSTEM_TIME
> > > > > > > > >> > > > > > AS
> > > > > > > > >> > > > > > > OF which is also used in your code example.
> > > Temporal
> > > > > > join
> > > > > > > > >> > performs
> > > > > > > > >> > > > the
> > > > > > > > >> > > > > > > lookup based on the processing time match.
> Only
> > > one
> > > > > row
> > > > > > > for
> > > > > > > > >> each
> > > > > > > > >> > > > > > > lookup(afaiu, I need to check the source code
> to
> > > > > double
> > > > > > > > >> confirm)
> > > > > > > > >> > > will
> > > > > > > > >> > > > > > > return for further enrichment. One the other
> > hand,
> > > > > > lateral
> > > > > > > > >> join
> > > > > > > > >> > > will
> > > > > > > > >> > > > > have
> > > > > > > > >> > > > > > > sub-queries correlated with every individual
> > value
> > > > of
> > > > > > the
> > > > > > > > >> > reference
> > > > > > > > >> > > > > table
> > > > > > > > >> > > > > > > from the preceding part of the query and each
> > sub
> > > > > query
> > > > > > > will
> > > > > > > > >> > return
> > > > > > > > >> > > > > > > multiple rows. If we use async calls with
> > lateral
> > > > > join,
> > > > > > my
> > > > > > > > gut
> > > > > > > > >> > > > feeling
> > > > > > > > >> > > > > is
> > > > > > > > >> > > > > > > that we might have many more async calls than
> > > lookup
> > > > > > > join. I
> > > > > > > > >> am
> > > > > > > > >> > not
> > > > > > > > >> > > > > > really
> > > > > > > > >> > > > > > > sure if we will be facing potential issues in
> > this
> > > > > case
> > > > > > or
> > > > > > > > >> not.
> > > > > > > > >> > > > > Possible
> > > > > > > > >> > > > > > > issues I can think of now e.g. too many PRC
> > calls,
> > > > too
> > > > > > > many
> > > > > > > > >> async
> > > > > > > > >> > > > calls
> > > > > > > > >> > > > > > > processing, the sub query will return a table
> > > which
> > > > > > might
> > > > > > > be
> > > > > > > > >> > (too)
> > > > > > > > >> > > > big,
> > > > > > > > >> > > > > > and
> > > > > > > > >> > > > > > > might cause performance issues. I would
> suggest
> > > > > > preparing
> > > > > > > > some
> > > > > > > > >> > use
> > > > > > > > >> > > > > cases
> > > > > > > > >> > > > > > > and running some performance tests to check
> it.
> > > > These
> > > > > > are
> > > > > > > my
> > > > > > > > >> > > concerns
> > > > > > > > >> > > > > > about
> > > > > > > > >> > > > > > > using async calls with lateral join and I'd
> like
> > > to
> > > > > > share
> > > > > > > > with
> > > > > > > > >> > you,
> > > > > > > > >> > > > > happy
> > > > > > > > >> > > > > > > to discuss with you and hear different
> opinions,
> > > > > > hopefully
> > > > > > > > the
> > > > > > > > >> > > > > > > discussion could help me understand it more
> > > deeply.
> > > > > > Please
> > > > > > > > >> > correct
> > > > > > > > >> > > me
> > > > > > > > >> > > > > if
> > > > > > > > >> > > > > > I
> > > > > > > > >> > > > > > > am wrong.
> > > > > > > > >> > > > > > >
> > > > > > > > >> > > > > > > Best regards,
> > > > > > > > >> > > > > > > Jing
> > > > > > > > >> > > > > > >
> > > > > > > > >> > > > > > >
> > > > > > > > >> > > > > > > On Thu, Jun 8, 2023 at 7:22 AM Aitozi <
> > > > > > > gjying1314@gmail.com
> > > > > > > > >
> > > > > > > > >> > > wrote:
> > > > > > > > >> > > > > > >
> > > > > > > > >> > > > > > > > Hi Mason,
> > > > > > > > >> > > > > > > >     Thanks for your input. I think if we
> > support
> > > > the
> > > > > > > user
> > > > > > > > >> > defined
> > > > > > > > >> > > > > async
> > > > > > > > >> > > > > > > > table function,
> > > > > > > > >> > > > > > > > user will be able to use it to hold a batch
> > data
> > > > > then
> > > > > > > > >> handle it
> > > > > > > > >> > > at
> > > > > > > > >> > > > > one
> > > > > > > > >> > > > > > > time
> > > > > > > > >> > > > > > > > in the customized function.
> > > > > > > > >> > > > > > > >
> > > > > > > > >> > > > > > > > AsyncSink is meant for the sink operator. I
> > have
> > > > not
> > > > > > > > figure
> > > > > > > > >> out
> > > > > > > > >> > > how
> > > > > > > > >> > > > > to
> > > > > > > > >> > > > > > > > integrate in this case.
> > > > > > > > >> > > > > > > >
> > > > > > > > >> > > > > > > > Thanks,
> > > > > > > > >> > > > > > > > Atiozi.
> > > > > > > > >> > > > > > > >
> > > > > > > > >> > > > > > > >
> > > > > > > > >> > > > > > > > Mason Chen <ma...@gmail.com>
> > > 于2023年6月8日周四
> > > > > > > 12:40写道：
> > > > > > > > >> > > > > > > >
> > > > > > > > >> > > > > > > > > Hi Aitozi,
> > > > > > > > >> > > > > > > > >
> > > > > > > > >> > > > > > > > > I think it makes sense to make it easier
> for
> > > SQL
> > > > > > users
> > > > > > > > to
> > > > > > > > >> > make
> > > > > > > > >> > > > > RPCs.
> > > > > > > > >> > > > > > Do
> > > > > > > > >> > > > > > > > you
> > > > > > > > >> > > > > > > > > think your proposal can extend to the
> > ability
> > > to
> > > > > > batch
> > > > > > > > >> data
> > > > > > > > >> > for
> > > > > > > > >> > > > the
> > > > > > > > >> > > > > > > RPC?
> > > > > > > > >> > > > > > > > > This is also another common strategy to
> > > increase
> > > > > > > > >> throughput.
> > > > > > > > >> > > > Also,
> > > > > > > > >> > > > > > have
> > > > > > > > >> > > > > > > > you
> > > > > > > > >> > > > > > > > > considered solving this a bit differently
> by
> > > > > > > leveraging
> > > > > > > > >> > Flink's
> > > > > > > > >> > > > > > > > AsyncSink?
> > > > > > > > >> > > > > > > > >
> > > > > > > > >> > > > > > > > > Best,
> > > > > > > > >> > > > > > > > > Mason
> > > > > > > > >> > > > > > > > >
> > > > > > > > >> > > > > > > > > On Mon, Jun 5, 2023 at 1:50 AM Aitozi <
> > > > > > > > >> gjying1314@gmail.com>
> > > > > > > > >> > > > > wrote:
> > > > > > > > >> > > > > > > > >
> > > > > > > > >> > > > > > > > > > One more thing for discussion:
> > > > > > > > >> > > > > > > > > >
> > > > > > > > >> > > > > > > > > > In our internal implementation, we reuse
> > the
> > > > > > option
> > > > > > > > >> > > > > > > > > >
> `table.exec.async-lookup.buffer-capacity`
> > > and
> > > > > > > > >> > > > > > > > > > `table.exec.async-lookup.timeout` to
> > config
> > > > > > > > >> > > > > > > > > > the async udtf. Do you think we should
> add
> > > two
> > > > > > extra
> > > > > > > > >> option
> > > > > > > > >> > > to
> > > > > > > > >> > > > > > > > > distinguish
> > > > > > > > >> > > > > > > > > > from the lookup option such as
> > > > > > > > >> > > > > > > > > >
> > > > > > > > >> > > > > > > > > > `table.exec.async-udtf.buffer-capacity`
> > > > > > > > >> > > > > > > > > > `table.exec.async-udtf.timeout`
> > > > > > > > >> > > > > > > > > >
> > > > > > > > >> > > > > > > > > >
> > > > > > > > >> > > > > > > > > > Best,
> > > > > > > > >> > > > > > > > > > Aitozi.
> > > > > > > > >> > > > > > > > > >
> > > > > > > > >> > > > > > > > > >
> > > > > > > > >> > > > > > > > > >
> > > > > > > > >> > > > > > > > > > Aitozi <gj...@gmail.com>
> > 于2023年6月5日周一
> > > > > > 12:20写道：
> > > > > > > > >> > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > Hi Jing,
> > > > > > > > >> > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > >     > what is the difference between
> the
> > > RPC
> > > > > > call
> > > > > > > or
> > > > > > > > >> > query
> > > > > > > > >> > > > you
> > > > > > > > >> > > > > > > > > mentioned
> > > > > > > > >> > > > > > > > > > > and the lookup in a very
> > > > > > > > >> > > > > > > > > > > general way
> > > > > > > > >> > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > I think the RPC call or query service
> is
> > > > quite
> > > > > > > > >> similar to
> > > > > > > > >> > > the
> > > > > > > > >> > > > > > > lookup
> > > > > > > > >> > > > > > > > > > join.
> > > > > > > > >> > > > > > > > > > > But lookup join should work
> > > > > > > > >> > > > > > > > > > > with `LookupTableSource`.
> > > > > > > > >> > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > Let's see how we can perform an async
> > RPC
> > > > call
> > > > > > > with
> > > > > > > > >> > lookup
> > > > > > > > >> > > > > join:
> > > > > > > > >> > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > (1) Implement an AsyncTableFunction
> with
> > > RPC
> > > > > > call
> > > > > > > > >> logic.
> > > > > > > > >> > > > > > > > > > > (2) Implement a `LookupTableSource`
> > > > connector
> > > > > > run
> > > > > > > > with
> > > > > > > > >> > the
> > > > > > > > >> > > > > async
> > > > > > > > >> > > > > > > udtf
> > > > > > > > >> > > > > > > > > > > defined in (1).
> > > > > > > > >> > > > > > > > > > > (3) Then define a DDL of this look up
> > > table
> > > > in
> > > > > > SQL
> > > > > > > > >> > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > CREATE TEMPORARY TABLE Customers (
> > > > > > > > >> > > > > > > > > > >   id INT,
> > > > > > > > >> > > > > > > > > > >   name STRING,
> > > > > > > > >> > > > > > > > > > >   country STRING,
> > > > > > > > >> > > > > > > > > > >   zip STRING
> > > > > > > > >> > > > > > > > > > > ) WITH (
> > > > > > > > >> > > > > > > > > > >   'connector' = 'custom'
> > > > > > > > >> > > > > > > > > > > );
> > > > > > > > >> > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > (4) Run with the query as below:
> > > > > > > > >> > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > SELECT o.order_id, o.total, c.country,
> > > c.zip
> > > > > > > > >> > > > > > > > > > > FROM Orders AS o
> > > > > > > > >> > > > > > > > > > >   JOIN Customers FOR SYSTEM_TIME AS OF
> > > > > > o.proc_time
> > > > > > > > AS
> > > > > > > > >> c
> > > > > > > > >> > > > > > > > > > >     ON o.customer_id = c.id;
> > > > > > > > >> > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > This example is from doc
> > > > > > > > >> > > > > > > > > > > <
> > > > > > > > >> > > > > > > > > >
> > > > > > > > >> > > > > > > > >
> > > > > > > > >> > > > > > > >
> > > > > > > > >> > > > > > >
> > > > > > > > >> > > > > >
> > > > > > > > >> > > > >
> > > > > > > > >> > > >
> > > > > > > > >> > >
> > > > > > > > >> >
> > > > > > > > >>
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/dev/table/sql/queries/joins/#lookup-join
> > > > > > > > >> > > > > > > > > > >.You
> > > > > > > > >> > > > > > > > > > > can image the look up process as an
> > async
> > > > RPC
> > > > > > call
> > > > > > > > >> > process.
> > > > > > > > >> > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > Let's see how we can perform an async
> > RPC
> > > > call
> > > > > > > with
> > > > > > > > >> > lateral
> > > > > > > > >> > > > > join:
> > > > > > > > >> > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > (1) Implement an AsyncTableFunction
> with
> > > RPC
> > > > > > call
> > > > > > > > >> logic.
> > > > > > > > >> > > > > > > > > > > (2) Run query with
> > > > > > > > >> > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > Create function f1 as '...' ;
> > > > > > > > >> > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > SELECT o.order_id, o.total, c.country,
> > > c.zip
> > > > > > FROM
> > > > > > > > >> Orders
> > > > > > > > >> > > > > lateral
> > > > > > > > >> > > > > > > > table
> > > > > > > > >> > > > > > > > > > > (f1(order_id)) as T(...);
> > > > > > > > >> > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > As you can see, the lateral join
> version
> > > is
> > > > > more
> > > > > > > > >> simple
> > > > > > > > >> > and
> > > > > > > > >> > > > > > > intuitive
> > > > > > > > >> > > > > > > > > to
> > > > > > > > >> > > > > > > > > > > users. Users do not have to wrap a
> > > > > > > > >> > > > > > > > > > > LookupTableSource for the purpose of
> > using
> > > > > async
> > > > > > > > udtf.
> > > > > > > > >> > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > In the end, We can also see the user
> > > defined
> > > > > > async
> > > > > > > > >> table
> > > > > > > > >> > > > > function
> > > > > > > > >> > > > > > > is
> > > > > > > > >> > > > > > > > an
> > > > > > > > >> > > > > > > > > > > enhancement of the current lateral
> table
> > > > join
> > > > > > > > >> > > > > > > > > > > which only supports sync lateral join
> > now.
> > > > > > > > >> > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > Best,
> > > > > > > > >> > > > > > > > > > > Aitozi.
> > > > > > > > >> > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > Jing Ge <ji...@ververica.com.invalid>
> > > > > > 于2023年6月2日周五
> > > > > > > > >> > 19:37写道：
> > > > > > > > >> > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > >> Hi Aitozi,
> > > > > > > > >> > > > > > > > > > >>
> > > > > > > > >> > > > > > > > > > >> Thanks for the update. Just out of
> > > > curiosity,
> > > > > > > what
> > > > > > > > is
> > > > > > > > >> > the
> > > > > > > > >> > > > > > > difference
> > > > > > > > >> > > > > > > > > > >> between the RPC call or query you
> > > mentioned
> > > > > and
> > > > > > > the
> > > > > > > > >> > lookup
> > > > > > > > >> > > > in
> > > > > > > > >> > > > > a
> > > > > > > > >> > > > > > > very
> > > > > > > > >> > > > > > > > > > >> general way? Since Lateral join is
> used
> > > in
> > > > > the
> > > > > > > > FLIP.
> > > > > > > > >> Is
> > > > > > > > >> > > > there
> > > > > > > > >> > > > > > any
> > > > > > > > >> > > > > > > > > > special
> > > > > > > > >> > > > > > > > > > >> thought for that? Sorry for asking so
> > > many
> > > > > > > > questions.
> > > > > > > > >> > The
> > > > > > > > >> > > > FLIP
> > > > > > > > >> > > > > > > > > contains
> > > > > > > > >> > > > > > > > > > >> limited information to understand the
> > > > > > motivation.
> > > > > > > > >> > > > > > > > > > >>
> > > > > > > > >> > > > > > > > > > >> Best regards,
> > > > > > > > >> > > > > > > > > > >> Jing
> > > > > > > > >> > > > > > > > > > >>
> > > > > > > > >> > > > > > > > > > >> On Fri, Jun 2, 2023 at 3:48 AM
> Aitozi <
> > > > > > > > >> > > gjying1314@gmail.com
> > > > > > > > >> > > > >
> > > > > > > > >> > > > > > > wrote:
> > > > > > > > >> > > > > > > > > > >>
> > > > > > > > >> > > > > > > > > > >> > Hi Jing,
> > > > > > > > >> > > > > > > > > > >> >     I have updated the proposed
> > changes
> > > > to
> > > > > > the
> > > > > > > > >> FLIP.
> > > > > > > > >> > > IMO,
> > > > > > > > >> > > > > > lookup
> > > > > > > > >> > > > > > > > has
> > > > > > > > >> > > > > > > > > > its
> > > > > > > > >> > > > > > > > > > >> > clear
> > > > > > > > >> > > > > > > > > > >> > async call requirement is due to
> its
> > IO
> > > > > heavy
> > > > > > > > >> > operator.
> > > > > > > > >> > > In
> > > > > > > > >> > > > > our
> > > > > > > > >> > > > > > > > > usage,
> > > > > > > > >> > > > > > > > > > >> sql
> > > > > > > > >> > > > > > > > > > >> > users have
> > > > > > > > >> > > > > > > > > > >> > logic to do some RPC call or query
> > the
> > > > > > > > third-party
> > > > > > > > >> > > service
> > > > > > > > >> > > > > > which
> > > > > > > > >> > > > > > > > is
> > > > > > > > >> > > > > > > > > > >> also IO
> > > > > > > > >> > > > > > > > > > >> > intensive.
> > > > > > > > >> > > > > > > > > > >> > In these case, we'd like to
> leverage
> > > the
> > > > > > async
> > > > > > > > >> > function
> > > > > > > > >> > > to
> > > > > > > > >> > > > > > > improve
> > > > > > > > >> > > > > > > > > the
> > > > > > > > >> > > > > > > > > > >> > throughput.
> > > > > > > > >> > > > > > > > > > >> >
> > > > > > > > >> > > > > > > > > > >> > Thanks,
> > > > > > > > >> > > > > > > > > > >> > Aitozi.
> > > > > > > > >> > > > > > > > > > >> >
> > > > > > > > >> > > > > > > > > > >> > Jing Ge <jing@ververica.com.invalid
> >
> > > > > > > > 于2023年6月1日周四
> > > > > > > > >> > > > 22:55写道：
> > > > > > > > >> > > > > > > > > > >> >
> > > > > > > > >> > > > > > > > > > >> > > Hi Aitozi,
> > > > > > > > >> > > > > > > > > > >> > >
> > > > > > > > >> > > > > > > > > > >> > > Sorry for the late reply. Would
> you
> > > > like
> > > > > to
> > > > > > > > >> update
> > > > > > > > >> > the
> > > > > > > > >> > > > > > > proposed
> > > > > > > > >> > > > > > > > > > >> changes
> > > > > > > > >> > > > > > > > > > >> > > with more details into the FLIP
> > too?
> > > > > > > > >> > > > > > > > > > >> > > I got your point. It looks like a
> > > > > rational
> > > > > > > > idea.
> > > > > > > > >> > > > However,
> > > > > > > > >> > > > > > > since
> > > > > > > > >> > > > > > > > > > lookup
> > > > > > > > >> > > > > > > > > > >> > has
> > > > > > > > >> > > > > > > > > > >> > > its clear async call requirement,
> > are
> > > > > there
> > > > > > > any
> > > > > > > > >> real
> > > > > > > > >> > > use
> > > > > > > > >> > > > > > cases
> > > > > > > > >> > > > > > > > > that
> > > > > > > > >> > > > > > > > > > >> > > need this change? This will help
> us
> > > > > > > understand
> > > > > > > > >> the
> > > > > > > > >> > > > > > motivation.
> > > > > > > > >> > > > > > > > > After
> > > > > > > > >> > > > > > > > > > >> all,
> > > > > > > > >> > > > > > > > > > >> > > lateral join and temporal lookup
> > > > join[1]
> > > > > > are
> > > > > > > > >> quite
> > > > > > > > >> > > > > > different.
> > > > > > > > >> > > > > > > > > > >> > >
> > > > > > > > >> > > > > > > > > > >> > > Best regards,
> > > > > > > > >> > > > > > > > > > >> > > Jing
> > > > > > > > >> > > > > > > > > > >> > >
> > > > > > > > >> > > > > > > > > > >> > >
> > > > > > > > >> > > > > > > > > > >> > > [1]
> > > > > > > > >> > > > > > > > > > >> > >
> > > > > > > > >> > > > > > > > > > >> > >
> > > > > > > > >> > > > > > > > > > >> >
> > > > > > > > >> > > > > > > > > > >>
> > > > > > > > >> > > > > > > > > >
> > > > > > > > >> > > > > > > > >
> > > > > > > > >> > > > > > > >
> > > > > > > > >> > > > > > >
> > > > > > > > >> > > > > >
> > > > > > > > >> > > > >
> > > > > > > > >> > > >
> > > > > > > > >> > >
> > > > > > > > >> >
> > > > > > > > >>
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/flink/blob/d90a72da2fd601ca4e2a46700e91ec5b348de2ad/flink-table/flink-table-common/src/main/java/org/apache/flink/table/functions/AsyncTableFunction.java#L54
> > > > > > > > >> > > > > > > > > > >> > >
> > > > > > > > >> > > > > > > > > > >> > > On Wed, May 31, 2023 at 8:53 AM
> > > Aitozi
> > > > <
> > > > > > > > >> > > > > > gjying1314@gmail.com>
> > > > > > > > >> > > > > > > > > > wrote:
> > > > > > > > >> > > > > > > > > > >> > >
> > > > > > > > >> > > > > > > > > > >> > > > Hi Jing,
> > > > > > > > >> > > > > > > > > > >> > > >     What do you think about it?
> > Can
> > > > we
> > > > > > move
> > > > > > > > >> > forward
> > > > > > > > >> > > > this
> > > > > > > > >> > > > > > > > > feature?
> > > > > > > > >> > > > > > > > > > >> > > >
> > > > > > > > >> > > > > > > > > > >> > > > Thanks,
> > > > > > > > >> > > > > > > > > > >> > > > Aitozi.
> > > > > > > > >> > > > > > > > > > >> > > >
> > > > > > > > >> > > > > > > > > > >> > > > Aitozi <gj...@gmail.com>
> > > > > > > 于2023年5月29日周一
> > > > > > > > >> > > 09:56写道：
> > > > > > > > >> > > > > > > > > > >> > > >
> > > > > > > > >> > > > > > > > > > >> > > > > Hi Jing,
> > > > > > > > >> > > > > > > > > > >> > > > >     > "Do you mean to support
> > the
> > > > > > > > >> > > AyncTableFunction
> > > > > > > > >> > > > > > beyond
> > > > > > > > >> > > > > > > > the
> > > > > > > > >> > > > > > > > > > >> > > > > LookupTableSource?"
> > > > > > > > >> > > > > > > > > > >> > > > > Yes, I mean to support the
> > > > > > > > AyncTableFunction
> > > > > > > > >> > > beyond
> > > > > > > > >> > > > > the
> > > > > > > > >> > > > > > > > > > >> > > > LookupTableSource.
> > > > > > > > >> > > > > > > > > > >> > > > >
> > > > > > > > >> > > > > > > > > > >> > > > > The "AsyncTableFunction" is
> the
> > > > > > function
> > > > > > > > with
> > > > > > > > >> > > > ability
> > > > > > > > >> > > > > to
> > > > > > > > >> > > > > > > be
> > > > > > > > >> > > > > > > > > > >> executed
> > > > > > > > >> > > > > > > > > > >> > > > async
> > > > > > > > >> > > > > > > > > > >> > > > > (with AsyncWaitOperator).
> > > > > > > > >> > > > > > > > > > >> > > > > The async lookup join is a
> one
> > of
> > > > > usage
> > > > > > > of
> > > > > > > > >> this.
> > > > > > > > >> > > So,
> > > > > > > > >> > > > > we
> > > > > > > > >> > > > > > > > don't
> > > > > > > > >> > > > > > > > > > >> have to
> > > > > > > > >> > > > > > > > > > >> > > > bind
> > > > > > > > >> > > > > > > > > > >> > > > > the AyncTableFunction with
> > > > > > > > LookupTableSource.
> > > > > > > > >> > > > > > > > > > >> > > > > If User-defined
> > > AsyncTableFunction
> > > > is
> > > > > > > > >> supported,
> > > > > > > > >> > > > user
> > > > > > > > >> > > > > > can
> > > > > > > > >> > > > > > > > > > directly
> > > > > > > > >> > > > > > > > > > >> > use
> > > > > > > > >> > > > > > > > > > >> > > > > lateral table syntax to
> perform
> > > > async
> > > > > > > > >> operation.
> > > > > > > > >> > > > > > > > > > >> > > > >
> > > > > > > > >> > > > > > > > > > >> > > > > > "It would be better if you
> > > could
> > > > > > > > elaborate
> > > > > > > > >> the
> > > > > > > > >> > > > > > proposed
> > > > > > > > >> > > > > > > > > > changes
> > > > > > > > >> > > > > > > > > > >> wrt
> > > > > > > > >> > > > > > > > > > >> > > the
> > > > > > > > >> > > > > > > > > > >> > > > > CorrelatedCodeGenerator with
> > more
> > > > > > > details"
> > > > > > > > >> > > > > > > > > > >> > > > >
> > > > > > > > >> > > > > > > > > > >> > > > > In the proposal, we use
> lateral
> > > > table
> > > > > > > > syntax
> > > > > > > > >> to
> > > > > > > > >> > > > > support
> > > > > > > > >> > > > > > > the
> > > > > > > > >> > > > > > > > > > async
> > > > > > > > >> > > > > > > > > > >> > table
> > > > > > > > >> > > > > > > > > > >> > > > > function. So the planner will
> > > also
> > > > > > treat
> > > > > > > > this
> > > > > > > > >> > > > > statement
> > > > > > > > >> > > > > > > to a
> > > > > > > > >> > > > > > > > > > >> > > > > CommonExecCorrelate node. So
> > the
> > > > > > runtime
> > > > > > > > code
> > > > > > > > >> > > should
> > > > > > > > >> > > > > be
> > > > > > > > >> > > > > > > > > > generated
> > > > > > > > >> > > > > > > > > > >> in
> > > > > > > > >> > > > > > > > > > >> > > > > CorrelatedCodeGenerator.
> > > > > > > > >> > > > > > > > > > >> > > > > In CorrelatedCodeGenerator,
> we
> > > will
> > > > > > know
> > > > > > > > the
> > > > > > > > >> > > > > > > TableFunction's
> > > > > > > > >> > > > > > > > > > Kind
> > > > > > > > >> > > > > > > > > > >> of
> > > > > > > > >> > > > > > > > > > >> > > > > `FunctionKind.Table` or
> > > > > > > > >> > `FunctionKind.ASYNC_TABLE`
> > > > > > > > >> > > > > > > > > > >> > > > > For
> `FunctionKind.ASYNC_TABLE`
> > > we
> > > > > can
> > > > > > > > >> generate
> > > > > > > > >> > a
> > > > > > > > >> > > > > > > > > > >> AsyncWaitOperator
> > > > > > > > >> > > > > > > > > > >> > to
> > > > > > > > >> > > > > > > > > > >> > > > > execute the async table
> > function.
> > > > > > > > >> > > > > > > > > > >> > > > >
> > > > > > > > >> > > > > > > > > > >> > > > >
> > > > > > > > >> > > > > > > > > > >> > > > > Thanks,
> > > > > > > > >> > > > > > > > > > >> > > > > Aitozi.
> > > > > > > > >> > > > > > > > > > >> > > > >
> > > > > > > > >> > > > > > > > > > >> > > > >
> > > > > > > > >> > > > > > > > > > >> > > > > Jing Ge
> > > <jing@ververica.com.invalid
> > > > >
> > > > > > > > >> > > 于2023年5月29日周一
> > > > > > > > >> > > > > > > 03:22写道：
> > > > > > > > >> > > > > > > > > > >> > > > >
> > > > > > > > >> > > > > > > > > > >> > > > >> Hi Aitozi,
> > > > > > > > >> > > > > > > > > > >> > > > >>
> > > > > > > > >> > > > > > > > > > >> > > > >> Thanks for the
> clarification.
> > > The
> > > > > > naming
> > > > > > > > >> > "Lookup"
> > > > > > > > >> > > > > might
> > > > > > > > >> > > > > > > > > suggest
> > > > > > > > >> > > > > > > > > > >> > using
> > > > > > > > >> > > > > > > > > > >> > > it
> > > > > > > > >> > > > > > > > > > >> > > > >> for table look up. But
> > > > conceptually
> > > > > > what
> > > > > > > > the
> > > > > > > > >> > > eval()
> > > > > > > > >> > > > > > > method
> > > > > > > > >> > > > > > > > > will
> > > > > > > > >> > > > > > > > > > >> do
> > > > > > > > >> > > > > > > > > > >> > is
> > > > > > > > >> > > > > > > > > > >> > > to
> > > > > > > > >> > > > > > > > > > >> > > > >> get a collection of
> > results(Row,
> > > > > > > RowData)
> > > > > > > > >> from
> > > > > > > > >> > > the
> > > > > > > > >> > > > > > given
> > > > > > > > >> > > > > > > > > keys.
> > > > > > > > >> > > > > > > > > > >> How
> > > > > > > > >> > > > > > > > > > >> > it
> > > > > > > > >> > > > > > > > > > >> > > > will
> > > > > > > > >> > > > > > > > > > >> > > > >> be done depends on the
> > > > > implementation,
> > > > > > > > i.e.
> > > > > > > > >> you
> > > > > > > > >> > > can
> > > > > > > > >> > > > > > > > implement
> > > > > > > > >> > > > > > > > > > >> your
> > > > > > > > >> > > > > > > > > > >> > own
> > > > > > > > >> > > > > > > > > > >> > > > >> Source[1][2]. The example in
> > the
> > > > > FLIP
> > > > > > > > >> should be
> > > > > > > > >> > > > able
> > > > > > > > >> > > > > to
> > > > > > > > >> > > > > > > be
> > > > > > > > >> > > > > > > > > > >> handled
> > > > > > > > >> > > > > > > > > > >> > in
> > > > > > > > >> > > > > > > > > > >> > > > this
> > > > > > > > >> > > > > > > > > > >> > > > >> way.
> > > > > > > > >> > > > > > > > > > >> > > > >>
> > > > > > > > >> > > > > > > > > > >> > > > >> Do you mean to support the
> > > > > > > > AyncTableFunction
> > > > > > > > >> > > beyond
> > > > > > > > >> > > > > the
> > > > > > > > >> > > > > > > > > > >> > > > LookupTableSource?
> > > > > > > > >> > > > > > > > > > >> > > > >> It would be better if you
> > could
> > > > > > > elaborate
> > > > > > > > >> the
> > > > > > > > >> > > > > proposed
> > > > > > > > >> > > > > > > > > changes
> > > > > > > > >> > > > > > > > > > >> wrt
> > > > > > > > >> > > > > > > > > > >> > the
> > > > > > > > >> > > > > > > > > > >> > > > >> CorrelatedCodeGenerator with
> > > more
> > > > > > > details.
> > > > > > > > >> > > Thanks!
> > > > > > > > >> > > > > > > > > > >> > > > >>
> > > > > > > > >> > > > > > > > > > >> > > > >> Best regards,
> > > > > > > > >> > > > > > > > > > >> > > > >> Jing
> > > > > > > > >> > > > > > > > > > >> > > > >>
> > > > > > > > >> > > > > > > > > > >> > > > >> [1]
> > > > > > > > >> > > > > > > > > > >> > > > >>
> > > > > > > > >> > > > > > > > > > >> > > > >>
> > > > > > > > >> > > > > > > > > > >> > > >
> > > > > > > > >> > > > > > > > > > >> > >
> > > > > > > > >> > > > > > > > > > >> >
> > > > > > > > >> > > > > > > > > > >>
> > > > > > > > >> > > > > > > > > >
> > > > > > > > >> > > > > > > > >
> > > > > > > > >> > > > > > > >
> > > > > > > > >> > > > > > >
> > > > > > > > >> > > > > >
> > > > > > > > >> > > > >
> > > > > > > > >> > > >
> > > > > > > > >> > >
> > > > > > > > >> >
> > > > > > > > >>
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/flink/blob/678370b18e1b6c4a23e5ce08f8efd05675a0cc17/flink-table/flink-table-common/src/main/java/org/apache/flink/table/connector/source/LookupTableSource.java#L64
> > > > > > > > >> > > > > > > > > > >> > > > >> [2]
> > > > > > > > >> > > > > > > > > > >> > > > >>
> > > > > > > > >> > > > > > > > > > >> > > > >>
> > > > > > > > >> > > > > > > > > > >> > > >
> > > > > > > > >> > > > > > > > > > >> > >
> > > > > > > > >> > > > > > > > > > >> >
> > > > > > > > >> > > > > > > > > > >>
> > > > > > > > >> > > > > > > > > >
> > > > > > > > >> > > > > > > > >
> > > > > > > > >> > > > > > > >
> > > > > > > > >> > > > > > >
> > > > > > > > >> > > > > >
> > > > > > > > >> > > > >
> > > > > > > > >> > > >
> > > > > > > > >> > >
> > > > > > > > >> >
> > > > > > > > >>
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/flink/blob/678370b18e1b6c4a23e5ce08f8efd05675a0cc17/flink-table/flink-table-common/src/main/java/org/apache/flink/table/connector/source/AsyncTableFunctionProvider.java#L49
> > > > > > > > >> > > > > > > > > > >> > > > >>
> > > > > > > > >> > > > > > > > > > >> > > > >> On Sat, May 27, 2023 at
> > 9:48 AM
> > > > > > Aitozi <
> > > > > > > > >> > > > > > > > gjying1314@gmail.com
> > > > > > > > >> > > > > > > > > >
> > > > > > > > >> > > > > > > > > > >> > wrote:
> > > > > > > > >> > > > > > > > > > >> > > > >>
> > > > > > > > >> > > > > > > > > > >> > > > >> > Hi Jing,
> > > > > > > > >> > > > > > > > > > >> > > > >> >     Thanks for your
> > response.
> > > As
> > > > > > > stated
> > > > > > > > in
> > > > > > > > >> > the
> > > > > > > > >> > > > > FLIP,
> > > > > > > > >> > > > > > > the
> > > > > > > > >> > > > > > > > > > >> purpose
> > > > > > > > >> > > > > > > > > > >> > of
> > > > > > > > >> > > > > > > > > > >> > > > this
> > > > > > > > >> > > > > > > > > > >> > > > >> > FLIP is meant to support
> > > > > > > > >> > > > > > > > > > >> > > > >> > user-defined async table
> > > > function.
> > > > > > As
> > > > > > > > >> > described
> > > > > > > > >> > > > in
> > > > > > > > >> > > > > > > flink
> > > > > > > > >> > > > > > > > > > >> document
> > > > > > > > >> > > > > > > > > > >> > > [1]
> > > > > > > > >> > > > > > > > > > >> > > > >> >
> > > > > > > > >> > > > > > > > > > >> > > > >> > Async table functions are
> > > > special
> > > > > > > > >> functions
> > > > > > > > >> > for
> > > > > > > > >> > > > > table
> > > > > > > > >> > > > > > > > > sources
> > > > > > > > >> > > > > > > > > > >> that
> > > > > > > > >> > > > > > > > > > >> > > > >> perform
> > > > > > > > >> > > > > > > > > > >> > > > >> > > a lookup.
> > > > > > > > >> > > > > > > > > > >> > > > >> > >
> > > > > > > > >> > > > > > > > > > >> > > > >> >
> > > > > > > > >> > > > > > > > > > >> > > > >> > So end user can not
> directly
> > > > > define
> > > > > > > and
> > > > > > > > >> use
> > > > > > > > >> > > async
> > > > > > > > >> > > > > > table
> > > > > > > > >> > > > > > > > > > >> function
> > > > > > > > >> > > > > > > > > > >> > > now.
> > > > > > > > >> > > > > > > > > > >> > > > An
> > > > > > > > >> > > > > > > > > > >> > > > >> > user case is reported in
> [2]
> > > > > > > > >> > > > > > > > > > >> > > > >> >
> > > > > > > > >> > > > > > > > > > >> > > > >> > So, in conclusion, no new
> > > > > interface
> > > > > > is
> > > > > > > > >> > > > introduced,
> > > > > > > > >> > > > > > but
> > > > > > > > >> > > > > > > we
> > > > > > > > >> > > > > > > > > > >> extend
> > > > > > > > >> > > > > > > > > > >> > the
> > > > > > > > >> > > > > > > > > > >> > > > >> > ability to support
> > > user-defined
> > > > > > async
> > > > > > > > >> table
> > > > > > > > >> > > > > function.
> > > > > > > > >> > > > > > > > > > >> > > > >> >
> > > > > > > > >> > > > > > > > > > >> > > > >> > [1]:
> > > > > > > > >> > > > > > > > > > >> > > > >> >
> > > > > > > > >> > > > > > > > > > >> > > > >> >
> > > > > > > > >> > > > > > > > > > >> > > > >>
> > > > > > > > >> > > > > > > > > > >> > > >
> > > > > > > > >> > > > > > > > > > >> > >
> > > > > > > > >> > > > > > > > > > >> >
> > > > > > > > >> > > > > > > > > > >>
> > > > > > > > >> > > > > > > > > >
> > > > > > > > >> > > > > > > > >
> > > > > > > > >> > > > > > > >
> > > > > > > > >> > > > > > >
> > > > > > > > >> > > > > >
> > > > > > > > >> > > > >
> > > > > > > > >> > > >
> > > > > > > > >> > >
> > > > > > > > >> >
> > > > > > > > >>
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/dev/table/functions/udfs/
> > > > > > > > >> > > > > > > > > > >> > > > >> > [2]:
> > > > > > > > >> > > > > > > > > > >> > >
> > > > > > > > >> > > > > > > >
> > > > > > > > >> >
> > > > > https://lists.apache.org/thread/qljwd40v5ntz6733cwcdr8s4z97b343b
> > > > > > > > >> > > > > > > > > > >> > > > >> >
> > > > > > > > >> > > > > > > > > > >> > > > >> > Thanks.
> > > > > > > > >> > > > > > > > > > >> > > > >> > Aitozi.
> > > > > > > > >> > > > > > > > > > >> > > > >> >
> > > > > > > > >> > > > > > > > > > >> > > > >> >
> > > > > > > > >> > > > > > > > > > >> > > > >> > Jing Ge
> > > > > <jing@ververica.com.invalid
> > > > > > >
> > > > > > > > >> > > > 于2023年5月27日周六
> > > > > > > > >> > > > > > > > > 06:40写道：
> > > > > > > > >> > > > > > > > > > >> > > > >> >
> > > > > > > > >> > > > > > > > > > >> > > > >> > > Hi Aitozi,
> > > > > > > > >> > > > > > > > > > >> > > > >> > >
> > > > > > > > >> > > > > > > > > > >> > > > >> > > Thanks for your
> proposal.
> > I
> > > am
> > > > > not
> > > > > > > > quite
> > > > > > > > >> > sure
> > > > > > > > >> > > > if
> > > > > > > > >> > > > > I
> > > > > > > > >> > > > > > > > > > understood
> > > > > > > > >> > > > > > > > > > >> > your
> > > > > > > > >> > > > > > > > > > >> > > > >> > thoughts
> > > > > > > > >> > > > > > > > > > >> > > > >> > > correctly. You
> described a
> > > > > special
> > > > > > > > case
> > > > > > > > >> > > > > > > implementation
> > > > > > > > >> > > > > > > > of
> > > > > > > > >> > > > > > > > > > the
> > > > > > > > >> > > > > > > > > > >> > > > >> > > AsyncTableFunction with
> on
> > > > > public
> > > > > > > API
> > > > > > > > >> > > changes.
> > > > > > > > >> > > > > > Would
> > > > > > > > >> > > > > > > > you
> > > > > > > > >> > > > > > > > > > >> please
> > > > > > > > >> > > > > > > > > > >> > > > >> elaborate
> > > > > > > > >> > > > > > > > > > >> > > > >> > > your purpose of writing
> a
> > > FLIP
> > > > > > > > >> according to
> > > > > > > > >> > > the
> > > > > > > > >> > > > > > FLIP
> > > > > > > > >> > > > > > > > > > >> > > > documentation[1]?
> > > > > > > > >> > > > > > > > > > >> > > > >> > > Thanks!
> > > > > > > > >> > > > > > > > > > >> > > > >> > >
> > > > > > > > >> > > > > > > > > > >> > > > >> > > [1]
> > > > > > > > >> > > > > > > > > > >> > > > >> > >
> > > > > > > > >> > > > > > > > > > >> > > > >> > >
> > > > > > > > >> > > > > > > > > > >> > > > >> >
> > > > > > > > >> > > > > > > > > > >> > > > >>
> > > > > > > > >> > > > > > > > > > >> > > >
> > > > > > > > >> > > > > > > > > > >> > >
> > > > > > > > >> > > > > > > > > > >> >
> > > > > > > > >> > > > > > > > > > >>
> > > > > > > > >> > > > > > > > > >
> > > > > > > > >> > > > > > > > >
> > > > > > > > >> > > > > > > >
> > > > > > > > >> > > > > > >
> > > > > > > > >> > > > > >
> > > > > > > > >> > > > >
> > > > > > > > >> > > >
> > > > > > > > >> > >
> > > > > > > > >> >
> > > > > > > > >>
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/Flink+Improvement+Proposals
> > > > > > > > >> > > > > > > > > > >> > > > >> > >
> > > > > > > > >> > > > > > > > > > >> > > > >> > > Best regards,
> > > > > > > > >> > > > > > > > > > >> > > > >> > > Jing
> > > > > > > > >> > > > > > > > > > >> > > > >> > >
> > > > > > > > >> > > > > > > > > > >> > > > >> > > On Wed, May 24, 2023 at
> > > > 1:07 PM
> > > > > > > > Aitozi <
> > > > > > > > >> > > > > > > > > > gjying1314@gmail.com
> > > > > > > > >> > > > > > > > > > >> >
> > > > > > > > >> > > > > > > > > > >> > > > wrote:
> > > > > > > > >> > > > > > > > > > >> > > > >> > >
> > > > > > > > >> > > > > > > > > > >> > > > >> > > > May I ask for some
> > > feedback
> > > > > :D
> > > > > > > > >> > > > > > > > > > >> > > > >> > > >
> > > > > > > > >> > > > > > > > > > >> > > > >> > > > Thanks,
> > > > > > > > >> > > > > > > > > > >> > > > >> > > > Aitozi
> > > > > > > > >> > > > > > > > > > >> > > > >> > > >
> > > > > > > > >> > > > > > > > > > >> > > > >> > > > Aitozi <
> > > > gjying1314@gmail.com>
> > > > > > > > >> > > 于2023年5月23日周二
> > > > > > > > >> > > > > > > 19:14写道：
> > > > > > > > >> > > > > > > > > > >> > > > >> > > > >
> > > > > > > > >> > > > > > > > > > >> > > > >> > > > > Just catch an user
> > case
> > > > > report
> > > > > > > > from
> > > > > > > > >> > > Giannis
> > > > > > > > >> > > > > > > Polyzos
> > > > > > > > >> > > > > > > > > for
> > > > > > > > >> > > > > > > > > > >> this
> > > > > > > > >> > > > > > > > > > >> > > > >> usage:
> > > > > > > > >> > > > > > > > > > >> > > > >> > > > >
> > > > > > > > >> > > > > > > > > > >> > > > >> > > > >
> > > > > > > > >> > > > > > > > > > >> > > >
> > > > > > > > >> > > > > > > > >
> > > > > > > > >> > >
> > > > > > https://lists.apache.org/thread/qljwd40v5ntz6733cwcdr8s4z97b343b
> > > > > > > > >> > > > > > > > > > >> > > > >> > > > >
> > > > > > > > >> > > > > > > > > > >> > > > >> > > > > Aitozi <
> > > > > gjying1314@gmail.com>
> > > > > > > > >> > > > 于2023年5月23日周二
> > > > > > > > >> > > > > > > > 17:45写道：
> > > > > > > > >> > > > > > > > > > >> > > > >> > > > > >
> > > > > > > > >> > > > > > > > > > >> > > > >> > > > > > Hi guys,
> > > > > > > > >> > > > > > > > > > >> > > > >> > > > > >     I want to
> bring
> > > up a
> > > > > > > > >> discussion
> > > > > > > > >> > > about
> > > > > > > > >> > > > > > > adding
> > > > > > > > >> > > > > > > > > > >> support
> > > > > > > > >> > > > > > > > > > >> > of
> > > > > > > > >> > > > > > > > > > >> > > > User
> > > > > > > > >> > > > > > > > > > >> > > > >> > > > > > Defined
> > > > AsyncTableFunction
> > > > > > in
> > > > > > > > >> Flink.
> > > > > > > > >> > > > > > > > > > >> > > > >> > > > > > Currently, async
> > table
> > > > > > > function
> > > > > > > > >> are
> > > > > > > > >> > > > special
> > > > > > > > >> > > > > > > > > functions
> > > > > > > > >> > > > > > > > > > >> for
> > > > > > > > >> > > > > > > > > > >> > > > table
> > > > > > > > >> > > > > > > > > > >> > > > >> > > source
> > > > > > > > >> > > > > > > > > > >> > > > >> > > > > > to perform
> > > > > > > > >> > > > > > > > > > >> > > > >> > > > > > async lookup.
> > However,
> > > > > it's
> > > > > > > > worth
> > > > > > > > >> to
> > > > > > > > >> > > > > support
> > > > > > > > >> > > > > > > the
> > > > > > > > >> > > > > > > > > user
> > > > > > > > >> > > > > > > > > > >> > > defined
> > > > > > > > >> > > > > > > > > > >> > > > >> async
> > > > > > > > >> > > > > > > > > > >> > > > >> > > > > > table function.
> > > > > > > > >> > > > > > > > > > >> > > > >> > > > > > Because, in this
> > way,
> > > > the
> > > > > > end
> > > > > > > > SQL
> > > > > > > > >> > user
> > > > > > > > >> > > > can
> > > > > > > > >> > > > > > > > leverage
> > > > > > > > >> > > > > > > > > > it
> > > > > > > > >> > > > > > > > > > >> to
> > > > > > > > >> > > > > > > > > > >> > > > >> perform
> > > > > > > > >> > > > > > > > > > >> > > > >> > the
> > > > > > > > >> > > > > > > > > > >> > > > >> > > > > > async operation
> > > > > > > > >> > > > > > > > > > >> > > > >> > > > > > which is useful to
> > > > maximum
> > > > > > the
> > > > > > > > >> system
> > > > > > > > >> > > > > > > throughput
> > > > > > > > >> > > > > > > > > > >> > especially
> > > > > > > > >> > > > > > > > > > >> > > > for
> > > > > > > > >> > > > > > > > > > >> > > > >> IO
> > > > > > > > >> > > > > > > > > > >> > > > >> > > > > > bottleneck case.
> > > > > > > > >> > > > > > > > > > >> > > > >> > > > > >
> > > > > > > > >> > > > > > > > > > >> > > > >> > > > > > You can find some
> > more
> > > > > > detail
> > > > > > > in
> > > > > > > > >> [1].
> > > > > > > > >> > > > > > > > > > >> > > > >> > > > > >
> > > > > > > > >> > > > > > > > > > >> > > > >> > > > > > Looking forward to
> > > > > feedback
> > > > > > > > >> > > > > > > > > > >> > > > >> > > > > >
> > > > > > > > >> > > > > > > > > > >> > > > >> > > > > >
> > > > > > > > >> > > > > > > > > > >> > > > >> > > > > > [1]:
> > > > > > > > >> > > > > > > > > > >> > > > >> > > >
> > > > > > > > >> > > > > > > > > > >> > > > >> > >
> > > > > > > > >> > > > > > > > > > >> > > > >> >
> > > > > > > > >> > > > > > > > > > >> > > > >>
> > > > > > > > >> > > > > > > > > > >> > > >
> > > > > > > > >> > > > > > > > > > >> > >
> > > > > > > > >> > > > > > > > > > >> >
> > > > > > > > >> > > > > > > > > > >>
> > > > > > > > >> > > > > > > > > >
> > > > > > > > >> > > > > > > > >
> > > > > > > > >> > > > > > > >
> > > > > > > > >> > > > > > >
> > > > > > > > >> > > > > >
> > > > > > > > >> > > > >
> > > > > > > > >> > > >
> > > > > > > > >> > >
> > > > > > > > >> >
> > > > > > > > >>
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/%5BFLIP-313%5D+Add+support+of+User+Defined+AsyncTableFunction
> > > > > > > > >> > > > > > > > > > >> > > > >> > > > > >
> > > > > > > > >> > > > > > > > > > >> > > > >> > > > > > Thanks,
> > > > > > > > >> > > > > > > > > > >> > > > >> > > > > > Aitozi.
> > > > > > > > >> > > > > > > > > > >> > > > >> > > >
> > > > > > > > >> > > > > > > > > > >> > > > >> > >
> > > > > > > > >> > > > > > > > > > >> > > > >> >
> > > > > > > > >> > > > > > > > > > >> > > > >>
> > > > > > > > >> > > > > > > > > > >> > > > >
> > > > > > > > >> > > > > > > > > > >> > > >
> > > > > > > > >> > > > > > > > > > >> > >
> > > > > > > > >> > > > > > > > > > >> >
> > > > > > > > >> > > > > > > > > > >>
> > > > > > > > >> > > > > > > > > > >
> > > > > > > > >> > > > > > > > > >
> > > > > > > > >> > > > > > > > >
> > > > > > > > >> > > > > > > >
> > > > > > > > >> > > > > > >
> > > > > > > > >> > > > > >
> > > > > > > > >> > > > >
> > > > > > > > >> > > >
> > > > > > > > >> > >
> > > > > > > > >> >
> > > > > > > > >>
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] FLIP-313 Add support of User Defined AsyncTableFunction

Posted by Aitozi <gj...@gmail.com>.

Hi Lincoln

    Very thanks for your valuable question. I will try to answer your
questions inline.

>Does the async udtf bring any additional benefits besides a
lighter implementation？

IMO, async udtf is more than a lighter implementation. It can act as a
general way for sql users to use the async operator. And they don't have to
bind the async function with a table (a LookupTable), and they are not
forced to join on an equality join condition, and they can use it to do
more than enrich data.

The async lookup join is more like a subset/specific usage of async udtf.
The specific version has more opportunity to be optimized (like push down)
is acceptable. Async table function should be categorized to used-defined
function.

>Should users

migrate to the lookup source when they encounter similar requirements or

problems, or should we develop an additional set of similar mechanisms？

As I clarified above, the lookup join is a specific usage of async udtf. So
it deserves more refined optimization like caching / retryable. But it may
not all

suitable for the async udtf. As function, it can be deterministic/or
non-deterministic. So caching is not suitable, and we also do not have a
common cache for the udf now. So I think optimization like caching/retry
should be handed over to the function implementor.

> the newly added query hint need a different name that
can be easier related to the lateral operation as the current join hints[5]
do.


What about using LATERAL?

as below

SELECT /*+ LATERAL('output-mode' = 'ordered', 'capacity' = '200', timeout =
'180s') */ a, c1, c2

FROM T1

LEFT JOIN lateral TABLE (async_split(b)) AS T(c1, c2) ON true

>For the async func example, since the target scenario is an external io
operation, it's better to add the `close` method to actively release
resources as a good example for users


Make sense to me, will update the FLIP

Best,

Aitozi.

Lincoln Lee <li...@gmail.com> 于2023年6月14日周三 14:24写道：

> Hi Aitozi,
>
> Sorry for the lately reply here!  Supports async udtf(`AsyncTableFunction`)
> directly in sql seems like an attractive feature, but there're two issues
> that need to be addressed before we can be sure to add it:
> 1. As mentioned in the flip[1], the current lookup function can already
> implement the requirements, but it requires implementing an extra
> `LookupTableSource` and explicitly declaring the table schema (which can
> help implementers the various push-down optimizations supported by the
> planner). Does the async udtf bring any additional benefits besides a
> lighter implementation?
> 2. FLIP-221[2] abstracts a reusable cache and metric infrastructure for
> lookup sources, which are important to improve performance and
> observability for high overhead external io scenarios, how do we integrate
> and reuse these capabilities after introducing async udtf? Should users
> migrate to the lookup source when they encounter similar requirements or
> problems, or should we develop an additional set of similar mechanisms? (a
> similarly case:  FLIP-234[3] introduced the retryable capability for lookup
> join)
>
> For the flip itself,
> 1. Considering the 'options' is already used as the dynamic table
> options[4] in flink, the newly added query hint need a different name that
> can be easier related to the lateral operation as the current join hints[5]
> do.
> 2. For the async func example, since the target scenario is an external io
> operation, it's better to add the `close` method to actively release
> resources as a good example for users. Also in terms of the determinism of
> a function, it is important to remind users that unless the behavior of the
> function is deterministic, it needs to be explicitly declared as
> non-deterministic.
>
> [1].
>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-313%3A+Add+support+of+User+Defined+AsyncTableFunction?src=contextnavpagetreemode
> [2].
>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-221%3A+Abstraction+for+lookup+source+cache+and+metric?src=contextnavpagetreemode
> [3].
>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-234%3A+Support+Retryable+Lookup+Join+To+Solve+Delayed+Updates+Issue+In+External+Systems?src=contextnavpagetreemode
> [4].
>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-113%3A+Supports+Dynamic+Table+Options+for+Flink+SQL?src=contextnavpagetreemode
> [5].
>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-229%3A+Introduces+Join+Hint+for+Flink+SQL+Batch+Job?src=contextnavpagetreemode
>
> Best,
> Lincoln Lee
>
>
> Aitozi <gj...@gmail.com> 于2023年6月13日周二 11:30写道：
>
> > Get your meaning now, thanks :)
> >
> > Best,
> > Aitozi.
> >
> > Feng Jin <ji...@gmail.com> 于2023年6月13日周二 11:16写道：
> >
> > > Hi Aitozi,
> > >
> > > Sorry for the confusing description.
> > >
> > > What I meant was that if we need to remind users about tire safety
> > issues,
> > > we should introduce the new UDTF interface instead of executing the
> > > original UDTF asynchronously. Therefore, I agree with introducing the
> > > AsyncTableFunction.
> > >
> > > Best,
> > > Feng
> > >
> > > On Tue, Jun 13, 2023 at 10:42 AM Aitozi <gj...@gmail.com> wrote:
> > >
> > > > Hi Feng,
> > > >     Thanks for your question. We do not provide a way to switch the
> > UDTF
> > > > between sync and async way,
> > > > So there should be no thread safety problem here.
> > > >
> > > > Best,
> > > > Aitozi
> > > >
> > > > Feng Jin <ji...@gmail.com> 于2023年6月13日周二 10:31写道：
> > > >
> > > > > Hi Aitozi, We do need to remind users about thread safety issues.
> > Thank
> > > > you
> > > > > for your efforts on this FLIP. I have no further questions.
> > > > > Best, Feng
> > > > >
> > > > >
> > > > > On Tue, Jun 13, 2023 at 6:05 AM Jing Ge <jing@ververica.com.invalid
> >
> > > > > wrote:
> > > > >
> > > > > > Hi Aitozi,
> > > > > >
> > > > > > Thanks for taking care of that part. I have no other concern.
> > > > > >
> > > > > > Best regards,
> > > > > > Jing
> > > > > >
> > > > > >
> > > > > > On Mon, Jun 12, 2023 at 5:38 PM Aitozi <gj...@gmail.com>
> > wrote:
> > > > > >
> > > > > > > BTW, If there are no other more blocking issue / comments, I
> > would
> > > > like
> > > > > > to
> > > > > > > start a VOTE in another thread this wednesday 6.14
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Aitozi.
> > > > > > >
> > > > > > > Aitozi <gj...@gmail.com> 于2023年6月12日周一 23:34写道：
> > > > > > >
> > > > > > > > Hi, Jing,
> > > > > > > >     Thanks for your explanation. I get your point now.
> > > > > > > >
> > > > > > > > For the performance part, I think it's a good idea to run
> with
> > > > > > returning
> > > > > > > a
> > > > > > > > big table case, the memory consumption
> > > > > > > > should be a point to be taken care about. Because in the
> > ordered
> > > > > mode,
> > > > > > > the
> > > > > > > > head element in buffer may affect the
> > > > > > > > total memory consumption.
> > > > > > > >
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > Aitozi.
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > Jing Ge <ji...@ververica.com.invalid> 于2023年6月12日周一 20:28写道：
> > > > > > > >
> > > > > > > >> Hi Aitozi,
> > > > > > > >>
> > > > > > > >> Which key will be used for lookup is not an issue, only one
> > row
> > > > will
> > > > > > be
> > > > > > > >> required for each key in order to enrich it. True, it
> depends
> > on
> > > > the
> > > > > > > >> implementation whether multiple rows or single row for each
> > key
> > > > will
> > > > > > be
> > > > > > > >> returned. However, for the lookup & enrichment scenario, one
> > > > row/key
> > > > > > is
> > > > > > > >> recommended, otherwise, like I mentioned previously,
> > enrichment
> > > > > won't
> > > > > > > >> work.
> > > > > > > >>
> > > > > > > >> I am a little bit concerned about returning a big table for
> > each
> > > > > key,
> > > > > > > >> since
> > > > > > > >> it will take the async call longer to return and need more
> > > memory.
> > > > > The
> > > > > > > >> performance tests should cover this scenario. This is not a
> > > > blocking
> > > > > > > issue
> > > > > > > >> for this FLIP.
> > > > > > > >>
> > > > > > > >> Best regards,
> > > > > > > >> Jing
> > > > > > > >>
> > > > > > > >> On Sat, Jun 10, 2023 at 4:11 AM Aitozi <
> gjying1314@gmail.com>
> > > > > wrote:
> > > > > > > >>
> > > > > > > >> > Hi Jing,
> > > > > > > >> >     I means the join key is not necessary to be the
> primary
> > > key
> > > > or
> > > > > > > >> unique
> > > > > > > >> > index of the database.
> > > > > > > >> > In this situation, we may queried out multi rows for one
> > join
> > > > > key. I
> > > > > > > >> think
> > > > > > > >> > that's why the
> > > > > > > >> > LookupFunction#lookup will return a collection of RowData.
> > > > > > > >> >
> > > > > > > >> > BTW, I think the behavior of lookup join will not affect
> the
> > > > > > semantic
> > > > > > > of
> > > > > > > >> > the async udtf.
> > > > > > > >> > We use the Async TableFunction here and the table function
> > can
> > > > > > collect
> > > > > > > >> > multiple rows.
> > > > > > > >> >
> > > > > > > >> > Thanks,
> > > > > > > >> > Atiozi.
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >> > Jing Ge <ji...@ververica.com.invalid> 于2023年6月10日周六
> 00:15写道：
> > > > > > > >> >
> > > > > > > >> > > Hi Aitozi,
> > > > > > > >> > >
> > > > > > > >> > > The keyRow used in this case contains all keys[1].
> > > > > > > >> > >
> > > > > > > >> > > Best regards,
> > > > > > > >> > > Jing
> > > > > > > >> > >
> > > > > > > >> > > [1]
> > > > > > > >> > >
> > > > > > > >> > >
> > > > > > > >> >
> > > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/flink/blob/191ec6ca3943d7119f14837efe112e074d815c47/flink-table/flink-table-common/src/main/java/org/apache/flink/table/functions/LookupFunction.java#L49
> > > > > > > >> > >
> > > > > > > >> > >
> > > > > > > >> > > On Fri, Jun 9, 2023 at 3:42 PM Aitozi <
> > gjying1314@gmail.com
> > > >
> > > > > > wrote:
> > > > > > > >> > >
> > > > > > > >> > > > Hi Jing,
> > > > > > > >> > > >
> > > > > > > >> > > >      The performance test is added to the FLIP.
> > > > > > > >> > > >
> > > > > > > >> > > >      As I know, The lookup join can return multi rows,
> > it
> > > > > > depends
> > > > > > > on
> > > > > > > >> > > > whether  the join key
> > > > > > > >> > > > is the primary key of the external database or not.
> The
> > > > > `lookup`
> > > > > > > [1]
> > > > > > > >> > will
> > > > > > > >> > > > return a collection of
> > > > > > > >> > > > joined result, and each of them will be collected
> > > > > > > >> > > >
> > > > > > > >> > > >
> > > > > > > >> > > > [1]:
> > > > > > > >> > > >
> > > > > > > >> > > >
> > > > > > > >> > >
> > > > > > > >> >
> > > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/flink/blob/191ec6ca3943d7119f14837efe112e074d815c47/flink-table/flink-table-common/src/main/java/org/apache/flink/table/functions/LookupFunction.java#L52
> > > > > > > >> > > >
> > > > > > > >> > > >
> > > > > > > >> > > > Thanks,
> > > > > > > >> > > > Aitozi.
> > > > > > > >> > > >
> > > > > > > >> > > > Jing Ge <ji...@ververica.com.invalid> 于2023年6月9日周五
> > > 17:05写道：
> > > > > > > >> > > >
> > > > > > > >> > > > > Hi Aitozi,
> > > > > > > >> > > > >
> > > > > > > >> > > > > Thanks for the feedback. Looking forward to the
> > > > performance
> > > > > > > tests.
> > > > > > > >> > > > >
> > > > > > > >> > > > > Afaik, lookup returns one row for each key [1] [2].
> > > > > > > Conceptually,
> > > > > > > >> the
> > > > > > > >> > > > > lookup function is used to enrich column(s) from the
> > > > > dimension
> > > > > > > >> table.
> > > > > > > >> > > If,
> > > > > > > >> > > > > for the given key, there will be more than one row,
> > > there
> > > > > will
> > > > > > > be
> > > > > > > >> no
> > > > > > > >> > > way
> > > > > > > >> > > > to
> > > > > > > >> > > > > know which row will be used to enrich the key.
> > > > > > > >> > > > >
> > > > > > > >> > > > > [1]
> > > > > > > >> > > > >
> > > > > > > >> > > > >
> > > > > > > >> > > >
> > > > > > > >> > >
> > > > > > > >> >
> > > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/flink/blob/191ec6ca3943d7119f14837efe112e074d815c47/flink-table/flink-table-common/src/main/java/org/apache/flink/table/functions/LookupFunction.java#L49
> > > > > > > >> > > > > [2]
> > > > > > > >> > > > >
> > > > > > > >> > > > >
> > > > > > > >> > > >
> > > > > > > >> > >
> > > > > > > >> >
> > > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/flink/blob/191ec6ca3943d7119f14837efe112e074d815c47/flink-table/flink-table-common/src/main/java/org/apache/flink/table/functions/TableFunction.java#L196
> > > > > > > >> > > > >
> > > > > > > >> > > > > Best regards,
> > > > > > > >> > > > > Jing
> > > > > > > >> > > > >
> > > > > > > >> > > > > On Fri, Jun 9, 2023 at 5:18 AM Aitozi <
> > > > gjying1314@gmail.com
> > > > > >
> > > > > > > >> wrote:
> > > > > > > >> > > > >
> > > > > > > >> > > > > > Hi Jing
> > > > > > > >> > > > > >     Thanks for your good questions. I have updated
> > the
> > > > > > example
> > > > > > > >> to
> > > > > > > >> > the
> > > > > > > >> > > > > FLIP.
> > > > > > > >> > > > > >
> > > > > > > >> > > > > > > Only one row for each lookup
> > > > > > > >> > > > > > lookup can also return multi rows, based on the
> > query
> > > > > > result.
> > > > > > > >> [1]
> > > > > > > >> > > > > >
> > > > > > > >> > > > > > [1]:
> > > > > > > >> > > > > >
> > > > > > > >> > > > > >
> > > > > > > >> > > > >
> > > > > > > >> > > >
> > > > > > > >> > >
> > > > > > > >> >
> > > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/flink/blob/191ec6ca3943d7119f14837efe112e074d815c47/flink-table/flink-table-common/src/main/java/org/apache/flink/table/functions/LookupFunction.java#L56
> > > > > > > >> > > > > >
> > > > > > > >> > > > > > > If we use async calls with lateral join, my gut
> > > > feeling
> > > > > is
> > > > > > > >> > > > > > that we might have many more async calls than
> lookup
> > > > > join. I
> > > > > > > am
> > > > > > > >> not
> > > > > > > >> > > > > really
> > > > > > > >> > > > > > sure if we will be facing potential issues in this
> > > case
> > > > or
> > > > > > > not.
> > > > > > > >> > > > > >
> > > > > > > >> > > > > > IMO, the work pattern is similar to the lookup
> > > function,
> > > > > for
> > > > > > > >> each
> > > > > > > >> > row
> > > > > > > >> > > > > from
> > > > > > > >> > > > > > the left table,
> > > > > > > >> > > > > > it will evaluate the eval method once, so the
> async
> > > call
> > > > > > > numbers
> > > > > > > >> > will
> > > > > > > >> > > > not
> > > > > > > >> > > > > > change.
> > > > > > > >> > > > > > and the maximum calls in flight is limited by the
> > > Async
> > > > > > > >> operators
> > > > > > > >> > > > buffer
> > > > > > > >> > > > > > capacity
> > > > > > > >> > > > > > which will be controlled by the option.
> > > > > > > >> > > > > >
> > > > > > > >> > > > > > BTW, for the naming of these option, I updated the
> > > FLIP
> > > > > > about
> > > > > > > >> this
> > > > > > > >> > > you
> > > > > > > >> > > > > can
> > > > > > > >> > > > > > refer to
> > > > > > > >> > > > > > the section of "ConfigOption" and "Rejected
> > > > Alternatives"
> > > > > > > >> > > > > >
> > > > > > > >> > > > > > In the end, for the performance evaluation, I'd
> like
> > > to
> > > > do
> > > > > > > some
> > > > > > > >> > tests
> > > > > > > >> > > > and
> > > > > > > >> > > > > > will update it to the FLIP doc
> > > > > > > >> > > > > >
> > > > > > > >> > > > > > Thanks,
> > > > > > > >> > > > > > Aitozi.
> > > > > > > >> > > > > >
> > > > > > > >> > > > > >
> > > > > > > >> > > > > > Jing Ge <ji...@ververica.com.invalid> 于2023年6月9日周五
> > > > > 07:23写道：
> > > > > > > >> > > > > >
> > > > > > > >> > > > > > > Hi Aitozi,
> > > > > > > >> > > > > > >
> > > > > > > >> > > > > > > Thanks for the clarification. The code example
> > looks
> > > > > > > >> > interesting. I
> > > > > > > >> > > > > would
> > > > > > > >> > > > > > > suggest adding them into the FLIP. The
> description
> > > > with
> > > > > > code
> > > > > > > >> > > examples
> > > > > > > >> > > > > > will
> > > > > > > >> > > > > > > help readers understand the motivation and how
> to
> > > use
> > > > > it.
> > > > > > > >> Afaiac,
> > > > > > > >> > > it
> > > > > > > >> > > > > is a
> > > > > > > >> > > > > > > valid feature for Flink users.
> > > > > > > >> > > > > > >
> > > > > > > >> > > > > > > As we knew, lookup join is based on temporal
> join,
> > > > i.e.
> > > > > > FOR
> > > > > > > >> > > > SYSTEM_TIME
> > > > > > > >> > > > > > AS
> > > > > > > >> > > > > > > OF which is also used in your code example.
> > Temporal
> > > > > join
> > > > > > > >> > performs
> > > > > > > >> > > > the
> > > > > > > >> > > > > > > lookup based on the processing time match. Only
> > one
> > > > row
> > > > > > for
> > > > > > > >> each
> > > > > > > >> > > > > > > lookup(afaiu, I need to check the source code to
> > > > double
> > > > > > > >> confirm)
> > > > > > > >> > > will
> > > > > > > >> > > > > > > return for further enrichment. One the other
> hand,
> > > > > lateral
> > > > > > > >> join
> > > > > > > >> > > will
> > > > > > > >> > > > > have
> > > > > > > >> > > > > > > sub-queries correlated with every individual
> value
> > > of
> > > > > the
> > > > > > > >> > reference
> > > > > > > >> > > > > table
> > > > > > > >> > > > > > > from the preceding part of the query and each
> sub
> > > > query
> > > > > > will
> > > > > > > >> > return
> > > > > > > >> > > > > > > multiple rows. If we use async calls with
> lateral
> > > > join,
> > > > > my
> > > > > > > gut
> > > > > > > >> > > > feeling
> > > > > > > >> > > > > is
> > > > > > > >> > > > > > > that we might have many more async calls than
> > lookup
> > > > > > join. I
> > > > > > > >> am
> > > > > > > >> > not
> > > > > > > >> > > > > > really
> > > > > > > >> > > > > > > sure if we will be facing potential issues in
> this
> > > > case
> > > > > or
> > > > > > > >> not.
> > > > > > > >> > > > > Possible
> > > > > > > >> > > > > > > issues I can think of now e.g. too many PRC
> calls,
> > > too
> > > > > > many
> > > > > > > >> async
> > > > > > > >> > > > calls
> > > > > > > >> > > > > > > processing, the sub query will return a table
> > which
> > > > > might
> > > > > > be
> > > > > > > >> > (too)
> > > > > > > >> > > > big,
> > > > > > > >> > > > > > and
> > > > > > > >> > > > > > > might cause performance issues. I would suggest
> > > > > preparing
> > > > > > > some
> > > > > > > >> > use
> > > > > > > >> > > > > cases
> > > > > > > >> > > > > > > and running some performance tests to check it.
> > > These
> > > > > are
> > > > > > my
> > > > > > > >> > > concerns
> > > > > > > >> > > > > > about
> > > > > > > >> > > > > > > using async calls with lateral join and I'd like
> > to
> > > > > share
> > > > > > > with
> > > > > > > >> > you,
> > > > > > > >> > > > > happy
> > > > > > > >> > > > > > > to discuss with you and hear different opinions,
> > > > > hopefully
> > > > > > > the
> > > > > > > >> > > > > > > discussion could help me understand it more
> > deeply.
> > > > > Please
> > > > > > > >> > correct
> > > > > > > >> > > me
> > > > > > > >> > > > > if
> > > > > > > >> > > > > > I
> > > > > > > >> > > > > > > am wrong.
> > > > > > > >> > > > > > >
> > > > > > > >> > > > > > > Best regards,
> > > > > > > >> > > > > > > Jing
> > > > > > > >> > > > > > >
> > > > > > > >> > > > > > >
> > > > > > > >> > > > > > > On Thu, Jun 8, 2023 at 7:22 AM Aitozi <
> > > > > > gjying1314@gmail.com
> > > > > > > >
> > > > > > > >> > > wrote:
> > > > > > > >> > > > > > >
> > > > > > > >> > > > > > > > Hi Mason,
> > > > > > > >> > > > > > > >     Thanks for your input. I think if we
> support
> > > the
> > > > > > user
> > > > > > > >> > defined
> > > > > > > >> > > > > async
> > > > > > > >> > > > > > > > table function,
> > > > > > > >> > > > > > > > user will be able to use it to hold a batch
> data
> > > > then
> > > > > > > >> handle it
> > > > > > > >> > > at
> > > > > > > >> > > > > one
> > > > > > > >> > > > > > > time
> > > > > > > >> > > > > > > > in the customized function.
> > > > > > > >> > > > > > > >
> > > > > > > >> > > > > > > > AsyncSink is meant for the sink operator. I
> have
> > > not
> > > > > > > figure
> > > > > > > >> out
> > > > > > > >> > > how
> > > > > > > >> > > > > to
> > > > > > > >> > > > > > > > integrate in this case.
> > > > > > > >> > > > > > > >
> > > > > > > >> > > > > > > > Thanks,
> > > > > > > >> > > > > > > > Atiozi.
> > > > > > > >> > > > > > > >
> > > > > > > >> > > > > > > >
> > > > > > > >> > > > > > > > Mason Chen <ma...@gmail.com>
> > 于2023年6月8日周四
> > > > > > 12:40写道：
> > > > > > > >> > > > > > > >
> > > > > > > >> > > > > > > > > Hi Aitozi,
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > > > I think it makes sense to make it easier for
> > SQL
> > > > > users
> > > > > > > to
> > > > > > > >> > make
> > > > > > > >> > > > > RPCs.
> > > > > > > >> > > > > > Do
> > > > > > > >> > > > > > > > you
> > > > > > > >> > > > > > > > > think your proposal can extend to the
> ability
> > to
> > > > > batch
> > > > > > > >> data
> > > > > > > >> > for
> > > > > > > >> > > > the
> > > > > > > >> > > > > > > RPC?
> > > > > > > >> > > > > > > > > This is also another common strategy to
> > increase
> > > > > > > >> throughput.
> > > > > > > >> > > > Also,
> > > > > > > >> > > > > > have
> > > > > > > >> > > > > > > > you
> > > > > > > >> > > > > > > > > considered solving this a bit differently by
> > > > > > leveraging
> > > > > > > >> > Flink's
> > > > > > > >> > > > > > > > AsyncSink?
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > > > Best,
> > > > > > > >> > > > > > > > > Mason
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > > > On Mon, Jun 5, 2023 at 1:50 AM Aitozi <
> > > > > > > >> gjying1314@gmail.com>
> > > > > > > >> > > > > wrote:
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > > > > One more thing for discussion:
> > > > > > > >> > > > > > > > > >
> > > > > > > >> > > > > > > > > > In our internal implementation, we reuse
> the
> > > > > option
> > > > > > > >> > > > > > > > > > `table.exec.async-lookup.buffer-capacity`
> > and
> > > > > > > >> > > > > > > > > > `table.exec.async-lookup.timeout` to
> config
> > > > > > > >> > > > > > > > > > the async udtf. Do you think we should add
> > two
> > > > > extra
> > > > > > > >> option
> > > > > > > >> > > to
> > > > > > > >> > > > > > > > > distinguish
> > > > > > > >> > > > > > > > > > from the lookup option such as
> > > > > > > >> > > > > > > > > >
> > > > > > > >> > > > > > > > > > `table.exec.async-udtf.buffer-capacity`
> > > > > > > >> > > > > > > > > > `table.exec.async-udtf.timeout`
> > > > > > > >> > > > > > > > > >
> > > > > > > >> > > > > > > > > >
> > > > > > > >> > > > > > > > > > Best,
> > > > > > > >> > > > > > > > > > Aitozi.
> > > > > > > >> > > > > > > > > >
> > > > > > > >> > > > > > > > > >
> > > > > > > >> > > > > > > > > >
> > > > > > > >> > > > > > > > > > Aitozi <gj...@gmail.com>
> 于2023年6月5日周一
> > > > > 12:20写道：
> > > > > > > >> > > > > > > > > >
> > > > > > > >> > > > > > > > > > > Hi Jing,
> > > > > > > >> > > > > > > > > > >
> > > > > > > >> > > > > > > > > > >     > what is the difference between the
> > RPC
> > > > > call
> > > > > > or
> > > > > > > >> > query
> > > > > > > >> > > > you
> > > > > > > >> > > > > > > > > mentioned
> > > > > > > >> > > > > > > > > > > and the lookup in a very
> > > > > > > >> > > > > > > > > > > general way
> > > > > > > >> > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > I think the RPC call or query service is
> > > quite
> > > > > > > >> similar to
> > > > > > > >> > > the
> > > > > > > >> > > > > > > lookup
> > > > > > > >> > > > > > > > > > join.
> > > > > > > >> > > > > > > > > > > But lookup join should work
> > > > > > > >> > > > > > > > > > > with `LookupTableSource`.
> > > > > > > >> > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > Let's see how we can perform an async
> RPC
> > > call
> > > > > > with
> > > > > > > >> > lookup
> > > > > > > >> > > > > join:
> > > > > > > >> > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > (1) Implement an AsyncTableFunction with
> > RPC
> > > > > call
> > > > > > > >> logic.
> > > > > > > >> > > > > > > > > > > (2) Implement a `LookupTableSource`
> > > connector
> > > > > run
> > > > > > > with
> > > > > > > >> > the
> > > > > > > >> > > > > async
> > > > > > > >> > > > > > > udtf
> > > > > > > >> > > > > > > > > > > defined in (1).
> > > > > > > >> > > > > > > > > > > (3) Then define a DDL of this look up
> > table
> > > in
> > > > > SQL
> > > > > > > >> > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > CREATE TEMPORARY TABLE Customers (
> > > > > > > >> > > > > > > > > > >   id INT,
> > > > > > > >> > > > > > > > > > >   name STRING,
> > > > > > > >> > > > > > > > > > >   country STRING,
> > > > > > > >> > > > > > > > > > >   zip STRING
> > > > > > > >> > > > > > > > > > > ) WITH (
> > > > > > > >> > > > > > > > > > >   'connector' = 'custom'
> > > > > > > >> > > > > > > > > > > );
> > > > > > > >> > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > (4) Run with the query as below:
> > > > > > > >> > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > SELECT o.order_id, o.total, c.country,
> > c.zip
> > > > > > > >> > > > > > > > > > > FROM Orders AS o
> > > > > > > >> > > > > > > > > > >   JOIN Customers FOR SYSTEM_TIME AS OF
> > > > > o.proc_time
> > > > > > > AS
> > > > > > > >> c
> > > > > > > >> > > > > > > > > > >     ON o.customer_id = c.id;
> > > > > > > >> > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > This example is from doc
> > > > > > > >> > > > > > > > > > > <
> > > > > > > >> > > > > > > > > >
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > >
> > > > > > > >> > > > > > >
> > > > > > > >> > > > > >
> > > > > > > >> > > > >
> > > > > > > >> > > >
> > > > > > > >> > >
> > > > > > > >> >
> > > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/dev/table/sql/queries/joins/#lookup-join
> > > > > > > >> > > > > > > > > > >.You
> > > > > > > >> > > > > > > > > > > can image the look up process as an
> async
> > > RPC
> > > > > call
> > > > > > > >> > process.
> > > > > > > >> > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > Let's see how we can perform an async
> RPC
> > > call
> > > > > > with
> > > > > > > >> > lateral
> > > > > > > >> > > > > join:
> > > > > > > >> > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > (1) Implement an AsyncTableFunction with
> > RPC
> > > > > call
> > > > > > > >> logic.
> > > > > > > >> > > > > > > > > > > (2) Run query with
> > > > > > > >> > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > Create function f1 as '...' ;
> > > > > > > >> > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > SELECT o.order_id, o.total, c.country,
> > c.zip
> > > > > FROM
> > > > > > > >> Orders
> > > > > > > >> > > > > lateral
> > > > > > > >> > > > > > > > table
> > > > > > > >> > > > > > > > > > > (f1(order_id)) as T(...);
> > > > > > > >> > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > As you can see, the lateral join version
> > is
> > > > more
> > > > > > > >> simple
> > > > > > > >> > and
> > > > > > > >> > > > > > > intuitive
> > > > > > > >> > > > > > > > > to
> > > > > > > >> > > > > > > > > > > users. Users do not have to wrap a
> > > > > > > >> > > > > > > > > > > LookupTableSource for the purpose of
> using
> > > > async
> > > > > > > udtf.
> > > > > > > >> > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > In the end, We can also see the user
> > defined
> > > > > async
> > > > > > > >> table
> > > > > > > >> > > > > function
> > > > > > > >> > > > > > > is
> > > > > > > >> > > > > > > > an
> > > > > > > >> > > > > > > > > > > enhancement of the current lateral table
> > > join
> > > > > > > >> > > > > > > > > > > which only supports sync lateral join
> now.
> > > > > > > >> > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > Best,
> > > > > > > >> > > > > > > > > > > Aitozi.
> > > > > > > >> > > > > > > > > > >
> > > > > > > >> > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > Jing Ge <ji...@ververica.com.invalid>
> > > > > 于2023年6月2日周五
> > > > > > > >> > 19:37写道：
> > > > > > > >> > > > > > > > > > >
> > > > > > > >> > > > > > > > > > >> Hi Aitozi,
> > > > > > > >> > > > > > > > > > >>
> > > > > > > >> > > > > > > > > > >> Thanks for the update. Just out of
> > > curiosity,
> > > > > > what
> > > > > > > is
> > > > > > > >> > the
> > > > > > > >> > > > > > > difference
> > > > > > > >> > > > > > > > > > >> between the RPC call or query you
> > mentioned
> > > > and
> > > > > > the
> > > > > > > >> > lookup
> > > > > > > >> > > > in
> > > > > > > >> > > > > a
> > > > > > > >> > > > > > > very
> > > > > > > >> > > > > > > > > > >> general way? Since Lateral join is used
> > in
> > > > the
> > > > > > > FLIP.
> > > > > > > >> Is
> > > > > > > >> > > > there
> > > > > > > >> > > > > > any
> > > > > > > >> > > > > > > > > > special
> > > > > > > >> > > > > > > > > > >> thought for that? Sorry for asking so
> > many
> > > > > > > questions.
> > > > > > > >> > The
> > > > > > > >> > > > FLIP
> > > > > > > >> > > > > > > > > contains
> > > > > > > >> > > > > > > > > > >> limited information to understand the
> > > > > motivation.
> > > > > > > >> > > > > > > > > > >>
> > > > > > > >> > > > > > > > > > >> Best regards,
> > > > > > > >> > > > > > > > > > >> Jing
> > > > > > > >> > > > > > > > > > >>
> > > > > > > >> > > > > > > > > > >> On Fri, Jun 2, 2023 at 3:48 AM Aitozi <
> > > > > > > >> > > gjying1314@gmail.com
> > > > > > > >> > > > >
> > > > > > > >> > > > > > > wrote:
> > > > > > > >> > > > > > > > > > >>
> > > > > > > >> > > > > > > > > > >> > Hi Jing,
> > > > > > > >> > > > > > > > > > >> >     I have updated the proposed
> changes
> > > to
> > > > > the
> > > > > > > >> FLIP.
> > > > > > > >> > > IMO,
> > > > > > > >> > > > > > lookup
> > > > > > > >> > > > > > > > has
> > > > > > > >> > > > > > > > > > its
> > > > > > > >> > > > > > > > > > >> > clear
> > > > > > > >> > > > > > > > > > >> > async call requirement is due to its
> IO
> > > > heavy
> > > > > > > >> > operator.
> > > > > > > >> > > In
> > > > > > > >> > > > > our
> > > > > > > >> > > > > > > > > usage,
> > > > > > > >> > > > > > > > > > >> sql
> > > > > > > >> > > > > > > > > > >> > users have
> > > > > > > >> > > > > > > > > > >> > logic to do some RPC call or query
> the
> > > > > > > third-party
> > > > > > > >> > > service
> > > > > > > >> > > > > > which
> > > > > > > >> > > > > > > > is
> > > > > > > >> > > > > > > > > > >> also IO
> > > > > > > >> > > > > > > > > > >> > intensive.
> > > > > > > >> > > > > > > > > > >> > In these case, we'd like to leverage
> > the
> > > > > async
> > > > > > > >> > function
> > > > > > > >> > > to
> > > > > > > >> > > > > > > improve
> > > > > > > >> > > > > > > > > the
> > > > > > > >> > > > > > > > > > >> > throughput.
> > > > > > > >> > > > > > > > > > >> >
> > > > > > > >> > > > > > > > > > >> > Thanks,
> > > > > > > >> > > > > > > > > > >> > Aitozi.
> > > > > > > >> > > > > > > > > > >> >
> > > > > > > >> > > > > > > > > > >> > Jing Ge <ji...@ververica.com.invalid>
> > > > > > > 于2023年6月1日周四
> > > > > > > >> > > > 22:55写道：
> > > > > > > >> > > > > > > > > > >> >
> > > > > > > >> > > > > > > > > > >> > > Hi Aitozi,
> > > > > > > >> > > > > > > > > > >> > >
> > > > > > > >> > > > > > > > > > >> > > Sorry for the late reply. Would you
> > > like
> > > > to
> > > > > > > >> update
> > > > > > > >> > the
> > > > > > > >> > > > > > > proposed
> > > > > > > >> > > > > > > > > > >> changes
> > > > > > > >> > > > > > > > > > >> > > with more details into the FLIP
> too?
> > > > > > > >> > > > > > > > > > >> > > I got your point. It looks like a
> > > > rational
> > > > > > > idea.
> > > > > > > >> > > > However,
> > > > > > > >> > > > > > > since
> > > > > > > >> > > > > > > > > > lookup
> > > > > > > >> > > > > > > > > > >> > has
> > > > > > > >> > > > > > > > > > >> > > its clear async call requirement,
> are
> > > > there
> > > > > > any
> > > > > > > >> real
> > > > > > > >> > > use
> > > > > > > >> > > > > > cases
> > > > > > > >> > > > > > > > > that
> > > > > > > >> > > > > > > > > > >> > > need this change? This will help us
> > > > > > understand
> > > > > > > >> the
> > > > > > > >> > > > > > motivation.
> > > > > > > >> > > > > > > > > After
> > > > > > > >> > > > > > > > > > >> all,
> > > > > > > >> > > > > > > > > > >> > > lateral join and temporal lookup
> > > join[1]
> > > > > are
> > > > > > > >> quite
> > > > > > > >> > > > > > different.
> > > > > > > >> > > > > > > > > > >> > >
> > > > > > > >> > > > > > > > > > >> > > Best regards,
> > > > > > > >> > > > > > > > > > >> > > Jing
> > > > > > > >> > > > > > > > > > >> > >
> > > > > > > >> > > > > > > > > > >> > >
> > > > > > > >> > > > > > > > > > >> > > [1]
> > > > > > > >> > > > > > > > > > >> > >
> > > > > > > >> > > > > > > > > > >> > >
> > > > > > > >> > > > > > > > > > >> >
> > > > > > > >> > > > > > > > > > >>
> > > > > > > >> > > > > > > > > >
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > >
> > > > > > > >> > > > > > >
> > > > > > > >> > > > > >
> > > > > > > >> > > > >
> > > > > > > >> > > >
> > > > > > > >> > >
> > > > > > > >> >
> > > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/flink/blob/d90a72da2fd601ca4e2a46700e91ec5b348de2ad/flink-table/flink-table-common/src/main/java/org/apache/flink/table/functions/AsyncTableFunction.java#L54
> > > > > > > >> > > > > > > > > > >> > >
> > > > > > > >> > > > > > > > > > >> > > On Wed, May 31, 2023 at 8:53 AM
> > Aitozi
> > > <
> > > > > > > >> > > > > > gjying1314@gmail.com>
> > > > > > > >> > > > > > > > > > wrote:
> > > > > > > >> > > > > > > > > > >> > >
> > > > > > > >> > > > > > > > > > >> > > > Hi Jing,
> > > > > > > >> > > > > > > > > > >> > > >     What do you think about it?
> Can
> > > we
> > > > > move
> > > > > > > >> > forward
> > > > > > > >> > > > this
> > > > > > > >> > > > > > > > > feature?
> > > > > > > >> > > > > > > > > > >> > > >
> > > > > > > >> > > > > > > > > > >> > > > Thanks,
> > > > > > > >> > > > > > > > > > >> > > > Aitozi.
> > > > > > > >> > > > > > > > > > >> > > >
> > > > > > > >> > > > > > > > > > >> > > > Aitozi <gj...@gmail.com>
> > > > > > 于2023年5月29日周一
> > > > > > > >> > > 09:56写道：
> > > > > > > >> > > > > > > > > > >> > > >
> > > > > > > >> > > > > > > > > > >> > > > > Hi Jing,
> > > > > > > >> > > > > > > > > > >> > > > >     > "Do you mean to support
> the
> > > > > > > >> > > AyncTableFunction
> > > > > > > >> > > > > > beyond
> > > > > > > >> > > > > > > > the
> > > > > > > >> > > > > > > > > > >> > > > > LookupTableSource?"
> > > > > > > >> > > > > > > > > > >> > > > > Yes, I mean to support the
> > > > > > > AyncTableFunction
> > > > > > > >> > > beyond
> > > > > > > >> > > > > the
> > > > > > > >> > > > > > > > > > >> > > > LookupTableSource.
> > > > > > > >> > > > > > > > > > >> > > > >
> > > > > > > >> > > > > > > > > > >> > > > > The "AsyncTableFunction" is the
> > > > > function
> > > > > > > with
> > > > > > > >> > > > ability
> > > > > > > >> > > > > to
> > > > > > > >> > > > > > > be
> > > > > > > >> > > > > > > > > > >> executed
> > > > > > > >> > > > > > > > > > >> > > > async
> > > > > > > >> > > > > > > > > > >> > > > > (with AsyncWaitOperator).
> > > > > > > >> > > > > > > > > > >> > > > > The async lookup join is a one
> of
> > > > usage
> > > > > > of
> > > > > > > >> this.
> > > > > > > >> > > So,
> > > > > > > >> > > > > we
> > > > > > > >> > > > > > > > don't
> > > > > > > >> > > > > > > > > > >> have to
> > > > > > > >> > > > > > > > > > >> > > > bind
> > > > > > > >> > > > > > > > > > >> > > > > the AyncTableFunction with
> > > > > > > LookupTableSource.
> > > > > > > >> > > > > > > > > > >> > > > > If User-defined
> > AsyncTableFunction
> > > is
> > > > > > > >> supported,
> > > > > > > >> > > > user
> > > > > > > >> > > > > > can
> > > > > > > >> > > > > > > > > > directly
> > > > > > > >> > > > > > > > > > >> > use
> > > > > > > >> > > > > > > > > > >> > > > > lateral table syntax to perform
> > > async
> > > > > > > >> operation.
> > > > > > > >> > > > > > > > > > >> > > > >
> > > > > > > >> > > > > > > > > > >> > > > > > "It would be better if you
> > could
> > > > > > > elaborate
> > > > > > > >> the
> > > > > > > >> > > > > > proposed
> > > > > > > >> > > > > > > > > > changes
> > > > > > > >> > > > > > > > > > >> wrt
> > > > > > > >> > > > > > > > > > >> > > the
> > > > > > > >> > > > > > > > > > >> > > > > CorrelatedCodeGenerator with
> more
> > > > > > details"
> > > > > > > >> > > > > > > > > > >> > > > >
> > > > > > > >> > > > > > > > > > >> > > > > In the proposal, we use lateral
> > > table
> > > > > > > syntax
> > > > > > > >> to
> > > > > > > >> > > > > support
> > > > > > > >> > > > > > > the
> > > > > > > >> > > > > > > > > > async
> > > > > > > >> > > > > > > > > > >> > table
> > > > > > > >> > > > > > > > > > >> > > > > function. So the planner will
> > also
> > > > > treat
> > > > > > > this
> > > > > > > >> > > > > statement
> > > > > > > >> > > > > > > to a
> > > > > > > >> > > > > > > > > > >> > > > > CommonExecCorrelate node. So
> the
> > > > > runtime
> > > > > > > code
> > > > > > > >> > > should
> > > > > > > >> > > > > be
> > > > > > > >> > > > > > > > > > generated
> > > > > > > >> > > > > > > > > > >> in
> > > > > > > >> > > > > > > > > > >> > > > > CorrelatedCodeGenerator.
> > > > > > > >> > > > > > > > > > >> > > > > In CorrelatedCodeGenerator, we
> > will
> > > > > know
> > > > > > > the
> > > > > > > >> > > > > > > TableFunction's
> > > > > > > >> > > > > > > > > > Kind
> > > > > > > >> > > > > > > > > > >> of
> > > > > > > >> > > > > > > > > > >> > > > > `FunctionKind.Table` or
> > > > > > > >> > `FunctionKind.ASYNC_TABLE`
> > > > > > > >> > > > > > > > > > >> > > > > For  `FunctionKind.ASYNC_TABLE`
> > we
> > > > can
> > > > > > > >> generate
> > > > > > > >> > a
> > > > > > > >> > > > > > > > > > >> AsyncWaitOperator
> > > > > > > >> > > > > > > > > > >> > to
> > > > > > > >> > > > > > > > > > >> > > > > execute the async table
> function.
> > > > > > > >> > > > > > > > > > >> > > > >
> > > > > > > >> > > > > > > > > > >> > > > >
> > > > > > > >> > > > > > > > > > >> > > > > Thanks,
> > > > > > > >> > > > > > > > > > >> > > > > Aitozi.
> > > > > > > >> > > > > > > > > > >> > > > >
> > > > > > > >> > > > > > > > > > >> > > > >
> > > > > > > >> > > > > > > > > > >> > > > > Jing Ge
> > <jing@ververica.com.invalid
> > > >
> > > > > > > >> > > 于2023年5月29日周一
> > > > > > > >> > > > > > > 03:22写道：
> > > > > > > >> > > > > > > > > > >> > > > >
> > > > > > > >> > > > > > > > > > >> > > > >> Hi Aitozi,
> > > > > > > >> > > > > > > > > > >> > > > >>
> > > > > > > >> > > > > > > > > > >> > > > >> Thanks for the clarification.
> > The
> > > > > naming
> > > > > > > >> > "Lookup"
> > > > > > > >> > > > > might
> > > > > > > >> > > > > > > > > suggest
> > > > > > > >> > > > > > > > > > >> > using
> > > > > > > >> > > > > > > > > > >> > > it
> > > > > > > >> > > > > > > > > > >> > > > >> for table look up. But
> > > conceptually
> > > > > what
> > > > > > > the
> > > > > > > >> > > eval()
> > > > > > > >> > > > > > > method
> > > > > > > >> > > > > > > > > will
> > > > > > > >> > > > > > > > > > >> do
> > > > > > > >> > > > > > > > > > >> > is
> > > > > > > >> > > > > > > > > > >> > > to
> > > > > > > >> > > > > > > > > > >> > > > >> get a collection of
> results(Row,
> > > > > > RowData)
> > > > > > > >> from
> > > > > > > >> > > the
> > > > > > > >> > > > > > given
> > > > > > > >> > > > > > > > > keys.
> > > > > > > >> > > > > > > > > > >> How
> > > > > > > >> > > > > > > > > > >> > it
> > > > > > > >> > > > > > > > > > >> > > > will
> > > > > > > >> > > > > > > > > > >> > > > >> be done depends on the
> > > > implementation,
> > > > > > > i.e.
> > > > > > > >> you
> > > > > > > >> > > can
> > > > > > > >> > > > > > > > implement
> > > > > > > >> > > > > > > > > > >> your
> > > > > > > >> > > > > > > > > > >> > own
> > > > > > > >> > > > > > > > > > >> > > > >> Source[1][2]. The example in
> the
> > > > FLIP
> > > > > > > >> should be
> > > > > > > >> > > > able
> > > > > > > >> > > > > to
> > > > > > > >> > > > > > > be
> > > > > > > >> > > > > > > > > > >> handled
> > > > > > > >> > > > > > > > > > >> > in
> > > > > > > >> > > > > > > > > > >> > > > this
> > > > > > > >> > > > > > > > > > >> > > > >> way.
> > > > > > > >> > > > > > > > > > >> > > > >>
> > > > > > > >> > > > > > > > > > >> > > > >> Do you mean to support the
> > > > > > > AyncTableFunction
> > > > > > > >> > > beyond
> > > > > > > >> > > > > the
> > > > > > > >> > > > > > > > > > >> > > > LookupTableSource?
> > > > > > > >> > > > > > > > > > >> > > > >> It would be better if you
> could
> > > > > > elaborate
> > > > > > > >> the
> > > > > > > >> > > > > proposed
> > > > > > > >> > > > > > > > > changes
> > > > > > > >> > > > > > > > > > >> wrt
> > > > > > > >> > > > > > > > > > >> > the
> > > > > > > >> > > > > > > > > > >> > > > >> CorrelatedCodeGenerator with
> > more
> > > > > > details.
> > > > > > > >> > > Thanks!
> > > > > > > >> > > > > > > > > > >> > > > >>
> > > > > > > >> > > > > > > > > > >> > > > >> Best regards,
> > > > > > > >> > > > > > > > > > >> > > > >> Jing
> > > > > > > >> > > > > > > > > > >> > > > >>
> > > > > > > >> > > > > > > > > > >> > > > >> [1]
> > > > > > > >> > > > > > > > > > >> > > > >>
> > > > > > > >> > > > > > > > > > >> > > > >>
> > > > > > > >> > > > > > > > > > >> > > >
> > > > > > > >> > > > > > > > > > >> > >
> > > > > > > >> > > > > > > > > > >> >
> > > > > > > >> > > > > > > > > > >>
> > > > > > > >> > > > > > > > > >
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > >
> > > > > > > >> > > > > > >
> > > > > > > >> > > > > >
> > > > > > > >> > > > >
> > > > > > > >> > > >
> > > > > > > >> > >
> > > > > > > >> >
> > > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/flink/blob/678370b18e1b6c4a23e5ce08f8efd05675a0cc17/flink-table/flink-table-common/src/main/java/org/apache/flink/table/connector/source/LookupTableSource.java#L64
> > > > > > > >> > > > > > > > > > >> > > > >> [2]
> > > > > > > >> > > > > > > > > > >> > > > >>
> > > > > > > >> > > > > > > > > > >> > > > >>
> > > > > > > >> > > > > > > > > > >> > > >
> > > > > > > >> > > > > > > > > > >> > >
> > > > > > > >> > > > > > > > > > >> >
> > > > > > > >> > > > > > > > > > >>
> > > > > > > >> > > > > > > > > >
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > >
> > > > > > > >> > > > > > >
> > > > > > > >> > > > > >
> > > > > > > >> > > > >
> > > > > > > >> > > >
> > > > > > > >> > >
> > > > > > > >> >
> > > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/flink/blob/678370b18e1b6c4a23e5ce08f8efd05675a0cc17/flink-table/flink-table-common/src/main/java/org/apache/flink/table/connector/source/AsyncTableFunctionProvider.java#L49
> > > > > > > >> > > > > > > > > > >> > > > >>
> > > > > > > >> > > > > > > > > > >> > > > >> On Sat, May 27, 2023 at
> 9:48 AM
> > > > > Aitozi <
> > > > > > > >> > > > > > > > gjying1314@gmail.com
> > > > > > > >> > > > > > > > > >
> > > > > > > >> > > > > > > > > > >> > wrote:
> > > > > > > >> > > > > > > > > > >> > > > >>
> > > > > > > >> > > > > > > > > > >> > > > >> > Hi Jing,
> > > > > > > >> > > > > > > > > > >> > > > >> >     Thanks for your
> response.
> > As
> > > > > > stated
> > > > > > > in
> > > > > > > >> > the
> > > > > > > >> > > > > FLIP,
> > > > > > > >> > > > > > > the
> > > > > > > >> > > > > > > > > > >> purpose
> > > > > > > >> > > > > > > > > > >> > of
> > > > > > > >> > > > > > > > > > >> > > > this
> > > > > > > >> > > > > > > > > > >> > > > >> > FLIP is meant to support
> > > > > > > >> > > > > > > > > > >> > > > >> > user-defined async table
> > > function.
> > > > > As
> > > > > > > >> > described
> > > > > > > >> > > > in
> > > > > > > >> > > > > > > flink
> > > > > > > >> > > > > > > > > > >> document
> > > > > > > >> > > > > > > > > > >> > > [1]
> > > > > > > >> > > > > > > > > > >> > > > >> >
> > > > > > > >> > > > > > > > > > >> > > > >> > Async table functions are
> > > special
> > > > > > > >> functions
> > > > > > > >> > for
> > > > > > > >> > > > > table
> > > > > > > >> > > > > > > > > sources
> > > > > > > >> > > > > > > > > > >> that
> > > > > > > >> > > > > > > > > > >> > > > >> perform
> > > > > > > >> > > > > > > > > > >> > > > >> > > a lookup.
> > > > > > > >> > > > > > > > > > >> > > > >> > >
> > > > > > > >> > > > > > > > > > >> > > > >> >
> > > > > > > >> > > > > > > > > > >> > > > >> > So end user can not directly
> > > > define
> > > > > > and
> > > > > > > >> use
> > > > > > > >> > > async
> > > > > > > >> > > > > > table
> > > > > > > >> > > > > > > > > > >> function
> > > > > > > >> > > > > > > > > > >> > > now.
> > > > > > > >> > > > > > > > > > >> > > > An
> > > > > > > >> > > > > > > > > > >> > > > >> > user case is reported in [2]
> > > > > > > >> > > > > > > > > > >> > > > >> >
> > > > > > > >> > > > > > > > > > >> > > > >> > So, in conclusion, no new
> > > > interface
> > > > > is
> > > > > > > >> > > > introduced,
> > > > > > > >> > > > > > but
> > > > > > > >> > > > > > > we
> > > > > > > >> > > > > > > > > > >> extend
> > > > > > > >> > > > > > > > > > >> > the
> > > > > > > >> > > > > > > > > > >> > > > >> > ability to support
> > user-defined
> > > > > async
> > > > > > > >> table
> > > > > > > >> > > > > function.
> > > > > > > >> > > > > > > > > > >> > > > >> >
> > > > > > > >> > > > > > > > > > >> > > > >> > [1]:
> > > > > > > >> > > > > > > > > > >> > > > >> >
> > > > > > > >> > > > > > > > > > >> > > > >> >
> > > > > > > >> > > > > > > > > > >> > > > >>
> > > > > > > >> > > > > > > > > > >> > > >
> > > > > > > >> > > > > > > > > > >> > >
> > > > > > > >> > > > > > > > > > >> >
> > > > > > > >> > > > > > > > > > >>
> > > > > > > >> > > > > > > > > >
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > >
> > > > > > > >> > > > > > >
> > > > > > > >> > > > > >
> > > > > > > >> > > > >
> > > > > > > >> > > >
> > > > > > > >> > >
> > > > > > > >> >
> > > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/dev/table/functions/udfs/
> > > > > > > >> > > > > > > > > > >> > > > >> > [2]:
> > > > > > > >> > > > > > > > > > >> > >
> > > > > > > >> > > > > > > >
> > > > > > > >> >
> > > > https://lists.apache.org/thread/qljwd40v5ntz6733cwcdr8s4z97b343b
> > > > > > > >> > > > > > > > > > >> > > > >> >
> > > > > > > >> > > > > > > > > > >> > > > >> > Thanks.
> > > > > > > >> > > > > > > > > > >> > > > >> > Aitozi.
> > > > > > > >> > > > > > > > > > >> > > > >> >
> > > > > > > >> > > > > > > > > > >> > > > >> >
> > > > > > > >> > > > > > > > > > >> > > > >> > Jing Ge
> > > > <jing@ververica.com.invalid
> > > > > >
> > > > > > > >> > > > 于2023年5月27日周六
> > > > > > > >> > > > > > > > > 06:40写道：
> > > > > > > >> > > > > > > > > > >> > > > >> >
> > > > > > > >> > > > > > > > > > >> > > > >> > > Hi Aitozi,
> > > > > > > >> > > > > > > > > > >> > > > >> > >
> > > > > > > >> > > > > > > > > > >> > > > >> > > Thanks for your proposal.
> I
> > am
> > > > not
> > > > > > > quite
> > > > > > > >> > sure
> > > > > > > >> > > > if
> > > > > > > >> > > > > I
> > > > > > > >> > > > > > > > > > understood
> > > > > > > >> > > > > > > > > > >> > your
> > > > > > > >> > > > > > > > > > >> > > > >> > thoughts
> > > > > > > >> > > > > > > > > > >> > > > >> > > correctly. You described a
> > > > special
> > > > > > > case
> > > > > > > >> > > > > > > implementation
> > > > > > > >> > > > > > > > of
> > > > > > > >> > > > > > > > > > the
> > > > > > > >> > > > > > > > > > >> > > > >> > > AsyncTableFunction with on
> > > > public
> > > > > > API
> > > > > > > >> > > changes.
> > > > > > > >> > > > > > Would
> > > > > > > >> > > > > > > > you
> > > > > > > >> > > > > > > > > > >> please
> > > > > > > >> > > > > > > > > > >> > > > >> elaborate
> > > > > > > >> > > > > > > > > > >> > > > >> > > your purpose of writing a
> > FLIP
> > > > > > > >> according to
> > > > > > > >> > > the
> > > > > > > >> > > > > > FLIP
> > > > > > > >> > > > > > > > > > >> > > > documentation[1]?
> > > > > > > >> > > > > > > > > > >> > > > >> > > Thanks!
> > > > > > > >> > > > > > > > > > >> > > > >> > >
> > > > > > > >> > > > > > > > > > >> > > > >> > > [1]
> > > > > > > >> > > > > > > > > > >> > > > >> > >
> > > > > > > >> > > > > > > > > > >> > > > >> > >
> > > > > > > >> > > > > > > > > > >> > > > >> >
> > > > > > > >> > > > > > > > > > >> > > > >>
> > > > > > > >> > > > > > > > > > >> > > >
> > > > > > > >> > > > > > > > > > >> > >
> > > > > > > >> > > > > > > > > > >> >
> > > > > > > >> > > > > > > > > > >>
> > > > > > > >> > > > > > > > > >
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > >
> > > > > > > >> > > > > > >
> > > > > > > >> > > > > >
> > > > > > > >> > > > >
> > > > > > > >> > > >
> > > > > > > >> > >
> > > > > > > >> >
> > > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/Flink+Improvement+Proposals
> > > > > > > >> > > > > > > > > > >> > > > >> > >
> > > > > > > >> > > > > > > > > > >> > > > >> > > Best regards,
> > > > > > > >> > > > > > > > > > >> > > > >> > > Jing
> > > > > > > >> > > > > > > > > > >> > > > >> > >
> > > > > > > >> > > > > > > > > > >> > > > >> > > On Wed, May 24, 2023 at
> > > 1:07 PM
> > > > > > > Aitozi <
> > > > > > > >> > > > > > > > > > gjying1314@gmail.com
> > > > > > > >> > > > > > > > > > >> >
> > > > > > > >> > > > > > > > > > >> > > > wrote:
> > > > > > > >> > > > > > > > > > >> > > > >> > >
> > > > > > > >> > > > > > > > > > >> > > > >> > > > May I ask for some
> > feedback
> > > > :D
> > > > > > > >> > > > > > > > > > >> > > > >> > > >
> > > > > > > >> > > > > > > > > > >> > > > >> > > > Thanks,
> > > > > > > >> > > > > > > > > > >> > > > >> > > > Aitozi
> > > > > > > >> > > > > > > > > > >> > > > >> > > >
> > > > > > > >> > > > > > > > > > >> > > > >> > > > Aitozi <
> > > gjying1314@gmail.com>
> > > > > > > >> > > 于2023年5月23日周二
> > > > > > > >> > > > > > > 19:14写道：
> > > > > > > >> > > > > > > > > > >> > > > >> > > > >
> > > > > > > >> > > > > > > > > > >> > > > >> > > > > Just catch an user
> case
> > > > report
> > > > > > > from
> > > > > > > >> > > Giannis
> > > > > > > >> > > > > > > Polyzos
> > > > > > > >> > > > > > > > > for
> > > > > > > >> > > > > > > > > > >> this
> > > > > > > >> > > > > > > > > > >> > > > >> usage:
> > > > > > > >> > > > > > > > > > >> > > > >> > > > >
> > > > > > > >> > > > > > > > > > >> > > > >> > > > >
> > > > > > > >> > > > > > > > > > >> > > >
> > > > > > > >> > > > > > > > >
> > > > > > > >> > >
> > > > > https://lists.apache.org/thread/qljwd40v5ntz6733cwcdr8s4z97b343b
> > > > > > > >> > > > > > > > > > >> > > > >> > > > >
> > > > > > > >> > > > > > > > > > >> > > > >> > > > > Aitozi <
> > > > gjying1314@gmail.com>
> > > > > > > >> > > > 于2023年5月23日周二
> > > > > > > >> > > > > > > > 17:45写道：
> > > > > > > >> > > > > > > > > > >> > > > >> > > > > >
> > > > > > > >> > > > > > > > > > >> > > > >> > > > > > Hi guys,
> > > > > > > >> > > > > > > > > > >> > > > >> > > > > >     I want to bring
> > up a
> > > > > > > >> discussion
> > > > > > > >> > > about
> > > > > > > >> > > > > > > adding
> > > > > > > >> > > > > > > > > > >> support
> > > > > > > >> > > > > > > > > > >> > of
> > > > > > > >> > > > > > > > > > >> > > > User
> > > > > > > >> > > > > > > > > > >> > > > >> > > > > > Defined
> > > AsyncTableFunction
> > > > > in
> > > > > > > >> Flink.
> > > > > > > >> > > > > > > > > > >> > > > >> > > > > > Currently, async
> table
> > > > > > function
> > > > > > > >> are
> > > > > > > >> > > > special
> > > > > > > >> > > > > > > > > functions
> > > > > > > >> > > > > > > > > > >> for
> > > > > > > >> > > > > > > > > > >> > > > table
> > > > > > > >> > > > > > > > > > >> > > > >> > > source
> > > > > > > >> > > > > > > > > > >> > > > >> > > > > > to perform
> > > > > > > >> > > > > > > > > > >> > > > >> > > > > > async lookup.
> However,
> > > > it's
> > > > > > > worth
> > > > > > > >> to
> > > > > > > >> > > > > support
> > > > > > > >> > > > > > > the
> > > > > > > >> > > > > > > > > user
> > > > > > > >> > > > > > > > > > >> > > defined
> > > > > > > >> > > > > > > > > > >> > > > >> async
> > > > > > > >> > > > > > > > > > >> > > > >> > > > > > table function.
> > > > > > > >> > > > > > > > > > >> > > > >> > > > > > Because, in this
> way,
> > > the
> > > > > end
> > > > > > > SQL
> > > > > > > >> > user
> > > > > > > >> > > > can
> > > > > > > >> > > > > > > > leverage
> > > > > > > >> > > > > > > > > > it
> > > > > > > >> > > > > > > > > > >> to
> > > > > > > >> > > > > > > > > > >> > > > >> perform
> > > > > > > >> > > > > > > > > > >> > > > >> > the
> > > > > > > >> > > > > > > > > > >> > > > >> > > > > > async operation
> > > > > > > >> > > > > > > > > > >> > > > >> > > > > > which is useful to
> > > maximum
> > > > > the
> > > > > > > >> system
> > > > > > > >> > > > > > > throughput
> > > > > > > >> > > > > > > > > > >> > especially
> > > > > > > >> > > > > > > > > > >> > > > for
> > > > > > > >> > > > > > > > > > >> > > > >> IO
> > > > > > > >> > > > > > > > > > >> > > > >> > > > > > bottleneck case.
> > > > > > > >> > > > > > > > > > >> > > > >> > > > > >
> > > > > > > >> > > > > > > > > > >> > > > >> > > > > > You can find some
> more
> > > > > detail
> > > > > > in
> > > > > > > >> [1].
> > > > > > > >> > > > > > > > > > >> > > > >> > > > > >
> > > > > > > >> > > > > > > > > > >> > > > >> > > > > > Looking forward to
> > > > feedback
> > > > > > > >> > > > > > > > > > >> > > > >> > > > > >
> > > > > > > >> > > > > > > > > > >> > > > >> > > > > >
> > > > > > > >> > > > > > > > > > >> > > > >> > > > > > [1]:
> > > > > > > >> > > > > > > > > > >> > > > >> > > >
> > > > > > > >> > > > > > > > > > >> > > > >> > >
> > > > > > > >> > > > > > > > > > >> > > > >> >
> > > > > > > >> > > > > > > > > > >> > > > >>
> > > > > > > >> > > > > > > > > > >> > > >
> > > > > > > >> > > > > > > > > > >> > >
> > > > > > > >> > > > > > > > > > >> >
> > > > > > > >> > > > > > > > > > >>
> > > > > > > >> > > > > > > > > >
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > >
> > > > > > > >> > > > > > >
> > > > > > > >> > > > > >
> > > > > > > >> > > > >
> > > > > > > >> > > >
> > > > > > > >> > >
> > > > > > > >> >
> > > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/%5BFLIP-313%5D+Add+support+of+User+Defined+AsyncTableFunction
> > > > > > > >> > > > > > > > > > >> > > > >> > > > > >
> > > > > > > >> > > > > > > > > > >> > > > >> > > > > > Thanks,
> > > > > > > >> > > > > > > > > > >> > > > >> > > > > > Aitozi.
> > > > > > > >> > > > > > > > > > >> > > > >> > > >
> > > > > > > >> > > > > > > > > > >> > > > >> > >
> > > > > > > >> > > > > > > > > > >> > > > >> >
> > > > > > > >> > > > > > > > > > >> > > > >>
> > > > > > > >> > > > > > > > > > >> > > > >
> > > > > > > >> > > > > > > > > > >> > > >
> > > > > > > >> > > > > > > > > > >> > >
> > > > > > > >> > > > > > > > > > >> >
> > > > > > > >> > > > > > > > > > >>
> > > > > > > >> > > > > > > > > > >
> > > > > > > >> > > > > > > > > >
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > >
> > > > > > > >> > > > > > >
> > > > > > > >> > > > > >
> > > > > > > >> > > > >
> > > > > > > >> > > >
> > > > > > > >> > >
> > > > > > > >> >
> > > > > > > >>
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] FLIP-313 Add support of User Defined AsyncTableFunction

Posted by Lincoln Lee <li...@gmail.com>.

Hi Aitozi,

Sorry for the lately reply here!  Supports async udtf(`AsyncTableFunction`)
directly in sql seems like an attractive feature, but there're two issues
that need to be addressed before we can be sure to add it:
1. As mentioned in the flip[1], the current lookup function can already
implement the requirements, but it requires implementing an extra
`LookupTableSource` and explicitly declaring the table schema (which can
help implementers the various push-down optimizations supported by the
planner). Does the async udtf bring any additional benefits besides a
lighter implementation?
2. FLIP-221[2] abstracts a reusable cache and metric infrastructure for
lookup sources, which are important to improve performance and
observability for high overhead external io scenarios, how do we integrate
and reuse these capabilities after introducing async udtf? Should users
migrate to the lookup source when they encounter similar requirements or
problems, or should we develop an additional set of similar mechanisms? (a
similarly case:  FLIP-234[3] introduced the retryable capability for lookup
join)

For the flip itself,
1. Considering the 'options' is already used as the dynamic table
options[4] in flink, the newly added query hint need a different name that
can be easier related to the lateral operation as the current join hints[5]
do.
2. For the async func example, since the target scenario is an external io
operation, it's better to add the `close` method to actively release
resources as a good example for users. Also in terms of the determinism of
a function, it is important to remind users that unless the behavior of the
function is deterministic, it needs to be explicitly declared as
non-deterministic.

[1].
https://cwiki.apache.org/confluence/display/FLINK/FLIP-313%3A+Add+support+of+User+Defined+AsyncTableFunction?src=contextnavpagetreemode
[2].
https://cwiki.apache.org/confluence/display/FLINK/FLIP-221%3A+Abstraction+for+lookup+source+cache+and+metric?src=contextnavpagetreemode
[3].
https://cwiki.apache.org/confluence/display/FLINK/FLIP-234%3A+Support+Retryable+Lookup+Join+To+Solve+Delayed+Updates+Issue+In+External+Systems?src=contextnavpagetreemode
[4].
https://cwiki.apache.org/confluence/display/FLINK/FLIP-113%3A+Supports+Dynamic+Table+Options+for+Flink+SQL?src=contextnavpagetreemode
[5].
https://cwiki.apache.org/confluence/display/FLINK/FLIP-229%3A+Introduces+Join+Hint+for+Flink+SQL+Batch+Job?src=contextnavpagetreemode

Best,
Lincoln Lee


Aitozi <gj...@gmail.com> 于2023年6月13日周二 11:30写道：

> Get your meaning now, thanks :)
>
> Best,
> Aitozi.
>
> Feng Jin <ji...@gmail.com> 于2023年6月13日周二 11:16写道：
>
> > Hi Aitozi,
> >
> > Sorry for the confusing description.
> >
> > What I meant was that if we need to remind users about tire safety
> issues,
> > we should introduce the new UDTF interface instead of executing the
> > original UDTF asynchronously. Therefore, I agree with introducing the
> > AsyncTableFunction.
> >
> > Best,
> > Feng
> >
> > On Tue, Jun 13, 2023 at 10:42 AM Aitozi <gj...@gmail.com> wrote:
> >
> > > Hi Feng,
> > >     Thanks for your question. We do not provide a way to switch the
> UDTF
> > > between sync and async way,
> > > So there should be no thread safety problem here.
> > >
> > > Best,
> > > Aitozi
> > >
> > > Feng Jin <ji...@gmail.com> 于2023年6月13日周二 10:31写道：
> > >
> > > > Hi Aitozi, We do need to remind users about thread safety issues.
> Thank
> > > you
> > > > for your efforts on this FLIP. I have no further questions.
> > > > Best, Feng
> > > >
> > > >
> > > > On Tue, Jun 13, 2023 at 6:05 AM Jing Ge <ji...@ververica.com.invalid>
> > > > wrote:
> > > >
> > > > > Hi Aitozi,
> > > > >
> > > > > Thanks for taking care of that part. I have no other concern.
> > > > >
> > > > > Best regards,
> > > > > Jing
> > > > >
> > > > >
> > > > > On Mon, Jun 12, 2023 at 5:38 PM Aitozi <gj...@gmail.com>
> wrote:
> > > > >
> > > > > > BTW, If there are no other more blocking issue / comments, I
> would
> > > like
> > > > > to
> > > > > > start a VOTE in another thread this wednesday 6.14
> > > > > >
> > > > > > Thanks,
> > > > > > Aitozi.
> > > > > >
> > > > > > Aitozi <gj...@gmail.com> 于2023年6月12日周一 23:34写道：
> > > > > >
> > > > > > > Hi, Jing,
> > > > > > >     Thanks for your explanation. I get your point now.
> > > > > > >
> > > > > > > For the performance part, I think it's a good idea to run with
> > > > > returning
> > > > > > a
> > > > > > > big table case, the memory consumption
> > > > > > > should be a point to be taken care about. Because in the
> ordered
> > > > mode,
> > > > > > the
> > > > > > > head element in buffer may affect the
> > > > > > > total memory consumption.
> > > > > > >
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Aitozi.
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > Jing Ge <ji...@ververica.com.invalid> 于2023年6月12日周一 20:28写道：
> > > > > > >
> > > > > > >> Hi Aitozi,
> > > > > > >>
> > > > > > >> Which key will be used for lookup is not an issue, only one
> row
> > > will
> > > > > be
> > > > > > >> required for each key in order to enrich it. True, it depends
> on
> > > the
> > > > > > >> implementation whether multiple rows or single row for each
> key
> > > will
> > > > > be
> > > > > > >> returned. However, for the lookup & enrichment scenario, one
> > > row/key
> > > > > is
> > > > > > >> recommended, otherwise, like I mentioned previously,
> enrichment
> > > > won't
> > > > > > >> work.
> > > > > > >>
> > > > > > >> I am a little bit concerned about returning a big table for
> each
> > > > key,
> > > > > > >> since
> > > > > > >> it will take the async call longer to return and need more
> > memory.
> > > > The
> > > > > > >> performance tests should cover this scenario. This is not a
> > > blocking
> > > > > > issue
> > > > > > >> for this FLIP.
> > > > > > >>
> > > > > > >> Best regards,
> > > > > > >> Jing
> > > > > > >>
> > > > > > >> On Sat, Jun 10, 2023 at 4:11 AM Aitozi <gj...@gmail.com>
> > > > wrote:
> > > > > > >>
> > > > > > >> > Hi Jing,
> > > > > > >> >     I means the join key is not necessary to be the primary
> > key
> > > or
> > > > > > >> unique
> > > > > > >> > index of the database.
> > > > > > >> > In this situation, we may queried out multi rows for one
> join
> > > > key. I
> > > > > > >> think
> > > > > > >> > that's why the
> > > > > > >> > LookupFunction#lookup will return a collection of RowData.
> > > > > > >> >
> > > > > > >> > BTW, I think the behavior of lookup join will not affect the
> > > > > semantic
> > > > > > of
> > > > > > >> > the async udtf.
> > > > > > >> > We use the Async TableFunction here and the table function
> can
> > > > > collect
> > > > > > >> > multiple rows.
> > > > > > >> >
> > > > > > >> > Thanks,
> > > > > > >> > Atiozi.
> > > > > > >> >
> > > > > > >> >
> > > > > > >> >
> > > > > > >> > Jing Ge <ji...@ververica.com.invalid> 于2023年6月10日周六 00:15写道：
> > > > > > >> >
> > > > > > >> > > Hi Aitozi,
> > > > > > >> > >
> > > > > > >> > > The keyRow used in this case contains all keys[1].
> > > > > > >> > >
> > > > > > >> > > Best regards,
> > > > > > >> > > Jing
> > > > > > >> > >
> > > > > > >> > > [1]
> > > > > > >> > >
> > > > > > >> > >
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/flink/blob/191ec6ca3943d7119f14837efe112e074d815c47/flink-table/flink-table-common/src/main/java/org/apache/flink/table/functions/LookupFunction.java#L49
> > > > > > >> > >
> > > > > > >> > >
> > > > > > >> > > On Fri, Jun 9, 2023 at 3:42 PM Aitozi <
> gjying1314@gmail.com
> > >
> > > > > wrote:
> > > > > > >> > >
> > > > > > >> > > > Hi Jing,
> > > > > > >> > > >
> > > > > > >> > > >      The performance test is added to the FLIP.
> > > > > > >> > > >
> > > > > > >> > > >      As I know, The lookup join can return multi rows,
> it
> > > > > depends
> > > > > > on
> > > > > > >> > > > whether  the join key
> > > > > > >> > > > is the primary key of the external database or not. The
> > > > `lookup`
> > > > > > [1]
> > > > > > >> > will
> > > > > > >> > > > return a collection of
> > > > > > >> > > > joined result, and each of them will be collected
> > > > > > >> > > >
> > > > > > >> > > >
> > > > > > >> > > > [1]:
> > > > > > >> > > >
> > > > > > >> > > >
> > > > > > >> > >
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/flink/blob/191ec6ca3943d7119f14837efe112e074d815c47/flink-table/flink-table-common/src/main/java/org/apache/flink/table/functions/LookupFunction.java#L52
> > > > > > >> > > >
> > > > > > >> > > >
> > > > > > >> > > > Thanks,
> > > > > > >> > > > Aitozi.
> > > > > > >> > > >
> > > > > > >> > > > Jing Ge <ji...@ververica.com.invalid> 于2023年6月9日周五
> > 17:05写道：
> > > > > > >> > > >
> > > > > > >> > > > > Hi Aitozi,
> > > > > > >> > > > >
> > > > > > >> > > > > Thanks for the feedback. Looking forward to the
> > > performance
> > > > > > tests.
> > > > > > >> > > > >
> > > > > > >> > > > > Afaik, lookup returns one row for each key [1] [2].
> > > > > > Conceptually,
> > > > > > >> the
> > > > > > >> > > > > lookup function is used to enrich column(s) from the
> > > > dimension
> > > > > > >> table.
> > > > > > >> > > If,
> > > > > > >> > > > > for the given key, there will be more than one row,
> > there
> > > > will
> > > > > > be
> > > > > > >> no
> > > > > > >> > > way
> > > > > > >> > > > to
> > > > > > >> > > > > know which row will be used to enrich the key.
> > > > > > >> > > > >
> > > > > > >> > > > > [1]
> > > > > > >> > > > >
> > > > > > >> > > > >
> > > > > > >> > > >
> > > > > > >> > >
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/flink/blob/191ec6ca3943d7119f14837efe112e074d815c47/flink-table/flink-table-common/src/main/java/org/apache/flink/table/functions/LookupFunction.java#L49
> > > > > > >> > > > > [2]
> > > > > > >> > > > >
> > > > > > >> > > > >
> > > > > > >> > > >
> > > > > > >> > >
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/flink/blob/191ec6ca3943d7119f14837efe112e074d815c47/flink-table/flink-table-common/src/main/java/org/apache/flink/table/functions/TableFunction.java#L196
> > > > > > >> > > > >
> > > > > > >> > > > > Best regards,
> > > > > > >> > > > > Jing
> > > > > > >> > > > >
> > > > > > >> > > > > On Fri, Jun 9, 2023 at 5:18 AM Aitozi <
> > > gjying1314@gmail.com
> > > > >
> > > > > > >> wrote:
> > > > > > >> > > > >
> > > > > > >> > > > > > Hi Jing
> > > > > > >> > > > > >     Thanks for your good questions. I have updated
> the
> > > > > example
> > > > > > >> to
> > > > > > >> > the
> > > > > > >> > > > > FLIP.
> > > > > > >> > > > > >
> > > > > > >> > > > > > > Only one row for each lookup
> > > > > > >> > > > > > lookup can also return multi rows, based on the
> query
> > > > > result.
> > > > > > >> [1]
> > > > > > >> > > > > >
> > > > > > >> > > > > > [1]:
> > > > > > >> > > > > >
> > > > > > >> > > > > >
> > > > > > >> > > > >
> > > > > > >> > > >
> > > > > > >> > >
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/flink/blob/191ec6ca3943d7119f14837efe112e074d815c47/flink-table/flink-table-common/src/main/java/org/apache/flink/table/functions/LookupFunction.java#L56
> > > > > > >> > > > > >
> > > > > > >> > > > > > > If we use async calls with lateral join, my gut
> > > feeling
> > > > is
> > > > > > >> > > > > > that we might have many more async calls than lookup
> > > > join. I
> > > > > > am
> > > > > > >> not
> > > > > > >> > > > > really
> > > > > > >> > > > > > sure if we will be facing potential issues in this
> > case
> > > or
> > > > > > not.
> > > > > > >> > > > > >
> > > > > > >> > > > > > IMO, the work pattern is similar to the lookup
> > function,
> > > > for
> > > > > > >> each
> > > > > > >> > row
> > > > > > >> > > > > from
> > > > > > >> > > > > > the left table,
> > > > > > >> > > > > > it will evaluate the eval method once, so the async
> > call
> > > > > > numbers
> > > > > > >> > will
> > > > > > >> > > > not
> > > > > > >> > > > > > change.
> > > > > > >> > > > > > and the maximum calls in flight is limited by the
> > Async
> > > > > > >> operators
> > > > > > >> > > > buffer
> > > > > > >> > > > > > capacity
> > > > > > >> > > > > > which will be controlled by the option.
> > > > > > >> > > > > >
> > > > > > >> > > > > > BTW, for the naming of these option, I updated the
> > FLIP
> > > > > about
> > > > > > >> this
> > > > > > >> > > you
> > > > > > >> > > > > can
> > > > > > >> > > > > > refer to
> > > > > > >> > > > > > the section of "ConfigOption" and "Rejected
> > > Alternatives"
> > > > > > >> > > > > >
> > > > > > >> > > > > > In the end, for the performance evaluation, I'd like
> > to
> > > do
> > > > > > some
> > > > > > >> > tests
> > > > > > >> > > > and
> > > > > > >> > > > > > will update it to the FLIP doc
> > > > > > >> > > > > >
> > > > > > >> > > > > > Thanks,
> > > > > > >> > > > > > Aitozi.
> > > > > > >> > > > > >
> > > > > > >> > > > > >
> > > > > > >> > > > > > Jing Ge <ji...@ververica.com.invalid> 于2023年6月9日周五
> > > > 07:23写道：
> > > > > > >> > > > > >
> > > > > > >> > > > > > > Hi Aitozi,
> > > > > > >> > > > > > >
> > > > > > >> > > > > > > Thanks for the clarification. The code example
> looks
> > > > > > >> > interesting. I
> > > > > > >> > > > > would
> > > > > > >> > > > > > > suggest adding them into the FLIP. The description
> > > with
> > > > > code
> > > > > > >> > > examples
> > > > > > >> > > > > > will
> > > > > > >> > > > > > > help readers understand the motivation and how to
> > use
> > > > it.
> > > > > > >> Afaiac,
> > > > > > >> > > it
> > > > > > >> > > > > is a
> > > > > > >> > > > > > > valid feature for Flink users.
> > > > > > >> > > > > > >
> > > > > > >> > > > > > > As we knew, lookup join is based on temporal join,
> > > i.e.
> > > > > FOR
> > > > > > >> > > > SYSTEM_TIME
> > > > > > >> > > > > > AS
> > > > > > >> > > > > > > OF which is also used in your code example.
> Temporal
> > > > join
> > > > > > >> > performs
> > > > > > >> > > > the
> > > > > > >> > > > > > > lookup based on the processing time match. Only
> one
> > > row
> > > > > for
> > > > > > >> each
> > > > > > >> > > > > > > lookup(afaiu, I need to check the source code to
> > > double
> > > > > > >> confirm)
> > > > > > >> > > will
> > > > > > >> > > > > > > return for further enrichment. One the other hand,
> > > > lateral
> > > > > > >> join
> > > > > > >> > > will
> > > > > > >> > > > > have
> > > > > > >> > > > > > > sub-queries correlated with every individual value
> > of
> > > > the
> > > > > > >> > reference
> > > > > > >> > > > > table
> > > > > > >> > > > > > > from the preceding part of the query and each sub
> > > query
> > > > > will
> > > > > > >> > return
> > > > > > >> > > > > > > multiple rows. If we use async calls with lateral
> > > join,
> > > > my
> > > > > > gut
> > > > > > >> > > > feeling
> > > > > > >> > > > > is
> > > > > > >> > > > > > > that we might have many more async calls than
> lookup
> > > > > join. I
> > > > > > >> am
> > > > > > >> > not
> > > > > > >> > > > > > really
> > > > > > >> > > > > > > sure if we will be facing potential issues in this
> > > case
> > > > or
> > > > > > >> not.
> > > > > > >> > > > > Possible
> > > > > > >> > > > > > > issues I can think of now e.g. too many PRC calls,
> > too
> > > > > many
> > > > > > >> async
> > > > > > >> > > > calls
> > > > > > >> > > > > > > processing, the sub query will return a table
> which
> > > > might
> > > > > be
> > > > > > >> > (too)
> > > > > > >> > > > big,
> > > > > > >> > > > > > and
> > > > > > >> > > > > > > might cause performance issues. I would suggest
> > > > preparing
> > > > > > some
> > > > > > >> > use
> > > > > > >> > > > > cases
> > > > > > >> > > > > > > and running some performance tests to check it.
> > These
> > > > are
> > > > > my
> > > > > > >> > > concerns
> > > > > > >> > > > > > about
> > > > > > >> > > > > > > using async calls with lateral join and I'd like
> to
> > > > share
> > > > > > with
> > > > > > >> > you,
> > > > > > >> > > > > happy
> > > > > > >> > > > > > > to discuss with you and hear different opinions,
> > > > hopefully
> > > > > > the
> > > > > > >> > > > > > > discussion could help me understand it more
> deeply.
> > > > Please
> > > > > > >> > correct
> > > > > > >> > > me
> > > > > > >> > > > > if
> > > > > > >> > > > > > I
> > > > > > >> > > > > > > am wrong.
> > > > > > >> > > > > > >
> > > > > > >> > > > > > > Best regards,
> > > > > > >> > > > > > > Jing
> > > > > > >> > > > > > >
> > > > > > >> > > > > > >
> > > > > > >> > > > > > > On Thu, Jun 8, 2023 at 7:22 AM Aitozi <
> > > > > gjying1314@gmail.com
> > > > > > >
> > > > > > >> > > wrote:
> > > > > > >> > > > > > >
> > > > > > >> > > > > > > > Hi Mason,
> > > > > > >> > > > > > > >     Thanks for your input. I think if we support
> > the
> > > > > user
> > > > > > >> > defined
> > > > > > >> > > > > async
> > > > > > >> > > > > > > > table function,
> > > > > > >> > > > > > > > user will be able to use it to hold a batch data
> > > then
> > > > > > >> handle it
> > > > > > >> > > at
> > > > > > >> > > > > one
> > > > > > >> > > > > > > time
> > > > > > >> > > > > > > > in the customized function.
> > > > > > >> > > > > > > >
> > > > > > >> > > > > > > > AsyncSink is meant for the sink operator. I have
> > not
> > > > > > figure
> > > > > > >> out
> > > > > > >> > > how
> > > > > > >> > > > > to
> > > > > > >> > > > > > > > integrate in this case.
> > > > > > >> > > > > > > >
> > > > > > >> > > > > > > > Thanks,
> > > > > > >> > > > > > > > Atiozi.
> > > > > > >> > > > > > > >
> > > > > > >> > > > > > > >
> > > > > > >> > > > > > > > Mason Chen <ma...@gmail.com>
> 于2023年6月8日周四
> > > > > 12:40写道：
> > > > > > >> > > > > > > >
> > > > > > >> > > > > > > > > Hi Aitozi,
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > > I think it makes sense to make it easier for
> SQL
> > > > users
> > > > > > to
> > > > > > >> > make
> > > > > > >> > > > > RPCs.
> > > > > > >> > > > > > Do
> > > > > > >> > > > > > > > you
> > > > > > >> > > > > > > > > think your proposal can extend to the ability
> to
> > > > batch
> > > > > > >> data
> > > > > > >> > for
> > > > > > >> > > > the
> > > > > > >> > > > > > > RPC?
> > > > > > >> > > > > > > > > This is also another common strategy to
> increase
> > > > > > >> throughput.
> > > > > > >> > > > Also,
> > > > > > >> > > > > > have
> > > > > > >> > > > > > > > you
> > > > > > >> > > > > > > > > considered solving this a bit differently by
> > > > > leveraging
> > > > > > >> > Flink's
> > > > > > >> > > > > > > > AsyncSink?
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > > Best,
> > > > > > >> > > > > > > > > Mason
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > > On Mon, Jun 5, 2023 at 1:50 AM Aitozi <
> > > > > > >> gjying1314@gmail.com>
> > > > > > >> > > > > wrote:
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > > > One more thing for discussion:
> > > > > > >> > > > > > > > > >
> > > > > > >> > > > > > > > > > In our internal implementation, we reuse the
> > > > option
> > > > > > >> > > > > > > > > > `table.exec.async-lookup.buffer-capacity`
> and
> > > > > > >> > > > > > > > > > `table.exec.async-lookup.timeout` to config
> > > > > > >> > > > > > > > > > the async udtf. Do you think we should add
> two
> > > > extra
> > > > > > >> option
> > > > > > >> > > to
> > > > > > >> > > > > > > > > distinguish
> > > > > > >> > > > > > > > > > from the lookup option such as
> > > > > > >> > > > > > > > > >
> > > > > > >> > > > > > > > > > `table.exec.async-udtf.buffer-capacity`
> > > > > > >> > > > > > > > > > `table.exec.async-udtf.timeout`
> > > > > > >> > > > > > > > > >
> > > > > > >> > > > > > > > > >
> > > > > > >> > > > > > > > > > Best,
> > > > > > >> > > > > > > > > > Aitozi.
> > > > > > >> > > > > > > > > >
> > > > > > >> > > > > > > > > >
> > > > > > >> > > > > > > > > >
> > > > > > >> > > > > > > > > > Aitozi <gj...@gmail.com> 于2023年6月5日周一
> > > > 12:20写道：
> > > > > > >> > > > > > > > > >
> > > > > > >> > > > > > > > > > > Hi Jing,
> > > > > > >> > > > > > > > > > >
> > > > > > >> > > > > > > > > > >     > what is the difference between the
> RPC
> > > > call
> > > > > or
> > > > > > >> > query
> > > > > > >> > > > you
> > > > > > >> > > > > > > > > mentioned
> > > > > > >> > > > > > > > > > > and the lookup in a very
> > > > > > >> > > > > > > > > > > general way
> > > > > > >> > > > > > > > > > >
> > > > > > >> > > > > > > > > > > I think the RPC call or query service is
> > quite
> > > > > > >> similar to
> > > > > > >> > > the
> > > > > > >> > > > > > > lookup
> > > > > > >> > > > > > > > > > join.
> > > > > > >> > > > > > > > > > > But lookup join should work
> > > > > > >> > > > > > > > > > > with `LookupTableSource`.
> > > > > > >> > > > > > > > > > >
> > > > > > >> > > > > > > > > > > Let's see how we can perform an async RPC
> > call
> > > > > with
> > > > > > >> > lookup
> > > > > > >> > > > > join:
> > > > > > >> > > > > > > > > > >
> > > > > > >> > > > > > > > > > > (1) Implement an AsyncTableFunction with
> RPC
> > > > call
> > > > > > >> logic.
> > > > > > >> > > > > > > > > > > (2) Implement a `LookupTableSource`
> > connector
> > > > run
> > > > > > with
> > > > > > >> > the
> > > > > > >> > > > > async
> > > > > > >> > > > > > > udtf
> > > > > > >> > > > > > > > > > > defined in (1).
> > > > > > >> > > > > > > > > > > (3) Then define a DDL of this look up
> table
> > in
> > > > SQL
> > > > > > >> > > > > > > > > > >
> > > > > > >> > > > > > > > > > > CREATE TEMPORARY TABLE Customers (
> > > > > > >> > > > > > > > > > >   id INT,
> > > > > > >> > > > > > > > > > >   name STRING,
> > > > > > >> > > > > > > > > > >   country STRING,
> > > > > > >> > > > > > > > > > >   zip STRING
> > > > > > >> > > > > > > > > > > ) WITH (
> > > > > > >> > > > > > > > > > >   'connector' = 'custom'
> > > > > > >> > > > > > > > > > > );
> > > > > > >> > > > > > > > > > >
> > > > > > >> > > > > > > > > > > (4) Run with the query as below:
> > > > > > >> > > > > > > > > > >
> > > > > > >> > > > > > > > > > > SELECT o.order_id, o.total, c.country,
> c.zip
> > > > > > >> > > > > > > > > > > FROM Orders AS o
> > > > > > >> > > > > > > > > > >   JOIN Customers FOR SYSTEM_TIME AS OF
> > > > o.proc_time
> > > > > > AS
> > > > > > >> c
> > > > > > >> > > > > > > > > > >     ON o.customer_id = c.id;
> > > > > > >> > > > > > > > > > >
> > > > > > >> > > > > > > > > > > This example is from doc
> > > > > > >> > > > > > > > > > > <
> > > > > > >> > > > > > > > > >
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > >
> > > > > > >> > > > > > >
> > > > > > >> > > > > >
> > > > > > >> > > > >
> > > > > > >> > > >
> > > > > > >> > >
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/dev/table/sql/queries/joins/#lookup-join
> > > > > > >> > > > > > > > > > >.You
> > > > > > >> > > > > > > > > > > can image the look up process as an async
> > RPC
> > > > call
> > > > > > >> > process.
> > > > > > >> > > > > > > > > > >
> > > > > > >> > > > > > > > > > > Let's see how we can perform an async RPC
> > call
> > > > > with
> > > > > > >> > lateral
> > > > > > >> > > > > join:
> > > > > > >> > > > > > > > > > >
> > > > > > >> > > > > > > > > > > (1) Implement an AsyncTableFunction with
> RPC
> > > > call
> > > > > > >> logic.
> > > > > > >> > > > > > > > > > > (2) Run query with
> > > > > > >> > > > > > > > > > >
> > > > > > >> > > > > > > > > > > Create function f1 as '...' ;
> > > > > > >> > > > > > > > > > >
> > > > > > >> > > > > > > > > > > SELECT o.order_id, o.total, c.country,
> c.zip
> > > > FROM
> > > > > > >> Orders
> > > > > > >> > > > > lateral
> > > > > > >> > > > > > > > table
> > > > > > >> > > > > > > > > > > (f1(order_id)) as T(...);
> > > > > > >> > > > > > > > > > >
> > > > > > >> > > > > > > > > > > As you can see, the lateral join version
> is
> > > more
> > > > > > >> simple
> > > > > > >> > and
> > > > > > >> > > > > > > intuitive
> > > > > > >> > > > > > > > > to
> > > > > > >> > > > > > > > > > > users. Users do not have to wrap a
> > > > > > >> > > > > > > > > > > LookupTableSource for the purpose of using
> > > async
> > > > > > udtf.
> > > > > > >> > > > > > > > > > >
> > > > > > >> > > > > > > > > > > In the end, We can also see the user
> defined
> > > > async
> > > > > > >> table
> > > > > > >> > > > > function
> > > > > > >> > > > > > > is
> > > > > > >> > > > > > > > an
> > > > > > >> > > > > > > > > > > enhancement of the current lateral table
> > join
> > > > > > >> > > > > > > > > > > which only supports sync lateral join now.
> > > > > > >> > > > > > > > > > >
> > > > > > >> > > > > > > > > > > Best,
> > > > > > >> > > > > > > > > > > Aitozi.
> > > > > > >> > > > > > > > > > >
> > > > > > >> > > > > > > > > > >
> > > > > > >> > > > > > > > > > > Jing Ge <ji...@ververica.com.invalid>
> > > > 于2023年6月2日周五
> > > > > > >> > 19:37写道：
> > > > > > >> > > > > > > > > > >
> > > > > > >> > > > > > > > > > >> Hi Aitozi,
> > > > > > >> > > > > > > > > > >>
> > > > > > >> > > > > > > > > > >> Thanks for the update. Just out of
> > curiosity,
> > > > > what
> > > > > > is
> > > > > > >> > the
> > > > > > >> > > > > > > difference
> > > > > > >> > > > > > > > > > >> between the RPC call or query you
> mentioned
> > > and
> > > > > the
> > > > > > >> > lookup
> > > > > > >> > > > in
> > > > > > >> > > > > a
> > > > > > >> > > > > > > very
> > > > > > >> > > > > > > > > > >> general way? Since Lateral join is used
> in
> > > the
> > > > > > FLIP.
> > > > > > >> Is
> > > > > > >> > > > there
> > > > > > >> > > > > > any
> > > > > > >> > > > > > > > > > special
> > > > > > >> > > > > > > > > > >> thought for that? Sorry for asking so
> many
> > > > > > questions.
> > > > > > >> > The
> > > > > > >> > > > FLIP
> > > > > > >> > > > > > > > > contains
> > > > > > >> > > > > > > > > > >> limited information to understand the
> > > > motivation.
> > > > > > >> > > > > > > > > > >>
> > > > > > >> > > > > > > > > > >> Best regards,
> > > > > > >> > > > > > > > > > >> Jing
> > > > > > >> > > > > > > > > > >>
> > > > > > >> > > > > > > > > > >> On Fri, Jun 2, 2023 at 3:48 AM Aitozi <
> > > > > > >> > > gjying1314@gmail.com
> > > > > > >> > > > >
> > > > > > >> > > > > > > wrote:
> > > > > > >> > > > > > > > > > >>
> > > > > > >> > > > > > > > > > >> > Hi Jing,
> > > > > > >> > > > > > > > > > >> >     I have updated the proposed changes
> > to
> > > > the
> > > > > > >> FLIP.
> > > > > > >> > > IMO,
> > > > > > >> > > > > > lookup
> > > > > > >> > > > > > > > has
> > > > > > >> > > > > > > > > > its
> > > > > > >> > > > > > > > > > >> > clear
> > > > > > >> > > > > > > > > > >> > async call requirement is due to its IO
> > > heavy
> > > > > > >> > operator.
> > > > > > >> > > In
> > > > > > >> > > > > our
> > > > > > >> > > > > > > > > usage,
> > > > > > >> > > > > > > > > > >> sql
> > > > > > >> > > > > > > > > > >> > users have
> > > > > > >> > > > > > > > > > >> > logic to do some RPC call or query the
> > > > > > third-party
> > > > > > >> > > service
> > > > > > >> > > > > > which
> > > > > > >> > > > > > > > is
> > > > > > >> > > > > > > > > > >> also IO
> > > > > > >> > > > > > > > > > >> > intensive.
> > > > > > >> > > > > > > > > > >> > In these case, we'd like to leverage
> the
> > > > async
> > > > > > >> > function
> > > > > > >> > > to
> > > > > > >> > > > > > > improve
> > > > > > >> > > > > > > > > the
> > > > > > >> > > > > > > > > > >> > throughput.
> > > > > > >> > > > > > > > > > >> >
> > > > > > >> > > > > > > > > > >> > Thanks,
> > > > > > >> > > > > > > > > > >> > Aitozi.
> > > > > > >> > > > > > > > > > >> >
> > > > > > >> > > > > > > > > > >> > Jing Ge <ji...@ververica.com.invalid>
> > > > > > 于2023年6月1日周四
> > > > > > >> > > > 22:55写道：
> > > > > > >> > > > > > > > > > >> >
> > > > > > >> > > > > > > > > > >> > > Hi Aitozi,
> > > > > > >> > > > > > > > > > >> > >
> > > > > > >> > > > > > > > > > >> > > Sorry for the late reply. Would you
> > like
> > > to
> > > > > > >> update
> > > > > > >> > the
> > > > > > >> > > > > > > proposed
> > > > > > >> > > > > > > > > > >> changes
> > > > > > >> > > > > > > > > > >> > > with more details into the FLIP too?
> > > > > > >> > > > > > > > > > >> > > I got your point. It looks like a
> > > rational
> > > > > > idea.
> > > > > > >> > > > However,
> > > > > > >> > > > > > > since
> > > > > > >> > > > > > > > > > lookup
> > > > > > >> > > > > > > > > > >> > has
> > > > > > >> > > > > > > > > > >> > > its clear async call requirement, are
> > > there
> > > > > any
> > > > > > >> real
> > > > > > >> > > use
> > > > > > >> > > > > > cases
> > > > > > >> > > > > > > > > that
> > > > > > >> > > > > > > > > > >> > > need this change? This will help us
> > > > > understand
> > > > > > >> the
> > > > > > >> > > > > > motivation.
> > > > > > >> > > > > > > > > After
> > > > > > >> > > > > > > > > > >> all,
> > > > > > >> > > > > > > > > > >> > > lateral join and temporal lookup
> > join[1]
> > > > are
> > > > > > >> quite
> > > > > > >> > > > > > different.
> > > > > > >> > > > > > > > > > >> > >
> > > > > > >> > > > > > > > > > >> > > Best regards,
> > > > > > >> > > > > > > > > > >> > > Jing
> > > > > > >> > > > > > > > > > >> > >
> > > > > > >> > > > > > > > > > >> > >
> > > > > > >> > > > > > > > > > >> > > [1]
> > > > > > >> > > > > > > > > > >> > >
> > > > > > >> > > > > > > > > > >> > >
> > > > > > >> > > > > > > > > > >> >
> > > > > > >> > > > > > > > > > >>
> > > > > > >> > > > > > > > > >
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > >
> > > > > > >> > > > > > >
> > > > > > >> > > > > >
> > > > > > >> > > > >
> > > > > > >> > > >
> > > > > > >> > >
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/flink/blob/d90a72da2fd601ca4e2a46700e91ec5b348de2ad/flink-table/flink-table-common/src/main/java/org/apache/flink/table/functions/AsyncTableFunction.java#L54
> > > > > > >> > > > > > > > > > >> > >
> > > > > > >> > > > > > > > > > >> > > On Wed, May 31, 2023 at 8:53 AM
> Aitozi
> > <
> > > > > > >> > > > > > gjying1314@gmail.com>
> > > > > > >> > > > > > > > > > wrote:
> > > > > > >> > > > > > > > > > >> > >
> > > > > > >> > > > > > > > > > >> > > > Hi Jing,
> > > > > > >> > > > > > > > > > >> > > >     What do you think about it? Can
> > we
> > > > move
> > > > > > >> > forward
> > > > > > >> > > > this
> > > > > > >> > > > > > > > > feature?
> > > > > > >> > > > > > > > > > >> > > >
> > > > > > >> > > > > > > > > > >> > > > Thanks,
> > > > > > >> > > > > > > > > > >> > > > Aitozi.
> > > > > > >> > > > > > > > > > >> > > >
> > > > > > >> > > > > > > > > > >> > > > Aitozi <gj...@gmail.com>
> > > > > 于2023年5月29日周一
> > > > > > >> > > 09:56写道：
> > > > > > >> > > > > > > > > > >> > > >
> > > > > > >> > > > > > > > > > >> > > > > Hi Jing,
> > > > > > >> > > > > > > > > > >> > > > >     > "Do you mean to support the
> > > > > > >> > > AyncTableFunction
> > > > > > >> > > > > > beyond
> > > > > > >> > > > > > > > the
> > > > > > >> > > > > > > > > > >> > > > > LookupTableSource?"
> > > > > > >> > > > > > > > > > >> > > > > Yes, I mean to support the
> > > > > > AyncTableFunction
> > > > > > >> > > beyond
> > > > > > >> > > > > the
> > > > > > >> > > > > > > > > > >> > > > LookupTableSource.
> > > > > > >> > > > > > > > > > >> > > > >
> > > > > > >> > > > > > > > > > >> > > > > The "AsyncTableFunction" is the
> > > > function
> > > > > > with
> > > > > > >> > > > ability
> > > > > > >> > > > > to
> > > > > > >> > > > > > > be
> > > > > > >> > > > > > > > > > >> executed
> > > > > > >> > > > > > > > > > >> > > > async
> > > > > > >> > > > > > > > > > >> > > > > (with AsyncWaitOperator).
> > > > > > >> > > > > > > > > > >> > > > > The async lookup join is a one of
> > > usage
> > > > > of
> > > > > > >> this.
> > > > > > >> > > So,
> > > > > > >> > > > > we
> > > > > > >> > > > > > > > don't
> > > > > > >> > > > > > > > > > >> have to
> > > > > > >> > > > > > > > > > >> > > > bind
> > > > > > >> > > > > > > > > > >> > > > > the AyncTableFunction with
> > > > > > LookupTableSource.
> > > > > > >> > > > > > > > > > >> > > > > If User-defined
> AsyncTableFunction
> > is
> > > > > > >> supported,
> > > > > > >> > > > user
> > > > > > >> > > > > > can
> > > > > > >> > > > > > > > > > directly
> > > > > > >> > > > > > > > > > >> > use
> > > > > > >> > > > > > > > > > >> > > > > lateral table syntax to perform
> > async
> > > > > > >> operation.
> > > > > > >> > > > > > > > > > >> > > > >
> > > > > > >> > > > > > > > > > >> > > > > > "It would be better if you
> could
> > > > > > elaborate
> > > > > > >> the
> > > > > > >> > > > > > proposed
> > > > > > >> > > > > > > > > > changes
> > > > > > >> > > > > > > > > > >> wrt
> > > > > > >> > > > > > > > > > >> > > the
> > > > > > >> > > > > > > > > > >> > > > > CorrelatedCodeGenerator with more
> > > > > details"
> > > > > > >> > > > > > > > > > >> > > > >
> > > > > > >> > > > > > > > > > >> > > > > In the proposal, we use lateral
> > table
> > > > > > syntax
> > > > > > >> to
> > > > > > >> > > > > support
> > > > > > >> > > > > > > the
> > > > > > >> > > > > > > > > > async
> > > > > > >> > > > > > > > > > >> > table
> > > > > > >> > > > > > > > > > >> > > > > function. So the planner will
> also
> > > > treat
> > > > > > this
> > > > > > >> > > > > statement
> > > > > > >> > > > > > > to a
> > > > > > >> > > > > > > > > > >> > > > > CommonExecCorrelate node. So the
> > > > runtime
> > > > > > code
> > > > > > >> > > should
> > > > > > >> > > > > be
> > > > > > >> > > > > > > > > > generated
> > > > > > >> > > > > > > > > > >> in
> > > > > > >> > > > > > > > > > >> > > > > CorrelatedCodeGenerator.
> > > > > > >> > > > > > > > > > >> > > > > In CorrelatedCodeGenerator, we
> will
> > > > know
> > > > > > the
> > > > > > >> > > > > > > TableFunction's
> > > > > > >> > > > > > > > > > Kind
> > > > > > >> > > > > > > > > > >> of
> > > > > > >> > > > > > > > > > >> > > > > `FunctionKind.Table` or
> > > > > > >> > `FunctionKind.ASYNC_TABLE`
> > > > > > >> > > > > > > > > > >> > > > > For  `FunctionKind.ASYNC_TABLE`
> we
> > > can
> > > > > > >> generate
> > > > > > >> > a
> > > > > > >> > > > > > > > > > >> AsyncWaitOperator
> > > > > > >> > > > > > > > > > >> > to
> > > > > > >> > > > > > > > > > >> > > > > execute the async table function.
> > > > > > >> > > > > > > > > > >> > > > >
> > > > > > >> > > > > > > > > > >> > > > >
> > > > > > >> > > > > > > > > > >> > > > > Thanks,
> > > > > > >> > > > > > > > > > >> > > > > Aitozi.
> > > > > > >> > > > > > > > > > >> > > > >
> > > > > > >> > > > > > > > > > >> > > > >
> > > > > > >> > > > > > > > > > >> > > > > Jing Ge
> <jing@ververica.com.invalid
> > >
> > > > > > >> > > 于2023年5月29日周一
> > > > > > >> > > > > > > 03:22写道：
> > > > > > >> > > > > > > > > > >> > > > >
> > > > > > >> > > > > > > > > > >> > > > >> Hi Aitozi,
> > > > > > >> > > > > > > > > > >> > > > >>
> > > > > > >> > > > > > > > > > >> > > > >> Thanks for the clarification.
> The
> > > > naming
> > > > > > >> > "Lookup"
> > > > > > >> > > > > might
> > > > > > >> > > > > > > > > suggest
> > > > > > >> > > > > > > > > > >> > using
> > > > > > >> > > > > > > > > > >> > > it
> > > > > > >> > > > > > > > > > >> > > > >> for table look up. But
> > conceptually
> > > > what
> > > > > > the
> > > > > > >> > > eval()
> > > > > > >> > > > > > > method
> > > > > > >> > > > > > > > > will
> > > > > > >> > > > > > > > > > >> do
> > > > > > >> > > > > > > > > > >> > is
> > > > > > >> > > > > > > > > > >> > > to
> > > > > > >> > > > > > > > > > >> > > > >> get a collection of results(Row,
> > > > > RowData)
> > > > > > >> from
> > > > > > >> > > the
> > > > > > >> > > > > > given
> > > > > > >> > > > > > > > > keys.
> > > > > > >> > > > > > > > > > >> How
> > > > > > >> > > > > > > > > > >> > it
> > > > > > >> > > > > > > > > > >> > > > will
> > > > > > >> > > > > > > > > > >> > > > >> be done depends on the
> > > implementation,
> > > > > > i.e.
> > > > > > >> you
> > > > > > >> > > can
> > > > > > >> > > > > > > > implement
> > > > > > >> > > > > > > > > > >> your
> > > > > > >> > > > > > > > > > >> > own
> > > > > > >> > > > > > > > > > >> > > > >> Source[1][2]. The example in the
> > > FLIP
> > > > > > >> should be
> > > > > > >> > > > able
> > > > > > >> > > > > to
> > > > > > >> > > > > > > be
> > > > > > >> > > > > > > > > > >> handled
> > > > > > >> > > > > > > > > > >> > in
> > > > > > >> > > > > > > > > > >> > > > this
> > > > > > >> > > > > > > > > > >> > > > >> way.
> > > > > > >> > > > > > > > > > >> > > > >>
> > > > > > >> > > > > > > > > > >> > > > >> Do you mean to support the
> > > > > > AyncTableFunction
> > > > > > >> > > beyond
> > > > > > >> > > > > the
> > > > > > >> > > > > > > > > > >> > > > LookupTableSource?
> > > > > > >> > > > > > > > > > >> > > > >> It would be better if you could
> > > > > elaborate
> > > > > > >> the
> > > > > > >> > > > > proposed
> > > > > > >> > > > > > > > > changes
> > > > > > >> > > > > > > > > > >> wrt
> > > > > > >> > > > > > > > > > >> > the
> > > > > > >> > > > > > > > > > >> > > > >> CorrelatedCodeGenerator with
> more
> > > > > details.
> > > > > > >> > > Thanks!
> > > > > > >> > > > > > > > > > >> > > > >>
> > > > > > >> > > > > > > > > > >> > > > >> Best regards,
> > > > > > >> > > > > > > > > > >> > > > >> Jing
> > > > > > >> > > > > > > > > > >> > > > >>
> > > > > > >> > > > > > > > > > >> > > > >> [1]
> > > > > > >> > > > > > > > > > >> > > > >>
> > > > > > >> > > > > > > > > > >> > > > >>
> > > > > > >> > > > > > > > > > >> > > >
> > > > > > >> > > > > > > > > > >> > >
> > > > > > >> > > > > > > > > > >> >
> > > > > > >> > > > > > > > > > >>
> > > > > > >> > > > > > > > > >
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > >
> > > > > > >> > > > > > >
> > > > > > >> > > > > >
> > > > > > >> > > > >
> > > > > > >> > > >
> > > > > > >> > >
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/flink/blob/678370b18e1b6c4a23e5ce08f8efd05675a0cc17/flink-table/flink-table-common/src/main/java/org/apache/flink/table/connector/source/LookupTableSource.java#L64
> > > > > > >> > > > > > > > > > >> > > > >> [2]
> > > > > > >> > > > > > > > > > >> > > > >>
> > > > > > >> > > > > > > > > > >> > > > >>
> > > > > > >> > > > > > > > > > >> > > >
> > > > > > >> > > > > > > > > > >> > >
> > > > > > >> > > > > > > > > > >> >
> > > > > > >> > > > > > > > > > >>
> > > > > > >> > > > > > > > > >
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > >
> > > > > > >> > > > > > >
> > > > > > >> > > > > >
> > > > > > >> > > > >
> > > > > > >> > > >
> > > > > > >> > >
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/flink/blob/678370b18e1b6c4a23e5ce08f8efd05675a0cc17/flink-table/flink-table-common/src/main/java/org/apache/flink/table/connector/source/AsyncTableFunctionProvider.java#L49
> > > > > > >> > > > > > > > > > >> > > > >>
> > > > > > >> > > > > > > > > > >> > > > >> On Sat, May 27, 2023 at 9:48 AM
> > > > Aitozi <
> > > > > > >> > > > > > > > gjying1314@gmail.com
> > > > > > >> > > > > > > > > >
> > > > > > >> > > > > > > > > > >> > wrote:
> > > > > > >> > > > > > > > > > >> > > > >>
> > > > > > >> > > > > > > > > > >> > > > >> > Hi Jing,
> > > > > > >> > > > > > > > > > >> > > > >> >     Thanks for your response.
> As
> > > > > stated
> > > > > > in
> > > > > > >> > the
> > > > > > >> > > > > FLIP,
> > > > > > >> > > > > > > the
> > > > > > >> > > > > > > > > > >> purpose
> > > > > > >> > > > > > > > > > >> > of
> > > > > > >> > > > > > > > > > >> > > > this
> > > > > > >> > > > > > > > > > >> > > > >> > FLIP is meant to support
> > > > > > >> > > > > > > > > > >> > > > >> > user-defined async table
> > function.
> > > > As
> > > > > > >> > described
> > > > > > >> > > > in
> > > > > > >> > > > > > > flink
> > > > > > >> > > > > > > > > > >> document
> > > > > > >> > > > > > > > > > >> > > [1]
> > > > > > >> > > > > > > > > > >> > > > >> >
> > > > > > >> > > > > > > > > > >> > > > >> > Async table functions are
> > special
> > > > > > >> functions
> > > > > > >> > for
> > > > > > >> > > > > table
> > > > > > >> > > > > > > > > sources
> > > > > > >> > > > > > > > > > >> that
> > > > > > >> > > > > > > > > > >> > > > >> perform
> > > > > > >> > > > > > > > > > >> > > > >> > > a lookup.
> > > > > > >> > > > > > > > > > >> > > > >> > >
> > > > > > >> > > > > > > > > > >> > > > >> >
> > > > > > >> > > > > > > > > > >> > > > >> > So end user can not directly
> > > define
> > > > > and
> > > > > > >> use
> > > > > > >> > > async
> > > > > > >> > > > > > table
> > > > > > >> > > > > > > > > > >> function
> > > > > > >> > > > > > > > > > >> > > now.
> > > > > > >> > > > > > > > > > >> > > > An
> > > > > > >> > > > > > > > > > >> > > > >> > user case is reported in [2]
> > > > > > >> > > > > > > > > > >> > > > >> >
> > > > > > >> > > > > > > > > > >> > > > >> > So, in conclusion, no new
> > > interface
> > > > is
> > > > > > >> > > > introduced,
> > > > > > >> > > > > > but
> > > > > > >> > > > > > > we
> > > > > > >> > > > > > > > > > >> extend
> > > > > > >> > > > > > > > > > >> > the
> > > > > > >> > > > > > > > > > >> > > > >> > ability to support
> user-defined
> > > > async
> > > > > > >> table
> > > > > > >> > > > > function.
> > > > > > >> > > > > > > > > > >> > > > >> >
> > > > > > >> > > > > > > > > > >> > > > >> > [1]:
> > > > > > >> > > > > > > > > > >> > > > >> >
> > > > > > >> > > > > > > > > > >> > > > >> >
> > > > > > >> > > > > > > > > > >> > > > >>
> > > > > > >> > > > > > > > > > >> > > >
> > > > > > >> > > > > > > > > > >> > >
> > > > > > >> > > > > > > > > > >> >
> > > > > > >> > > > > > > > > > >>
> > > > > > >> > > > > > > > > >
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > >
> > > > > > >> > > > > > >
> > > > > > >> > > > > >
> > > > > > >> > > > >
> > > > > > >> > > >
> > > > > > >> > >
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/dev/table/functions/udfs/
> > > > > > >> > > > > > > > > > >> > > > >> > [2]:
> > > > > > >> > > > > > > > > > >> > >
> > > > > > >> > > > > > > >
> > > > > > >> >
> > > https://lists.apache.org/thread/qljwd40v5ntz6733cwcdr8s4z97b343b
> > > > > > >> > > > > > > > > > >> > > > >> >
> > > > > > >> > > > > > > > > > >> > > > >> > Thanks.
> > > > > > >> > > > > > > > > > >> > > > >> > Aitozi.
> > > > > > >> > > > > > > > > > >> > > > >> >
> > > > > > >> > > > > > > > > > >> > > > >> >
> > > > > > >> > > > > > > > > > >> > > > >> > Jing Ge
> > > <jing@ververica.com.invalid
> > > > >
> > > > > > >> > > > 于2023年5月27日周六
> > > > > > >> > > > > > > > > 06:40写道：
> > > > > > >> > > > > > > > > > >> > > > >> >
> > > > > > >> > > > > > > > > > >> > > > >> > > Hi Aitozi,
> > > > > > >> > > > > > > > > > >> > > > >> > >
> > > > > > >> > > > > > > > > > >> > > > >> > > Thanks for your proposal. I
> am
> > > not
> > > > > > quite
> > > > > > >> > sure
> > > > > > >> > > > if
> > > > > > >> > > > > I
> > > > > > >> > > > > > > > > > understood
> > > > > > >> > > > > > > > > > >> > your
> > > > > > >> > > > > > > > > > >> > > > >> > thoughts
> > > > > > >> > > > > > > > > > >> > > > >> > > correctly. You described a
> > > special
> > > > > > case
> > > > > > >> > > > > > > implementation
> > > > > > >> > > > > > > > of
> > > > > > >> > > > > > > > > > the
> > > > > > >> > > > > > > > > > >> > > > >> > > AsyncTableFunction with on
> > > public
> > > > > API
> > > > > > >> > > changes.
> > > > > > >> > > > > > Would
> > > > > > >> > > > > > > > you
> > > > > > >> > > > > > > > > > >> please
> > > > > > >> > > > > > > > > > >> > > > >> elaborate
> > > > > > >> > > > > > > > > > >> > > > >> > > your purpose of writing a
> FLIP
> > > > > > >> according to
> > > > > > >> > > the
> > > > > > >> > > > > > FLIP
> > > > > > >> > > > > > > > > > >> > > > documentation[1]?
> > > > > > >> > > > > > > > > > >> > > > >> > > Thanks!
> > > > > > >> > > > > > > > > > >> > > > >> > >
> > > > > > >> > > > > > > > > > >> > > > >> > > [1]
> > > > > > >> > > > > > > > > > >> > > > >> > >
> > > > > > >> > > > > > > > > > >> > > > >> > >
> > > > > > >> > > > > > > > > > >> > > > >> >
> > > > > > >> > > > > > > > > > >> > > > >>
> > > > > > >> > > > > > > > > > >> > > >
> > > > > > >> > > > > > > > > > >> > >
> > > > > > >> > > > > > > > > > >> >
> > > > > > >> > > > > > > > > > >>
> > > > > > >> > > > > > > > > >
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > >
> > > > > > >> > > > > > >
> > > > > > >> > > > > >
> > > > > > >> > > > >
> > > > > > >> > > >
> > > > > > >> > >
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/Flink+Improvement+Proposals
> > > > > > >> > > > > > > > > > >> > > > >> > >
> > > > > > >> > > > > > > > > > >> > > > >> > > Best regards,
> > > > > > >> > > > > > > > > > >> > > > >> > > Jing
> > > > > > >> > > > > > > > > > >> > > > >> > >
> > > > > > >> > > > > > > > > > >> > > > >> > > On Wed, May 24, 2023 at
> > 1:07 PM
> > > > > > Aitozi <
> > > > > > >> > > > > > > > > > gjying1314@gmail.com
> > > > > > >> > > > > > > > > > >> >
> > > > > > >> > > > > > > > > > >> > > > wrote:
> > > > > > >> > > > > > > > > > >> > > > >> > >
> > > > > > >> > > > > > > > > > >> > > > >> > > > May I ask for some
> feedback
> > > :D
> > > > > > >> > > > > > > > > > >> > > > >> > > >
> > > > > > >> > > > > > > > > > >> > > > >> > > > Thanks,
> > > > > > >> > > > > > > > > > >> > > > >> > > > Aitozi
> > > > > > >> > > > > > > > > > >> > > > >> > > >
> > > > > > >> > > > > > > > > > >> > > > >> > > > Aitozi <
> > gjying1314@gmail.com>
> > > > > > >> > > 于2023年5月23日周二
> > > > > > >> > > > > > > 19:14写道：
> > > > > > >> > > > > > > > > > >> > > > >> > > > >
> > > > > > >> > > > > > > > > > >> > > > >> > > > > Just catch an user case
> > > report
> > > > > > from
> > > > > > >> > > Giannis
> > > > > > >> > > > > > > Polyzos
> > > > > > >> > > > > > > > > for
> > > > > > >> > > > > > > > > > >> this
> > > > > > >> > > > > > > > > > >> > > > >> usage:
> > > > > > >> > > > > > > > > > >> > > > >> > > > >
> > > > > > >> > > > > > > > > > >> > > > >> > > > >
> > > > > > >> > > > > > > > > > >> > > >
> > > > > > >> > > > > > > > >
> > > > > > >> > >
> > > > https://lists.apache.org/thread/qljwd40v5ntz6733cwcdr8s4z97b343b
> > > > > > >> > > > > > > > > > >> > > > >> > > > >
> > > > > > >> > > > > > > > > > >> > > > >> > > > > Aitozi <
> > > gjying1314@gmail.com>
> > > > > > >> > > > 于2023年5月23日周二
> > > > > > >> > > > > > > > 17:45写道：
> > > > > > >> > > > > > > > > > >> > > > >> > > > > >
> > > > > > >> > > > > > > > > > >> > > > >> > > > > > Hi guys,
> > > > > > >> > > > > > > > > > >> > > > >> > > > > >     I want to bring
> up a
> > > > > > >> discussion
> > > > > > >> > > about
> > > > > > >> > > > > > > adding
> > > > > > >> > > > > > > > > > >> support
> > > > > > >> > > > > > > > > > >> > of
> > > > > > >> > > > > > > > > > >> > > > User
> > > > > > >> > > > > > > > > > >> > > > >> > > > > > Defined
> > AsyncTableFunction
> > > > in
> > > > > > >> Flink.
> > > > > > >> > > > > > > > > > >> > > > >> > > > > > Currently, async table
> > > > > function
> > > > > > >> are
> > > > > > >> > > > special
> > > > > > >> > > > > > > > > functions
> > > > > > >> > > > > > > > > > >> for
> > > > > > >> > > > > > > > > > >> > > > table
> > > > > > >> > > > > > > > > > >> > > > >> > > source
> > > > > > >> > > > > > > > > > >> > > > >> > > > > > to perform
> > > > > > >> > > > > > > > > > >> > > > >> > > > > > async lookup. However,
> > > it's
> > > > > > worth
> > > > > > >> to
> > > > > > >> > > > > support
> > > > > > >> > > > > > > the
> > > > > > >> > > > > > > > > user
> > > > > > >> > > > > > > > > > >> > > defined
> > > > > > >> > > > > > > > > > >> > > > >> async
> > > > > > >> > > > > > > > > > >> > > > >> > > > > > table function.
> > > > > > >> > > > > > > > > > >> > > > >> > > > > > Because, in this way,
> > the
> > > > end
> > > > > > SQL
> > > > > > >> > user
> > > > > > >> > > > can
> > > > > > >> > > > > > > > leverage
> > > > > > >> > > > > > > > > > it
> > > > > > >> > > > > > > > > > >> to
> > > > > > >> > > > > > > > > > >> > > > >> perform
> > > > > > >> > > > > > > > > > >> > > > >> > the
> > > > > > >> > > > > > > > > > >> > > > >> > > > > > async operation
> > > > > > >> > > > > > > > > > >> > > > >> > > > > > which is useful to
> > maximum
> > > > the
> > > > > > >> system
> > > > > > >> > > > > > > throughput
> > > > > > >> > > > > > > > > > >> > especially
> > > > > > >> > > > > > > > > > >> > > > for
> > > > > > >> > > > > > > > > > >> > > > >> IO
> > > > > > >> > > > > > > > > > >> > > > >> > > > > > bottleneck case.
> > > > > > >> > > > > > > > > > >> > > > >> > > > > >
> > > > > > >> > > > > > > > > > >> > > > >> > > > > > You can find some more
> > > > detail
> > > > > in
> > > > > > >> [1].
> > > > > > >> > > > > > > > > > >> > > > >> > > > > >
> > > > > > >> > > > > > > > > > >> > > > >> > > > > > Looking forward to
> > > feedback
> > > > > > >> > > > > > > > > > >> > > > >> > > > > >
> > > > > > >> > > > > > > > > > >> > > > >> > > > > >
> > > > > > >> > > > > > > > > > >> > > > >> > > > > > [1]:
> > > > > > >> > > > > > > > > > >> > > > >> > > >
> > > > > > >> > > > > > > > > > >> > > > >> > >
> > > > > > >> > > > > > > > > > >> > > > >> >
> > > > > > >> > > > > > > > > > >> > > > >>
> > > > > > >> > > > > > > > > > >> > > >
> > > > > > >> > > > > > > > > > >> > >
> > > > > > >> > > > > > > > > > >> >
> > > > > > >> > > > > > > > > > >>
> > > > > > >> > > > > > > > > >
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > >
> > > > > > >> > > > > > >
> > > > > > >> > > > > >
> > > > > > >> > > > >
> > > > > > >> > > >
> > > > > > >> > >
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/%5BFLIP-313%5D+Add+support+of+User+Defined+AsyncTableFunction
> > > > > > >> > > > > > > > > > >> > > > >> > > > > >
> > > > > > >> > > > > > > > > > >> > > > >> > > > > > Thanks,
> > > > > > >> > > > > > > > > > >> > > > >> > > > > > Aitozi.
> > > > > > >> > > > > > > > > > >> > > > >> > > >
> > > > > > >> > > > > > > > > > >> > > > >> > >
> > > > > > >> > > > > > > > > > >> > > > >> >
> > > > > > >> > > > > > > > > > >> > > > >>
> > > > > > >> > > > > > > > > > >> > > > >
> > > > > > >> > > > > > > > > > >> > > >
> > > > > > >> > > > > > > > > > >> > >
> > > > > > >> > > > > > > > > > >> >
> > > > > > >> > > > > > > > > > >>
> > > > > > >> > > > > > > > > > >
> > > > > > >> > > > > > > > > >
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > >
> > > > > > >> > > > > > >
> > > > > > >> > > > > >
> > > > > > >> > > > >
> > > > > > >> > > >
> > > > > > >> > >
> > > > > > >> >
> > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] FLIP-313 Add support of User Defined AsyncTableFunction

Posted by Aitozi <gj...@gmail.com>.

Get your meaning now, thanks :)

Best,
Aitozi.

Feng Jin <ji...@gmail.com> 于2023年6月13日周二 11:16写道：

> Hi Aitozi,
>
> Sorry for the confusing description.
>
> What I meant was that if we need to remind users about tire safety issues,
> we should introduce the new UDTF interface instead of executing the
> original UDTF asynchronously. Therefore, I agree with introducing the
> AsyncTableFunction.
>
> Best,
> Feng
>
> On Tue, Jun 13, 2023 at 10:42 AM Aitozi <gj...@gmail.com> wrote:
>
> > Hi Feng,
> >     Thanks for your question. We do not provide a way to switch the UDTF
> > between sync and async way,
> > So there should be no thread safety problem here.
> >
> > Best,
> > Aitozi
> >
> > Feng Jin <ji...@gmail.com> 于2023年6月13日周二 10:31写道：
> >
> > > Hi Aitozi, We do need to remind users about thread safety issues. Thank
> > you
> > > for your efforts on this FLIP. I have no further questions.
> > > Best, Feng
> > >
> > >
> > > On Tue, Jun 13, 2023 at 6:05 AM Jing Ge <ji...@ververica.com.invalid>
> > > wrote:
> > >
> > > > Hi Aitozi,
> > > >
> > > > Thanks for taking care of that part. I have no other concern.
> > > >
> > > > Best regards,
> > > > Jing
> > > >
> > > >
> > > > On Mon, Jun 12, 2023 at 5:38 PM Aitozi <gj...@gmail.com> wrote:
> > > >
> > > > > BTW, If there are no other more blocking issue / comments, I would
> > like
> > > > to
> > > > > start a VOTE in another thread this wednesday 6.14
> > > > >
> > > > > Thanks,
> > > > > Aitozi.
> > > > >
> > > > > Aitozi <gj...@gmail.com> 于2023年6月12日周一 23:34写道：
> > > > >
> > > > > > Hi, Jing,
> > > > > >     Thanks for your explanation. I get your point now.
> > > > > >
> > > > > > For the performance part, I think it's a good idea to run with
> > > > returning
> > > > > a
> > > > > > big table case, the memory consumption
> > > > > > should be a point to be taken care about. Because in the ordered
> > > mode,
> > > > > the
> > > > > > head element in buffer may affect the
> > > > > > total memory consumption.
> > > > > >
> > > > > >
> > > > > > Thanks,
> > > > > > Aitozi.
> > > > > >
> > > > > >
> > > > > >
> > > > > > Jing Ge <ji...@ververica.com.invalid> 于2023年6月12日周一 20:28写道：
> > > > > >
> > > > > >> Hi Aitozi,
> > > > > >>
> > > > > >> Which key will be used for lookup is not an issue, only one row
> > will
> > > > be
> > > > > >> required for each key in order to enrich it. True, it depends on
> > the
> > > > > >> implementation whether multiple rows or single row for each key
> > will
> > > > be
> > > > > >> returned. However, for the lookup & enrichment scenario, one
> > row/key
> > > > is
> > > > > >> recommended, otherwise, like I mentioned previously, enrichment
> > > won't
> > > > > >> work.
> > > > > >>
> > > > > >> I am a little bit concerned about returning a big table for each
> > > key,
> > > > > >> since
> > > > > >> it will take the async call longer to return and need more
> memory.
> > > The
> > > > > >> performance tests should cover this scenario. This is not a
> > blocking
> > > > > issue
> > > > > >> for this FLIP.
> > > > > >>
> > > > > >> Best regards,
> > > > > >> Jing
> > > > > >>
> > > > > >> On Sat, Jun 10, 2023 at 4:11 AM Aitozi <gj...@gmail.com>
> > > wrote:
> > > > > >>
> > > > > >> > Hi Jing,
> > > > > >> >     I means the join key is not necessary to be the primary
> key
> > or
> > > > > >> unique
> > > > > >> > index of the database.
> > > > > >> > In this situation, we may queried out multi rows for one join
> > > key. I
> > > > > >> think
> > > > > >> > that's why the
> > > > > >> > LookupFunction#lookup will return a collection of RowData.
> > > > > >> >
> > > > > >> > BTW, I think the behavior of lookup join will not affect the
> > > > semantic
> > > > > of
> > > > > >> > the async udtf.
> > > > > >> > We use the Async TableFunction here and the table function can
> > > > collect
> > > > > >> > multiple rows.
> > > > > >> >
> > > > > >> > Thanks,
> > > > > >> > Atiozi.
> > > > > >> >
> > > > > >> >
> > > > > >> >
> > > > > >> > Jing Ge <ji...@ververica.com.invalid> 于2023年6月10日周六 00:15写道：
> > > > > >> >
> > > > > >> > > Hi Aitozi,
> > > > > >> > >
> > > > > >> > > The keyRow used in this case contains all keys[1].
> > > > > >> > >
> > > > > >> > > Best regards,
> > > > > >> > > Jing
> > > > > >> > >
> > > > > >> > > [1]
> > > > > >> > >
> > > > > >> > >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
> https://github.com/apache/flink/blob/191ec6ca3943d7119f14837efe112e074d815c47/flink-table/flink-table-common/src/main/java/org/apache/flink/table/functions/LookupFunction.java#L49
> > > > > >> > >
> > > > > >> > >
> > > > > >> > > On Fri, Jun 9, 2023 at 3:42 PM Aitozi <gjying1314@gmail.com
> >
> > > > wrote:
> > > > > >> > >
> > > > > >> > > > Hi Jing,
> > > > > >> > > >
> > > > > >> > > >      The performance test is added to the FLIP.
> > > > > >> > > >
> > > > > >> > > >      As I know, The lookup join can return multi rows, it
> > > > depends
> > > > > on
> > > > > >> > > > whether  the join key
> > > > > >> > > > is the primary key of the external database or not. The
> > > `lookup`
> > > > > [1]
> > > > > >> > will
> > > > > >> > > > return a collection of
> > > > > >> > > > joined result, and each of them will be collected
> > > > > >> > > >
> > > > > >> > > >
> > > > > >> > > > [1]:
> > > > > >> > > >
> > > > > >> > > >
> > > > > >> > >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
> https://github.com/apache/flink/blob/191ec6ca3943d7119f14837efe112e074d815c47/flink-table/flink-table-common/src/main/java/org/apache/flink/table/functions/LookupFunction.java#L52
> > > > > >> > > >
> > > > > >> > > >
> > > > > >> > > > Thanks,
> > > > > >> > > > Aitozi.
> > > > > >> > > >
> > > > > >> > > > Jing Ge <ji...@ververica.com.invalid> 于2023年6月9日周五
> 17:05写道：
> > > > > >> > > >
> > > > > >> > > > > Hi Aitozi,
> > > > > >> > > > >
> > > > > >> > > > > Thanks for the feedback. Looking forward to the
> > performance
> > > > > tests.
> > > > > >> > > > >
> > > > > >> > > > > Afaik, lookup returns one row for each key [1] [2].
> > > > > Conceptually,
> > > > > >> the
> > > > > >> > > > > lookup function is used to enrich column(s) from the
> > > dimension
> > > > > >> table.
> > > > > >> > > If,
> > > > > >> > > > > for the given key, there will be more than one row,
> there
> > > will
> > > > > be
> > > > > >> no
> > > > > >> > > way
> > > > > >> > > > to
> > > > > >> > > > > know which row will be used to enrich the key.
> > > > > >> > > > >
> > > > > >> > > > > [1]
> > > > > >> > > > >
> > > > > >> > > > >
> > > > > >> > > >
> > > > > >> > >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
> https://github.com/apache/flink/blob/191ec6ca3943d7119f14837efe112e074d815c47/flink-table/flink-table-common/src/main/java/org/apache/flink/table/functions/LookupFunction.java#L49
> > > > > >> > > > > [2]
> > > > > >> > > > >
> > > > > >> > > > >
> > > > > >> > > >
> > > > > >> > >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
> https://github.com/apache/flink/blob/191ec6ca3943d7119f14837efe112e074d815c47/flink-table/flink-table-common/src/main/java/org/apache/flink/table/functions/TableFunction.java#L196
> > > > > >> > > > >
> > > > > >> > > > > Best regards,
> > > > > >> > > > > Jing
> > > > > >> > > > >
> > > > > >> > > > > On Fri, Jun 9, 2023 at 5:18 AM Aitozi <
> > gjying1314@gmail.com
> > > >
> > > > > >> wrote:
> > > > > >> > > > >
> > > > > >> > > > > > Hi Jing
> > > > > >> > > > > >     Thanks for your good questions. I have updated the
> > > > example
> > > > > >> to
> > > > > >> > the
> > > > > >> > > > > FLIP.
> > > > > >> > > > > >
> > > > > >> > > > > > > Only one row for each lookup
> > > > > >> > > > > > lookup can also return multi rows, based on the query
> > > > result.
> > > > > >> [1]
> > > > > >> > > > > >
> > > > > >> > > > > > [1]:
> > > > > >> > > > > >
> > > > > >> > > > > >
> > > > > >> > > > >
> > > > > >> > > >
> > > > > >> > >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
> https://github.com/apache/flink/blob/191ec6ca3943d7119f14837efe112e074d815c47/flink-table/flink-table-common/src/main/java/org/apache/flink/table/functions/LookupFunction.java#L56
> > > > > >> > > > > >
> > > > > >> > > > > > > If we use async calls with lateral join, my gut
> > feeling
> > > is
> > > > > >> > > > > > that we might have many more async calls than lookup
> > > join. I
> > > > > am
> > > > > >> not
> > > > > >> > > > > really
> > > > > >> > > > > > sure if we will be facing potential issues in this
> case
> > or
> > > > > not.
> > > > > >> > > > > >
> > > > > >> > > > > > IMO, the work pattern is similar to the lookup
> function,
> > > for
> > > > > >> each
> > > > > >> > row
> > > > > >> > > > > from
> > > > > >> > > > > > the left table,
> > > > > >> > > > > > it will evaluate the eval method once, so the async
> call
> > > > > numbers
> > > > > >> > will
> > > > > >> > > > not
> > > > > >> > > > > > change.
> > > > > >> > > > > > and the maximum calls in flight is limited by the
> Async
> > > > > >> operators
> > > > > >> > > > buffer
> > > > > >> > > > > > capacity
> > > > > >> > > > > > which will be controlled by the option.
> > > > > >> > > > > >
> > > > > >> > > > > > BTW, for the naming of these option, I updated the
> FLIP
> > > > about
> > > > > >> this
> > > > > >> > > you
> > > > > >> > > > > can
> > > > > >> > > > > > refer to
> > > > > >> > > > > > the section of "ConfigOption" and "Rejected
> > Alternatives"
> > > > > >> > > > > >
> > > > > >> > > > > > In the end, for the performance evaluation, I'd like
> to
> > do
> > > > > some
> > > > > >> > tests
> > > > > >> > > > and
> > > > > >> > > > > > will update it to the FLIP doc
> > > > > >> > > > > >
> > > > > >> > > > > > Thanks,
> > > > > >> > > > > > Aitozi.
> > > > > >> > > > > >
> > > > > >> > > > > >
> > > > > >> > > > > > Jing Ge <ji...@ververica.com.invalid> 于2023年6月9日周五
> > > 07:23写道：
> > > > > >> > > > > >
> > > > > >> > > > > > > Hi Aitozi,
> > > > > >> > > > > > >
> > > > > >> > > > > > > Thanks for the clarification. The code example looks
> > > > > >> > interesting. I
> > > > > >> > > > > would
> > > > > >> > > > > > > suggest adding them into the FLIP. The description
> > with
> > > > code
> > > > > >> > > examples
> > > > > >> > > > > > will
> > > > > >> > > > > > > help readers understand the motivation and how to
> use
> > > it.
> > > > > >> Afaiac,
> > > > > >> > > it
> > > > > >> > > > > is a
> > > > > >> > > > > > > valid feature for Flink users.
> > > > > >> > > > > > >
> > > > > >> > > > > > > As we knew, lookup join is based on temporal join,
> > i.e.
> > > > FOR
> > > > > >> > > > SYSTEM_TIME
> > > > > >> > > > > > AS
> > > > > >> > > > > > > OF which is also used in your code example. Temporal
> > > join
> > > > > >> > performs
> > > > > >> > > > the
> > > > > >> > > > > > > lookup based on the processing time match. Only one
> > row
> > > > for
> > > > > >> each
> > > > > >> > > > > > > lookup(afaiu, I need to check the source code to
> > double
> > > > > >> confirm)
> > > > > >> > > will
> > > > > >> > > > > > > return for further enrichment. One the other hand,
> > > lateral
> > > > > >> join
> > > > > >> > > will
> > > > > >> > > > > have
> > > > > >> > > > > > > sub-queries correlated with every individual value
> of
> > > the
> > > > > >> > reference
> > > > > >> > > > > table
> > > > > >> > > > > > > from the preceding part of the query and each sub
> > query
> > > > will
> > > > > >> > return
> > > > > >> > > > > > > multiple rows. If we use async calls with lateral
> > join,
> > > my
> > > > > gut
> > > > > >> > > > feeling
> > > > > >> > > > > is
> > > > > >> > > > > > > that we might have many more async calls than lookup
> > > > join. I
> > > > > >> am
> > > > > >> > not
> > > > > >> > > > > > really
> > > > > >> > > > > > > sure if we will be facing potential issues in this
> > case
> > > or
> > > > > >> not.
> > > > > >> > > > > Possible
> > > > > >> > > > > > > issues I can think of now e.g. too many PRC calls,
> too
> > > > many
> > > > > >> async
> > > > > >> > > > calls
> > > > > >> > > > > > > processing, the sub query will return a table which
> > > might
> > > > be
> > > > > >> > (too)
> > > > > >> > > > big,
> > > > > >> > > > > > and
> > > > > >> > > > > > > might cause performance issues. I would suggest
> > > preparing
> > > > > some
> > > > > >> > use
> > > > > >> > > > > cases
> > > > > >> > > > > > > and running some performance tests to check it.
> These
> > > are
> > > > my
> > > > > >> > > concerns
> > > > > >> > > > > > about
> > > > > >> > > > > > > using async calls with lateral join and I'd like to
> > > share
> > > > > with
> > > > > >> > you,
> > > > > >> > > > > happy
> > > > > >> > > > > > > to discuss with you and hear different opinions,
> > > hopefully
> > > > > the
> > > > > >> > > > > > > discussion could help me understand it more deeply.
> > > Please
> > > > > >> > correct
> > > > > >> > > me
> > > > > >> > > > > if
> > > > > >> > > > > > I
> > > > > >> > > > > > > am wrong.
> > > > > >> > > > > > >
> > > > > >> > > > > > > Best regards,
> > > > > >> > > > > > > Jing
> > > > > >> > > > > > >
> > > > > >> > > > > > >
> > > > > >> > > > > > > On Thu, Jun 8, 2023 at 7:22 AM Aitozi <
> > > > gjying1314@gmail.com
> > > > > >
> > > > > >> > > wrote:
> > > > > >> > > > > > >
> > > > > >> > > > > > > > Hi Mason,
> > > > > >> > > > > > > >     Thanks for your input. I think if we support
> the
> > > > user
> > > > > >> > defined
> > > > > >> > > > > async
> > > > > >> > > > > > > > table function,
> > > > > >> > > > > > > > user will be able to use it to hold a batch data
> > then
> > > > > >> handle it
> > > > > >> > > at
> > > > > >> > > > > one
> > > > > >> > > > > > > time
> > > > > >> > > > > > > > in the customized function.
> > > > > >> > > > > > > >
> > > > > >> > > > > > > > AsyncSink is meant for the sink operator. I have
> not
> > > > > figure
> > > > > >> out
> > > > > >> > > how
> > > > > >> > > > > to
> > > > > >> > > > > > > > integrate in this case.
> > > > > >> > > > > > > >
> > > > > >> > > > > > > > Thanks,
> > > > > >> > > > > > > > Atiozi.
> > > > > >> > > > > > > >
> > > > > >> > > > > > > >
> > > > > >> > > > > > > > Mason Chen <ma...@gmail.com> 于2023年6月8日周四
> > > > 12:40写道：
> > > > > >> > > > > > > >
> > > > > >> > > > > > > > > Hi Aitozi,
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > > I think it makes sense to make it easier for SQL
> > > users
> > > > > to
> > > > > >> > make
> > > > > >> > > > > RPCs.
> > > > > >> > > > > > Do
> > > > > >> > > > > > > > you
> > > > > >> > > > > > > > > think your proposal can extend to the ability to
> > > batch
> > > > > >> data
> > > > > >> > for
> > > > > >> > > > the
> > > > > >> > > > > > > RPC?
> > > > > >> > > > > > > > > This is also another common strategy to increase
> > > > > >> throughput.
> > > > > >> > > > Also,
> > > > > >> > > > > > have
> > > > > >> > > > > > > > you
> > > > > >> > > > > > > > > considered solving this a bit differently by
> > > > leveraging
> > > > > >> > Flink's
> > > > > >> > > > > > > > AsyncSink?
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > > Best,
> > > > > >> > > > > > > > > Mason
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > > On Mon, Jun 5, 2023 at 1:50 AM Aitozi <
> > > > > >> gjying1314@gmail.com>
> > > > > >> > > > > wrote:
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > > > One more thing for discussion:
> > > > > >> > > > > > > > > >
> > > > > >> > > > > > > > > > In our internal implementation, we reuse the
> > > option
> > > > > >> > > > > > > > > > `table.exec.async-lookup.buffer-capacity` and
> > > > > >> > > > > > > > > > `table.exec.async-lookup.timeout` to config
> > > > > >> > > > > > > > > > the async udtf. Do you think we should add two
> > > extra
> > > > > >> option
> > > > > >> > > to
> > > > > >> > > > > > > > > distinguish
> > > > > >> > > > > > > > > > from the lookup option such as
> > > > > >> > > > > > > > > >
> > > > > >> > > > > > > > > > `table.exec.async-udtf.buffer-capacity`
> > > > > >> > > > > > > > > > `table.exec.async-udtf.timeout`
> > > > > >> > > > > > > > > >
> > > > > >> > > > > > > > > >
> > > > > >> > > > > > > > > > Best,
> > > > > >> > > > > > > > > > Aitozi.
> > > > > >> > > > > > > > > >
> > > > > >> > > > > > > > > >
> > > > > >> > > > > > > > > >
> > > > > >> > > > > > > > > > Aitozi <gj...@gmail.com> 于2023年6月5日周一
> > > 12:20写道：
> > > > > >> > > > > > > > > >
> > > > > >> > > > > > > > > > > Hi Jing,
> > > > > >> > > > > > > > > > >
> > > > > >> > > > > > > > > > >     > what is the difference between the RPC
> > > call
> > > > or
> > > > > >> > query
> > > > > >> > > > you
> > > > > >> > > > > > > > > mentioned
> > > > > >> > > > > > > > > > > and the lookup in a very
> > > > > >> > > > > > > > > > > general way
> > > > > >> > > > > > > > > > >
> > > > > >> > > > > > > > > > > I think the RPC call or query service is
> quite
> > > > > >> similar to
> > > > > >> > > the
> > > > > >> > > > > > > lookup
> > > > > >> > > > > > > > > > join.
> > > > > >> > > > > > > > > > > But lookup join should work
> > > > > >> > > > > > > > > > > with `LookupTableSource`.
> > > > > >> > > > > > > > > > >
> > > > > >> > > > > > > > > > > Let's see how we can perform an async RPC
> call
> > > > with
> > > > > >> > lookup
> > > > > >> > > > > join:
> > > > > >> > > > > > > > > > >
> > > > > >> > > > > > > > > > > (1) Implement an AsyncTableFunction with RPC
> > > call
> > > > > >> logic.
> > > > > >> > > > > > > > > > > (2) Implement a `LookupTableSource`
> connector
> > > run
> > > > > with
> > > > > >> > the
> > > > > >> > > > > async
> > > > > >> > > > > > > udtf
> > > > > >> > > > > > > > > > > defined in (1).
> > > > > >> > > > > > > > > > > (3) Then define a DDL of this look up table
> in
> > > SQL
> > > > > >> > > > > > > > > > >
> > > > > >> > > > > > > > > > > CREATE TEMPORARY TABLE Customers (
> > > > > >> > > > > > > > > > >   id INT,
> > > > > >> > > > > > > > > > >   name STRING,
> > > > > >> > > > > > > > > > >   country STRING,
> > > > > >> > > > > > > > > > >   zip STRING
> > > > > >> > > > > > > > > > > ) WITH (
> > > > > >> > > > > > > > > > >   'connector' = 'custom'
> > > > > >> > > > > > > > > > > );
> > > > > >> > > > > > > > > > >
> > > > > >> > > > > > > > > > > (4) Run with the query as below:
> > > > > >> > > > > > > > > > >
> > > > > >> > > > > > > > > > > SELECT o.order_id, o.total, c.country, c.zip
> > > > > >> > > > > > > > > > > FROM Orders AS o
> > > > > >> > > > > > > > > > >   JOIN Customers FOR SYSTEM_TIME AS OF
> > > o.proc_time
> > > > > AS
> > > > > >> c
> > > > > >> > > > > > > > > > >     ON o.customer_id = c.id;
> > > > > >> > > > > > > > > > >
> > > > > >> > > > > > > > > > > This example is from doc
> > > > > >> > > > > > > > > > > <
> > > > > >> > > > > > > > > >
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > >
> > > > > >> > > > > > >
> > > > > >> > > > > >
> > > > > >> > > > >
> > > > > >> > > >
> > > > > >> > >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
> https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/dev/table/sql/queries/joins/#lookup-join
> > > > > >> > > > > > > > > > >.You
> > > > > >> > > > > > > > > > > can image the look up process as an async
> RPC
> > > call
> > > > > >> > process.
> > > > > >> > > > > > > > > > >
> > > > > >> > > > > > > > > > > Let's see how we can perform an async RPC
> call
> > > > with
> > > > > >> > lateral
> > > > > >> > > > > join:
> > > > > >> > > > > > > > > > >
> > > > > >> > > > > > > > > > > (1) Implement an AsyncTableFunction with RPC
> > > call
> > > > > >> logic.
> > > > > >> > > > > > > > > > > (2) Run query with
> > > > > >> > > > > > > > > > >
> > > > > >> > > > > > > > > > > Create function f1 as '...' ;
> > > > > >> > > > > > > > > > >
> > > > > >> > > > > > > > > > > SELECT o.order_id, o.total, c.country, c.zip
> > > FROM
> > > > > >> Orders
> > > > > >> > > > > lateral
> > > > > >> > > > > > > > table
> > > > > >> > > > > > > > > > > (f1(order_id)) as T(...);
> > > > > >> > > > > > > > > > >
> > > > > >> > > > > > > > > > > As you can see, the lateral join version is
> > more
> > > > > >> simple
> > > > > >> > and
> > > > > >> > > > > > > intuitive
> > > > > >> > > > > > > > > to
> > > > > >> > > > > > > > > > > users. Users do not have to wrap a
> > > > > >> > > > > > > > > > > LookupTableSource for the purpose of using
> > async
> > > > > udtf.
> > > > > >> > > > > > > > > > >
> > > > > >> > > > > > > > > > > In the end, We can also see the user defined
> > > async
> > > > > >> table
> > > > > >> > > > > function
> > > > > >> > > > > > > is
> > > > > >> > > > > > > > an
> > > > > >> > > > > > > > > > > enhancement of the current lateral table
> join
> > > > > >> > > > > > > > > > > which only supports sync lateral join now.
> > > > > >> > > > > > > > > > >
> > > > > >> > > > > > > > > > > Best,
> > > > > >> > > > > > > > > > > Aitozi.
> > > > > >> > > > > > > > > > >
> > > > > >> > > > > > > > > > >
> > > > > >> > > > > > > > > > > Jing Ge <ji...@ververica.com.invalid>
> > > 于2023年6月2日周五
> > > > > >> > 19:37写道：
> > > > > >> > > > > > > > > > >
> > > > > >> > > > > > > > > > >> Hi Aitozi,
> > > > > >> > > > > > > > > > >>
> > > > > >> > > > > > > > > > >> Thanks for the update. Just out of
> curiosity,
> > > > what
> > > > > is
> > > > > >> > the
> > > > > >> > > > > > > difference
> > > > > >> > > > > > > > > > >> between the RPC call or query you mentioned
> > and
> > > > the
> > > > > >> > lookup
> > > > > >> > > > in
> > > > > >> > > > > a
> > > > > >> > > > > > > very
> > > > > >> > > > > > > > > > >> general way? Since Lateral join is used in
> > the
> > > > > FLIP.
> > > > > >> Is
> > > > > >> > > > there
> > > > > >> > > > > > any
> > > > > >> > > > > > > > > > special
> > > > > >> > > > > > > > > > >> thought for that? Sorry for asking so many
> > > > > questions.
> > > > > >> > The
> > > > > >> > > > FLIP
> > > > > >> > > > > > > > > contains
> > > > > >> > > > > > > > > > >> limited information to understand the
> > > motivation.
> > > > > >> > > > > > > > > > >>
> > > > > >> > > > > > > > > > >> Best regards,
> > > > > >> > > > > > > > > > >> Jing
> > > > > >> > > > > > > > > > >>
> > > > > >> > > > > > > > > > >> On Fri, Jun 2, 2023 at 3:48 AM Aitozi <
> > > > > >> > > gjying1314@gmail.com
> > > > > >> > > > >
> > > > > >> > > > > > > wrote:
> > > > > >> > > > > > > > > > >>
> > > > > >> > > > > > > > > > >> > Hi Jing,
> > > > > >> > > > > > > > > > >> >     I have updated the proposed changes
> to
> > > the
> > > > > >> FLIP.
> > > > > >> > > IMO,
> > > > > >> > > > > > lookup
> > > > > >> > > > > > > > has
> > > > > >> > > > > > > > > > its
> > > > > >> > > > > > > > > > >> > clear
> > > > > >> > > > > > > > > > >> > async call requirement is due to its IO
> > heavy
> > > > > >> > operator.
> > > > > >> > > In
> > > > > >> > > > > our
> > > > > >> > > > > > > > > usage,
> > > > > >> > > > > > > > > > >> sql
> > > > > >> > > > > > > > > > >> > users have
> > > > > >> > > > > > > > > > >> > logic to do some RPC call or query the
> > > > > third-party
> > > > > >> > > service
> > > > > >> > > > > > which
> > > > > >> > > > > > > > is
> > > > > >> > > > > > > > > > >> also IO
> > > > > >> > > > > > > > > > >> > intensive.
> > > > > >> > > > > > > > > > >> > In these case, we'd like to leverage the
> > > async
> > > > > >> > function
> > > > > >> > > to
> > > > > >> > > > > > > improve
> > > > > >> > > > > > > > > the
> > > > > >> > > > > > > > > > >> > throughput.
> > > > > >> > > > > > > > > > >> >
> > > > > >> > > > > > > > > > >> > Thanks,
> > > > > >> > > > > > > > > > >> > Aitozi.
> > > > > >> > > > > > > > > > >> >
> > > > > >> > > > > > > > > > >> > Jing Ge <ji...@ververica.com.invalid>
> > > > > 于2023年6月1日周四
> > > > > >> > > > 22:55写道：
> > > > > >> > > > > > > > > > >> >
> > > > > >> > > > > > > > > > >> > > Hi Aitozi,
> > > > > >> > > > > > > > > > >> > >
> > > > > >> > > > > > > > > > >> > > Sorry for the late reply. Would you
> like
> > to
> > > > > >> update
> > > > > >> > the
> > > > > >> > > > > > > proposed
> > > > > >> > > > > > > > > > >> changes
> > > > > >> > > > > > > > > > >> > > with more details into the FLIP too?
> > > > > >> > > > > > > > > > >> > > I got your point. It looks like a
> > rational
> > > > > idea.
> > > > > >> > > > However,
> > > > > >> > > > > > > since
> > > > > >> > > > > > > > > > lookup
> > > > > >> > > > > > > > > > >> > has
> > > > > >> > > > > > > > > > >> > > its clear async call requirement, are
> > there
> > > > any
> > > > > >> real
> > > > > >> > > use
> > > > > >> > > > > > cases
> > > > > >> > > > > > > > > that
> > > > > >> > > > > > > > > > >> > > need this change? This will help us
> > > > understand
> > > > > >> the
> > > > > >> > > > > > motivation.
> > > > > >> > > > > > > > > After
> > > > > >> > > > > > > > > > >> all,
> > > > > >> > > > > > > > > > >> > > lateral join and temporal lookup
> join[1]
> > > are
> > > > > >> quite
> > > > > >> > > > > > different.
> > > > > >> > > > > > > > > > >> > >
> > > > > >> > > > > > > > > > >> > > Best regards,
> > > > > >> > > > > > > > > > >> > > Jing
> > > > > >> > > > > > > > > > >> > >
> > > > > >> > > > > > > > > > >> > >
> > > > > >> > > > > > > > > > >> > > [1]
> > > > > >> > > > > > > > > > >> > >
> > > > > >> > > > > > > > > > >> > >
> > > > > >> > > > > > > > > > >> >
> > > > > >> > > > > > > > > > >>
> > > > > >> > > > > > > > > >
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > >
> > > > > >> > > > > > >
> > > > > >> > > > > >
> > > > > >> > > > >
> > > > > >> > > >
> > > > > >> > >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
> https://github.com/apache/flink/blob/d90a72da2fd601ca4e2a46700e91ec5b348de2ad/flink-table/flink-table-common/src/main/java/org/apache/flink/table/functions/AsyncTableFunction.java#L54
> > > > > >> > > > > > > > > > >> > >
> > > > > >> > > > > > > > > > >> > > On Wed, May 31, 2023 at 8:53 AM Aitozi
> <
> > > > > >> > > > > > gjying1314@gmail.com>
> > > > > >> > > > > > > > > > wrote:
> > > > > >> > > > > > > > > > >> > >
> > > > > >> > > > > > > > > > >> > > > Hi Jing,
> > > > > >> > > > > > > > > > >> > > >     What do you think about it? Can
> we
> > > move
> > > > > >> > forward
> > > > > >> > > > this
> > > > > >> > > > > > > > > feature?
> > > > > >> > > > > > > > > > >> > > >
> > > > > >> > > > > > > > > > >> > > > Thanks,
> > > > > >> > > > > > > > > > >> > > > Aitozi.
> > > > > >> > > > > > > > > > >> > > >
> > > > > >> > > > > > > > > > >> > > > Aitozi <gj...@gmail.com>
> > > > 于2023年5月29日周一
> > > > > >> > > 09:56写道：
> > > > > >> > > > > > > > > > >> > > >
> > > > > >> > > > > > > > > > >> > > > > Hi Jing,
> > > > > >> > > > > > > > > > >> > > > >     > "Do you mean to support the
> > > > > >> > > AyncTableFunction
> > > > > >> > > > > > beyond
> > > > > >> > > > > > > > the
> > > > > >> > > > > > > > > > >> > > > > LookupTableSource?"
> > > > > >> > > > > > > > > > >> > > > > Yes, I mean to support the
> > > > > AyncTableFunction
> > > > > >> > > beyond
> > > > > >> > > > > the
> > > > > >> > > > > > > > > > >> > > > LookupTableSource.
> > > > > >> > > > > > > > > > >> > > > >
> > > > > >> > > > > > > > > > >> > > > > The "AsyncTableFunction" is the
> > > function
> > > > > with
> > > > > >> > > > ability
> > > > > >> > > > > to
> > > > > >> > > > > > > be
> > > > > >> > > > > > > > > > >> executed
> > > > > >> > > > > > > > > > >> > > > async
> > > > > >> > > > > > > > > > >> > > > > (with AsyncWaitOperator).
> > > > > >> > > > > > > > > > >> > > > > The async lookup join is a one of
> > usage
> > > > of
> > > > > >> this.
> > > > > >> > > So,
> > > > > >> > > > > we
> > > > > >> > > > > > > > don't
> > > > > >> > > > > > > > > > >> have to
> > > > > >> > > > > > > > > > >> > > > bind
> > > > > >> > > > > > > > > > >> > > > > the AyncTableFunction with
> > > > > LookupTableSource.
> > > > > >> > > > > > > > > > >> > > > > If User-defined AsyncTableFunction
> is
> > > > > >> supported,
> > > > > >> > > > user
> > > > > >> > > > > > can
> > > > > >> > > > > > > > > > directly
> > > > > >> > > > > > > > > > >> > use
> > > > > >> > > > > > > > > > >> > > > > lateral table syntax to perform
> async
> > > > > >> operation.
> > > > > >> > > > > > > > > > >> > > > >
> > > > > >> > > > > > > > > > >> > > > > > "It would be better if you could
> > > > > elaborate
> > > > > >> the
> > > > > >> > > > > > proposed
> > > > > >> > > > > > > > > > changes
> > > > > >> > > > > > > > > > >> wrt
> > > > > >> > > > > > > > > > >> > > the
> > > > > >> > > > > > > > > > >> > > > > CorrelatedCodeGenerator with more
> > > > details"
> > > > > >> > > > > > > > > > >> > > > >
> > > > > >> > > > > > > > > > >> > > > > In the proposal, we use lateral
> table
> > > > > syntax
> > > > > >> to
> > > > > >> > > > > support
> > > > > >> > > > > > > the
> > > > > >> > > > > > > > > > async
> > > > > >> > > > > > > > > > >> > table
> > > > > >> > > > > > > > > > >> > > > > function. So the planner will also
> > > treat
> > > > > this
> > > > > >> > > > > statement
> > > > > >> > > > > > > to a
> > > > > >> > > > > > > > > > >> > > > > CommonExecCorrelate node. So the
> > > runtime
> > > > > code
> > > > > >> > > should
> > > > > >> > > > > be
> > > > > >> > > > > > > > > > generated
> > > > > >> > > > > > > > > > >> in
> > > > > >> > > > > > > > > > >> > > > > CorrelatedCodeGenerator.
> > > > > >> > > > > > > > > > >> > > > > In CorrelatedCodeGenerator, we will
> > > know
> > > > > the
> > > > > >> > > > > > > TableFunction's
> > > > > >> > > > > > > > > > Kind
> > > > > >> > > > > > > > > > >> of
> > > > > >> > > > > > > > > > >> > > > > `FunctionKind.Table` or
> > > > > >> > `FunctionKind.ASYNC_TABLE`
> > > > > >> > > > > > > > > > >> > > > > For  `FunctionKind.ASYNC_TABLE` we
> > can
> > > > > >> generate
> > > > > >> > a
> > > > > >> > > > > > > > > > >> AsyncWaitOperator
> > > > > >> > > > > > > > > > >> > to
> > > > > >> > > > > > > > > > >> > > > > execute the async table function.
> > > > > >> > > > > > > > > > >> > > > >
> > > > > >> > > > > > > > > > >> > > > >
> > > > > >> > > > > > > > > > >> > > > > Thanks,
> > > > > >> > > > > > > > > > >> > > > > Aitozi.
> > > > > >> > > > > > > > > > >> > > > >
> > > > > >> > > > > > > > > > >> > > > >
> > > > > >> > > > > > > > > > >> > > > > Jing Ge <jing@ververica.com.invalid
> >
> > > > > >> > > 于2023年5月29日周一
> > > > > >> > > > > > > 03:22写道：
> > > > > >> > > > > > > > > > >> > > > >
> > > > > >> > > > > > > > > > >> > > > >> Hi Aitozi,
> > > > > >> > > > > > > > > > >> > > > >>
> > > > > >> > > > > > > > > > >> > > > >> Thanks for the clarification. The
> > > naming
> > > > > >> > "Lookup"
> > > > > >> > > > > might
> > > > > >> > > > > > > > > suggest
> > > > > >> > > > > > > > > > >> > using
> > > > > >> > > > > > > > > > >> > > it
> > > > > >> > > > > > > > > > >> > > > >> for table look up. But
> conceptually
> > > what
> > > > > the
> > > > > >> > > eval()
> > > > > >> > > > > > > method
> > > > > >> > > > > > > > > will
> > > > > >> > > > > > > > > > >> do
> > > > > >> > > > > > > > > > >> > is
> > > > > >> > > > > > > > > > >> > > to
> > > > > >> > > > > > > > > > >> > > > >> get a collection of results(Row,
> > > > RowData)
> > > > > >> from
> > > > > >> > > the
> > > > > >> > > > > > given
> > > > > >> > > > > > > > > keys.
> > > > > >> > > > > > > > > > >> How
> > > > > >> > > > > > > > > > >> > it
> > > > > >> > > > > > > > > > >> > > > will
> > > > > >> > > > > > > > > > >> > > > >> be done depends on the
> > implementation,
> > > > > i.e.
> > > > > >> you
> > > > > >> > > can
> > > > > >> > > > > > > > implement
> > > > > >> > > > > > > > > > >> your
> > > > > >> > > > > > > > > > >> > own
> > > > > >> > > > > > > > > > >> > > > >> Source[1][2]. The example in the
> > FLIP
> > > > > >> should be
> > > > > >> > > > able
> > > > > >> > > > > to
> > > > > >> > > > > > > be
> > > > > >> > > > > > > > > > >> handled
> > > > > >> > > > > > > > > > >> > in
> > > > > >> > > > > > > > > > >> > > > this
> > > > > >> > > > > > > > > > >> > > > >> way.
> > > > > >> > > > > > > > > > >> > > > >>
> > > > > >> > > > > > > > > > >> > > > >> Do you mean to support the
> > > > > AyncTableFunction
> > > > > >> > > beyond
> > > > > >> > > > > the
> > > > > >> > > > > > > > > > >> > > > LookupTableSource?
> > > > > >> > > > > > > > > > >> > > > >> It would be better if you could
> > > > elaborate
> > > > > >> the
> > > > > >> > > > > proposed
> > > > > >> > > > > > > > > changes
> > > > > >> > > > > > > > > > >> wrt
> > > > > >> > > > > > > > > > >> > the
> > > > > >> > > > > > > > > > >> > > > >> CorrelatedCodeGenerator with more
> > > > details.
> > > > > >> > > Thanks!
> > > > > >> > > > > > > > > > >> > > > >>
> > > > > >> > > > > > > > > > >> > > > >> Best regards,
> > > > > >> > > > > > > > > > >> > > > >> Jing
> > > > > >> > > > > > > > > > >> > > > >>
> > > > > >> > > > > > > > > > >> > > > >> [1]
> > > > > >> > > > > > > > > > >> > > > >>
> > > > > >> > > > > > > > > > >> > > > >>
> > > > > >> > > > > > > > > > >> > > >
> > > > > >> > > > > > > > > > >> > >
> > > > > >> > > > > > > > > > >> >
> > > > > >> > > > > > > > > > >>
> > > > > >> > > > > > > > > >
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > >
> > > > > >> > > > > > >
> > > > > >> > > > > >
> > > > > >> > > > >
> > > > > >> > > >
> > > > > >> > >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
> https://github.com/apache/flink/blob/678370b18e1b6c4a23e5ce08f8efd05675a0cc17/flink-table/flink-table-common/src/main/java/org/apache/flink/table/connector/source/LookupTableSource.java#L64
> > > > > >> > > > > > > > > > >> > > > >> [2]
> > > > > >> > > > > > > > > > >> > > > >>
> > > > > >> > > > > > > > > > >> > > > >>
> > > > > >> > > > > > > > > > >> > > >
> > > > > >> > > > > > > > > > >> > >
> > > > > >> > > > > > > > > > >> >
> > > > > >> > > > > > > > > > >>
> > > > > >> > > > > > > > > >
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > >
> > > > > >> > > > > > >
> > > > > >> > > > > >
> > > > > >> > > > >
> > > > > >> > > >
> > > > > >> > >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
> https://github.com/apache/flink/blob/678370b18e1b6c4a23e5ce08f8efd05675a0cc17/flink-table/flink-table-common/src/main/java/org/apache/flink/table/connector/source/AsyncTableFunctionProvider.java#L49
> > > > > >> > > > > > > > > > >> > > > >>
> > > > > >> > > > > > > > > > >> > > > >> On Sat, May 27, 2023 at 9:48 AM
> > > Aitozi <
> > > > > >> > > > > > > > gjying1314@gmail.com
> > > > > >> > > > > > > > > >
> > > > > >> > > > > > > > > > >> > wrote:
> > > > > >> > > > > > > > > > >> > > > >>
> > > > > >> > > > > > > > > > >> > > > >> > Hi Jing,
> > > > > >> > > > > > > > > > >> > > > >> >     Thanks for your response. As
> > > > stated
> > > > > in
> > > > > >> > the
> > > > > >> > > > > FLIP,
> > > > > >> > > > > > > the
> > > > > >> > > > > > > > > > >> purpose
> > > > > >> > > > > > > > > > >> > of
> > > > > >> > > > > > > > > > >> > > > this
> > > > > >> > > > > > > > > > >> > > > >> > FLIP is meant to support
> > > > > >> > > > > > > > > > >> > > > >> > user-defined async table
> function.
> > > As
> > > > > >> > described
> > > > > >> > > > in
> > > > > >> > > > > > > flink
> > > > > >> > > > > > > > > > >> document
> > > > > >> > > > > > > > > > >> > > [1]
> > > > > >> > > > > > > > > > >> > > > >> >
> > > > > >> > > > > > > > > > >> > > > >> > Async table functions are
> special
> > > > > >> functions
> > > > > >> > for
> > > > > >> > > > > table
> > > > > >> > > > > > > > > sources
> > > > > >> > > > > > > > > > >> that
> > > > > >> > > > > > > > > > >> > > > >> perform
> > > > > >> > > > > > > > > > >> > > > >> > > a lookup.
> > > > > >> > > > > > > > > > >> > > > >> > >
> > > > > >> > > > > > > > > > >> > > > >> >
> > > > > >> > > > > > > > > > >> > > > >> > So end user can not directly
> > define
> > > > and
> > > > > >> use
> > > > > >> > > async
> > > > > >> > > > > > table
> > > > > >> > > > > > > > > > >> function
> > > > > >> > > > > > > > > > >> > > now.
> > > > > >> > > > > > > > > > >> > > > An
> > > > > >> > > > > > > > > > >> > > > >> > user case is reported in [2]
> > > > > >> > > > > > > > > > >> > > > >> >
> > > > > >> > > > > > > > > > >> > > > >> > So, in conclusion, no new
> > interface
> > > is
> > > > > >> > > > introduced,
> > > > > >> > > > > > but
> > > > > >> > > > > > > we
> > > > > >> > > > > > > > > > >> extend
> > > > > >> > > > > > > > > > >> > the
> > > > > >> > > > > > > > > > >> > > > >> > ability to support user-defined
> > > async
> > > > > >> table
> > > > > >> > > > > function.
> > > > > >> > > > > > > > > > >> > > > >> >
> > > > > >> > > > > > > > > > >> > > > >> > [1]:
> > > > > >> > > > > > > > > > >> > > > >> >
> > > > > >> > > > > > > > > > >> > > > >> >
> > > > > >> > > > > > > > > > >> > > > >>
> > > > > >> > > > > > > > > > >> > > >
> > > > > >> > > > > > > > > > >> > >
> > > > > >> > > > > > > > > > >> >
> > > > > >> > > > > > > > > > >>
> > > > > >> > > > > > > > > >
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > >
> > > > > >> > > > > > >
> > > > > >> > > > > >
> > > > > >> > > > >
> > > > > >> > > >
> > > > > >> > >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
> https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/dev/table/functions/udfs/
> > > > > >> > > > > > > > > > >> > > > >> > [2]:
> > > > > >> > > > > > > > > > >> > >
> > > > > >> > > > > > > >
> > > > > >> >
> > https://lists.apache.org/thread/qljwd40v5ntz6733cwcdr8s4z97b343b
> > > > > >> > > > > > > > > > >> > > > >> >
> > > > > >> > > > > > > > > > >> > > > >> > Thanks.
> > > > > >> > > > > > > > > > >> > > > >> > Aitozi.
> > > > > >> > > > > > > > > > >> > > > >> >
> > > > > >> > > > > > > > > > >> > > > >> >
> > > > > >> > > > > > > > > > >> > > > >> > Jing Ge
> > <jing@ververica.com.invalid
> > > >
> > > > > >> > > > 于2023年5月27日周六
> > > > > >> > > > > > > > > 06:40写道：
> > > > > >> > > > > > > > > > >> > > > >> >
> > > > > >> > > > > > > > > > >> > > > >> > > Hi Aitozi,
> > > > > >> > > > > > > > > > >> > > > >> > >
> > > > > >> > > > > > > > > > >> > > > >> > > Thanks for your proposal. I am
> > not
> > > > > quite
> > > > > >> > sure
> > > > > >> > > > if
> > > > > >> > > > > I
> > > > > >> > > > > > > > > > understood
> > > > > >> > > > > > > > > > >> > your
> > > > > >> > > > > > > > > > >> > > > >> > thoughts
> > > > > >> > > > > > > > > > >> > > > >> > > correctly. You described a
> > special
> > > > > case
> > > > > >> > > > > > > implementation
> > > > > >> > > > > > > > of
> > > > > >> > > > > > > > > > the
> > > > > >> > > > > > > > > > >> > > > >> > > AsyncTableFunction with on
> > public
> > > > API
> > > > > >> > > changes.
> > > > > >> > > > > > Would
> > > > > >> > > > > > > > you
> > > > > >> > > > > > > > > > >> please
> > > > > >> > > > > > > > > > >> > > > >> elaborate
> > > > > >> > > > > > > > > > >> > > > >> > > your purpose of writing a FLIP
> > > > > >> according to
> > > > > >> > > the
> > > > > >> > > > > > FLIP
> > > > > >> > > > > > > > > > >> > > > documentation[1]?
> > > > > >> > > > > > > > > > >> > > > >> > > Thanks!
> > > > > >> > > > > > > > > > >> > > > >> > >
> > > > > >> > > > > > > > > > >> > > > >> > > [1]
> > > > > >> > > > > > > > > > >> > > > >> > >
> > > > > >> > > > > > > > > > >> > > > >> > >
> > > > > >> > > > > > > > > > >> > > > >> >
> > > > > >> > > > > > > > > > >> > > > >>
> > > > > >> > > > > > > > > > >> > > >
> > > > > >> > > > > > > > > > >> > >
> > > > > >> > > > > > > > > > >> >
> > > > > >> > > > > > > > > > >>
> > > > > >> > > > > > > > > >
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > >
> > > > > >> > > > > > >
> > > > > >> > > > > >
> > > > > >> > > > >
> > > > > >> > > >
> > > > > >> > >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/Flink+Improvement+Proposals
> > > > > >> > > > > > > > > > >> > > > >> > >
> > > > > >> > > > > > > > > > >> > > > >> > > Best regards,
> > > > > >> > > > > > > > > > >> > > > >> > > Jing
> > > > > >> > > > > > > > > > >> > > > >> > >
> > > > > >> > > > > > > > > > >> > > > >> > > On Wed, May 24, 2023 at
> 1:07 PM
> > > > > Aitozi <
> > > > > >> > > > > > > > > > gjying1314@gmail.com
> > > > > >> > > > > > > > > > >> >
> > > > > >> > > > > > > > > > >> > > > wrote:
> > > > > >> > > > > > > > > > >> > > > >> > >
> > > > > >> > > > > > > > > > >> > > > >> > > > May I ask for some feedback
> > :D
> > > > > >> > > > > > > > > > >> > > > >> > > >
> > > > > >> > > > > > > > > > >> > > > >> > > > Thanks,
> > > > > >> > > > > > > > > > >> > > > >> > > > Aitozi
> > > > > >> > > > > > > > > > >> > > > >> > > >
> > > > > >> > > > > > > > > > >> > > > >> > > > Aitozi <
> gjying1314@gmail.com>
> > > > > >> > > 于2023年5月23日周二
> > > > > >> > > > > > > 19:14写道：
> > > > > >> > > > > > > > > > >> > > > >> > > > >
> > > > > >> > > > > > > > > > >> > > > >> > > > > Just catch an user case
> > report
> > > > > from
> > > > > >> > > Giannis
> > > > > >> > > > > > > Polyzos
> > > > > >> > > > > > > > > for
> > > > > >> > > > > > > > > > >> this
> > > > > >> > > > > > > > > > >> > > > >> usage:
> > > > > >> > > > > > > > > > >> > > > >> > > > >
> > > > > >> > > > > > > > > > >> > > > >> > > > >
> > > > > >> > > > > > > > > > >> > > >
> > > > > >> > > > > > > > >
> > > > > >> > >
> > > https://lists.apache.org/thread/qljwd40v5ntz6733cwcdr8s4z97b343b
> > > > > >> > > > > > > > > > >> > > > >> > > > >
> > > > > >> > > > > > > > > > >> > > > >> > > > > Aitozi <
> > gjying1314@gmail.com>
> > > > > >> > > > 于2023年5月23日周二
> > > > > >> > > > > > > > 17:45写道：
> > > > > >> > > > > > > > > > >> > > > >> > > > > >
> > > > > >> > > > > > > > > > >> > > > >> > > > > > Hi guys,
> > > > > >> > > > > > > > > > >> > > > >> > > > > >     I want to bring up a
> > > > > >> discussion
> > > > > >> > > about
> > > > > >> > > > > > > adding
> > > > > >> > > > > > > > > > >> support
> > > > > >> > > > > > > > > > >> > of
> > > > > >> > > > > > > > > > >> > > > User
> > > > > >> > > > > > > > > > >> > > > >> > > > > > Defined
> AsyncTableFunction
> > > in
> > > > > >> Flink.
> > > > > >> > > > > > > > > > >> > > > >> > > > > > Currently, async table
> > > > function
> > > > > >> are
> > > > > >> > > > special
> > > > > >> > > > > > > > > functions
> > > > > >> > > > > > > > > > >> for
> > > > > >> > > > > > > > > > >> > > > table
> > > > > >> > > > > > > > > > >> > > > >> > > source
> > > > > >> > > > > > > > > > >> > > > >> > > > > > to perform
> > > > > >> > > > > > > > > > >> > > > >> > > > > > async lookup. However,
> > it's
> > > > > worth
> > > > > >> to
> > > > > >> > > > > support
> > > > > >> > > > > > > the
> > > > > >> > > > > > > > > user
> > > > > >> > > > > > > > > > >> > > defined
> > > > > >> > > > > > > > > > >> > > > >> async
> > > > > >> > > > > > > > > > >> > > > >> > > > > > table function.
> > > > > >> > > > > > > > > > >> > > > >> > > > > > Because, in this way,
> the
> > > end
> > > > > SQL
> > > > > >> > user
> > > > > >> > > > can
> > > > > >> > > > > > > > leverage
> > > > > >> > > > > > > > > > it
> > > > > >> > > > > > > > > > >> to
> > > > > >> > > > > > > > > > >> > > > >> perform
> > > > > >> > > > > > > > > > >> > > > >> > the
> > > > > >> > > > > > > > > > >> > > > >> > > > > > async operation
> > > > > >> > > > > > > > > > >> > > > >> > > > > > which is useful to
> maximum
> > > the
> > > > > >> system
> > > > > >> > > > > > > throughput
> > > > > >> > > > > > > > > > >> > especially
> > > > > >> > > > > > > > > > >> > > > for
> > > > > >> > > > > > > > > > >> > > > >> IO
> > > > > >> > > > > > > > > > >> > > > >> > > > > > bottleneck case.
> > > > > >> > > > > > > > > > >> > > > >> > > > > >
> > > > > >> > > > > > > > > > >> > > > >> > > > > > You can find some more
> > > detail
> > > > in
> > > > > >> [1].
> > > > > >> > > > > > > > > > >> > > > >> > > > > >
> > > > > >> > > > > > > > > > >> > > > >> > > > > > Looking forward to
> > feedback
> > > > > >> > > > > > > > > > >> > > > >> > > > > >
> > > > > >> > > > > > > > > > >> > > > >> > > > > >
> > > > > >> > > > > > > > > > >> > > > >> > > > > > [1]:
> > > > > >> > > > > > > > > > >> > > > >> > > >
> > > > > >> > > > > > > > > > >> > > > >> > >
> > > > > >> > > > > > > > > > >> > > > >> >
> > > > > >> > > > > > > > > > >> > > > >>
> > > > > >> > > > > > > > > > >> > > >
> > > > > >> > > > > > > > > > >> > >
> > > > > >> > > > > > > > > > >> >
> > > > > >> > > > > > > > > > >>
> > > > > >> > > > > > > > > >
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > >
> > > > > >> > > > > > >
> > > > > >> > > > > >
> > > > > >> > > > >
> > > > > >> > > >
> > > > > >> > >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/%5BFLIP-313%5D+Add+support+of+User+Defined+AsyncTableFunction
> > > > > >> > > > > > > > > > >> > > > >> > > > > >
> > > > > >> > > > > > > > > > >> > > > >> > > > > > Thanks,
> > > > > >> > > > > > > > > > >> > > > >> > > > > > Aitozi.
> > > > > >> > > > > > > > > > >> > > > >> > > >
> > > > > >> > > > > > > > > > >> > > > >> > >
> > > > > >> > > > > > > > > > >> > > > >> >
> > > > > >> > > > > > > > > > >> > > > >>
> > > > > >> > > > > > > > > > >> > > > >
> > > > > >> > > > > > > > > > >> > > >
> > > > > >> > > > > > > > > > >> > >
> > > > > >> > > > > > > > > > >> >
> > > > > >> > > > > > > > > > >>
> > > > > >> > > > > > > > > > >
> > > > > >> > > > > > > > > >
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > >
> > > > > >> > > > > > >
> > > > > >> > > > > >
> > > > > >> > > > >
> > > > > >> > > >
> > > > > >> > >
> > > > > >> >
> > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] FLIP-313 Add support of User Defined AsyncTableFunction

Posted by Feng Jin <ji...@gmail.com>.

Hi Aitozi,

Sorry for the confusing description.

What I meant was that if we need to remind users about tire safety issues,
we should introduce the new UDTF interface instead of executing the
original UDTF asynchronously. Therefore, I agree with introducing the
AsyncTableFunction.

Best,
Feng

On Tue, Jun 13, 2023 at 10:42 AM Aitozi <gj...@gmail.com> wrote:

> Hi Feng,
>     Thanks for your question. We do not provide a way to switch the UDTF
> between sync and async way,
> So there should be no thread safety problem here.
>
> Best,
> Aitozi
>
> Feng Jin <ji...@gmail.com> 于2023年6月13日周二 10:31写道：
>
> > Hi Aitozi, We do need to remind users about thread safety issues. Thank
> you
> > for your efforts on this FLIP. I have no further questions.
> > Best, Feng
> >
> >
> > On Tue, Jun 13, 2023 at 6:05 AM Jing Ge <ji...@ververica.com.invalid>
> > wrote:
> >
> > > Hi Aitozi,
> > >
> > > Thanks for taking care of that part. I have no other concern.
> > >
> > > Best regards,
> > > Jing
> > >
> > >
> > > On Mon, Jun 12, 2023 at 5:38 PM Aitozi <gj...@gmail.com> wrote:
> > >
> > > > BTW, If there are no other more blocking issue / comments, I would
> like
> > > to
> > > > start a VOTE in another thread this wednesday 6.14
> > > >
> > > > Thanks,
> > > > Aitozi.
> > > >
> > > > Aitozi <gj...@gmail.com> 于2023年6月12日周一 23:34写道：
> > > >
> > > > > Hi, Jing,
> > > > >     Thanks for your explanation. I get your point now.
> > > > >
> > > > > For the performance part, I think it's a good idea to run with
> > > returning
> > > > a
> > > > > big table case, the memory consumption
> > > > > should be a point to be taken care about. Because in the ordered
> > mode,
> > > > the
> > > > > head element in buffer may affect the
> > > > > total memory consumption.
> > > > >
> > > > >
> > > > > Thanks,
> > > > > Aitozi.
> > > > >
> > > > >
> > > > >
> > > > > Jing Ge <ji...@ververica.com.invalid> 于2023年6月12日周一 20:28写道：
> > > > >
> > > > >> Hi Aitozi,
> > > > >>
> > > > >> Which key will be used for lookup is not an issue, only one row
> will
> > > be
> > > > >> required for each key in order to enrich it. True, it depends on
> the
> > > > >> implementation whether multiple rows or single row for each key
> will
> > > be
> > > > >> returned. However, for the lookup & enrichment scenario, one
> row/key
> > > is
> > > > >> recommended, otherwise, like I mentioned previously, enrichment
> > won't
> > > > >> work.
> > > > >>
> > > > >> I am a little bit concerned about returning a big table for each
> > key,
> > > > >> since
> > > > >> it will take the async call longer to return and need more memory.
> > The
> > > > >> performance tests should cover this scenario. This is not a
> blocking
> > > > issue
> > > > >> for this FLIP.
> > > > >>
> > > > >> Best regards,
> > > > >> Jing
> > > > >>
> > > > >> On Sat, Jun 10, 2023 at 4:11 AM Aitozi <gj...@gmail.com>
> > wrote:
> > > > >>
> > > > >> > Hi Jing,
> > > > >> >     I means the join key is not necessary to be the primary key
> or
> > > > >> unique
> > > > >> > index of the database.
> > > > >> > In this situation, we may queried out multi rows for one join
> > key. I
> > > > >> think
> > > > >> > that's why the
> > > > >> > LookupFunction#lookup will return a collection of RowData.
> > > > >> >
> > > > >> > BTW, I think the behavior of lookup join will not affect the
> > > semantic
> > > > of
> > > > >> > the async udtf.
> > > > >> > We use the Async TableFunction here and the table function can
> > > collect
> > > > >> > multiple rows.
> > > > >> >
> > > > >> > Thanks,
> > > > >> > Atiozi.
> > > > >> >
> > > > >> >
> > > > >> >
> > > > >> > Jing Ge <ji...@ververica.com.invalid> 于2023年6月10日周六 00:15写道：
> > > > >> >
> > > > >> > > Hi Aitozi,
> > > > >> > >
> > > > >> > > The keyRow used in this case contains all keys[1].
> > > > >> > >
> > > > >> > > Best regards,
> > > > >> > > Jing
> > > > >> > >
> > > > >> > > [1]
> > > > >> > >
> > > > >> > >
> > > > >> >
> > > > >>
> > > >
> > >
> >
> https://github.com/apache/flink/blob/191ec6ca3943d7119f14837efe112e074d815c47/flink-table/flink-table-common/src/main/java/org/apache/flink/table/functions/LookupFunction.java#L49
> > > > >> > >
> > > > >> > >
> > > > >> > > On Fri, Jun 9, 2023 at 3:42 PM Aitozi <gj...@gmail.com>
> > > wrote:
> > > > >> > >
> > > > >> > > > Hi Jing,
> > > > >> > > >
> > > > >> > > >      The performance test is added to the FLIP.
> > > > >> > > >
> > > > >> > > >      As I know, The lookup join can return multi rows, it
> > > depends
> > > > on
> > > > >> > > > whether  the join key
> > > > >> > > > is the primary key of the external database or not. The
> > `lookup`
> > > > [1]
> > > > >> > will
> > > > >> > > > return a collection of
> > > > >> > > > joined result, and each of them will be collected
> > > > >> > > >
> > > > >> > > >
> > > > >> > > > [1]:
> > > > >> > > >
> > > > >> > > >
> > > > >> > >
> > > > >> >
> > > > >>
> > > >
> > >
> >
> https://github.com/apache/flink/blob/191ec6ca3943d7119f14837efe112e074d815c47/flink-table/flink-table-common/src/main/java/org/apache/flink/table/functions/LookupFunction.java#L52
> > > > >> > > >
> > > > >> > > >
> > > > >> > > > Thanks,
> > > > >> > > > Aitozi.
> > > > >> > > >
> > > > >> > > > Jing Ge <ji...@ververica.com.invalid> 于2023年6月9日周五 17:05写道：
> > > > >> > > >
> > > > >> > > > > Hi Aitozi,
> > > > >> > > > >
> > > > >> > > > > Thanks for the feedback. Looking forward to the
> performance
> > > > tests.
> > > > >> > > > >
> > > > >> > > > > Afaik, lookup returns one row for each key [1] [2].
> > > > Conceptually,
> > > > >> the
> > > > >> > > > > lookup function is used to enrich column(s) from the
> > dimension
> > > > >> table.
> > > > >> > > If,
> > > > >> > > > > for the given key, there will be more than one row, there
> > will
> > > > be
> > > > >> no
> > > > >> > > way
> > > > >> > > > to
> > > > >> > > > > know which row will be used to enrich the key.
> > > > >> > > > >
> > > > >> > > > > [1]
> > > > >> > > > >
> > > > >> > > > >
> > > > >> > > >
> > > > >> > >
> > > > >> >
> > > > >>
> > > >
> > >
> >
> https://github.com/apache/flink/blob/191ec6ca3943d7119f14837efe112e074d815c47/flink-table/flink-table-common/src/main/java/org/apache/flink/table/functions/LookupFunction.java#L49
> > > > >> > > > > [2]
> > > > >> > > > >
> > > > >> > > > >
> > > > >> > > >
> > > > >> > >
> > > > >> >
> > > > >>
> > > >
> > >
> >
> https://github.com/apache/flink/blob/191ec6ca3943d7119f14837efe112e074d815c47/flink-table/flink-table-common/src/main/java/org/apache/flink/table/functions/TableFunction.java#L196
> > > > >> > > > >
> > > > >> > > > > Best regards,
> > > > >> > > > > Jing
> > > > >> > > > >
> > > > >> > > > > On Fri, Jun 9, 2023 at 5:18 AM Aitozi <
> gjying1314@gmail.com
> > >
> > > > >> wrote:
> > > > >> > > > >
> > > > >> > > > > > Hi Jing
> > > > >> > > > > >     Thanks for your good questions. I have updated the
> > > example
> > > > >> to
> > > > >> > the
> > > > >> > > > > FLIP.
> > > > >> > > > > >
> > > > >> > > > > > > Only one row for each lookup
> > > > >> > > > > > lookup can also return multi rows, based on the query
> > > result.
> > > > >> [1]
> > > > >> > > > > >
> > > > >> > > > > > [1]:
> > > > >> > > > > >
> > > > >> > > > > >
> > > > >> > > > >
> > > > >> > > >
> > > > >> > >
> > > > >> >
> > > > >>
> > > >
> > >
> >
> https://github.com/apache/flink/blob/191ec6ca3943d7119f14837efe112e074d815c47/flink-table/flink-table-common/src/main/java/org/apache/flink/table/functions/LookupFunction.java#L56
> > > > >> > > > > >
> > > > >> > > > > > > If we use async calls with lateral join, my gut
> feeling
> > is
> > > > >> > > > > > that we might have many more async calls than lookup
> > join. I
> > > > am
> > > > >> not
> > > > >> > > > > really
> > > > >> > > > > > sure if we will be facing potential issues in this case
> or
> > > > not.
> > > > >> > > > > >
> > > > >> > > > > > IMO, the work pattern is similar to the lookup function,
> > for
> > > > >> each
> > > > >> > row
> > > > >> > > > > from
> > > > >> > > > > > the left table,
> > > > >> > > > > > it will evaluate the eval method once, so the async call
> > > > numbers
> > > > >> > will
> > > > >> > > > not
> > > > >> > > > > > change.
> > > > >> > > > > > and the maximum calls in flight is limited by the Async
> > > > >> operators
> > > > >> > > > buffer
> > > > >> > > > > > capacity
> > > > >> > > > > > which will be controlled by the option.
> > > > >> > > > > >
> > > > >> > > > > > BTW, for the naming of these option, I updated the FLIP
> > > about
> > > > >> this
> > > > >> > > you
> > > > >> > > > > can
> > > > >> > > > > > refer to
> > > > >> > > > > > the section of "ConfigOption" and "Rejected
> Alternatives"
> > > > >> > > > > >
> > > > >> > > > > > In the end, for the performance evaluation, I'd like to
> do
> > > > some
> > > > >> > tests
> > > > >> > > > and
> > > > >> > > > > > will update it to the FLIP doc
> > > > >> > > > > >
> > > > >> > > > > > Thanks,
> > > > >> > > > > > Aitozi.
> > > > >> > > > > >
> > > > >> > > > > >
> > > > >> > > > > > Jing Ge <ji...@ververica.com.invalid> 于2023年6月9日周五
> > 07:23写道：
> > > > >> > > > > >
> > > > >> > > > > > > Hi Aitozi,
> > > > >> > > > > > >
> > > > >> > > > > > > Thanks for the clarification. The code example looks
> > > > >> > interesting. I
> > > > >> > > > > would
> > > > >> > > > > > > suggest adding them into the FLIP. The description
> with
> > > code
> > > > >> > > examples
> > > > >> > > > > > will
> > > > >> > > > > > > help readers understand the motivation and how to use
> > it.
> > > > >> Afaiac,
> > > > >> > > it
> > > > >> > > > > is a
> > > > >> > > > > > > valid feature for Flink users.
> > > > >> > > > > > >
> > > > >> > > > > > > As we knew, lookup join is based on temporal join,
> i.e.
> > > FOR
> > > > >> > > > SYSTEM_TIME
> > > > >> > > > > > AS
> > > > >> > > > > > > OF which is also used in your code example. Temporal
> > join
> > > > >> > performs
> > > > >> > > > the
> > > > >> > > > > > > lookup based on the processing time match. Only one
> row
> > > for
> > > > >> each
> > > > >> > > > > > > lookup(afaiu, I need to check the source code to
> double
> > > > >> confirm)
> > > > >> > > will
> > > > >> > > > > > > return for further enrichment. One the other hand,
> > lateral
> > > > >> join
> > > > >> > > will
> > > > >> > > > > have
> > > > >> > > > > > > sub-queries correlated with every individual value of
> > the
> > > > >> > reference
> > > > >> > > > > table
> > > > >> > > > > > > from the preceding part of the query and each sub
> query
> > > will
> > > > >> > return
> > > > >> > > > > > > multiple rows. If we use async calls with lateral
> join,
> > my
> > > > gut
> > > > >> > > > feeling
> > > > >> > > > > is
> > > > >> > > > > > > that we might have many more async calls than lookup
> > > join. I
> > > > >> am
> > > > >> > not
> > > > >> > > > > > really
> > > > >> > > > > > > sure if we will be facing potential issues in this
> case
> > or
> > > > >> not.
> > > > >> > > > > Possible
> > > > >> > > > > > > issues I can think of now e.g. too many PRC calls, too
> > > many
> > > > >> async
> > > > >> > > > calls
> > > > >> > > > > > > processing, the sub query will return a table which
> > might
> > > be
> > > > >> > (too)
> > > > >> > > > big,
> > > > >> > > > > > and
> > > > >> > > > > > > might cause performance issues. I would suggest
> > preparing
> > > > some
> > > > >> > use
> > > > >> > > > > cases
> > > > >> > > > > > > and running some performance tests to check it. These
> > are
> > > my
> > > > >> > > concerns
> > > > >> > > > > > about
> > > > >> > > > > > > using async calls with lateral join and I'd like to
> > share
> > > > with
> > > > >> > you,
> > > > >> > > > > happy
> > > > >> > > > > > > to discuss with you and hear different opinions,
> > hopefully
> > > > the
> > > > >> > > > > > > discussion could help me understand it more deeply.
> > Please
> > > > >> > correct
> > > > >> > > me
> > > > >> > > > > if
> > > > >> > > > > > I
> > > > >> > > > > > > am wrong.
> > > > >> > > > > > >
> > > > >> > > > > > > Best regards,
> > > > >> > > > > > > Jing
> > > > >> > > > > > >
> > > > >> > > > > > >
> > > > >> > > > > > > On Thu, Jun 8, 2023 at 7:22 AM Aitozi <
> > > gjying1314@gmail.com
> > > > >
> > > > >> > > wrote:
> > > > >> > > > > > >
> > > > >> > > > > > > > Hi Mason,
> > > > >> > > > > > > >     Thanks for your input. I think if we support the
> > > user
> > > > >> > defined
> > > > >> > > > > async
> > > > >> > > > > > > > table function,
> > > > >> > > > > > > > user will be able to use it to hold a batch data
> then
> > > > >> handle it
> > > > >> > > at
> > > > >> > > > > one
> > > > >> > > > > > > time
> > > > >> > > > > > > > in the customized function.
> > > > >> > > > > > > >
> > > > >> > > > > > > > AsyncSink is meant for the sink operator. I have not
> > > > figure
> > > > >> out
> > > > >> > > how
> > > > >> > > > > to
> > > > >> > > > > > > > integrate in this case.
> > > > >> > > > > > > >
> > > > >> > > > > > > > Thanks,
> > > > >> > > > > > > > Atiozi.
> > > > >> > > > > > > >
> > > > >> > > > > > > >
> > > > >> > > > > > > > Mason Chen <ma...@gmail.com> 于2023年6月8日周四
> > > 12:40写道：
> > > > >> > > > > > > >
> > > > >> > > > > > > > > Hi Aitozi,
> > > > >> > > > > > > > >
> > > > >> > > > > > > > > I think it makes sense to make it easier for SQL
> > users
> > > > to
> > > > >> > make
> > > > >> > > > > RPCs.
> > > > >> > > > > > Do
> > > > >> > > > > > > > you
> > > > >> > > > > > > > > think your proposal can extend to the ability to
> > batch
> > > > >> data
> > > > >> > for
> > > > >> > > > the
> > > > >> > > > > > > RPC?
> > > > >> > > > > > > > > This is also another common strategy to increase
> > > > >> throughput.
> > > > >> > > > Also,
> > > > >> > > > > > have
> > > > >> > > > > > > > you
> > > > >> > > > > > > > > considered solving this a bit differently by
> > > leveraging
> > > > >> > Flink's
> > > > >> > > > > > > > AsyncSink?
> > > > >> > > > > > > > >
> > > > >> > > > > > > > > Best,
> > > > >> > > > > > > > > Mason
> > > > >> > > > > > > > >
> > > > >> > > > > > > > > On Mon, Jun 5, 2023 at 1:50 AM Aitozi <
> > > > >> gjying1314@gmail.com>
> > > > >> > > > > wrote:
> > > > >> > > > > > > > >
> > > > >> > > > > > > > > > One more thing for discussion:
> > > > >> > > > > > > > > >
> > > > >> > > > > > > > > > In our internal implementation, we reuse the
> > option
> > > > >> > > > > > > > > > `table.exec.async-lookup.buffer-capacity` and
> > > > >> > > > > > > > > > `table.exec.async-lookup.timeout` to config
> > > > >> > > > > > > > > > the async udtf. Do you think we should add two
> > extra
> > > > >> option
> > > > >> > > to
> > > > >> > > > > > > > > distinguish
> > > > >> > > > > > > > > > from the lookup option such as
> > > > >> > > > > > > > > >
> > > > >> > > > > > > > > > `table.exec.async-udtf.buffer-capacity`
> > > > >> > > > > > > > > > `table.exec.async-udtf.timeout`
> > > > >> > > > > > > > > >
> > > > >> > > > > > > > > >
> > > > >> > > > > > > > > > Best,
> > > > >> > > > > > > > > > Aitozi.
> > > > >> > > > > > > > > >
> > > > >> > > > > > > > > >
> > > > >> > > > > > > > > >
> > > > >> > > > > > > > > > Aitozi <gj...@gmail.com> 于2023年6月5日周一
> > 12:20写道：
> > > > >> > > > > > > > > >
> > > > >> > > > > > > > > > > Hi Jing,
> > > > >> > > > > > > > > > >
> > > > >> > > > > > > > > > >     > what is the difference between the RPC
> > call
> > > or
> > > > >> > query
> > > > >> > > > you
> > > > >> > > > > > > > > mentioned
> > > > >> > > > > > > > > > > and the lookup in a very
> > > > >> > > > > > > > > > > general way
> > > > >> > > > > > > > > > >
> > > > >> > > > > > > > > > > I think the RPC call or query service is quite
> > > > >> similar to
> > > > >> > > the
> > > > >> > > > > > > lookup
> > > > >> > > > > > > > > > join.
> > > > >> > > > > > > > > > > But lookup join should work
> > > > >> > > > > > > > > > > with `LookupTableSource`.
> > > > >> > > > > > > > > > >
> > > > >> > > > > > > > > > > Let's see how we can perform an async RPC call
> > > with
> > > > >> > lookup
> > > > >> > > > > join:
> > > > >> > > > > > > > > > >
> > > > >> > > > > > > > > > > (1) Implement an AsyncTableFunction with RPC
> > call
> > > > >> logic.
> > > > >> > > > > > > > > > > (2) Implement a `LookupTableSource` connector
> > run
> > > > with
> > > > >> > the
> > > > >> > > > > async
> > > > >> > > > > > > udtf
> > > > >> > > > > > > > > > > defined in (1).
> > > > >> > > > > > > > > > > (3) Then define a DDL of this look up table in
> > SQL
> > > > >> > > > > > > > > > >
> > > > >> > > > > > > > > > > CREATE TEMPORARY TABLE Customers (
> > > > >> > > > > > > > > > >   id INT,
> > > > >> > > > > > > > > > >   name STRING,
> > > > >> > > > > > > > > > >   country STRING,
> > > > >> > > > > > > > > > >   zip STRING
> > > > >> > > > > > > > > > > ) WITH (
> > > > >> > > > > > > > > > >   'connector' = 'custom'
> > > > >> > > > > > > > > > > );
> > > > >> > > > > > > > > > >
> > > > >> > > > > > > > > > > (4) Run with the query as below:
> > > > >> > > > > > > > > > >
> > > > >> > > > > > > > > > > SELECT o.order_id, o.total, c.country, c.zip
> > > > >> > > > > > > > > > > FROM Orders AS o
> > > > >> > > > > > > > > > >   JOIN Customers FOR SYSTEM_TIME AS OF
> > o.proc_time
> > > > AS
> > > > >> c
> > > > >> > > > > > > > > > >     ON o.customer_id = c.id;
> > > > >> > > > > > > > > > >
> > > > >> > > > > > > > > > > This example is from doc
> > > > >> > > > > > > > > > > <
> > > > >> > > > > > > > > >
> > > > >> > > > > > > > >
> > > > >> > > > > > > >
> > > > >> > > > > > >
> > > > >> > > > > >
> > > > >> > > > >
> > > > >> > > >
> > > > >> > >
> > > > >> >
> > > > >>
> > > >
> > >
> >
> https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/dev/table/sql/queries/joins/#lookup-join
> > > > >> > > > > > > > > > >.You
> > > > >> > > > > > > > > > > can image the look up process as an async RPC
> > call
> > > > >> > process.
> > > > >> > > > > > > > > > >
> > > > >> > > > > > > > > > > Let's see how we can perform an async RPC call
> > > with
> > > > >> > lateral
> > > > >> > > > > join:
> > > > >> > > > > > > > > > >
> > > > >> > > > > > > > > > > (1) Implement an AsyncTableFunction with RPC
> > call
> > > > >> logic.
> > > > >> > > > > > > > > > > (2) Run query with
> > > > >> > > > > > > > > > >
> > > > >> > > > > > > > > > > Create function f1 as '...' ;
> > > > >> > > > > > > > > > >
> > > > >> > > > > > > > > > > SELECT o.order_id, o.total, c.country, c.zip
> > FROM
> > > > >> Orders
> > > > >> > > > > lateral
> > > > >> > > > > > > > table
> > > > >> > > > > > > > > > > (f1(order_id)) as T(...);
> > > > >> > > > > > > > > > >
> > > > >> > > > > > > > > > > As you can see, the lateral join version is
> more
> > > > >> simple
> > > > >> > and
> > > > >> > > > > > > intuitive
> > > > >> > > > > > > > > to
> > > > >> > > > > > > > > > > users. Users do not have to wrap a
> > > > >> > > > > > > > > > > LookupTableSource for the purpose of using
> async
> > > > udtf.
> > > > >> > > > > > > > > > >
> > > > >> > > > > > > > > > > In the end, We can also see the user defined
> > async
> > > > >> table
> > > > >> > > > > function
> > > > >> > > > > > > is
> > > > >> > > > > > > > an
> > > > >> > > > > > > > > > > enhancement of the current lateral table join
> > > > >> > > > > > > > > > > which only supports sync lateral join now.
> > > > >> > > > > > > > > > >
> > > > >> > > > > > > > > > > Best,
> > > > >> > > > > > > > > > > Aitozi.
> > > > >> > > > > > > > > > >
> > > > >> > > > > > > > > > >
> > > > >> > > > > > > > > > > Jing Ge <ji...@ververica.com.invalid>
> > 于2023年6月2日周五
> > > > >> > 19:37写道：
> > > > >> > > > > > > > > > >
> > > > >> > > > > > > > > > >> Hi Aitozi,
> > > > >> > > > > > > > > > >>
> > > > >> > > > > > > > > > >> Thanks for the update. Just out of curiosity,
> > > what
> > > > is
> > > > >> > the
> > > > >> > > > > > > difference
> > > > >> > > > > > > > > > >> between the RPC call or query you mentioned
> and
> > > the
> > > > >> > lookup
> > > > >> > > > in
> > > > >> > > > > a
> > > > >> > > > > > > very
> > > > >> > > > > > > > > > >> general way? Since Lateral join is used in
> the
> > > > FLIP.
> > > > >> Is
> > > > >> > > > there
> > > > >> > > > > > any
> > > > >> > > > > > > > > > special
> > > > >> > > > > > > > > > >> thought for that? Sorry for asking so many
> > > > questions.
> > > > >> > The
> > > > >> > > > FLIP
> > > > >> > > > > > > > > contains
> > > > >> > > > > > > > > > >> limited information to understand the
> > motivation.
> > > > >> > > > > > > > > > >>
> > > > >> > > > > > > > > > >> Best regards,
> > > > >> > > > > > > > > > >> Jing
> > > > >> > > > > > > > > > >>
> > > > >> > > > > > > > > > >> On Fri, Jun 2, 2023 at 3:48 AM Aitozi <
> > > > >> > > gjying1314@gmail.com
> > > > >> > > > >
> > > > >> > > > > > > wrote:
> > > > >> > > > > > > > > > >>
> > > > >> > > > > > > > > > >> > Hi Jing,
> > > > >> > > > > > > > > > >> >     I have updated the proposed changes to
> > the
> > > > >> FLIP.
> > > > >> > > IMO,
> > > > >> > > > > > lookup
> > > > >> > > > > > > > has
> > > > >> > > > > > > > > > its
> > > > >> > > > > > > > > > >> > clear
> > > > >> > > > > > > > > > >> > async call requirement is due to its IO
> heavy
> > > > >> > operator.
> > > > >> > > In
> > > > >> > > > > our
> > > > >> > > > > > > > > usage,
> > > > >> > > > > > > > > > >> sql
> > > > >> > > > > > > > > > >> > users have
> > > > >> > > > > > > > > > >> > logic to do some RPC call or query the
> > > > third-party
> > > > >> > > service
> > > > >> > > > > > which
> > > > >> > > > > > > > is
> > > > >> > > > > > > > > > >> also IO
> > > > >> > > > > > > > > > >> > intensive.
> > > > >> > > > > > > > > > >> > In these case, we'd like to leverage the
> > async
> > > > >> > function
> > > > >> > > to
> > > > >> > > > > > > improve
> > > > >> > > > > > > > > the
> > > > >> > > > > > > > > > >> > throughput.
> > > > >> > > > > > > > > > >> >
> > > > >> > > > > > > > > > >> > Thanks,
> > > > >> > > > > > > > > > >> > Aitozi.
> > > > >> > > > > > > > > > >> >
> > > > >> > > > > > > > > > >> > Jing Ge <ji...@ververica.com.invalid>
> > > > 于2023年6月1日周四
> > > > >> > > > 22:55写道：
> > > > >> > > > > > > > > > >> >
> > > > >> > > > > > > > > > >> > > Hi Aitozi,
> > > > >> > > > > > > > > > >> > >
> > > > >> > > > > > > > > > >> > > Sorry for the late reply. Would you like
> to
> > > > >> update
> > > > >> > the
> > > > >> > > > > > > proposed
> > > > >> > > > > > > > > > >> changes
> > > > >> > > > > > > > > > >> > > with more details into the FLIP too?
> > > > >> > > > > > > > > > >> > > I got your point. It looks like a
> rational
> > > > idea.
> > > > >> > > > However,
> > > > >> > > > > > > since
> > > > >> > > > > > > > > > lookup
> > > > >> > > > > > > > > > >> > has
> > > > >> > > > > > > > > > >> > > its clear async call requirement, are
> there
> > > any
> > > > >> real
> > > > >> > > use
> > > > >> > > > > > cases
> > > > >> > > > > > > > > that
> > > > >> > > > > > > > > > >> > > need this change? This will help us
> > > understand
> > > > >> the
> > > > >> > > > > > motivation.
> > > > >> > > > > > > > > After
> > > > >> > > > > > > > > > >> all,
> > > > >> > > > > > > > > > >> > > lateral join and temporal lookup join[1]
> > are
> > > > >> quite
> > > > >> > > > > > different.
> > > > >> > > > > > > > > > >> > >
> > > > >> > > > > > > > > > >> > > Best regards,
> > > > >> > > > > > > > > > >> > > Jing
> > > > >> > > > > > > > > > >> > >
> > > > >> > > > > > > > > > >> > >
> > > > >> > > > > > > > > > >> > > [1]
> > > > >> > > > > > > > > > >> > >
> > > > >> > > > > > > > > > >> > >
> > > > >> > > > > > > > > > >> >
> > > > >> > > > > > > > > > >>
> > > > >> > > > > > > > > >
> > > > >> > > > > > > > >
> > > > >> > > > > > > >
> > > > >> > > > > > >
> > > > >> > > > > >
> > > > >> > > > >
> > > > >> > > >
> > > > >> > >
> > > > >> >
> > > > >>
> > > >
> > >
> >
> https://github.com/apache/flink/blob/d90a72da2fd601ca4e2a46700e91ec5b348de2ad/flink-table/flink-table-common/src/main/java/org/apache/flink/table/functions/AsyncTableFunction.java#L54
> > > > >> > > > > > > > > > >> > >
> > > > >> > > > > > > > > > >> > > On Wed, May 31, 2023 at 8:53 AM Aitozi <
> > > > >> > > > > > gjying1314@gmail.com>
> > > > >> > > > > > > > > > wrote:
> > > > >> > > > > > > > > > >> > >
> > > > >> > > > > > > > > > >> > > > Hi Jing,
> > > > >> > > > > > > > > > >> > > >     What do you think about it? Can we
> > move
> > > > >> > forward
> > > > >> > > > this
> > > > >> > > > > > > > > feature?
> > > > >> > > > > > > > > > >> > > >
> > > > >> > > > > > > > > > >> > > > Thanks,
> > > > >> > > > > > > > > > >> > > > Aitozi.
> > > > >> > > > > > > > > > >> > > >
> > > > >> > > > > > > > > > >> > > > Aitozi <gj...@gmail.com>
> > > 于2023年5月29日周一
> > > > >> > > 09:56写道：
> > > > >> > > > > > > > > > >> > > >
> > > > >> > > > > > > > > > >> > > > > Hi Jing,
> > > > >> > > > > > > > > > >> > > > >     > "Do you mean to support the
> > > > >> > > AyncTableFunction
> > > > >> > > > > > beyond
> > > > >> > > > > > > > the
> > > > >> > > > > > > > > > >> > > > > LookupTableSource?"
> > > > >> > > > > > > > > > >> > > > > Yes, I mean to support the
> > > > AyncTableFunction
> > > > >> > > beyond
> > > > >> > > > > the
> > > > >> > > > > > > > > > >> > > > LookupTableSource.
> > > > >> > > > > > > > > > >> > > > >
> > > > >> > > > > > > > > > >> > > > > The "AsyncTableFunction" is the
> > function
> > > > with
> > > > >> > > > ability
> > > > >> > > > > to
> > > > >> > > > > > > be
> > > > >> > > > > > > > > > >> executed
> > > > >> > > > > > > > > > >> > > > async
> > > > >> > > > > > > > > > >> > > > > (with AsyncWaitOperator).
> > > > >> > > > > > > > > > >> > > > > The async lookup join is a one of
> usage
> > > of
> > > > >> this.
> > > > >> > > So,
> > > > >> > > > > we
> > > > >> > > > > > > > don't
> > > > >> > > > > > > > > > >> have to
> > > > >> > > > > > > > > > >> > > > bind
> > > > >> > > > > > > > > > >> > > > > the AyncTableFunction with
> > > > LookupTableSource.
> > > > >> > > > > > > > > > >> > > > > If User-defined AsyncTableFunction is
> > > > >> supported,
> > > > >> > > > user
> > > > >> > > > > > can
> > > > >> > > > > > > > > > directly
> > > > >> > > > > > > > > > >> > use
> > > > >> > > > > > > > > > >> > > > > lateral table syntax to perform async
> > > > >> operation.
> > > > >> > > > > > > > > > >> > > > >
> > > > >> > > > > > > > > > >> > > > > > "It would be better if you could
> > > > elaborate
> > > > >> the
> > > > >> > > > > > proposed
> > > > >> > > > > > > > > > changes
> > > > >> > > > > > > > > > >> wrt
> > > > >> > > > > > > > > > >> > > the
> > > > >> > > > > > > > > > >> > > > > CorrelatedCodeGenerator with more
> > > details"
> > > > >> > > > > > > > > > >> > > > >
> > > > >> > > > > > > > > > >> > > > > In the proposal, we use lateral table
> > > > syntax
> > > > >> to
> > > > >> > > > > support
> > > > >> > > > > > > the
> > > > >> > > > > > > > > > async
> > > > >> > > > > > > > > > >> > table
> > > > >> > > > > > > > > > >> > > > > function. So the planner will also
> > treat
> > > > this
> > > > >> > > > > statement
> > > > >> > > > > > > to a
> > > > >> > > > > > > > > > >> > > > > CommonExecCorrelate node. So the
> > runtime
> > > > code
> > > > >> > > should
> > > > >> > > > > be
> > > > >> > > > > > > > > > generated
> > > > >> > > > > > > > > > >> in
> > > > >> > > > > > > > > > >> > > > > CorrelatedCodeGenerator.
> > > > >> > > > > > > > > > >> > > > > In CorrelatedCodeGenerator, we will
> > know
> > > > the
> > > > >> > > > > > > TableFunction's
> > > > >> > > > > > > > > > Kind
> > > > >> > > > > > > > > > >> of
> > > > >> > > > > > > > > > >> > > > > `FunctionKind.Table` or
> > > > >> > `FunctionKind.ASYNC_TABLE`
> > > > >> > > > > > > > > > >> > > > > For  `FunctionKind.ASYNC_TABLE` we
> can
> > > > >> generate
> > > > >> > a
> > > > >> > > > > > > > > > >> AsyncWaitOperator
> > > > >> > > > > > > > > > >> > to
> > > > >> > > > > > > > > > >> > > > > execute the async table function.
> > > > >> > > > > > > > > > >> > > > >
> > > > >> > > > > > > > > > >> > > > >
> > > > >> > > > > > > > > > >> > > > > Thanks,
> > > > >> > > > > > > > > > >> > > > > Aitozi.
> > > > >> > > > > > > > > > >> > > > >
> > > > >> > > > > > > > > > >> > > > >
> > > > >> > > > > > > > > > >> > > > > Jing Ge <ji...@ververica.com.invalid>
> > > > >> > > 于2023年5月29日周一
> > > > >> > > > > > > 03:22写道：
> > > > >> > > > > > > > > > >> > > > >
> > > > >> > > > > > > > > > >> > > > >> Hi Aitozi,
> > > > >> > > > > > > > > > >> > > > >>
> > > > >> > > > > > > > > > >> > > > >> Thanks for the clarification. The
> > naming
> > > > >> > "Lookup"
> > > > >> > > > > might
> > > > >> > > > > > > > > suggest
> > > > >> > > > > > > > > > >> > using
> > > > >> > > > > > > > > > >> > > it
> > > > >> > > > > > > > > > >> > > > >> for table look up. But conceptually
> > what
> > > > the
> > > > >> > > eval()
> > > > >> > > > > > > method
> > > > >> > > > > > > > > will
> > > > >> > > > > > > > > > >> do
> > > > >> > > > > > > > > > >> > is
> > > > >> > > > > > > > > > >> > > to
> > > > >> > > > > > > > > > >> > > > >> get a collection of results(Row,
> > > RowData)
> > > > >> from
> > > > >> > > the
> > > > >> > > > > > given
> > > > >> > > > > > > > > keys.
> > > > >> > > > > > > > > > >> How
> > > > >> > > > > > > > > > >> > it
> > > > >> > > > > > > > > > >> > > > will
> > > > >> > > > > > > > > > >> > > > >> be done depends on the
> implementation,
> > > > i.e.
> > > > >> you
> > > > >> > > can
> > > > >> > > > > > > > implement
> > > > >> > > > > > > > > > >> your
> > > > >> > > > > > > > > > >> > own
> > > > >> > > > > > > > > > >> > > > >> Source[1][2]. The example in the
> FLIP
> > > > >> should be
> > > > >> > > > able
> > > > >> > > > > to
> > > > >> > > > > > > be
> > > > >> > > > > > > > > > >> handled
> > > > >> > > > > > > > > > >> > in
> > > > >> > > > > > > > > > >> > > > this
> > > > >> > > > > > > > > > >> > > > >> way.
> > > > >> > > > > > > > > > >> > > > >>
> > > > >> > > > > > > > > > >> > > > >> Do you mean to support the
> > > > AyncTableFunction
> > > > >> > > beyond
> > > > >> > > > > the
> > > > >> > > > > > > > > > >> > > > LookupTableSource?
> > > > >> > > > > > > > > > >> > > > >> It would be better if you could
> > > elaborate
> > > > >> the
> > > > >> > > > > proposed
> > > > >> > > > > > > > > changes
> > > > >> > > > > > > > > > >> wrt
> > > > >> > > > > > > > > > >> > the
> > > > >> > > > > > > > > > >> > > > >> CorrelatedCodeGenerator with more
> > > details.
> > > > >> > > Thanks!
> > > > >> > > > > > > > > > >> > > > >>
> > > > >> > > > > > > > > > >> > > > >> Best regards,
> > > > >> > > > > > > > > > >> > > > >> Jing
> > > > >> > > > > > > > > > >> > > > >>
> > > > >> > > > > > > > > > >> > > > >> [1]
> > > > >> > > > > > > > > > >> > > > >>
> > > > >> > > > > > > > > > >> > > > >>
> > > > >> > > > > > > > > > >> > > >
> > > > >> > > > > > > > > > >> > >
> > > > >> > > > > > > > > > >> >
> > > > >> > > > > > > > > > >>
> > > > >> > > > > > > > > >
> > > > >> > > > > > > > >
> > > > >> > > > > > > >
> > > > >> > > > > > >
> > > > >> > > > > >
> > > > >> > > > >
> > > > >> > > >
> > > > >> > >
> > > > >> >
> > > > >>
> > > >
> > >
> >
> https://github.com/apache/flink/blob/678370b18e1b6c4a23e5ce08f8efd05675a0cc17/flink-table/flink-table-common/src/main/java/org/apache/flink/table/connector/source/LookupTableSource.java#L64
> > > > >> > > > > > > > > > >> > > > >> [2]
> > > > >> > > > > > > > > > >> > > > >>
> > > > >> > > > > > > > > > >> > > > >>
> > > > >> > > > > > > > > > >> > > >
> > > > >> > > > > > > > > > >> > >
> > > > >> > > > > > > > > > >> >
> > > > >> > > > > > > > > > >>
> > > > >> > > > > > > > > >
> > > > >> > > > > > > > >
> > > > >> > > > > > > >
> > > > >> > > > > > >
> > > > >> > > > > >
> > > > >> > > > >
> > > > >> > > >
> > > > >> > >
> > > > >> >
> > > > >>
> > > >
> > >
> >
> https://github.com/apache/flink/blob/678370b18e1b6c4a23e5ce08f8efd05675a0cc17/flink-table/flink-table-common/src/main/java/org/apache/flink/table/connector/source/AsyncTableFunctionProvider.java#L49
> > > > >> > > > > > > > > > >> > > > >>
> > > > >> > > > > > > > > > >> > > > >> On Sat, May 27, 2023 at 9:48 AM
> > Aitozi <
> > > > >> > > > > > > > gjying1314@gmail.com
> > > > >> > > > > > > > > >
> > > > >> > > > > > > > > > >> > wrote:
> > > > >> > > > > > > > > > >> > > > >>
> > > > >> > > > > > > > > > >> > > > >> > Hi Jing,
> > > > >> > > > > > > > > > >> > > > >> >     Thanks for your response. As
> > > stated
> > > > in
> > > > >> > the
> > > > >> > > > > FLIP,
> > > > >> > > > > > > the
> > > > >> > > > > > > > > > >> purpose
> > > > >> > > > > > > > > > >> > of
> > > > >> > > > > > > > > > >> > > > this
> > > > >> > > > > > > > > > >> > > > >> > FLIP is meant to support
> > > > >> > > > > > > > > > >> > > > >> > user-defined async table function.
> > As
> > > > >> > described
> > > > >> > > > in
> > > > >> > > > > > > flink
> > > > >> > > > > > > > > > >> document
> > > > >> > > > > > > > > > >> > > [1]
> > > > >> > > > > > > > > > >> > > > >> >
> > > > >> > > > > > > > > > >> > > > >> > Async table functions are special
> > > > >> functions
> > > > >> > for
> > > > >> > > > > table
> > > > >> > > > > > > > > sources
> > > > >> > > > > > > > > > >> that
> > > > >> > > > > > > > > > >> > > > >> perform
> > > > >> > > > > > > > > > >> > > > >> > > a lookup.
> > > > >> > > > > > > > > > >> > > > >> > >
> > > > >> > > > > > > > > > >> > > > >> >
> > > > >> > > > > > > > > > >> > > > >> > So end user can not directly
> define
> > > and
> > > > >> use
> > > > >> > > async
> > > > >> > > > > > table
> > > > >> > > > > > > > > > >> function
> > > > >> > > > > > > > > > >> > > now.
> > > > >> > > > > > > > > > >> > > > An
> > > > >> > > > > > > > > > >> > > > >> > user case is reported in [2]
> > > > >> > > > > > > > > > >> > > > >> >
> > > > >> > > > > > > > > > >> > > > >> > So, in conclusion, no new
> interface
> > is
> > > > >> > > > introduced,
> > > > >> > > > > > but
> > > > >> > > > > > > we
> > > > >> > > > > > > > > > >> extend
> > > > >> > > > > > > > > > >> > the
> > > > >> > > > > > > > > > >> > > > >> > ability to support user-defined
> > async
> > > > >> table
> > > > >> > > > > function.
> > > > >> > > > > > > > > > >> > > > >> >
> > > > >> > > > > > > > > > >> > > > >> > [1]:
> > > > >> > > > > > > > > > >> > > > >> >
> > > > >> > > > > > > > > > >> > > > >> >
> > > > >> > > > > > > > > > >> > > > >>
> > > > >> > > > > > > > > > >> > > >
> > > > >> > > > > > > > > > >> > >
> > > > >> > > > > > > > > > >> >
> > > > >> > > > > > > > > > >>
> > > > >> > > > > > > > > >
> > > > >> > > > > > > > >
> > > > >> > > > > > > >
> > > > >> > > > > > >
> > > > >> > > > > >
> > > > >> > > > >
> > > > >> > > >
> > > > >> > >
> > > > >> >
> > > > >>
> > > >
> > >
> >
> https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/dev/table/functions/udfs/
> > > > >> > > > > > > > > > >> > > > >> > [2]:
> > > > >> > > > > > > > > > >> > >
> > > > >> > > > > > > >
> > > > >> >
> https://lists.apache.org/thread/qljwd40v5ntz6733cwcdr8s4z97b343b
> > > > >> > > > > > > > > > >> > > > >> >
> > > > >> > > > > > > > > > >> > > > >> > Thanks.
> > > > >> > > > > > > > > > >> > > > >> > Aitozi.
> > > > >> > > > > > > > > > >> > > > >> >
> > > > >> > > > > > > > > > >> > > > >> >
> > > > >> > > > > > > > > > >> > > > >> > Jing Ge
> <jing@ververica.com.invalid
> > >
> > > > >> > > > 于2023年5月27日周六
> > > > >> > > > > > > > > 06:40写道：
> > > > >> > > > > > > > > > >> > > > >> >
> > > > >> > > > > > > > > > >> > > > >> > > Hi Aitozi,
> > > > >> > > > > > > > > > >> > > > >> > >
> > > > >> > > > > > > > > > >> > > > >> > > Thanks for your proposal. I am
> not
> > > > quite
> > > > >> > sure
> > > > >> > > > if
> > > > >> > > > > I
> > > > >> > > > > > > > > > understood
> > > > >> > > > > > > > > > >> > your
> > > > >> > > > > > > > > > >> > > > >> > thoughts
> > > > >> > > > > > > > > > >> > > > >> > > correctly. You described a
> special
> > > > case
> > > > >> > > > > > > implementation
> > > > >> > > > > > > > of
> > > > >> > > > > > > > > > the
> > > > >> > > > > > > > > > >> > > > >> > > AsyncTableFunction with on
> public
> > > API
> > > > >> > > changes.
> > > > >> > > > > > Would
> > > > >> > > > > > > > you
> > > > >> > > > > > > > > > >> please
> > > > >> > > > > > > > > > >> > > > >> elaborate
> > > > >> > > > > > > > > > >> > > > >> > > your purpose of writing a FLIP
> > > > >> according to
> > > > >> > > the
> > > > >> > > > > > FLIP
> > > > >> > > > > > > > > > >> > > > documentation[1]?
> > > > >> > > > > > > > > > >> > > > >> > > Thanks!
> > > > >> > > > > > > > > > >> > > > >> > >
> > > > >> > > > > > > > > > >> > > > >> > > [1]
> > > > >> > > > > > > > > > >> > > > >> > >
> > > > >> > > > > > > > > > >> > > > >> > >
> > > > >> > > > > > > > > > >> > > > >> >
> > > > >> > > > > > > > > > >> > > > >>
> > > > >> > > > > > > > > > >> > > >
> > > > >> > > > > > > > > > >> > >
> > > > >> > > > > > > > > > >> >
> > > > >> > > > > > > > > > >>
> > > > >> > > > > > > > > >
> > > > >> > > > > > > > >
> > > > >> > > > > > > >
> > > > >> > > > > > >
> > > > >> > > > > >
> > > > >> > > > >
> > > > >> > > >
> > > > >> > >
> > > > >> >
> > > > >>
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/Flink+Improvement+Proposals
> > > > >> > > > > > > > > > >> > > > >> > >
> > > > >> > > > > > > > > > >> > > > >> > > Best regards,
> > > > >> > > > > > > > > > >> > > > >> > > Jing
> > > > >> > > > > > > > > > >> > > > >> > >
> > > > >> > > > > > > > > > >> > > > >> > > On Wed, May 24, 2023 at 1:07 PM
> > > > Aitozi <
> > > > >> > > > > > > > > > gjying1314@gmail.com
> > > > >> > > > > > > > > > >> >
> > > > >> > > > > > > > > > >> > > > wrote:
> > > > >> > > > > > > > > > >> > > > >> > >
> > > > >> > > > > > > > > > >> > > > >> > > > May I ask for some feedback
> :D
> > > > >> > > > > > > > > > >> > > > >> > > >
> > > > >> > > > > > > > > > >> > > > >> > > > Thanks,
> > > > >> > > > > > > > > > >> > > > >> > > > Aitozi
> > > > >> > > > > > > > > > >> > > > >> > > >
> > > > >> > > > > > > > > > >> > > > >> > > > Aitozi <gj...@gmail.com>
> > > > >> > > 于2023年5月23日周二
> > > > >> > > > > > > 19:14写道：
> > > > >> > > > > > > > > > >> > > > >> > > > >
> > > > >> > > > > > > > > > >> > > > >> > > > > Just catch an user case
> report
> > > > from
> > > > >> > > Giannis
> > > > >> > > > > > > Polyzos
> > > > >> > > > > > > > > for
> > > > >> > > > > > > > > > >> this
> > > > >> > > > > > > > > > >> > > > >> usage:
> > > > >> > > > > > > > > > >> > > > >> > > > >
> > > > >> > > > > > > > > > >> > > > >> > > > >
> > > > >> > > > > > > > > > >> > > >
> > > > >> > > > > > > > >
> > > > >> > >
> > https://lists.apache.org/thread/qljwd40v5ntz6733cwcdr8s4z97b343b
> > > > >> > > > > > > > > > >> > > > >> > > > >
> > > > >> > > > > > > > > > >> > > > >> > > > > Aitozi <
> gjying1314@gmail.com>
> > > > >> > > > 于2023年5月23日周二
> > > > >> > > > > > > > 17:45写道：
> > > > >> > > > > > > > > > >> > > > >> > > > > >
> > > > >> > > > > > > > > > >> > > > >> > > > > > Hi guys,
> > > > >> > > > > > > > > > >> > > > >> > > > > >     I want to bring up a
> > > > >> discussion
> > > > >> > > about
> > > > >> > > > > > > adding
> > > > >> > > > > > > > > > >> support
> > > > >> > > > > > > > > > >> > of
> > > > >> > > > > > > > > > >> > > > User
> > > > >> > > > > > > > > > >> > > > >> > > > > > Defined AsyncTableFunction
> > in
> > > > >> Flink.
> > > > >> > > > > > > > > > >> > > > >> > > > > > Currently, async table
> > > function
> > > > >> are
> > > > >> > > > special
> > > > >> > > > > > > > > functions
> > > > >> > > > > > > > > > >> for
> > > > >> > > > > > > > > > >> > > > table
> > > > >> > > > > > > > > > >> > > > >> > > source
> > > > >> > > > > > > > > > >> > > > >> > > > > > to perform
> > > > >> > > > > > > > > > >> > > > >> > > > > > async lookup. However,
> it's
> > > > worth
> > > > >> to
> > > > >> > > > > support
> > > > >> > > > > > > the
> > > > >> > > > > > > > > user
> > > > >> > > > > > > > > > >> > > defined
> > > > >> > > > > > > > > > >> > > > >> async
> > > > >> > > > > > > > > > >> > > > >> > > > > > table function.
> > > > >> > > > > > > > > > >> > > > >> > > > > > Because, in this way, the
> > end
> > > > SQL
> > > > >> > user
> > > > >> > > > can
> > > > >> > > > > > > > leverage
> > > > >> > > > > > > > > > it
> > > > >> > > > > > > > > > >> to
> > > > >> > > > > > > > > > >> > > > >> perform
> > > > >> > > > > > > > > > >> > > > >> > the
> > > > >> > > > > > > > > > >> > > > >> > > > > > async operation
> > > > >> > > > > > > > > > >> > > > >> > > > > > which is useful to maximum
> > the
> > > > >> system
> > > > >> > > > > > > throughput
> > > > >> > > > > > > > > > >> > especially
> > > > >> > > > > > > > > > >> > > > for
> > > > >> > > > > > > > > > >> > > > >> IO
> > > > >> > > > > > > > > > >> > > > >> > > > > > bottleneck case.
> > > > >> > > > > > > > > > >> > > > >> > > > > >
> > > > >> > > > > > > > > > >> > > > >> > > > > > You can find some more
> > detail
> > > in
> > > > >> [1].
> > > > >> > > > > > > > > > >> > > > >> > > > > >
> > > > >> > > > > > > > > > >> > > > >> > > > > > Looking forward to
> feedback
> > > > >> > > > > > > > > > >> > > > >> > > > > >
> > > > >> > > > > > > > > > >> > > > >> > > > > >
> > > > >> > > > > > > > > > >> > > > >> > > > > > [1]:
> > > > >> > > > > > > > > > >> > > > >> > > >
> > > > >> > > > > > > > > > >> > > > >> > >
> > > > >> > > > > > > > > > >> > > > >> >
> > > > >> > > > > > > > > > >> > > > >>
> > > > >> > > > > > > > > > >> > > >
> > > > >> > > > > > > > > > >> > >
> > > > >> > > > > > > > > > >> >
> > > > >> > > > > > > > > > >>
> > > > >> > > > > > > > > >
> > > > >> > > > > > > > >
> > > > >> > > > > > > >
> > > > >> > > > > > >
> > > > >> > > > > >
> > > > >> > > > >
> > > > >> > > >
> > > > >> > >
> > > > >> >
> > > > >>
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/%5BFLIP-313%5D+Add+support+of+User+Defined+AsyncTableFunction
> > > > >> > > > > > > > > > >> > > > >> > > > > >
> > > > >> > > > > > > > > > >> > > > >> > > > > > Thanks,
> > > > >> > > > > > > > > > >> > > > >> > > > > > Aitozi.
> > > > >> > > > > > > > > > >> > > > >> > > >
> > > > >> > > > > > > > > > >> > > > >> > >
> > > > >> > > > > > > > > > >> > > > >> >
> > > > >> > > > > > > > > > >> > > > >>
> > > > >> > > > > > > > > > >> > > > >
> > > > >> > > > > > > > > > >> > > >
> > > > >> > > > > > > > > > >> > >
> > > > >> > > > > > > > > > >> >
> > > > >> > > > > > > > > > >>
> > > > >> > > > > > > > > > >
> > > > >> > > > > > > > > >
> > > > >> > > > > > > > >
> > > > >> > > > > > > >
> > > > >> > > > > > >
> > > > >> > > > > >
> > > > >> > > > >
> > > > >> > > >
> > > > >> > >
> > > > >> >
> > > > >>
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] FLIP-313 Add support of User Defined AsyncTableFunction

Posted by Aitozi <gj...@gmail.com>.

Hi Feng,
    Thanks for your question. We do not provide a way to switch the UDTF
between sync and async way,
So there should be no thread safety problem here.

Best,
Aitozi

Feng Jin <ji...@gmail.com> 于2023年6月13日周二 10:31写道：

> Hi Aitozi, We do need to remind users about thread safety issues. Thank you
> for your efforts on this FLIP. I have no further questions.
> Best, Feng
>
>
> On Tue, Jun 13, 2023 at 6:05 AM Jing Ge <ji...@ververica.com.invalid>
> wrote:
>
> > Hi Aitozi,
> >
> > Thanks for taking care of that part. I have no other concern.
> >
> > Best regards,
> > Jing
> >
> >
> > On Mon, Jun 12, 2023 at 5:38 PM Aitozi <gj...@gmail.com> wrote:
> >
> > > BTW, If there are no other more blocking issue / comments, I would like
> > to
> > > start a VOTE in another thread this wednesday 6.14
> > >
> > > Thanks,
> > > Aitozi.
> > >
> > > Aitozi <gj...@gmail.com> 于2023年6月12日周一 23:34写道：
> > >
> > > > Hi, Jing,
> > > >     Thanks for your explanation. I get your point now.
> > > >
> > > > For the performance part, I think it's a good idea to run with
> > returning
> > > a
> > > > big table case, the memory consumption
> > > > should be a point to be taken care about. Because in the ordered
> mode,
> > > the
> > > > head element in buffer may affect the
> > > > total memory consumption.
> > > >
> > > >
> > > > Thanks,
> > > > Aitozi.
> > > >
> > > >
> > > >
> > > > Jing Ge <ji...@ververica.com.invalid> 于2023年6月12日周一 20:28写道：
> > > >
> > > >> Hi Aitozi,
> > > >>
> > > >> Which key will be used for lookup is not an issue, only one row will
> > be
> > > >> required for each key in order to enrich it. True, it depends on the
> > > >> implementation whether multiple rows or single row for each key will
> > be
> > > >> returned. However, for the lookup & enrichment scenario, one row/key
> > is
> > > >> recommended, otherwise, like I mentioned previously, enrichment
> won't
> > > >> work.
> > > >>
> > > >> I am a little bit concerned about returning a big table for each
> key,
> > > >> since
> > > >> it will take the async call longer to return and need more memory.
> The
> > > >> performance tests should cover this scenario. This is not a blocking
> > > issue
> > > >> for this FLIP.
> > > >>
> > > >> Best regards,
> > > >> Jing
> > > >>
> > > >> On Sat, Jun 10, 2023 at 4:11 AM Aitozi <gj...@gmail.com>
> wrote:
> > > >>
> > > >> > Hi Jing,
> > > >> >     I means the join key is not necessary to be the primary key or
> > > >> unique
> > > >> > index of the database.
> > > >> > In this situation, we may queried out multi rows for one join
> key. I
> > > >> think
> > > >> > that's why the
> > > >> > LookupFunction#lookup will return a collection of RowData.
> > > >> >
> > > >> > BTW, I think the behavior of lookup join will not affect the
> > semantic
> > > of
> > > >> > the async udtf.
> > > >> > We use the Async TableFunction here and the table function can
> > collect
> > > >> > multiple rows.
> > > >> >
> > > >> > Thanks,
> > > >> > Atiozi.
> > > >> >
> > > >> >
> > > >> >
> > > >> > Jing Ge <ji...@ververica.com.invalid> 于2023年6月10日周六 00:15写道：
> > > >> >
> > > >> > > Hi Aitozi,
> > > >> > >
> > > >> > > The keyRow used in this case contains all keys[1].
> > > >> > >
> > > >> > > Best regards,
> > > >> > > Jing
> > > >> > >
> > > >> > > [1]
> > > >> > >
> > > >> > >
> > > >> >
> > > >>
> > >
> >
> https://github.com/apache/flink/blob/191ec6ca3943d7119f14837efe112e074d815c47/flink-table/flink-table-common/src/main/java/org/apache/flink/table/functions/LookupFunction.java#L49
> > > >> > >
> > > >> > >
> > > >> > > On Fri, Jun 9, 2023 at 3:42 PM Aitozi <gj...@gmail.com>
> > wrote:
> > > >> > >
> > > >> > > > Hi Jing,
> > > >> > > >
> > > >> > > >      The performance test is added to the FLIP.
> > > >> > > >
> > > >> > > >      As I know, The lookup join can return multi rows, it
> > depends
> > > on
> > > >> > > > whether  the join key
> > > >> > > > is the primary key of the external database or not. The
> `lookup`
> > > [1]
> > > >> > will
> > > >> > > > return a collection of
> > > >> > > > joined result, and each of them will be collected
> > > >> > > >
> > > >> > > >
> > > >> > > > [1]:
> > > >> > > >
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > >
> >
> https://github.com/apache/flink/blob/191ec6ca3943d7119f14837efe112e074d815c47/flink-table/flink-table-common/src/main/java/org/apache/flink/table/functions/LookupFunction.java#L52
> > > >> > > >
> > > >> > > >
> > > >> > > > Thanks,
> > > >> > > > Aitozi.
> > > >> > > >
> > > >> > > > Jing Ge <ji...@ververica.com.invalid> 于2023年6月9日周五 17:05写道：
> > > >> > > >
> > > >> > > > > Hi Aitozi,
> > > >> > > > >
> > > >> > > > > Thanks for the feedback. Looking forward to the performance
> > > tests.
> > > >> > > > >
> > > >> > > > > Afaik, lookup returns one row for each key [1] [2].
> > > Conceptually,
> > > >> the
> > > >> > > > > lookup function is used to enrich column(s) from the
> dimension
> > > >> table.
> > > >> > > If,
> > > >> > > > > for the given key, there will be more than one row, there
> will
> > > be
> > > >> no
> > > >> > > way
> > > >> > > > to
> > > >> > > > > know which row will be used to enrich the key.
> > > >> > > > >
> > > >> > > > > [1]
> > > >> > > > >
> > > >> > > > >
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > >
> >
> https://github.com/apache/flink/blob/191ec6ca3943d7119f14837efe112e074d815c47/flink-table/flink-table-common/src/main/java/org/apache/flink/table/functions/LookupFunction.java#L49
> > > >> > > > > [2]
> > > >> > > > >
> > > >> > > > >
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > >
> >
> https://github.com/apache/flink/blob/191ec6ca3943d7119f14837efe112e074d815c47/flink-table/flink-table-common/src/main/java/org/apache/flink/table/functions/TableFunction.java#L196
> > > >> > > > >
> > > >> > > > > Best regards,
> > > >> > > > > Jing
> > > >> > > > >
> > > >> > > > > On Fri, Jun 9, 2023 at 5:18 AM Aitozi <gjying1314@gmail.com
> >
> > > >> wrote:
> > > >> > > > >
> > > >> > > > > > Hi Jing
> > > >> > > > > >     Thanks for your good questions. I have updated the
> > example
> > > >> to
> > > >> > the
> > > >> > > > > FLIP.
> > > >> > > > > >
> > > >> > > > > > > Only one row for each lookup
> > > >> > > > > > lookup can also return multi rows, based on the query
> > result.
> > > >> [1]
> > > >> > > > > >
> > > >> > > > > > [1]:
> > > >> > > > > >
> > > >> > > > > >
> > > >> > > > >
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > >
> >
> https://github.com/apache/flink/blob/191ec6ca3943d7119f14837efe112e074d815c47/flink-table/flink-table-common/src/main/java/org/apache/flink/table/functions/LookupFunction.java#L56
> > > >> > > > > >
> > > >> > > > > > > If we use async calls with lateral join, my gut feeling
> is
> > > >> > > > > > that we might have many more async calls than lookup
> join. I
> > > am
> > > >> not
> > > >> > > > > really
> > > >> > > > > > sure if we will be facing potential issues in this case or
> > > not.
> > > >> > > > > >
> > > >> > > > > > IMO, the work pattern is similar to the lookup function,
> for
> > > >> each
> > > >> > row
> > > >> > > > > from
> > > >> > > > > > the left table,
> > > >> > > > > > it will evaluate the eval method once, so the async call
> > > numbers
> > > >> > will
> > > >> > > > not
> > > >> > > > > > change.
> > > >> > > > > > and the maximum calls in flight is limited by the Async
> > > >> operators
> > > >> > > > buffer
> > > >> > > > > > capacity
> > > >> > > > > > which will be controlled by the option.
> > > >> > > > > >
> > > >> > > > > > BTW, for the naming of these option, I updated the FLIP
> > about
> > > >> this
> > > >> > > you
> > > >> > > > > can
> > > >> > > > > > refer to
> > > >> > > > > > the section of "ConfigOption" and "Rejected Alternatives"
> > > >> > > > > >
> > > >> > > > > > In the end, for the performance evaluation, I'd like to do
> > > some
> > > >> > tests
> > > >> > > > and
> > > >> > > > > > will update it to the FLIP doc
> > > >> > > > > >
> > > >> > > > > > Thanks,
> > > >> > > > > > Aitozi.
> > > >> > > > > >
> > > >> > > > > >
> > > >> > > > > > Jing Ge <ji...@ververica.com.invalid> 于2023年6月9日周五
> 07:23写道：
> > > >> > > > > >
> > > >> > > > > > > Hi Aitozi,
> > > >> > > > > > >
> > > >> > > > > > > Thanks for the clarification. The code example looks
> > > >> > interesting. I
> > > >> > > > > would
> > > >> > > > > > > suggest adding them into the FLIP. The description with
> > code
> > > >> > > examples
> > > >> > > > > > will
> > > >> > > > > > > help readers understand the motivation and how to use
> it.
> > > >> Afaiac,
> > > >> > > it
> > > >> > > > > is a
> > > >> > > > > > > valid feature for Flink users.
> > > >> > > > > > >
> > > >> > > > > > > As we knew, lookup join is based on temporal join, i.e.
> > FOR
> > > >> > > > SYSTEM_TIME
> > > >> > > > > > AS
> > > >> > > > > > > OF which is also used in your code example. Temporal
> join
> > > >> > performs
> > > >> > > > the
> > > >> > > > > > > lookup based on the processing time match. Only one row
> > for
> > > >> each
> > > >> > > > > > > lookup(afaiu, I need to check the source code to double
> > > >> confirm)
> > > >> > > will
> > > >> > > > > > > return for further enrichment. One the other hand,
> lateral
> > > >> join
> > > >> > > will
> > > >> > > > > have
> > > >> > > > > > > sub-queries correlated with every individual value of
> the
> > > >> > reference
> > > >> > > > > table
> > > >> > > > > > > from the preceding part of the query and each sub query
> > will
> > > >> > return
> > > >> > > > > > > multiple rows. If we use async calls with lateral join,
> my
> > > gut
> > > >> > > > feeling
> > > >> > > > > is
> > > >> > > > > > > that we might have many more async calls than lookup
> > join. I
> > > >> am
> > > >> > not
> > > >> > > > > > really
> > > >> > > > > > > sure if we will be facing potential issues in this case
> or
> > > >> not.
> > > >> > > > > Possible
> > > >> > > > > > > issues I can think of now e.g. too many PRC calls, too
> > many
> > > >> async
> > > >> > > > calls
> > > >> > > > > > > processing, the sub query will return a table which
> might
> > be
> > > >> > (too)
> > > >> > > > big,
> > > >> > > > > > and
> > > >> > > > > > > might cause performance issues. I would suggest
> preparing
> > > some
> > > >> > use
> > > >> > > > > cases
> > > >> > > > > > > and running some performance tests to check it. These
> are
> > my
> > > >> > > concerns
> > > >> > > > > > about
> > > >> > > > > > > using async calls with lateral join and I'd like to
> share
> > > with
> > > >> > you,
> > > >> > > > > happy
> > > >> > > > > > > to discuss with you and hear different opinions,
> hopefully
> > > the
> > > >> > > > > > > discussion could help me understand it more deeply.
> Please
> > > >> > correct
> > > >> > > me
> > > >> > > > > if
> > > >> > > > > > I
> > > >> > > > > > > am wrong.
> > > >> > > > > > >
> > > >> > > > > > > Best regards,
> > > >> > > > > > > Jing
> > > >> > > > > > >
> > > >> > > > > > >
> > > >> > > > > > > On Thu, Jun 8, 2023 at 7:22 AM Aitozi <
> > gjying1314@gmail.com
> > > >
> > > >> > > wrote:
> > > >> > > > > > >
> > > >> > > > > > > > Hi Mason,
> > > >> > > > > > > >     Thanks for your input. I think if we support the
> > user
> > > >> > defined
> > > >> > > > > async
> > > >> > > > > > > > table function,
> > > >> > > > > > > > user will be able to use it to hold a batch data then
> > > >> handle it
> > > >> > > at
> > > >> > > > > one
> > > >> > > > > > > time
> > > >> > > > > > > > in the customized function.
> > > >> > > > > > > >
> > > >> > > > > > > > AsyncSink is meant for the sink operator. I have not
> > > figure
> > > >> out
> > > >> > > how
> > > >> > > > > to
> > > >> > > > > > > > integrate in this case.
> > > >> > > > > > > >
> > > >> > > > > > > > Thanks,
> > > >> > > > > > > > Atiozi.
> > > >> > > > > > > >
> > > >> > > > > > > >
> > > >> > > > > > > > Mason Chen <ma...@gmail.com> 于2023年6月8日周四
> > 12:40写道：
> > > >> > > > > > > >
> > > >> > > > > > > > > Hi Aitozi,
> > > >> > > > > > > > >
> > > >> > > > > > > > > I think it makes sense to make it easier for SQL
> users
> > > to
> > > >> > make
> > > >> > > > > RPCs.
> > > >> > > > > > Do
> > > >> > > > > > > > you
> > > >> > > > > > > > > think your proposal can extend to the ability to
> batch
> > > >> data
> > > >> > for
> > > >> > > > the
> > > >> > > > > > > RPC?
> > > >> > > > > > > > > This is also another common strategy to increase
> > > >> throughput.
> > > >> > > > Also,
> > > >> > > > > > have
> > > >> > > > > > > > you
> > > >> > > > > > > > > considered solving this a bit differently by
> > leveraging
> > > >> > Flink's
> > > >> > > > > > > > AsyncSink?
> > > >> > > > > > > > >
> > > >> > > > > > > > > Best,
> > > >> > > > > > > > > Mason
> > > >> > > > > > > > >
> > > >> > > > > > > > > On Mon, Jun 5, 2023 at 1:50 AM Aitozi <
> > > >> gjying1314@gmail.com>
> > > >> > > > > wrote:
> > > >> > > > > > > > >
> > > >> > > > > > > > > > One more thing for discussion:
> > > >> > > > > > > > > >
> > > >> > > > > > > > > > In our internal implementation, we reuse the
> option
> > > >> > > > > > > > > > `table.exec.async-lookup.buffer-capacity` and
> > > >> > > > > > > > > > `table.exec.async-lookup.timeout` to config
> > > >> > > > > > > > > > the async udtf. Do you think we should add two
> extra
> > > >> option
> > > >> > > to
> > > >> > > > > > > > > distinguish
> > > >> > > > > > > > > > from the lookup option such as
> > > >> > > > > > > > > >
> > > >> > > > > > > > > > `table.exec.async-udtf.buffer-capacity`
> > > >> > > > > > > > > > `table.exec.async-udtf.timeout`
> > > >> > > > > > > > > >
> > > >> > > > > > > > > >
> > > >> > > > > > > > > > Best,
> > > >> > > > > > > > > > Aitozi.
> > > >> > > > > > > > > >
> > > >> > > > > > > > > >
> > > >> > > > > > > > > >
> > > >> > > > > > > > > > Aitozi <gj...@gmail.com> 于2023年6月5日周一
> 12:20写道：
> > > >> > > > > > > > > >
> > > >> > > > > > > > > > > Hi Jing,
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > > >     > what is the difference between the RPC
> call
> > or
> > > >> > query
> > > >> > > > you
> > > >> > > > > > > > > mentioned
> > > >> > > > > > > > > > > and the lookup in a very
> > > >> > > > > > > > > > > general way
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > > > I think the RPC call or query service is quite
> > > >> similar to
> > > >> > > the
> > > >> > > > > > > lookup
> > > >> > > > > > > > > > join.
> > > >> > > > > > > > > > > But lookup join should work
> > > >> > > > > > > > > > > with `LookupTableSource`.
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > > > Let's see how we can perform an async RPC call
> > with
> > > >> > lookup
> > > >> > > > > join:
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > > > (1) Implement an AsyncTableFunction with RPC
> call
> > > >> logic.
> > > >> > > > > > > > > > > (2) Implement a `LookupTableSource` connector
> run
> > > with
> > > >> > the
> > > >> > > > > async
> > > >> > > > > > > udtf
> > > >> > > > > > > > > > > defined in (1).
> > > >> > > > > > > > > > > (3) Then define a DDL of this look up table in
> SQL
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > > > CREATE TEMPORARY TABLE Customers (
> > > >> > > > > > > > > > >   id INT,
> > > >> > > > > > > > > > >   name STRING,
> > > >> > > > > > > > > > >   country STRING,
> > > >> > > > > > > > > > >   zip STRING
> > > >> > > > > > > > > > > ) WITH (
> > > >> > > > > > > > > > >   'connector' = 'custom'
> > > >> > > > > > > > > > > );
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > > > (4) Run with the query as below:
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > > > SELECT o.order_id, o.total, c.country, c.zip
> > > >> > > > > > > > > > > FROM Orders AS o
> > > >> > > > > > > > > > >   JOIN Customers FOR SYSTEM_TIME AS OF
> o.proc_time
> > > AS
> > > >> c
> > > >> > > > > > > > > > >     ON o.customer_id = c.id;
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > > > This example is from doc
> > > >> > > > > > > > > > > <
> > > >> > > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > >
> > > >> > > > > > >
> > > >> > > > > >
> > > >> > > > >
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > >
> >
> https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/dev/table/sql/queries/joins/#lookup-join
> > > >> > > > > > > > > > >.You
> > > >> > > > > > > > > > > can image the look up process as an async RPC
> call
> > > >> > process.
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > > > Let's see how we can perform an async RPC call
> > with
> > > >> > lateral
> > > >> > > > > join:
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > > > (1) Implement an AsyncTableFunction with RPC
> call
> > > >> logic.
> > > >> > > > > > > > > > > (2) Run query with
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > > > Create function f1 as '...' ;
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > > > SELECT o.order_id, o.total, c.country, c.zip
> FROM
> > > >> Orders
> > > >> > > > > lateral
> > > >> > > > > > > > table
> > > >> > > > > > > > > > > (f1(order_id)) as T(...);
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > > > As you can see, the lateral join version is more
> > > >> simple
> > > >> > and
> > > >> > > > > > > intuitive
> > > >> > > > > > > > > to
> > > >> > > > > > > > > > > users. Users do not have to wrap a
> > > >> > > > > > > > > > > LookupTableSource for the purpose of using async
> > > udtf.
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > > > In the end, We can also see the user defined
> async
> > > >> table
> > > >> > > > > function
> > > >> > > > > > > is
> > > >> > > > > > > > an
> > > >> > > > > > > > > > > enhancement of the current lateral table join
> > > >> > > > > > > > > > > which only supports sync lateral join now.
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > > > Best,
> > > >> > > > > > > > > > > Aitozi.
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > > > Jing Ge <ji...@ververica.com.invalid>
> 于2023年6月2日周五
> > > >> > 19:37写道：
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > > >> Hi Aitozi,
> > > >> > > > > > > > > > >>
> > > >> > > > > > > > > > >> Thanks for the update. Just out of curiosity,
> > what
> > > is
> > > >> > the
> > > >> > > > > > > difference
> > > >> > > > > > > > > > >> between the RPC call or query you mentioned and
> > the
> > > >> > lookup
> > > >> > > > in
> > > >> > > > > a
> > > >> > > > > > > very
> > > >> > > > > > > > > > >> general way? Since Lateral join is used in the
> > > FLIP.
> > > >> Is
> > > >> > > > there
> > > >> > > > > > any
> > > >> > > > > > > > > > special
> > > >> > > > > > > > > > >> thought for that? Sorry for asking so many
> > > questions.
> > > >> > The
> > > >> > > > FLIP
> > > >> > > > > > > > > contains
> > > >> > > > > > > > > > >> limited information to understand the
> motivation.
> > > >> > > > > > > > > > >>
> > > >> > > > > > > > > > >> Best regards,
> > > >> > > > > > > > > > >> Jing
> > > >> > > > > > > > > > >>
> > > >> > > > > > > > > > >> On Fri, Jun 2, 2023 at 3:48 AM Aitozi <
> > > >> > > gjying1314@gmail.com
> > > >> > > > >
> > > >> > > > > > > wrote:
> > > >> > > > > > > > > > >>
> > > >> > > > > > > > > > >> > Hi Jing,
> > > >> > > > > > > > > > >> >     I have updated the proposed changes to
> the
> > > >> FLIP.
> > > >> > > IMO,
> > > >> > > > > > lookup
> > > >> > > > > > > > has
> > > >> > > > > > > > > > its
> > > >> > > > > > > > > > >> > clear
> > > >> > > > > > > > > > >> > async call requirement is due to its IO heavy
> > > >> > operator.
> > > >> > > In
> > > >> > > > > our
> > > >> > > > > > > > > usage,
> > > >> > > > > > > > > > >> sql
> > > >> > > > > > > > > > >> > users have
> > > >> > > > > > > > > > >> > logic to do some RPC call or query the
> > > third-party
> > > >> > > service
> > > >> > > > > > which
> > > >> > > > > > > > is
> > > >> > > > > > > > > > >> also IO
> > > >> > > > > > > > > > >> > intensive.
> > > >> > > > > > > > > > >> > In these case, we'd like to leverage the
> async
> > > >> > function
> > > >> > > to
> > > >> > > > > > > improve
> > > >> > > > > > > > > the
> > > >> > > > > > > > > > >> > throughput.
> > > >> > > > > > > > > > >> >
> > > >> > > > > > > > > > >> > Thanks,
> > > >> > > > > > > > > > >> > Aitozi.
> > > >> > > > > > > > > > >> >
> > > >> > > > > > > > > > >> > Jing Ge <ji...@ververica.com.invalid>
> > > 于2023年6月1日周四
> > > >> > > > 22:55写道：
> > > >> > > > > > > > > > >> >
> > > >> > > > > > > > > > >> > > Hi Aitozi,
> > > >> > > > > > > > > > >> > >
> > > >> > > > > > > > > > >> > > Sorry for the late reply. Would you like to
> > > >> update
> > > >> > the
> > > >> > > > > > > proposed
> > > >> > > > > > > > > > >> changes
> > > >> > > > > > > > > > >> > > with more details into the FLIP too?
> > > >> > > > > > > > > > >> > > I got your point. It looks like a rational
> > > idea.
> > > >> > > > However,
> > > >> > > > > > > since
> > > >> > > > > > > > > > lookup
> > > >> > > > > > > > > > >> > has
> > > >> > > > > > > > > > >> > > its clear async call requirement, are there
> > any
> > > >> real
> > > >> > > use
> > > >> > > > > > cases
> > > >> > > > > > > > > that
> > > >> > > > > > > > > > >> > > need this change? This will help us
> > understand
> > > >> the
> > > >> > > > > > motivation.
> > > >> > > > > > > > > After
> > > >> > > > > > > > > > >> all,
> > > >> > > > > > > > > > >> > > lateral join and temporal lookup join[1]
> are
> > > >> quite
> > > >> > > > > > different.
> > > >> > > > > > > > > > >> > >
> > > >> > > > > > > > > > >> > > Best regards,
> > > >> > > > > > > > > > >> > > Jing
> > > >> > > > > > > > > > >> > >
> > > >> > > > > > > > > > >> > >
> > > >> > > > > > > > > > >> > > [1]
> > > >> > > > > > > > > > >> > >
> > > >> > > > > > > > > > >> > >
> > > >> > > > > > > > > > >> >
> > > >> > > > > > > > > > >>
> > > >> > > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > >
> > > >> > > > > > >
> > > >> > > > > >
> > > >> > > > >
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > >
> >
> https://github.com/apache/flink/blob/d90a72da2fd601ca4e2a46700e91ec5b348de2ad/flink-table/flink-table-common/src/main/java/org/apache/flink/table/functions/AsyncTableFunction.java#L54
> > > >> > > > > > > > > > >> > >
> > > >> > > > > > > > > > >> > > On Wed, May 31, 2023 at 8:53 AM Aitozi <
> > > >> > > > > > gjying1314@gmail.com>
> > > >> > > > > > > > > > wrote:
> > > >> > > > > > > > > > >> > >
> > > >> > > > > > > > > > >> > > > Hi Jing,
> > > >> > > > > > > > > > >> > > >     What do you think about it? Can we
> move
> > > >> > forward
> > > >> > > > this
> > > >> > > > > > > > > feature?
> > > >> > > > > > > > > > >> > > >
> > > >> > > > > > > > > > >> > > > Thanks,
> > > >> > > > > > > > > > >> > > > Aitozi.
> > > >> > > > > > > > > > >> > > >
> > > >> > > > > > > > > > >> > > > Aitozi <gj...@gmail.com>
> > 于2023年5月29日周一
> > > >> > > 09:56写道：
> > > >> > > > > > > > > > >> > > >
> > > >> > > > > > > > > > >> > > > > Hi Jing,
> > > >> > > > > > > > > > >> > > > >     > "Do you mean to support the
> > > >> > > AyncTableFunction
> > > >> > > > > > beyond
> > > >> > > > > > > > the
> > > >> > > > > > > > > > >> > > > > LookupTableSource?"
> > > >> > > > > > > > > > >> > > > > Yes, I mean to support the
> > > AyncTableFunction
> > > >> > > beyond
> > > >> > > > > the
> > > >> > > > > > > > > > >> > > > LookupTableSource.
> > > >> > > > > > > > > > >> > > > >
> > > >> > > > > > > > > > >> > > > > The "AsyncTableFunction" is the
> function
> > > with
> > > >> > > > ability
> > > >> > > > > to
> > > >> > > > > > > be
> > > >> > > > > > > > > > >> executed
> > > >> > > > > > > > > > >> > > > async
> > > >> > > > > > > > > > >> > > > > (with AsyncWaitOperator).
> > > >> > > > > > > > > > >> > > > > The async lookup join is a one of usage
> > of
> > > >> this.
> > > >> > > So,
> > > >> > > > > we
> > > >> > > > > > > > don't
> > > >> > > > > > > > > > >> have to
> > > >> > > > > > > > > > >> > > > bind
> > > >> > > > > > > > > > >> > > > > the AyncTableFunction with
> > > LookupTableSource.
> > > >> > > > > > > > > > >> > > > > If User-defined AsyncTableFunction is
> > > >> supported,
> > > >> > > > user
> > > >> > > > > > can
> > > >> > > > > > > > > > directly
> > > >> > > > > > > > > > >> > use
> > > >> > > > > > > > > > >> > > > > lateral table syntax to perform async
> > > >> operation.
> > > >> > > > > > > > > > >> > > > >
> > > >> > > > > > > > > > >> > > > > > "It would be better if you could
> > > elaborate
> > > >> the
> > > >> > > > > > proposed
> > > >> > > > > > > > > > changes
> > > >> > > > > > > > > > >> wrt
> > > >> > > > > > > > > > >> > > the
> > > >> > > > > > > > > > >> > > > > CorrelatedCodeGenerator with more
> > details"
> > > >> > > > > > > > > > >> > > > >
> > > >> > > > > > > > > > >> > > > > In the proposal, we use lateral table
> > > syntax
> > > >> to
> > > >> > > > > support
> > > >> > > > > > > the
> > > >> > > > > > > > > > async
> > > >> > > > > > > > > > >> > table
> > > >> > > > > > > > > > >> > > > > function. So the planner will also
> treat
> > > this
> > > >> > > > > statement
> > > >> > > > > > > to a
> > > >> > > > > > > > > > >> > > > > CommonExecCorrelate node. So the
> runtime
> > > code
> > > >> > > should
> > > >> > > > > be
> > > >> > > > > > > > > > generated
> > > >> > > > > > > > > > >> in
> > > >> > > > > > > > > > >> > > > > CorrelatedCodeGenerator.
> > > >> > > > > > > > > > >> > > > > In CorrelatedCodeGenerator, we will
> know
> > > the
> > > >> > > > > > > TableFunction's
> > > >> > > > > > > > > > Kind
> > > >> > > > > > > > > > >> of
> > > >> > > > > > > > > > >> > > > > `FunctionKind.Table` or
> > > >> > `FunctionKind.ASYNC_TABLE`
> > > >> > > > > > > > > > >> > > > > For  `FunctionKind.ASYNC_TABLE` we can
> > > >> generate
> > > >> > a
> > > >> > > > > > > > > > >> AsyncWaitOperator
> > > >> > > > > > > > > > >> > to
> > > >> > > > > > > > > > >> > > > > execute the async table function.
> > > >> > > > > > > > > > >> > > > >
> > > >> > > > > > > > > > >> > > > >
> > > >> > > > > > > > > > >> > > > > Thanks,
> > > >> > > > > > > > > > >> > > > > Aitozi.
> > > >> > > > > > > > > > >> > > > >
> > > >> > > > > > > > > > >> > > > >
> > > >> > > > > > > > > > >> > > > > Jing Ge <ji...@ververica.com.invalid>
> > > >> > > 于2023年5月29日周一
> > > >> > > > > > > 03:22写道：
> > > >> > > > > > > > > > >> > > > >
> > > >> > > > > > > > > > >> > > > >> Hi Aitozi,
> > > >> > > > > > > > > > >> > > > >>
> > > >> > > > > > > > > > >> > > > >> Thanks for the clarification. The
> naming
> > > >> > "Lookup"
> > > >> > > > > might
> > > >> > > > > > > > > suggest
> > > >> > > > > > > > > > >> > using
> > > >> > > > > > > > > > >> > > it
> > > >> > > > > > > > > > >> > > > >> for table look up. But conceptually
> what
> > > the
> > > >> > > eval()
> > > >> > > > > > > method
> > > >> > > > > > > > > will
> > > >> > > > > > > > > > >> do
> > > >> > > > > > > > > > >> > is
> > > >> > > > > > > > > > >> > > to
> > > >> > > > > > > > > > >> > > > >> get a collection of results(Row,
> > RowData)
> > > >> from
> > > >> > > the
> > > >> > > > > > given
> > > >> > > > > > > > > keys.
> > > >> > > > > > > > > > >> How
> > > >> > > > > > > > > > >> > it
> > > >> > > > > > > > > > >> > > > will
> > > >> > > > > > > > > > >> > > > >> be done depends on the implementation,
> > > i.e.
> > > >> you
> > > >> > > can
> > > >> > > > > > > > implement
> > > >> > > > > > > > > > >> your
> > > >> > > > > > > > > > >> > own
> > > >> > > > > > > > > > >> > > > >> Source[1][2]. The example in the FLIP
> > > >> should be
> > > >> > > > able
> > > >> > > > > to
> > > >> > > > > > > be
> > > >> > > > > > > > > > >> handled
> > > >> > > > > > > > > > >> > in
> > > >> > > > > > > > > > >> > > > this
> > > >> > > > > > > > > > >> > > > >> way.
> > > >> > > > > > > > > > >> > > > >>
> > > >> > > > > > > > > > >> > > > >> Do you mean to support the
> > > AyncTableFunction
> > > >> > > beyond
> > > >> > > > > the
> > > >> > > > > > > > > > >> > > > LookupTableSource?
> > > >> > > > > > > > > > >> > > > >> It would be better if you could
> > elaborate
> > > >> the
> > > >> > > > > proposed
> > > >> > > > > > > > > changes
> > > >> > > > > > > > > > >> wrt
> > > >> > > > > > > > > > >> > the
> > > >> > > > > > > > > > >> > > > >> CorrelatedCodeGenerator with more
> > details.
> > > >> > > Thanks!
> > > >> > > > > > > > > > >> > > > >>
> > > >> > > > > > > > > > >> > > > >> Best regards,
> > > >> > > > > > > > > > >> > > > >> Jing
> > > >> > > > > > > > > > >> > > > >>
> > > >> > > > > > > > > > >> > > > >> [1]
> > > >> > > > > > > > > > >> > > > >>
> > > >> > > > > > > > > > >> > > > >>
> > > >> > > > > > > > > > >> > > >
> > > >> > > > > > > > > > >> > >
> > > >> > > > > > > > > > >> >
> > > >> > > > > > > > > > >>
> > > >> > > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > >
> > > >> > > > > > >
> > > >> > > > > >
> > > >> > > > >
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > >
> >
> https://github.com/apache/flink/blob/678370b18e1b6c4a23e5ce08f8efd05675a0cc17/flink-table/flink-table-common/src/main/java/org/apache/flink/table/connector/source/LookupTableSource.java#L64
> > > >> > > > > > > > > > >> > > > >> [2]
> > > >> > > > > > > > > > >> > > > >>
> > > >> > > > > > > > > > >> > > > >>
> > > >> > > > > > > > > > >> > > >
> > > >> > > > > > > > > > >> > >
> > > >> > > > > > > > > > >> >
> > > >> > > > > > > > > > >>
> > > >> > > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > >
> > > >> > > > > > >
> > > >> > > > > >
> > > >> > > > >
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > >
> >
> https://github.com/apache/flink/blob/678370b18e1b6c4a23e5ce08f8efd05675a0cc17/flink-table/flink-table-common/src/main/java/org/apache/flink/table/connector/source/AsyncTableFunctionProvider.java#L49
> > > >> > > > > > > > > > >> > > > >>
> > > >> > > > > > > > > > >> > > > >> On Sat, May 27, 2023 at 9:48 AM
> Aitozi <
> > > >> > > > > > > > gjying1314@gmail.com
> > > >> > > > > > > > > >
> > > >> > > > > > > > > > >> > wrote:
> > > >> > > > > > > > > > >> > > > >>
> > > >> > > > > > > > > > >> > > > >> > Hi Jing,
> > > >> > > > > > > > > > >> > > > >> >     Thanks for your response. As
> > stated
> > > in
> > > >> > the
> > > >> > > > > FLIP,
> > > >> > > > > > > the
> > > >> > > > > > > > > > >> purpose
> > > >> > > > > > > > > > >> > of
> > > >> > > > > > > > > > >> > > > this
> > > >> > > > > > > > > > >> > > > >> > FLIP is meant to support
> > > >> > > > > > > > > > >> > > > >> > user-defined async table function.
> As
> > > >> > described
> > > >> > > > in
> > > >> > > > > > > flink
> > > >> > > > > > > > > > >> document
> > > >> > > > > > > > > > >> > > [1]
> > > >> > > > > > > > > > >> > > > >> >
> > > >> > > > > > > > > > >> > > > >> > Async table functions are special
> > > >> functions
> > > >> > for
> > > >> > > > > table
> > > >> > > > > > > > > sources
> > > >> > > > > > > > > > >> that
> > > >> > > > > > > > > > >> > > > >> perform
> > > >> > > > > > > > > > >> > > > >> > > a lookup.
> > > >> > > > > > > > > > >> > > > >> > >
> > > >> > > > > > > > > > >> > > > >> >
> > > >> > > > > > > > > > >> > > > >> > So end user can not directly define
> > and
> > > >> use
> > > >> > > async
> > > >> > > > > > table
> > > >> > > > > > > > > > >> function
> > > >> > > > > > > > > > >> > > now.
> > > >> > > > > > > > > > >> > > > An
> > > >> > > > > > > > > > >> > > > >> > user case is reported in [2]
> > > >> > > > > > > > > > >> > > > >> >
> > > >> > > > > > > > > > >> > > > >> > So, in conclusion, no new interface
> is
> > > >> > > > introduced,
> > > >> > > > > > but
> > > >> > > > > > > we
> > > >> > > > > > > > > > >> extend
> > > >> > > > > > > > > > >> > the
> > > >> > > > > > > > > > >> > > > >> > ability to support user-defined
> async
> > > >> table
> > > >> > > > > function.
> > > >> > > > > > > > > > >> > > > >> >
> > > >> > > > > > > > > > >> > > > >> > [1]:
> > > >> > > > > > > > > > >> > > > >> >
> > > >> > > > > > > > > > >> > > > >> >
> > > >> > > > > > > > > > >> > > > >>
> > > >> > > > > > > > > > >> > > >
> > > >> > > > > > > > > > >> > >
> > > >> > > > > > > > > > >> >
> > > >> > > > > > > > > > >>
> > > >> > > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > >
> > > >> > > > > > >
> > > >> > > > > >
> > > >> > > > >
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > >
> >
> https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/dev/table/functions/udfs/
> > > >> > > > > > > > > > >> > > > >> > [2]:
> > > >> > > > > > > > > > >> > >
> > > >> > > > > > > >
> > > >> > https://lists.apache.org/thread/qljwd40v5ntz6733cwcdr8s4z97b343b
> > > >> > > > > > > > > > >> > > > >> >
> > > >> > > > > > > > > > >> > > > >> > Thanks.
> > > >> > > > > > > > > > >> > > > >> > Aitozi.
> > > >> > > > > > > > > > >> > > > >> >
> > > >> > > > > > > > > > >> > > > >> >
> > > >> > > > > > > > > > >> > > > >> > Jing Ge <jing@ververica.com.invalid
> >
> > > >> > > > 于2023年5月27日周六
> > > >> > > > > > > > > 06:40写道：
> > > >> > > > > > > > > > >> > > > >> >
> > > >> > > > > > > > > > >> > > > >> > > Hi Aitozi,
> > > >> > > > > > > > > > >> > > > >> > >
> > > >> > > > > > > > > > >> > > > >> > > Thanks for your proposal. I am not
> > > quite
> > > >> > sure
> > > >> > > > if
> > > >> > > > > I
> > > >> > > > > > > > > > understood
> > > >> > > > > > > > > > >> > your
> > > >> > > > > > > > > > >> > > > >> > thoughts
> > > >> > > > > > > > > > >> > > > >> > > correctly. You described a special
> > > case
> > > >> > > > > > > implementation
> > > >> > > > > > > > of
> > > >> > > > > > > > > > the
> > > >> > > > > > > > > > >> > > > >> > > AsyncTableFunction with on public
> > API
> > > >> > > changes.
> > > >> > > > > > Would
> > > >> > > > > > > > you
> > > >> > > > > > > > > > >> please
> > > >> > > > > > > > > > >> > > > >> elaborate
> > > >> > > > > > > > > > >> > > > >> > > your purpose of writing a FLIP
> > > >> according to
> > > >> > > the
> > > >> > > > > > FLIP
> > > >> > > > > > > > > > >> > > > documentation[1]?
> > > >> > > > > > > > > > >> > > > >> > > Thanks!
> > > >> > > > > > > > > > >> > > > >> > >
> > > >> > > > > > > > > > >> > > > >> > > [1]
> > > >> > > > > > > > > > >> > > > >> > >
> > > >> > > > > > > > > > >> > > > >> > >
> > > >> > > > > > > > > > >> > > > >> >
> > > >> > > > > > > > > > >> > > > >>
> > > >> > > > > > > > > > >> > > >
> > > >> > > > > > > > > > >> > >
> > > >> > > > > > > > > > >> >
> > > >> > > > > > > > > > >>
> > > >> > > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > >
> > > >> > > > > > >
> > > >> > > > > >
> > > >> > > > >
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/Flink+Improvement+Proposals
> > > >> > > > > > > > > > >> > > > >> > >
> > > >> > > > > > > > > > >> > > > >> > > Best regards,
> > > >> > > > > > > > > > >> > > > >> > > Jing
> > > >> > > > > > > > > > >> > > > >> > >
> > > >> > > > > > > > > > >> > > > >> > > On Wed, May 24, 2023 at 1:07 PM
> > > Aitozi <
> > > >> > > > > > > > > > gjying1314@gmail.com
> > > >> > > > > > > > > > >> >
> > > >> > > > > > > > > > >> > > > wrote:
> > > >> > > > > > > > > > >> > > > >> > >
> > > >> > > > > > > > > > >> > > > >> > > > May I ask for some feedback  :D
> > > >> > > > > > > > > > >> > > > >> > > >
> > > >> > > > > > > > > > >> > > > >> > > > Thanks,
> > > >> > > > > > > > > > >> > > > >> > > > Aitozi
> > > >> > > > > > > > > > >> > > > >> > > >
> > > >> > > > > > > > > > >> > > > >> > > > Aitozi <gj...@gmail.com>
> > > >> > > 于2023年5月23日周二
> > > >> > > > > > > 19:14写道：
> > > >> > > > > > > > > > >> > > > >> > > > >
> > > >> > > > > > > > > > >> > > > >> > > > > Just catch an user case report
> > > from
> > > >> > > Giannis
> > > >> > > > > > > Polyzos
> > > >> > > > > > > > > for
> > > >> > > > > > > > > > >> this
> > > >> > > > > > > > > > >> > > > >> usage:
> > > >> > > > > > > > > > >> > > > >> > > > >
> > > >> > > > > > > > > > >> > > > >> > > > >
> > > >> > > > > > > > > > >> > > >
> > > >> > > > > > > > >
> > > >> > >
> https://lists.apache.org/thread/qljwd40v5ntz6733cwcdr8s4z97b343b
> > > >> > > > > > > > > > >> > > > >> > > > >
> > > >> > > > > > > > > > >> > > > >> > > > > Aitozi <gj...@gmail.com>
> > > >> > > > 于2023年5月23日周二
> > > >> > > > > > > > 17:45写道：
> > > >> > > > > > > > > > >> > > > >> > > > > >
> > > >> > > > > > > > > > >> > > > >> > > > > > Hi guys,
> > > >> > > > > > > > > > >> > > > >> > > > > >     I want to bring up a
> > > >> discussion
> > > >> > > about
> > > >> > > > > > > adding
> > > >> > > > > > > > > > >> support
> > > >> > > > > > > > > > >> > of
> > > >> > > > > > > > > > >> > > > User
> > > >> > > > > > > > > > >> > > > >> > > > > > Defined AsyncTableFunction
> in
> > > >> Flink.
> > > >> > > > > > > > > > >> > > > >> > > > > > Currently, async table
> > function
> > > >> are
> > > >> > > > special
> > > >> > > > > > > > > functions
> > > >> > > > > > > > > > >> for
> > > >> > > > > > > > > > >> > > > table
> > > >> > > > > > > > > > >> > > > >> > > source
> > > >> > > > > > > > > > >> > > > >> > > > > > to perform
> > > >> > > > > > > > > > >> > > > >> > > > > > async lookup. However, it's
> > > worth
> > > >> to
> > > >> > > > > support
> > > >> > > > > > > the
> > > >> > > > > > > > > user
> > > >> > > > > > > > > > >> > > defined
> > > >> > > > > > > > > > >> > > > >> async
> > > >> > > > > > > > > > >> > > > >> > > > > > table function.
> > > >> > > > > > > > > > >> > > > >> > > > > > Because, in this way, the
> end
> > > SQL
> > > >> > user
> > > >> > > > can
> > > >> > > > > > > > leverage
> > > >> > > > > > > > > > it
> > > >> > > > > > > > > > >> to
> > > >> > > > > > > > > > >> > > > >> perform
> > > >> > > > > > > > > > >> > > > >> > the
> > > >> > > > > > > > > > >> > > > >> > > > > > async operation
> > > >> > > > > > > > > > >> > > > >> > > > > > which is useful to maximum
> the
> > > >> system
> > > >> > > > > > > throughput
> > > >> > > > > > > > > > >> > especially
> > > >> > > > > > > > > > >> > > > for
> > > >> > > > > > > > > > >> > > > >> IO
> > > >> > > > > > > > > > >> > > > >> > > > > > bottleneck case.
> > > >> > > > > > > > > > >> > > > >> > > > > >
> > > >> > > > > > > > > > >> > > > >> > > > > > You can find some more
> detail
> > in
> > > >> [1].
> > > >> > > > > > > > > > >> > > > >> > > > > >
> > > >> > > > > > > > > > >> > > > >> > > > > > Looking forward to feedback
> > > >> > > > > > > > > > >> > > > >> > > > > >
> > > >> > > > > > > > > > >> > > > >> > > > > >
> > > >> > > > > > > > > > >> > > > >> > > > > > [1]:
> > > >> > > > > > > > > > >> > > > >> > > >
> > > >> > > > > > > > > > >> > > > >> > >
> > > >> > > > > > > > > > >> > > > >> >
> > > >> > > > > > > > > > >> > > > >>
> > > >> > > > > > > > > > >> > > >
> > > >> > > > > > > > > > >> > >
> > > >> > > > > > > > > > >> >
> > > >> > > > > > > > > > >>
> > > >> > > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > >
> > > >> > > > > > >
> > > >> > > > > >
> > > >> > > > >
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/%5BFLIP-313%5D+Add+support+of+User+Defined+AsyncTableFunction
> > > >> > > > > > > > > > >> > > > >> > > > > >
> > > >> > > > > > > > > > >> > > > >> > > > > > Thanks,
> > > >> > > > > > > > > > >> > > > >> > > > > > Aitozi.
> > > >> > > > > > > > > > >> > > > >> > > >
> > > >> > > > > > > > > > >> > > > >> > >
> > > >> > > > > > > > > > >> > > > >> >
> > > >> > > > > > > > > > >> > > > >>
> > > >> > > > > > > > > > >> > > > >
> > > >> > > > > > > > > > >> > > >
> > > >> > > > > > > > > > >> > >
> > > >> > > > > > > > > > >> >
> > > >> > > > > > > > > > >>
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > >
> > > >> > > > > > >
> > > >> > > > > >
> > > >> > > > >
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > > >
> > >
> >
>

Re: [DISCUSS] FLIP-313 Add support of User Defined AsyncTableFunction

Posted by Feng Jin <ji...@gmail.com>.

Hi Aitozi, We do need to remind users about thread safety issues. Thank you
for your efforts on this FLIP. I have no further questions.
Best, Feng


On Tue, Jun 13, 2023 at 6:05 AM Jing Ge <ji...@ververica.com.invalid> wrote:

> Hi Aitozi,
>
> Thanks for taking care of that part. I have no other concern.
>
> Best regards,
> Jing
>
>
> On Mon, Jun 12, 2023 at 5:38 PM Aitozi <gj...@gmail.com> wrote:
>
> > BTW, If there are no other more blocking issue / comments, I would like
> to
> > start a VOTE in another thread this wednesday 6.14
> >
> > Thanks,
> > Aitozi.
> >
> > Aitozi <gj...@gmail.com> 于2023年6月12日周一 23:34写道：
> >
> > > Hi, Jing,
> > >     Thanks for your explanation. I get your point now.
> > >
> > > For the performance part, I think it's a good idea to run with
> returning
> > a
> > > big table case, the memory consumption
> > > should be a point to be taken care about. Because in the ordered mode,
> > the
> > > head element in buffer may affect the
> > > total memory consumption.
> > >
> > >
> > > Thanks,
> > > Aitozi.
> > >
> > >
> > >
> > > Jing Ge <ji...@ververica.com.invalid> 于2023年6月12日周一 20:28写道：
> > >
> > >> Hi Aitozi,
> > >>
> > >> Which key will be used for lookup is not an issue, only one row will
> be
> > >> required for each key in order to enrich it. True, it depends on the
> > >> implementation whether multiple rows or single row for each key will
> be
> > >> returned. However, for the lookup & enrichment scenario, one row/key
> is
> > >> recommended, otherwise, like I mentioned previously, enrichment won't
> > >> work.
> > >>
> > >> I am a little bit concerned about returning a big table for each key,
> > >> since
> > >> it will take the async call longer to return and need more memory. The
> > >> performance tests should cover this scenario. This is not a blocking
> > issue
> > >> for this FLIP.
> > >>
> > >> Best regards,
> > >> Jing
> > >>
> > >> On Sat, Jun 10, 2023 at 4:11 AM Aitozi <gj...@gmail.com> wrote:
> > >>
> > >> > Hi Jing,
> > >> >     I means the join key is not necessary to be the primary key or
> > >> unique
> > >> > index of the database.
> > >> > In this situation, we may queried out multi rows for one join key. I
> > >> think
> > >> > that's why the
> > >> > LookupFunction#lookup will return a collection of RowData.
> > >> >
> > >> > BTW, I think the behavior of lookup join will not affect the
> semantic
> > of
> > >> > the async udtf.
> > >> > We use the Async TableFunction here and the table function can
> collect
> > >> > multiple rows.
> > >> >
> > >> > Thanks,
> > >> > Atiozi.
> > >> >
> > >> >
> > >> >
> > >> > Jing Ge <ji...@ververica.com.invalid> 于2023年6月10日周六 00:15写道：
> > >> >
> > >> > > Hi Aitozi,
> > >> > >
> > >> > > The keyRow used in this case contains all keys[1].
> > >> > >
> > >> > > Best regards,
> > >> > > Jing
> > >> > >
> > >> > > [1]
> > >> > >
> > >> > >
> > >> >
> > >>
> >
> https://github.com/apache/flink/blob/191ec6ca3943d7119f14837efe112e074d815c47/flink-table/flink-table-common/src/main/java/org/apache/flink/table/functions/LookupFunction.java#L49
> > >> > >
> > >> > >
> > >> > > On Fri, Jun 9, 2023 at 3:42 PM Aitozi <gj...@gmail.com>
> wrote:
> > >> > >
> > >> > > > Hi Jing,
> > >> > > >
> > >> > > >      The performance test is added to the FLIP.
> > >> > > >
> > >> > > >      As I know, The lookup join can return multi rows, it
> depends
> > on
> > >> > > > whether  the join key
> > >> > > > is the primary key of the external database or not. The `lookup`
> > [1]
> > >> > will
> > >> > > > return a collection of
> > >> > > > joined result, and each of them will be collected
> > >> > > >
> > >> > > >
> > >> > > > [1]:
> > >> > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> >
> https://github.com/apache/flink/blob/191ec6ca3943d7119f14837efe112e074d815c47/flink-table/flink-table-common/src/main/java/org/apache/flink/table/functions/LookupFunction.java#L52
> > >> > > >
> > >> > > >
> > >> > > > Thanks,
> > >> > > > Aitozi.
> > >> > > >
> > >> > > > Jing Ge <ji...@ververica.com.invalid> 于2023年6月9日周五 17:05写道：
> > >> > > >
> > >> > > > > Hi Aitozi,
> > >> > > > >
> > >> > > > > Thanks for the feedback. Looking forward to the performance
> > tests.
> > >> > > > >
> > >> > > > > Afaik, lookup returns one row for each key [1] [2].
> > Conceptually,
> > >> the
> > >> > > > > lookup function is used to enrich column(s) from the dimension
> > >> table.
> > >> > > If,
> > >> > > > > for the given key, there will be more than one row, there will
> > be
> > >> no
> > >> > > way
> > >> > > > to
> > >> > > > > know which row will be used to enrich the key.
> > >> > > > >
> > >> > > > > [1]
> > >> > > > >
> > >> > > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> >
> https://github.com/apache/flink/blob/191ec6ca3943d7119f14837efe112e074d815c47/flink-table/flink-table-common/src/main/java/org/apache/flink/table/functions/LookupFunction.java#L49
> > >> > > > > [2]
> > >> > > > >
> > >> > > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> >
> https://github.com/apache/flink/blob/191ec6ca3943d7119f14837efe112e074d815c47/flink-table/flink-table-common/src/main/java/org/apache/flink/table/functions/TableFunction.java#L196
> > >> > > > >
> > >> > > > > Best regards,
> > >> > > > > Jing
> > >> > > > >
> > >> > > > > On Fri, Jun 9, 2023 at 5:18 AM Aitozi <gj...@gmail.com>
> > >> wrote:
> > >> > > > >
> > >> > > > > > Hi Jing
> > >> > > > > >     Thanks for your good questions. I have updated the
> example
> > >> to
> > >> > the
> > >> > > > > FLIP.
> > >> > > > > >
> > >> > > > > > > Only one row for each lookup
> > >> > > > > > lookup can also return multi rows, based on the query
> result.
> > >> [1]
> > >> > > > > >
> > >> > > > > > [1]:
> > >> > > > > >
> > >> > > > > >
> > >> > > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> >
> https://github.com/apache/flink/blob/191ec6ca3943d7119f14837efe112e074d815c47/flink-table/flink-table-common/src/main/java/org/apache/flink/table/functions/LookupFunction.java#L56
> > >> > > > > >
> > >> > > > > > > If we use async calls with lateral join, my gut feeling is
> > >> > > > > > that we might have many more async calls than lookup join. I
> > am
> > >> not
> > >> > > > > really
> > >> > > > > > sure if we will be facing potential issues in this case or
> > not.
> > >> > > > > >
> > >> > > > > > IMO, the work pattern is similar to the lookup function, for
> > >> each
> > >> > row
> > >> > > > > from
> > >> > > > > > the left table,
> > >> > > > > > it will evaluate the eval method once, so the async call
> > numbers
> > >> > will
> > >> > > > not
> > >> > > > > > change.
> > >> > > > > > and the maximum calls in flight is limited by the Async
> > >> operators
> > >> > > > buffer
> > >> > > > > > capacity
> > >> > > > > > which will be controlled by the option.
> > >> > > > > >
> > >> > > > > > BTW, for the naming of these option, I updated the FLIP
> about
> > >> this
> > >> > > you
> > >> > > > > can
> > >> > > > > > refer to
> > >> > > > > > the section of "ConfigOption" and "Rejected Alternatives"
> > >> > > > > >
> > >> > > > > > In the end, for the performance evaluation, I'd like to do
> > some
> > >> > tests
> > >> > > > and
> > >> > > > > > will update it to the FLIP doc
> > >> > > > > >
> > >> > > > > > Thanks,
> > >> > > > > > Aitozi.
> > >> > > > > >
> > >> > > > > >
> > >> > > > > > Jing Ge <ji...@ververica.com.invalid> 于2023年6月9日周五 07:23写道：
> > >> > > > > >
> > >> > > > > > > Hi Aitozi,
> > >> > > > > > >
> > >> > > > > > > Thanks for the clarification. The code example looks
> > >> > interesting. I
> > >> > > > > would
> > >> > > > > > > suggest adding them into the FLIP. The description with
> code
> > >> > > examples
> > >> > > > > > will
> > >> > > > > > > help readers understand the motivation and how to use it.
> > >> Afaiac,
> > >> > > it
> > >> > > > > is a
> > >> > > > > > > valid feature for Flink users.
> > >> > > > > > >
> > >> > > > > > > As we knew, lookup join is based on temporal join, i.e.
> FOR
> > >> > > > SYSTEM_TIME
> > >> > > > > > AS
> > >> > > > > > > OF which is also used in your code example. Temporal join
> > >> > performs
> > >> > > > the
> > >> > > > > > > lookup based on the processing time match. Only one row
> for
> > >> each
> > >> > > > > > > lookup(afaiu, I need to check the source code to double
> > >> confirm)
> > >> > > will
> > >> > > > > > > return for further enrichment. One the other hand, lateral
> > >> join
> > >> > > will
> > >> > > > > have
> > >> > > > > > > sub-queries correlated with every individual value of the
> > >> > reference
> > >> > > > > table
> > >> > > > > > > from the preceding part of the query and each sub query
> will
> > >> > return
> > >> > > > > > > multiple rows. If we use async calls with lateral join, my
> > gut
> > >> > > > feeling
> > >> > > > > is
> > >> > > > > > > that we might have many more async calls than lookup
> join. I
> > >> am
> > >> > not
> > >> > > > > > really
> > >> > > > > > > sure if we will be facing potential issues in this case or
> > >> not.
> > >> > > > > Possible
> > >> > > > > > > issues I can think of now e.g. too many PRC calls, too
> many
> > >> async
> > >> > > > calls
> > >> > > > > > > processing, the sub query will return a table which might
> be
> > >> > (too)
> > >> > > > big,
> > >> > > > > > and
> > >> > > > > > > might cause performance issues. I would suggest preparing
> > some
> > >> > use
> > >> > > > > cases
> > >> > > > > > > and running some performance tests to check it. These are
> my
> > >> > > concerns
> > >> > > > > > about
> > >> > > > > > > using async calls with lateral join and I'd like to share
> > with
> > >> > you,
> > >> > > > > happy
> > >> > > > > > > to discuss with you and hear different opinions, hopefully
> > the
> > >> > > > > > > discussion could help me understand it more deeply. Please
> > >> > correct
> > >> > > me
> > >> > > > > if
> > >> > > > > > I
> > >> > > > > > > am wrong.
> > >> > > > > > >
> > >> > > > > > > Best regards,
> > >> > > > > > > Jing
> > >> > > > > > >
> > >> > > > > > >
> > >> > > > > > > On Thu, Jun 8, 2023 at 7:22 AM Aitozi <
> gjying1314@gmail.com
> > >
> > >> > > wrote:
> > >> > > > > > >
> > >> > > > > > > > Hi Mason,
> > >> > > > > > > >     Thanks for your input. I think if we support the
> user
> > >> > defined
> > >> > > > > async
> > >> > > > > > > > table function,
> > >> > > > > > > > user will be able to use it to hold a batch data then
> > >> handle it
> > >> > > at
> > >> > > > > one
> > >> > > > > > > time
> > >> > > > > > > > in the customized function.
> > >> > > > > > > >
> > >> > > > > > > > AsyncSink is meant for the sink operator. I have not
> > figure
> > >> out
> > >> > > how
> > >> > > > > to
> > >> > > > > > > > integrate in this case.
> > >> > > > > > > >
> > >> > > > > > > > Thanks,
> > >> > > > > > > > Atiozi.
> > >> > > > > > > >
> > >> > > > > > > >
> > >> > > > > > > > Mason Chen <ma...@gmail.com> 于2023年6月8日周四
> 12:40写道：
> > >> > > > > > > >
> > >> > > > > > > > > Hi Aitozi,
> > >> > > > > > > > >
> > >> > > > > > > > > I think it makes sense to make it easier for SQL users
> > to
> > >> > make
> > >> > > > > RPCs.
> > >> > > > > > Do
> > >> > > > > > > > you
> > >> > > > > > > > > think your proposal can extend to the ability to batch
> > >> data
> > >> > for
> > >> > > > the
> > >> > > > > > > RPC?
> > >> > > > > > > > > This is also another common strategy to increase
> > >> throughput.
> > >> > > > Also,
> > >> > > > > > have
> > >> > > > > > > > you
> > >> > > > > > > > > considered solving this a bit differently by
> leveraging
> > >> > Flink's
> > >> > > > > > > > AsyncSink?
> > >> > > > > > > > >
> > >> > > > > > > > > Best,
> > >> > > > > > > > > Mason
> > >> > > > > > > > >
> > >> > > > > > > > > On Mon, Jun 5, 2023 at 1:50 AM Aitozi <
> > >> gjying1314@gmail.com>
> > >> > > > > wrote:
> > >> > > > > > > > >
> > >> > > > > > > > > > One more thing for discussion:
> > >> > > > > > > > > >
> > >> > > > > > > > > > In our internal implementation, we reuse the option
> > >> > > > > > > > > > `table.exec.async-lookup.buffer-capacity` and
> > >> > > > > > > > > > `table.exec.async-lookup.timeout` to config
> > >> > > > > > > > > > the async udtf. Do you think we should add two extra
> > >> option
> > >> > > to
> > >> > > > > > > > > distinguish
> > >> > > > > > > > > > from the lookup option such as
> > >> > > > > > > > > >
> > >> > > > > > > > > > `table.exec.async-udtf.buffer-capacity`
> > >> > > > > > > > > > `table.exec.async-udtf.timeout`
> > >> > > > > > > > > >
> > >> > > > > > > > > >
> > >> > > > > > > > > > Best,
> > >> > > > > > > > > > Aitozi.
> > >> > > > > > > > > >
> > >> > > > > > > > > >
> > >> > > > > > > > > >
> > >> > > > > > > > > > Aitozi <gj...@gmail.com> 于2023年6月5日周一 12:20写道：
> > >> > > > > > > > > >
> > >> > > > > > > > > > > Hi Jing,
> > >> > > > > > > > > > >
> > >> > > > > > > > > > >     > what is the difference between the RPC call
> or
> > >> > query
> > >> > > > you
> > >> > > > > > > > > mentioned
> > >> > > > > > > > > > > and the lookup in a very
> > >> > > > > > > > > > > general way
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > I think the RPC call or query service is quite
> > >> similar to
> > >> > > the
> > >> > > > > > > lookup
> > >> > > > > > > > > > join.
> > >> > > > > > > > > > > But lookup join should work
> > >> > > > > > > > > > > with `LookupTableSource`.
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > Let's see how we can perform an async RPC call
> with
> > >> > lookup
> > >> > > > > join:
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > (1) Implement an AsyncTableFunction with RPC call
> > >> logic.
> > >> > > > > > > > > > > (2) Implement a `LookupTableSource` connector run
> > with
> > >> > the
> > >> > > > > async
> > >> > > > > > > udtf
> > >> > > > > > > > > > > defined in (1).
> > >> > > > > > > > > > > (3) Then define a DDL of this look up table in SQL
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > CREATE TEMPORARY TABLE Customers (
> > >> > > > > > > > > > >   id INT,
> > >> > > > > > > > > > >   name STRING,
> > >> > > > > > > > > > >   country STRING,
> > >> > > > > > > > > > >   zip STRING
> > >> > > > > > > > > > > ) WITH (
> > >> > > > > > > > > > >   'connector' = 'custom'
> > >> > > > > > > > > > > );
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > (4) Run with the query as below:
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > SELECT o.order_id, o.total, c.country, c.zip
> > >> > > > > > > > > > > FROM Orders AS o
> > >> > > > > > > > > > >   JOIN Customers FOR SYSTEM_TIME AS OF o.proc_time
> > AS
> > >> c
> > >> > > > > > > > > > >     ON o.customer_id = c.id;
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > This example is from doc
> > >> > > > > > > > > > > <
> > >> > > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > >
> > >> > > > > > >
> > >> > > > > >
> > >> > > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> >
> https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/dev/table/sql/queries/joins/#lookup-join
> > >> > > > > > > > > > >.You
> > >> > > > > > > > > > > can image the look up process as an async RPC call
> > >> > process.
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > Let's see how we can perform an async RPC call
> with
> > >> > lateral
> > >> > > > > join:
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > (1) Implement an AsyncTableFunction with RPC call
> > >> logic.
> > >> > > > > > > > > > > (2) Run query with
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > Create function f1 as '...' ;
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > SELECT o.order_id, o.total, c.country, c.zip FROM
> > >> Orders
> > >> > > > > lateral
> > >> > > > > > > > table
> > >> > > > > > > > > > > (f1(order_id)) as T(...);
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > As you can see, the lateral join version is more
> > >> simple
> > >> > and
> > >> > > > > > > intuitive
> > >> > > > > > > > > to
> > >> > > > > > > > > > > users. Users do not have to wrap a
> > >> > > > > > > > > > > LookupTableSource for the purpose of using async
> > udtf.
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > In the end, We can also see the user defined async
> > >> table
> > >> > > > > function
> > >> > > > > > > is
> > >> > > > > > > > an
> > >> > > > > > > > > > > enhancement of the current lateral table join
> > >> > > > > > > > > > > which only supports sync lateral join now.
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > Best,
> > >> > > > > > > > > > > Aitozi.
> > >> > > > > > > > > > >
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > Jing Ge <ji...@ververica.com.invalid> 于2023年6月2日周五
> > >> > 19:37写道：
> > >> > > > > > > > > > >
> > >> > > > > > > > > > >> Hi Aitozi,
> > >> > > > > > > > > > >>
> > >> > > > > > > > > > >> Thanks for the update. Just out of curiosity,
> what
> > is
> > >> > the
> > >> > > > > > > difference
> > >> > > > > > > > > > >> between the RPC call or query you mentioned and
> the
> > >> > lookup
> > >> > > > in
> > >> > > > > a
> > >> > > > > > > very
> > >> > > > > > > > > > >> general way? Since Lateral join is used in the
> > FLIP.
> > >> Is
> > >> > > > there
> > >> > > > > > any
> > >> > > > > > > > > > special
> > >> > > > > > > > > > >> thought for that? Sorry for asking so many
> > questions.
> > >> > The
> > >> > > > FLIP
> > >> > > > > > > > > contains
> > >> > > > > > > > > > >> limited information to understand the motivation.
> > >> > > > > > > > > > >>
> > >> > > > > > > > > > >> Best regards,
> > >> > > > > > > > > > >> Jing
> > >> > > > > > > > > > >>
> > >> > > > > > > > > > >> On Fri, Jun 2, 2023 at 3:48 AM Aitozi <
> > >> > > gjying1314@gmail.com
> > >> > > > >
> > >> > > > > > > wrote:
> > >> > > > > > > > > > >>
> > >> > > > > > > > > > >> > Hi Jing,
> > >> > > > > > > > > > >> >     I have updated the proposed changes to the
> > >> FLIP.
> > >> > > IMO,
> > >> > > > > > lookup
> > >> > > > > > > > has
> > >> > > > > > > > > > its
> > >> > > > > > > > > > >> > clear
> > >> > > > > > > > > > >> > async call requirement is due to its IO heavy
> > >> > operator.
> > >> > > In
> > >> > > > > our
> > >> > > > > > > > > usage,
> > >> > > > > > > > > > >> sql
> > >> > > > > > > > > > >> > users have
> > >> > > > > > > > > > >> > logic to do some RPC call or query the
> > third-party
> > >> > > service
> > >> > > > > > which
> > >> > > > > > > > is
> > >> > > > > > > > > > >> also IO
> > >> > > > > > > > > > >> > intensive.
> > >> > > > > > > > > > >> > In these case, we'd like to leverage the async
> > >> > function
> > >> > > to
> > >> > > > > > > improve
> > >> > > > > > > > > the
> > >> > > > > > > > > > >> > throughput.
> > >> > > > > > > > > > >> >
> > >> > > > > > > > > > >> > Thanks,
> > >> > > > > > > > > > >> > Aitozi.
> > >> > > > > > > > > > >> >
> > >> > > > > > > > > > >> > Jing Ge <ji...@ververica.com.invalid>
> > 于2023年6月1日周四
> > >> > > > 22:55写道：
> > >> > > > > > > > > > >> >
> > >> > > > > > > > > > >> > > Hi Aitozi,
> > >> > > > > > > > > > >> > >
> > >> > > > > > > > > > >> > > Sorry for the late reply. Would you like to
> > >> update
> > >> > the
> > >> > > > > > > proposed
> > >> > > > > > > > > > >> changes
> > >> > > > > > > > > > >> > > with more details into the FLIP too?
> > >> > > > > > > > > > >> > > I got your point. It looks like a rational
> > idea.
> > >> > > > However,
> > >> > > > > > > since
> > >> > > > > > > > > > lookup
> > >> > > > > > > > > > >> > has
> > >> > > > > > > > > > >> > > its clear async call requirement, are there
> any
> > >> real
> > >> > > use
> > >> > > > > > cases
> > >> > > > > > > > > that
> > >> > > > > > > > > > >> > > need this change? This will help us
> understand
> > >> the
> > >> > > > > > motivation.
> > >> > > > > > > > > After
> > >> > > > > > > > > > >> all,
> > >> > > > > > > > > > >> > > lateral join and temporal lookup join[1] are
> > >> quite
> > >> > > > > > different.
> > >> > > > > > > > > > >> > >
> > >> > > > > > > > > > >> > > Best regards,
> > >> > > > > > > > > > >> > > Jing
> > >> > > > > > > > > > >> > >
> > >> > > > > > > > > > >> > >
> > >> > > > > > > > > > >> > > [1]
> > >> > > > > > > > > > >> > >
> > >> > > > > > > > > > >> > >
> > >> > > > > > > > > > >> >
> > >> > > > > > > > > > >>
> > >> > > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > >
> > >> > > > > > >
> > >> > > > > >
> > >> > > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> >
> https://github.com/apache/flink/blob/d90a72da2fd601ca4e2a46700e91ec5b348de2ad/flink-table/flink-table-common/src/main/java/org/apache/flink/table/functions/AsyncTableFunction.java#L54
> > >> > > > > > > > > > >> > >
> > >> > > > > > > > > > >> > > On Wed, May 31, 2023 at 8:53 AM Aitozi <
> > >> > > > > > gjying1314@gmail.com>
> > >> > > > > > > > > > wrote:
> > >> > > > > > > > > > >> > >
> > >> > > > > > > > > > >> > > > Hi Jing,
> > >> > > > > > > > > > >> > > >     What do you think about it? Can we move
> > >> > forward
> > >> > > > this
> > >> > > > > > > > > feature?
> > >> > > > > > > > > > >> > > >
> > >> > > > > > > > > > >> > > > Thanks,
> > >> > > > > > > > > > >> > > > Aitozi.
> > >> > > > > > > > > > >> > > >
> > >> > > > > > > > > > >> > > > Aitozi <gj...@gmail.com>
> 于2023年5月29日周一
> > >> > > 09:56写道：
> > >> > > > > > > > > > >> > > >
> > >> > > > > > > > > > >> > > > > Hi Jing,
> > >> > > > > > > > > > >> > > > >     > "Do you mean to support the
> > >> > > AyncTableFunction
> > >> > > > > > beyond
> > >> > > > > > > > the
> > >> > > > > > > > > > >> > > > > LookupTableSource?"
> > >> > > > > > > > > > >> > > > > Yes, I mean to support the
> > AyncTableFunction
> > >> > > beyond
> > >> > > > > the
> > >> > > > > > > > > > >> > > > LookupTableSource.
> > >> > > > > > > > > > >> > > > >
> > >> > > > > > > > > > >> > > > > The "AsyncTableFunction" is the function
> > with
> > >> > > > ability
> > >> > > > > to
> > >> > > > > > > be
> > >> > > > > > > > > > >> executed
> > >> > > > > > > > > > >> > > > async
> > >> > > > > > > > > > >> > > > > (with AsyncWaitOperator).
> > >> > > > > > > > > > >> > > > > The async lookup join is a one of usage
> of
> > >> this.
> > >> > > So,
> > >> > > > > we
> > >> > > > > > > > don't
> > >> > > > > > > > > > >> have to
> > >> > > > > > > > > > >> > > > bind
> > >> > > > > > > > > > >> > > > > the AyncTableFunction with
> > LookupTableSource.
> > >> > > > > > > > > > >> > > > > If User-defined AsyncTableFunction is
> > >> supported,
> > >> > > > user
> > >> > > > > > can
> > >> > > > > > > > > > directly
> > >> > > > > > > > > > >> > use
> > >> > > > > > > > > > >> > > > > lateral table syntax to perform async
> > >> operation.
> > >> > > > > > > > > > >> > > > >
> > >> > > > > > > > > > >> > > > > > "It would be better if you could
> > elaborate
> > >> the
> > >> > > > > > proposed
> > >> > > > > > > > > > changes
> > >> > > > > > > > > > >> wrt
> > >> > > > > > > > > > >> > > the
> > >> > > > > > > > > > >> > > > > CorrelatedCodeGenerator with more
> details"
> > >> > > > > > > > > > >> > > > >
> > >> > > > > > > > > > >> > > > > In the proposal, we use lateral table
> > syntax
> > >> to
> > >> > > > > support
> > >> > > > > > > the
> > >> > > > > > > > > > async
> > >> > > > > > > > > > >> > table
> > >> > > > > > > > > > >> > > > > function. So the planner will also treat
> > this
> > >> > > > > statement
> > >> > > > > > > to a
> > >> > > > > > > > > > >> > > > > CommonExecCorrelate node. So the runtime
> > code
> > >> > > should
> > >> > > > > be
> > >> > > > > > > > > > generated
> > >> > > > > > > > > > >> in
> > >> > > > > > > > > > >> > > > > CorrelatedCodeGenerator.
> > >> > > > > > > > > > >> > > > > In CorrelatedCodeGenerator, we will know
> > the
> > >> > > > > > > TableFunction's
> > >> > > > > > > > > > Kind
> > >> > > > > > > > > > >> of
> > >> > > > > > > > > > >> > > > > `FunctionKind.Table` or
> > >> > `FunctionKind.ASYNC_TABLE`
> > >> > > > > > > > > > >> > > > > For  `FunctionKind.ASYNC_TABLE` we can
> > >> generate
> > >> > a
> > >> > > > > > > > > > >> AsyncWaitOperator
> > >> > > > > > > > > > >> > to
> > >> > > > > > > > > > >> > > > > execute the async table function.
> > >> > > > > > > > > > >> > > > >
> > >> > > > > > > > > > >> > > > >
> > >> > > > > > > > > > >> > > > > Thanks,
> > >> > > > > > > > > > >> > > > > Aitozi.
> > >> > > > > > > > > > >> > > > >
> > >> > > > > > > > > > >> > > > >
> > >> > > > > > > > > > >> > > > > Jing Ge <ji...@ververica.com.invalid>
> > >> > > 于2023年5月29日周一
> > >> > > > > > > 03:22写道：
> > >> > > > > > > > > > >> > > > >
> > >> > > > > > > > > > >> > > > >> Hi Aitozi,
> > >> > > > > > > > > > >> > > > >>
> > >> > > > > > > > > > >> > > > >> Thanks for the clarification. The naming
> > >> > "Lookup"
> > >> > > > > might
> > >> > > > > > > > > suggest
> > >> > > > > > > > > > >> > using
> > >> > > > > > > > > > >> > > it
> > >> > > > > > > > > > >> > > > >> for table look up. But conceptually what
> > the
> > >> > > eval()
> > >> > > > > > > method
> > >> > > > > > > > > will
> > >> > > > > > > > > > >> do
> > >> > > > > > > > > > >> > is
> > >> > > > > > > > > > >> > > to
> > >> > > > > > > > > > >> > > > >> get a collection of results(Row,
> RowData)
> > >> from
> > >> > > the
> > >> > > > > > given
> > >> > > > > > > > > keys.
> > >> > > > > > > > > > >> How
> > >> > > > > > > > > > >> > it
> > >> > > > > > > > > > >> > > > will
> > >> > > > > > > > > > >> > > > >> be done depends on the implementation,
> > i.e.
> > >> you
> > >> > > can
> > >> > > > > > > > implement
> > >> > > > > > > > > > >> your
> > >> > > > > > > > > > >> > own
> > >> > > > > > > > > > >> > > > >> Source[1][2]. The example in the FLIP
> > >> should be
> > >> > > > able
> > >> > > > > to
> > >> > > > > > > be
> > >> > > > > > > > > > >> handled
> > >> > > > > > > > > > >> > in
> > >> > > > > > > > > > >> > > > this
> > >> > > > > > > > > > >> > > > >> way.
> > >> > > > > > > > > > >> > > > >>
> > >> > > > > > > > > > >> > > > >> Do you mean to support the
> > AyncTableFunction
> > >> > > beyond
> > >> > > > > the
> > >> > > > > > > > > > >> > > > LookupTableSource?
> > >> > > > > > > > > > >> > > > >> It would be better if you could
> elaborate
> > >> the
> > >> > > > > proposed
> > >> > > > > > > > > changes
> > >> > > > > > > > > > >> wrt
> > >> > > > > > > > > > >> > the
> > >> > > > > > > > > > >> > > > >> CorrelatedCodeGenerator with more
> details.
> > >> > > Thanks!
> > >> > > > > > > > > > >> > > > >>
> > >> > > > > > > > > > >> > > > >> Best regards,
> > >> > > > > > > > > > >> > > > >> Jing
> > >> > > > > > > > > > >> > > > >>
> > >> > > > > > > > > > >> > > > >> [1]
> > >> > > > > > > > > > >> > > > >>
> > >> > > > > > > > > > >> > > > >>
> > >> > > > > > > > > > >> > > >
> > >> > > > > > > > > > >> > >
> > >> > > > > > > > > > >> >
> > >> > > > > > > > > > >>
> > >> > > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > >
> > >> > > > > > >
> > >> > > > > >
> > >> > > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> >
> https://github.com/apache/flink/blob/678370b18e1b6c4a23e5ce08f8efd05675a0cc17/flink-table/flink-table-common/src/main/java/org/apache/flink/table/connector/source/LookupTableSource.java#L64
> > >> > > > > > > > > > >> > > > >> [2]
> > >> > > > > > > > > > >> > > > >>
> > >> > > > > > > > > > >> > > > >>
> > >> > > > > > > > > > >> > > >
> > >> > > > > > > > > > >> > >
> > >> > > > > > > > > > >> >
> > >> > > > > > > > > > >>
> > >> > > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > >
> > >> > > > > > >
> > >> > > > > >
> > >> > > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> >
> https://github.com/apache/flink/blob/678370b18e1b6c4a23e5ce08f8efd05675a0cc17/flink-table/flink-table-common/src/main/java/org/apache/flink/table/connector/source/AsyncTableFunctionProvider.java#L49
> > >> > > > > > > > > > >> > > > >>
> > >> > > > > > > > > > >> > > > >> On Sat, May 27, 2023 at 9:48 AM Aitozi <
> > >> > > > > > > > gjying1314@gmail.com
> > >> > > > > > > > > >
> > >> > > > > > > > > > >> > wrote:
> > >> > > > > > > > > > >> > > > >>
> > >> > > > > > > > > > >> > > > >> > Hi Jing,
> > >> > > > > > > > > > >> > > > >> >     Thanks for your response. As
> stated
> > in
> > >> > the
> > >> > > > > FLIP,
> > >> > > > > > > the
> > >> > > > > > > > > > >> purpose
> > >> > > > > > > > > > >> > of
> > >> > > > > > > > > > >> > > > this
> > >> > > > > > > > > > >> > > > >> > FLIP is meant to support
> > >> > > > > > > > > > >> > > > >> > user-defined async table function. As
> > >> > described
> > >> > > > in
> > >> > > > > > > flink
> > >> > > > > > > > > > >> document
> > >> > > > > > > > > > >> > > [1]
> > >> > > > > > > > > > >> > > > >> >
> > >> > > > > > > > > > >> > > > >> > Async table functions are special
> > >> functions
> > >> > for
> > >> > > > > table
> > >> > > > > > > > > sources
> > >> > > > > > > > > > >> that
> > >> > > > > > > > > > >> > > > >> perform
> > >> > > > > > > > > > >> > > > >> > > a lookup.
> > >> > > > > > > > > > >> > > > >> > >
> > >> > > > > > > > > > >> > > > >> >
> > >> > > > > > > > > > >> > > > >> > So end user can not directly define
> and
> > >> use
> > >> > > async
> > >> > > > > > table
> > >> > > > > > > > > > >> function
> > >> > > > > > > > > > >> > > now.
> > >> > > > > > > > > > >> > > > An
> > >> > > > > > > > > > >> > > > >> > user case is reported in [2]
> > >> > > > > > > > > > >> > > > >> >
> > >> > > > > > > > > > >> > > > >> > So, in conclusion, no new interface is
> > >> > > > introduced,
> > >> > > > > > but
> > >> > > > > > > we
> > >> > > > > > > > > > >> extend
> > >> > > > > > > > > > >> > the
> > >> > > > > > > > > > >> > > > >> > ability to support user-defined async
> > >> table
> > >> > > > > function.
> > >> > > > > > > > > > >> > > > >> >
> > >> > > > > > > > > > >> > > > >> > [1]:
> > >> > > > > > > > > > >> > > > >> >
> > >> > > > > > > > > > >> > > > >> >
> > >> > > > > > > > > > >> > > > >>
> > >> > > > > > > > > > >> > > >
> > >> > > > > > > > > > >> > >
> > >> > > > > > > > > > >> >
> > >> > > > > > > > > > >>
> > >> > > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > >
> > >> > > > > > >
> > >> > > > > >
> > >> > > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> >
> https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/dev/table/functions/udfs/
> > >> > > > > > > > > > >> > > > >> > [2]:
> > >> > > > > > > > > > >> > >
> > >> > > > > > > >
> > >> > https://lists.apache.org/thread/qljwd40v5ntz6733cwcdr8s4z97b343b
> > >> > > > > > > > > > >> > > > >> >
> > >> > > > > > > > > > >> > > > >> > Thanks.
> > >> > > > > > > > > > >> > > > >> > Aitozi.
> > >> > > > > > > > > > >> > > > >> >
> > >> > > > > > > > > > >> > > > >> >
> > >> > > > > > > > > > >> > > > >> > Jing Ge <ji...@ververica.com.invalid>
> > >> > > > 于2023年5月27日周六
> > >> > > > > > > > > 06:40写道：
> > >> > > > > > > > > > >> > > > >> >
> > >> > > > > > > > > > >> > > > >> > > Hi Aitozi,
> > >> > > > > > > > > > >> > > > >> > >
> > >> > > > > > > > > > >> > > > >> > > Thanks for your proposal. I am not
> > quite
> > >> > sure
> > >> > > > if
> > >> > > > > I
> > >> > > > > > > > > > understood
> > >> > > > > > > > > > >> > your
> > >> > > > > > > > > > >> > > > >> > thoughts
> > >> > > > > > > > > > >> > > > >> > > correctly. You described a special
> > case
> > >> > > > > > > implementation
> > >> > > > > > > > of
> > >> > > > > > > > > > the
> > >> > > > > > > > > > >> > > > >> > > AsyncTableFunction with on public
> API
> > >> > > changes.
> > >> > > > > > Would
> > >> > > > > > > > you
> > >> > > > > > > > > > >> please
> > >> > > > > > > > > > >> > > > >> elaborate
> > >> > > > > > > > > > >> > > > >> > > your purpose of writing a FLIP
> > >> according to
> > >> > > the
> > >> > > > > > FLIP
> > >> > > > > > > > > > >> > > > documentation[1]?
> > >> > > > > > > > > > >> > > > >> > > Thanks!
> > >> > > > > > > > > > >> > > > >> > >
> > >> > > > > > > > > > >> > > > >> > > [1]
> > >> > > > > > > > > > >> > > > >> > >
> > >> > > > > > > > > > >> > > > >> > >
> > >> > > > > > > > > > >> > > > >> >
> > >> > > > > > > > > > >> > > > >>
> > >> > > > > > > > > > >> > > >
> > >> > > > > > > > > > >> > >
> > >> > > > > > > > > > >> >
> > >> > > > > > > > > > >>
> > >> > > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > >
> > >> > > > > > >
> > >> > > > > >
> > >> > > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> >
> https://cwiki.apache.org/confluence/display/FLINK/Flink+Improvement+Proposals
> > >> > > > > > > > > > >> > > > >> > >
> > >> > > > > > > > > > >> > > > >> > > Best regards,
> > >> > > > > > > > > > >> > > > >> > > Jing
> > >> > > > > > > > > > >> > > > >> > >
> > >> > > > > > > > > > >> > > > >> > > On Wed, May 24, 2023 at 1:07 PM
> > Aitozi <
> > >> > > > > > > > > > gjying1314@gmail.com
> > >> > > > > > > > > > >> >
> > >> > > > > > > > > > >> > > > wrote:
> > >> > > > > > > > > > >> > > > >> > >
> > >> > > > > > > > > > >> > > > >> > > > May I ask for some feedback  :D
> > >> > > > > > > > > > >> > > > >> > > >
> > >> > > > > > > > > > >> > > > >> > > > Thanks,
> > >> > > > > > > > > > >> > > > >> > > > Aitozi
> > >> > > > > > > > > > >> > > > >> > > >
> > >> > > > > > > > > > >> > > > >> > > > Aitozi <gj...@gmail.com>
> > >> > > 于2023年5月23日周二
> > >> > > > > > > 19:14写道：
> > >> > > > > > > > > > >> > > > >> > > > >
> > >> > > > > > > > > > >> > > > >> > > > > Just catch an user case report
> > from
> > >> > > Giannis
> > >> > > > > > > Polyzos
> > >> > > > > > > > > for
> > >> > > > > > > > > > >> this
> > >> > > > > > > > > > >> > > > >> usage:
> > >> > > > > > > > > > >> > > > >> > > > >
> > >> > > > > > > > > > >> > > > >> > > > >
> > >> > > > > > > > > > >> > > >
> > >> > > > > > > > >
> > >> > > https://lists.apache.org/thread/qljwd40v5ntz6733cwcdr8s4z97b343b
> > >> > > > > > > > > > >> > > > >> > > > >
> > >> > > > > > > > > > >> > > > >> > > > > Aitozi <gj...@gmail.com>
> > >> > > > 于2023年5月23日周二
> > >> > > > > > > > 17:45写道：
> > >> > > > > > > > > > >> > > > >> > > > > >
> > >> > > > > > > > > > >> > > > >> > > > > > Hi guys,
> > >> > > > > > > > > > >> > > > >> > > > > >     I want to bring up a
> > >> discussion
> > >> > > about
> > >> > > > > > > adding
> > >> > > > > > > > > > >> support
> > >> > > > > > > > > > >> > of
> > >> > > > > > > > > > >> > > > User
> > >> > > > > > > > > > >> > > > >> > > > > > Defined AsyncTableFunction in
> > >> Flink.
> > >> > > > > > > > > > >> > > > >> > > > > > Currently, async table
> function
> > >> are
> > >> > > > special
> > >> > > > > > > > > functions
> > >> > > > > > > > > > >> for
> > >> > > > > > > > > > >> > > > table
> > >> > > > > > > > > > >> > > > >> > > source
> > >> > > > > > > > > > >> > > > >> > > > > > to perform
> > >> > > > > > > > > > >> > > > >> > > > > > async lookup. However, it's
> > worth
> > >> to
> > >> > > > > support
> > >> > > > > > > the
> > >> > > > > > > > > user
> > >> > > > > > > > > > >> > > defined
> > >> > > > > > > > > > >> > > > >> async
> > >> > > > > > > > > > >> > > > >> > > > > > table function.
> > >> > > > > > > > > > >> > > > >> > > > > > Because, in this way, the end
> > SQL
> > >> > user
> > >> > > > can
> > >> > > > > > > > leverage
> > >> > > > > > > > > > it
> > >> > > > > > > > > > >> to
> > >> > > > > > > > > > >> > > > >> perform
> > >> > > > > > > > > > >> > > > >> > the
> > >> > > > > > > > > > >> > > > >> > > > > > async operation
> > >> > > > > > > > > > >> > > > >> > > > > > which is useful to maximum the
> > >> system
> > >> > > > > > > throughput
> > >> > > > > > > > > > >> > especially
> > >> > > > > > > > > > >> > > > for
> > >> > > > > > > > > > >> > > > >> IO
> > >> > > > > > > > > > >> > > > >> > > > > > bottleneck case.
> > >> > > > > > > > > > >> > > > >> > > > > >
> > >> > > > > > > > > > >> > > > >> > > > > > You can find some more detail
> in
> > >> [1].
> > >> > > > > > > > > > >> > > > >> > > > > >
> > >> > > > > > > > > > >> > > > >> > > > > > Looking forward to feedback
> > >> > > > > > > > > > >> > > > >> > > > > >
> > >> > > > > > > > > > >> > > > >> > > > > >
> > >> > > > > > > > > > >> > > > >> > > > > > [1]:
> > >> > > > > > > > > > >> > > > >> > > >
> > >> > > > > > > > > > >> > > > >> > >
> > >> > > > > > > > > > >> > > > >> >
> > >> > > > > > > > > > >> > > > >>
> > >> > > > > > > > > > >> > > >
> > >> > > > > > > > > > >> > >
> > >> > > > > > > > > > >> >
> > >> > > > > > > > > > >>
> > >> > > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > >
> > >> > > > > > >
> > >> > > > > >
> > >> > > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> >
> https://cwiki.apache.org/confluence/display/FLINK/%5BFLIP-313%5D+Add+support+of+User+Defined+AsyncTableFunction
> > >> > > > > > > > > > >> > > > >> > > > > >
> > >> > > > > > > > > > >> > > > >> > > > > > Thanks,
> > >> > > > > > > > > > >> > > > >> > > > > > Aitozi.
> > >> > > > > > > > > > >> > > > >> > > >
> > >> > > > > > > > > > >> > > > >> > >
> > >> > > > > > > > > > >> > > > >> >
> > >> > > > > > > > > > >> > > > >>
> > >> > > > > > > > > > >> > > > >
> > >> > > > > > > > > > >> > > >
> > >> > > > > > > > > > >> > >
> > >> > > > > > > > > > >> >
> > >> > > > > > > > > > >>
> > >> > > > > > > > > > >
> > >> > > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > >
> > >> > > > > > >
> > >> > > > > >
> > >> > > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> > >
> >
>

Re: [DISCUSS] FLIP-313 Add support of User Defined AsyncTableFunction

Posted by Jing Ge <ji...@ververica.com.INVALID>.

Hi Aitozi,

Thanks for taking care of that part. I have no other concern.

Best regards,
Jing


On Mon, Jun 12, 2023 at 5:38 PM Aitozi <gj...@gmail.com> wrote:

> BTW, If there are no other more blocking issue / comments, I would like to
> start a VOTE in another thread this wednesday 6.14
>
> Thanks,
> Aitozi.
>
> Aitozi <gj...@gmail.com> 于2023年6月12日周一 23:34写道：
>
> > Hi, Jing,
> >     Thanks for your explanation. I get your point now.
> >
> > For the performance part, I think it's a good idea to run with returning
> a
> > big table case, the memory consumption
> > should be a point to be taken care about. Because in the ordered mode,
> the
> > head element in buffer may affect the
> > total memory consumption.
> >
> >
> > Thanks,
> > Aitozi.
> >
> >
> >
> > Jing Ge <ji...@ververica.com.invalid> 于2023年6月12日周一 20:28写道：
> >
> >> Hi Aitozi,
> >>
> >> Which key will be used for lookup is not an issue, only one row will be
> >> required for each key in order to enrich it. True, it depends on the
> >> implementation whether multiple rows or single row for each key will be
> >> returned. However, for the lookup & enrichment scenario, one row/key is
> >> recommended, otherwise, like I mentioned previously, enrichment won't
> >> work.
> >>
> >> I am a little bit concerned about returning a big table for each key,
> >> since
> >> it will take the async call longer to return and need more memory. The
> >> performance tests should cover this scenario. This is not a blocking
> issue
> >> for this FLIP.
> >>
> >> Best regards,
> >> Jing
> >>
> >> On Sat, Jun 10, 2023 at 4:11 AM Aitozi <gj...@gmail.com> wrote:
> >>
> >> > Hi Jing,
> >> >     I means the join key is not necessary to be the primary key or
> >> unique
> >> > index of the database.
> >> > In this situation, we may queried out multi rows for one join key. I
> >> think
> >> > that's why the
> >> > LookupFunction#lookup will return a collection of RowData.
> >> >
> >> > BTW, I think the behavior of lookup join will not affect the semantic
> of
> >> > the async udtf.
> >> > We use the Async TableFunction here and the table function can collect
> >> > multiple rows.
> >> >
> >> > Thanks,
> >> > Atiozi.
> >> >
> >> >
> >> >
> >> > Jing Ge <ji...@ververica.com.invalid> 于2023年6月10日周六 00:15写道：
> >> >
> >> > > Hi Aitozi,
> >> > >
> >> > > The keyRow used in this case contains all keys[1].
> >> > >
> >> > > Best regards,
> >> > > Jing
> >> > >
> >> > > [1]
> >> > >
> >> > >
> >> >
> >>
> https://github.com/apache/flink/blob/191ec6ca3943d7119f14837efe112e074d815c47/flink-table/flink-table-common/src/main/java/org/apache/flink/table/functions/LookupFunction.java#L49
> >> > >
> >> > >
> >> > > On Fri, Jun 9, 2023 at 3:42 PM Aitozi <gj...@gmail.com> wrote:
> >> > >
> >> > > > Hi Jing,
> >> > > >
> >> > > >      The performance test is added to the FLIP.
> >> > > >
> >> > > >      As I know, The lookup join can return multi rows, it depends
> on
> >> > > > whether  the join key
> >> > > > is the primary key of the external database or not. The `lookup`
> [1]
> >> > will
> >> > > > return a collection of
> >> > > > joined result, and each of them will be collected
> >> > > >
> >> > > >
> >> > > > [1]:
> >> > > >
> >> > > >
> >> > >
> >> >
> >>
> https://github.com/apache/flink/blob/191ec6ca3943d7119f14837efe112e074d815c47/flink-table/flink-table-common/src/main/java/org/apache/flink/table/functions/LookupFunction.java#L52
> >> > > >
> >> > > >
> >> > > > Thanks,
> >> > > > Aitozi.
> >> > > >
> >> > > > Jing Ge <ji...@ververica.com.invalid> 于2023年6月9日周五 17:05写道：
> >> > > >
> >> > > > > Hi Aitozi,
> >> > > > >
> >> > > > > Thanks for the feedback. Looking forward to the performance
> tests.
> >> > > > >
> >> > > > > Afaik, lookup returns one row for each key [1] [2].
> Conceptually,
> >> the
> >> > > > > lookup function is used to enrich column(s) from the dimension
> >> table.
> >> > > If,
> >> > > > > for the given key, there will be more than one row, there will
> be
> >> no
> >> > > way
> >> > > > to
> >> > > > > know which row will be used to enrich the key.
> >> > > > >
> >> > > > > [1]
> >> > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
> https://github.com/apache/flink/blob/191ec6ca3943d7119f14837efe112e074d815c47/flink-table/flink-table-common/src/main/java/org/apache/flink/table/functions/LookupFunction.java#L49
> >> > > > > [2]
> >> > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
> https://github.com/apache/flink/blob/191ec6ca3943d7119f14837efe112e074d815c47/flink-table/flink-table-common/src/main/java/org/apache/flink/table/functions/TableFunction.java#L196
> >> > > > >
> >> > > > > Best regards,
> >> > > > > Jing
> >> > > > >
> >> > > > > On Fri, Jun 9, 2023 at 5:18 AM Aitozi <gj...@gmail.com>
> >> wrote:
> >> > > > >
> >> > > > > > Hi Jing
> >> > > > > >     Thanks for your good questions. I have updated the example
> >> to
> >> > the
> >> > > > > FLIP.
> >> > > > > >
> >> > > > > > > Only one row for each lookup
> >> > > > > > lookup can also return multi rows, based on the query result.
> >> [1]
> >> > > > > >
> >> > > > > > [1]:
> >> > > > > >
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
> https://github.com/apache/flink/blob/191ec6ca3943d7119f14837efe112e074d815c47/flink-table/flink-table-common/src/main/java/org/apache/flink/table/functions/LookupFunction.java#L56
> >> > > > > >
> >> > > > > > > If we use async calls with lateral join, my gut feeling is
> >> > > > > > that we might have many more async calls than lookup join. I
> am
> >> not
> >> > > > > really
> >> > > > > > sure if we will be facing potential issues in this case or
> not.
> >> > > > > >
> >> > > > > > IMO, the work pattern is similar to the lookup function, for
> >> each
> >> > row
> >> > > > > from
> >> > > > > > the left table,
> >> > > > > > it will evaluate the eval method once, so the async call
> numbers
> >> > will
> >> > > > not
> >> > > > > > change.
> >> > > > > > and the maximum calls in flight is limited by the Async
> >> operators
> >> > > > buffer
> >> > > > > > capacity
> >> > > > > > which will be controlled by the option.
> >> > > > > >
> >> > > > > > BTW, for the naming of these option, I updated the FLIP about
> >> this
> >> > > you
> >> > > > > can
> >> > > > > > refer to
> >> > > > > > the section of "ConfigOption" and "Rejected Alternatives"
> >> > > > > >
> >> > > > > > In the end, for the performance evaluation, I'd like to do
> some
> >> > tests
> >> > > > and
> >> > > > > > will update it to the FLIP doc
> >> > > > > >
> >> > > > > > Thanks,
> >> > > > > > Aitozi.
> >> > > > > >
> >> > > > > >
> >> > > > > > Jing Ge <ji...@ververica.com.invalid> 于2023年6月9日周五 07:23写道：
> >> > > > > >
> >> > > > > > > Hi Aitozi,
> >> > > > > > >
> >> > > > > > > Thanks for the clarification. The code example looks
> >> > interesting. I
> >> > > > > would
> >> > > > > > > suggest adding them into the FLIP. The description with code
> >> > > examples
> >> > > > > > will
> >> > > > > > > help readers understand the motivation and how to use it.
> >> Afaiac,
> >> > > it
> >> > > > > is a
> >> > > > > > > valid feature for Flink users.
> >> > > > > > >
> >> > > > > > > As we knew, lookup join is based on temporal join, i.e. FOR
> >> > > > SYSTEM_TIME
> >> > > > > > AS
> >> > > > > > > OF which is also used in your code example. Temporal join
> >> > performs
> >> > > > the
> >> > > > > > > lookup based on the processing time match. Only one row for
> >> each
> >> > > > > > > lookup(afaiu, I need to check the source code to double
> >> confirm)
> >> > > will
> >> > > > > > > return for further enrichment. One the other hand, lateral
> >> join
> >> > > will
> >> > > > > have
> >> > > > > > > sub-queries correlated with every individual value of the
> >> > reference
> >> > > > > table
> >> > > > > > > from the preceding part of the query and each sub query will
> >> > return
> >> > > > > > > multiple rows. If we use async calls with lateral join, my
> gut
> >> > > > feeling
> >> > > > > is
> >> > > > > > > that we might have many more async calls than lookup join. I
> >> am
> >> > not
> >> > > > > > really
> >> > > > > > > sure if we will be facing potential issues in this case or
> >> not.
> >> > > > > Possible
> >> > > > > > > issues I can think of now e.g. too many PRC calls, too many
> >> async
> >> > > > calls
> >> > > > > > > processing, the sub query will return a table which might be
> >> > (too)
> >> > > > big,
> >> > > > > > and
> >> > > > > > > might cause performance issues. I would suggest preparing
> some
> >> > use
> >> > > > > cases
> >> > > > > > > and running some performance tests to check it. These are my
> >> > > concerns
> >> > > > > > about
> >> > > > > > > using async calls with lateral join and I'd like to share
> with
> >> > you,
> >> > > > > happy
> >> > > > > > > to discuss with you and hear different opinions, hopefully
> the
> >> > > > > > > discussion could help me understand it more deeply. Please
> >> > correct
> >> > > me
> >> > > > > if
> >> > > > > > I
> >> > > > > > > am wrong.
> >> > > > > > >
> >> > > > > > > Best regards,
> >> > > > > > > Jing
> >> > > > > > >
> >> > > > > > >
> >> > > > > > > On Thu, Jun 8, 2023 at 7:22 AM Aitozi <gjying1314@gmail.com
> >
> >> > > wrote:
> >> > > > > > >
> >> > > > > > > > Hi Mason,
> >> > > > > > > >     Thanks for your input. I think if we support the user
> >> > defined
> >> > > > > async
> >> > > > > > > > table function,
> >> > > > > > > > user will be able to use it to hold a batch data then
> >> handle it
> >> > > at
> >> > > > > one
> >> > > > > > > time
> >> > > > > > > > in the customized function.
> >> > > > > > > >
> >> > > > > > > > AsyncSink is meant for the sink operator. I have not
> figure
> >> out
> >> > > how
> >> > > > > to
> >> > > > > > > > integrate in this case.
> >> > > > > > > >
> >> > > > > > > > Thanks,
> >> > > > > > > > Atiozi.
> >> > > > > > > >
> >> > > > > > > >
> >> > > > > > > > Mason Chen <ma...@gmail.com> 于2023年6月8日周四 12:40写道：
> >> > > > > > > >
> >> > > > > > > > > Hi Aitozi,
> >> > > > > > > > >
> >> > > > > > > > > I think it makes sense to make it easier for SQL users
> to
> >> > make
> >> > > > > RPCs.
> >> > > > > > Do
> >> > > > > > > > you
> >> > > > > > > > > think your proposal can extend to the ability to batch
> >> data
> >> > for
> >> > > > the
> >> > > > > > > RPC?
> >> > > > > > > > > This is also another common strategy to increase
> >> throughput.
> >> > > > Also,
> >> > > > > > have
> >> > > > > > > > you
> >> > > > > > > > > considered solving this a bit differently by leveraging
> >> > Flink's
> >> > > > > > > > AsyncSink?
> >> > > > > > > > >
> >> > > > > > > > > Best,
> >> > > > > > > > > Mason
> >> > > > > > > > >
> >> > > > > > > > > On Mon, Jun 5, 2023 at 1:50 AM Aitozi <
> >> gjying1314@gmail.com>
> >> > > > > wrote:
> >> > > > > > > > >
> >> > > > > > > > > > One more thing for discussion:
> >> > > > > > > > > >
> >> > > > > > > > > > In our internal implementation, we reuse the option
> >> > > > > > > > > > `table.exec.async-lookup.buffer-capacity` and
> >> > > > > > > > > > `table.exec.async-lookup.timeout` to config
> >> > > > > > > > > > the async udtf. Do you think we should add two extra
> >> option
> >> > > to
> >> > > > > > > > > distinguish
> >> > > > > > > > > > from the lookup option such as
> >> > > > > > > > > >
> >> > > > > > > > > > `table.exec.async-udtf.buffer-capacity`
> >> > > > > > > > > > `table.exec.async-udtf.timeout`
> >> > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > > > Best,
> >> > > > > > > > > > Aitozi.
> >> > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > > > Aitozi <gj...@gmail.com> 于2023年6月5日周一 12:20写道：
> >> > > > > > > > > >
> >> > > > > > > > > > > Hi Jing,
> >> > > > > > > > > > >
> >> > > > > > > > > > >     > what is the difference between the RPC call or
> >> > query
> >> > > > you
> >> > > > > > > > > mentioned
> >> > > > > > > > > > > and the lookup in a very
> >> > > > > > > > > > > general way
> >> > > > > > > > > > >
> >> > > > > > > > > > > I think the RPC call or query service is quite
> >> similar to
> >> > > the
> >> > > > > > > lookup
> >> > > > > > > > > > join.
> >> > > > > > > > > > > But lookup join should work
> >> > > > > > > > > > > with `LookupTableSource`.
> >> > > > > > > > > > >
> >> > > > > > > > > > > Let's see how we can perform an async RPC call with
> >> > lookup
> >> > > > > join:
> >> > > > > > > > > > >
> >> > > > > > > > > > > (1) Implement an AsyncTableFunction with RPC call
> >> logic.
> >> > > > > > > > > > > (2) Implement a `LookupTableSource` connector run
> with
> >> > the
> >> > > > > async
> >> > > > > > > udtf
> >> > > > > > > > > > > defined in (1).
> >> > > > > > > > > > > (3) Then define a DDL of this look up table in SQL
> >> > > > > > > > > > >
> >> > > > > > > > > > > CREATE TEMPORARY TABLE Customers (
> >> > > > > > > > > > >   id INT,
> >> > > > > > > > > > >   name STRING,
> >> > > > > > > > > > >   country STRING,
> >> > > > > > > > > > >   zip STRING
> >> > > > > > > > > > > ) WITH (
> >> > > > > > > > > > >   'connector' = 'custom'
> >> > > > > > > > > > > );
> >> > > > > > > > > > >
> >> > > > > > > > > > > (4) Run with the query as below:
> >> > > > > > > > > > >
> >> > > > > > > > > > > SELECT o.order_id, o.total, c.country, c.zip
> >> > > > > > > > > > > FROM Orders AS o
> >> > > > > > > > > > >   JOIN Customers FOR SYSTEM_TIME AS OF o.proc_time
> AS
> >> c
> >> > > > > > > > > > >     ON o.customer_id = c.id;
> >> > > > > > > > > > >
> >> > > > > > > > > > > This example is from doc
> >> > > > > > > > > > > <
> >> > > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > >
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
> https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/dev/table/sql/queries/joins/#lookup-join
> >> > > > > > > > > > >.You
> >> > > > > > > > > > > can image the look up process as an async RPC call
> >> > process.
> >> > > > > > > > > > >
> >> > > > > > > > > > > Let's see how we can perform an async RPC call with
> >> > lateral
> >> > > > > join:
> >> > > > > > > > > > >
> >> > > > > > > > > > > (1) Implement an AsyncTableFunction with RPC call
> >> logic.
> >> > > > > > > > > > > (2) Run query with
> >> > > > > > > > > > >
> >> > > > > > > > > > > Create function f1 as '...' ;
> >> > > > > > > > > > >
> >> > > > > > > > > > > SELECT o.order_id, o.total, c.country, c.zip FROM
> >> Orders
> >> > > > > lateral
> >> > > > > > > > table
> >> > > > > > > > > > > (f1(order_id)) as T(...);
> >> > > > > > > > > > >
> >> > > > > > > > > > > As you can see, the lateral join version is more
> >> simple
> >> > and
> >> > > > > > > intuitive
> >> > > > > > > > > to
> >> > > > > > > > > > > users. Users do not have to wrap a
> >> > > > > > > > > > > LookupTableSource for the purpose of using async
> udtf.
> >> > > > > > > > > > >
> >> > > > > > > > > > > In the end, We can also see the user defined async
> >> table
> >> > > > > function
> >> > > > > > > is
> >> > > > > > > > an
> >> > > > > > > > > > > enhancement of the current lateral table join
> >> > > > > > > > > > > which only supports sync lateral join now.
> >> > > > > > > > > > >
> >> > > > > > > > > > > Best,
> >> > > > > > > > > > > Aitozi.
> >> > > > > > > > > > >
> >> > > > > > > > > > >
> >> > > > > > > > > > > Jing Ge <ji...@ververica.com.invalid> 于2023年6月2日周五
> >> > 19:37写道：
> >> > > > > > > > > > >
> >> > > > > > > > > > >> Hi Aitozi,
> >> > > > > > > > > > >>
> >> > > > > > > > > > >> Thanks for the update. Just out of curiosity, what
> is
> >> > the
> >> > > > > > > difference
> >> > > > > > > > > > >> between the RPC call or query you mentioned and the
> >> > lookup
> >> > > > in
> >> > > > > a
> >> > > > > > > very
> >> > > > > > > > > > >> general way? Since Lateral join is used in the
> FLIP.
> >> Is
> >> > > > there
> >> > > > > > any
> >> > > > > > > > > > special
> >> > > > > > > > > > >> thought for that? Sorry for asking so many
> questions.
> >> > The
> >> > > > FLIP
> >> > > > > > > > > contains
> >> > > > > > > > > > >> limited information to understand the motivation.
> >> > > > > > > > > > >>
> >> > > > > > > > > > >> Best regards,
> >> > > > > > > > > > >> Jing
> >> > > > > > > > > > >>
> >> > > > > > > > > > >> On Fri, Jun 2, 2023 at 3:48 AM Aitozi <
> >> > > gjying1314@gmail.com
> >> > > > >
> >> > > > > > > wrote:
> >> > > > > > > > > > >>
> >> > > > > > > > > > >> > Hi Jing,
> >> > > > > > > > > > >> >     I have updated the proposed changes to the
> >> FLIP.
> >> > > IMO,
> >> > > > > > lookup
> >> > > > > > > > has
> >> > > > > > > > > > its
> >> > > > > > > > > > >> > clear
> >> > > > > > > > > > >> > async call requirement is due to its IO heavy
> >> > operator.
> >> > > In
> >> > > > > our
> >> > > > > > > > > usage,
> >> > > > > > > > > > >> sql
> >> > > > > > > > > > >> > users have
> >> > > > > > > > > > >> > logic to do some RPC call or query the
> third-party
> >> > > service
> >> > > > > > which
> >> > > > > > > > is
> >> > > > > > > > > > >> also IO
> >> > > > > > > > > > >> > intensive.
> >> > > > > > > > > > >> > In these case, we'd like to leverage the async
> >> > function
> >> > > to
> >> > > > > > > improve
> >> > > > > > > > > the
> >> > > > > > > > > > >> > throughput.
> >> > > > > > > > > > >> >
> >> > > > > > > > > > >> > Thanks,
> >> > > > > > > > > > >> > Aitozi.
> >> > > > > > > > > > >> >
> >> > > > > > > > > > >> > Jing Ge <ji...@ververica.com.invalid>
> 于2023年6月1日周四
> >> > > > 22:55写道：
> >> > > > > > > > > > >> >
> >> > > > > > > > > > >> > > Hi Aitozi,
> >> > > > > > > > > > >> > >
> >> > > > > > > > > > >> > > Sorry for the late reply. Would you like to
> >> update
> >> > the
> >> > > > > > > proposed
> >> > > > > > > > > > >> changes
> >> > > > > > > > > > >> > > with more details into the FLIP too?
> >> > > > > > > > > > >> > > I got your point. It looks like a rational
> idea.
> >> > > > However,
> >> > > > > > > since
> >> > > > > > > > > > lookup
> >> > > > > > > > > > >> > has
> >> > > > > > > > > > >> > > its clear async call requirement, are there any
> >> real
> >> > > use
> >> > > > > > cases
> >> > > > > > > > > that
> >> > > > > > > > > > >> > > need this change? This will help us understand
> >> the
> >> > > > > > motivation.
> >> > > > > > > > > After
> >> > > > > > > > > > >> all,
> >> > > > > > > > > > >> > > lateral join and temporal lookup join[1] are
> >> quite
> >> > > > > > different.
> >> > > > > > > > > > >> > >
> >> > > > > > > > > > >> > > Best regards,
> >> > > > > > > > > > >> > > Jing
> >> > > > > > > > > > >> > >
> >> > > > > > > > > > >> > >
> >> > > > > > > > > > >> > > [1]
> >> > > > > > > > > > >> > >
> >> > > > > > > > > > >> > >
> >> > > > > > > > > > >> >
> >> > > > > > > > > > >>
> >> > > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > >
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
> https://github.com/apache/flink/blob/d90a72da2fd601ca4e2a46700e91ec5b348de2ad/flink-table/flink-table-common/src/main/java/org/apache/flink/table/functions/AsyncTableFunction.java#L54
> >> > > > > > > > > > >> > >
> >> > > > > > > > > > >> > > On Wed, May 31, 2023 at 8:53 AM Aitozi <
> >> > > > > > gjying1314@gmail.com>
> >> > > > > > > > > > wrote:
> >> > > > > > > > > > >> > >
> >> > > > > > > > > > >> > > > Hi Jing,
> >> > > > > > > > > > >> > > >     What do you think about it? Can we move
> >> > forward
> >> > > > this
> >> > > > > > > > > feature?
> >> > > > > > > > > > >> > > >
> >> > > > > > > > > > >> > > > Thanks,
> >> > > > > > > > > > >> > > > Aitozi.
> >> > > > > > > > > > >> > > >
> >> > > > > > > > > > >> > > > Aitozi <gj...@gmail.com> 于2023年5月29日周一
> >> > > 09:56写道：
> >> > > > > > > > > > >> > > >
> >> > > > > > > > > > >> > > > > Hi Jing,
> >> > > > > > > > > > >> > > > >     > "Do you mean to support the
> >> > > AyncTableFunction
> >> > > > > > beyond
> >> > > > > > > > the
> >> > > > > > > > > > >> > > > > LookupTableSource?"
> >> > > > > > > > > > >> > > > > Yes, I mean to support the
> AyncTableFunction
> >> > > beyond
> >> > > > > the
> >> > > > > > > > > > >> > > > LookupTableSource.
> >> > > > > > > > > > >> > > > >
> >> > > > > > > > > > >> > > > > The "AsyncTableFunction" is the function
> with
> >> > > > ability
> >> > > > > to
> >> > > > > > > be
> >> > > > > > > > > > >> executed
> >> > > > > > > > > > >> > > > async
> >> > > > > > > > > > >> > > > > (with AsyncWaitOperator).
> >> > > > > > > > > > >> > > > > The async lookup join is a one of usage of
> >> this.
> >> > > So,
> >> > > > > we
> >> > > > > > > > don't
> >> > > > > > > > > > >> have to
> >> > > > > > > > > > >> > > > bind
> >> > > > > > > > > > >> > > > > the AyncTableFunction with
> LookupTableSource.
> >> > > > > > > > > > >> > > > > If User-defined AsyncTableFunction is
> >> supported,
> >> > > > user
> >> > > > > > can
> >> > > > > > > > > > directly
> >> > > > > > > > > > >> > use
> >> > > > > > > > > > >> > > > > lateral table syntax to perform async
> >> operation.
> >> > > > > > > > > > >> > > > >
> >> > > > > > > > > > >> > > > > > "It would be better if you could
> elaborate
> >> the
> >> > > > > > proposed
> >> > > > > > > > > > changes
> >> > > > > > > > > > >> wrt
> >> > > > > > > > > > >> > > the
> >> > > > > > > > > > >> > > > > CorrelatedCodeGenerator with more details"
> >> > > > > > > > > > >> > > > >
> >> > > > > > > > > > >> > > > > In the proposal, we use lateral table
> syntax
> >> to
> >> > > > > support
> >> > > > > > > the
> >> > > > > > > > > > async
> >> > > > > > > > > > >> > table
> >> > > > > > > > > > >> > > > > function. So the planner will also treat
> this
> >> > > > > statement
> >> > > > > > > to a
> >> > > > > > > > > > >> > > > > CommonExecCorrelate node. So the runtime
> code
> >> > > should
> >> > > > > be
> >> > > > > > > > > > generated
> >> > > > > > > > > > >> in
> >> > > > > > > > > > >> > > > > CorrelatedCodeGenerator.
> >> > > > > > > > > > >> > > > > In CorrelatedCodeGenerator, we will know
> the
> >> > > > > > > TableFunction's
> >> > > > > > > > > > Kind
> >> > > > > > > > > > >> of
> >> > > > > > > > > > >> > > > > `FunctionKind.Table` or
> >> > `FunctionKind.ASYNC_TABLE`
> >> > > > > > > > > > >> > > > > For  `FunctionKind.ASYNC_TABLE` we can
> >> generate
> >> > a
> >> > > > > > > > > > >> AsyncWaitOperator
> >> > > > > > > > > > >> > to
> >> > > > > > > > > > >> > > > > execute the async table function.
> >> > > > > > > > > > >> > > > >
> >> > > > > > > > > > >> > > > >
> >> > > > > > > > > > >> > > > > Thanks,
> >> > > > > > > > > > >> > > > > Aitozi.
> >> > > > > > > > > > >> > > > >
> >> > > > > > > > > > >> > > > >
> >> > > > > > > > > > >> > > > > Jing Ge <ji...@ververica.com.invalid>
> >> > > 于2023年5月29日周一
> >> > > > > > > 03:22写道：
> >> > > > > > > > > > >> > > > >
> >> > > > > > > > > > >> > > > >> Hi Aitozi,
> >> > > > > > > > > > >> > > > >>
> >> > > > > > > > > > >> > > > >> Thanks for the clarification. The naming
> >> > "Lookup"
> >> > > > > might
> >> > > > > > > > > suggest
> >> > > > > > > > > > >> > using
> >> > > > > > > > > > >> > > it
> >> > > > > > > > > > >> > > > >> for table look up. But conceptually what
> the
> >> > > eval()
> >> > > > > > > method
> >> > > > > > > > > will
> >> > > > > > > > > > >> do
> >> > > > > > > > > > >> > is
> >> > > > > > > > > > >> > > to
> >> > > > > > > > > > >> > > > >> get a collection of results(Row, RowData)
> >> from
> >> > > the
> >> > > > > > given
> >> > > > > > > > > keys.
> >> > > > > > > > > > >> How
> >> > > > > > > > > > >> > it
> >> > > > > > > > > > >> > > > will
> >> > > > > > > > > > >> > > > >> be done depends on the implementation,
> i.e.
> >> you
> >> > > can
> >> > > > > > > > implement
> >> > > > > > > > > > >> your
> >> > > > > > > > > > >> > own
> >> > > > > > > > > > >> > > > >> Source[1][2]. The example in the FLIP
> >> should be
> >> > > > able
> >> > > > > to
> >> > > > > > > be
> >> > > > > > > > > > >> handled
> >> > > > > > > > > > >> > in
> >> > > > > > > > > > >> > > > this
> >> > > > > > > > > > >> > > > >> way.
> >> > > > > > > > > > >> > > > >>
> >> > > > > > > > > > >> > > > >> Do you mean to support the
> AyncTableFunction
> >> > > beyond
> >> > > > > the
> >> > > > > > > > > > >> > > > LookupTableSource?
> >> > > > > > > > > > >> > > > >> It would be better if you could elaborate
> >> the
> >> > > > > proposed
> >> > > > > > > > > changes
> >> > > > > > > > > > >> wrt
> >> > > > > > > > > > >> > the
> >> > > > > > > > > > >> > > > >> CorrelatedCodeGenerator with more details.
> >> > > Thanks!
> >> > > > > > > > > > >> > > > >>
> >> > > > > > > > > > >> > > > >> Best regards,
> >> > > > > > > > > > >> > > > >> Jing
> >> > > > > > > > > > >> > > > >>
> >> > > > > > > > > > >> > > > >> [1]
> >> > > > > > > > > > >> > > > >>
> >> > > > > > > > > > >> > > > >>
> >> > > > > > > > > > >> > > >
> >> > > > > > > > > > >> > >
> >> > > > > > > > > > >> >
> >> > > > > > > > > > >>
> >> > > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > >
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
> https://github.com/apache/flink/blob/678370b18e1b6c4a23e5ce08f8efd05675a0cc17/flink-table/flink-table-common/src/main/java/org/apache/flink/table/connector/source/LookupTableSource.java#L64
> >> > > > > > > > > > >> > > > >> [2]
> >> > > > > > > > > > >> > > > >>
> >> > > > > > > > > > >> > > > >>
> >> > > > > > > > > > >> > > >
> >> > > > > > > > > > >> > >
> >> > > > > > > > > > >> >
> >> > > > > > > > > > >>
> >> > > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > >
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
> https://github.com/apache/flink/blob/678370b18e1b6c4a23e5ce08f8efd05675a0cc17/flink-table/flink-table-common/src/main/java/org/apache/flink/table/connector/source/AsyncTableFunctionProvider.java#L49
> >> > > > > > > > > > >> > > > >>
> >> > > > > > > > > > >> > > > >> On Sat, May 27, 2023 at 9:48 AM Aitozi <
> >> > > > > > > > gjying1314@gmail.com
> >> > > > > > > > > >
> >> > > > > > > > > > >> > wrote:
> >> > > > > > > > > > >> > > > >>
> >> > > > > > > > > > >> > > > >> > Hi Jing,
> >> > > > > > > > > > >> > > > >> >     Thanks for your response. As stated
> in
> >> > the
> >> > > > > FLIP,
> >> > > > > > > the
> >> > > > > > > > > > >> purpose
> >> > > > > > > > > > >> > of
> >> > > > > > > > > > >> > > > this
> >> > > > > > > > > > >> > > > >> > FLIP is meant to support
> >> > > > > > > > > > >> > > > >> > user-defined async table function. As
> >> > described
> >> > > > in
> >> > > > > > > flink
> >> > > > > > > > > > >> document
> >> > > > > > > > > > >> > > [1]
> >> > > > > > > > > > >> > > > >> >
> >> > > > > > > > > > >> > > > >> > Async table functions are special
> >> functions
> >> > for
> >> > > > > table
> >> > > > > > > > > sources
> >> > > > > > > > > > >> that
> >> > > > > > > > > > >> > > > >> perform
> >> > > > > > > > > > >> > > > >> > > a lookup.
> >> > > > > > > > > > >> > > > >> > >
> >> > > > > > > > > > >> > > > >> >
> >> > > > > > > > > > >> > > > >> > So end user can not directly define and
> >> use
> >> > > async
> >> > > > > > table
> >> > > > > > > > > > >> function
> >> > > > > > > > > > >> > > now.
> >> > > > > > > > > > >> > > > An
> >> > > > > > > > > > >> > > > >> > user case is reported in [2]
> >> > > > > > > > > > >> > > > >> >
> >> > > > > > > > > > >> > > > >> > So, in conclusion, no new interface is
> >> > > > introduced,
> >> > > > > > but
> >> > > > > > > we
> >> > > > > > > > > > >> extend
> >> > > > > > > > > > >> > the
> >> > > > > > > > > > >> > > > >> > ability to support user-defined async
> >> table
> >> > > > > function.
> >> > > > > > > > > > >> > > > >> >
> >> > > > > > > > > > >> > > > >> > [1]:
> >> > > > > > > > > > >> > > > >> >
> >> > > > > > > > > > >> > > > >> >
> >> > > > > > > > > > >> > > > >>
> >> > > > > > > > > > >> > > >
> >> > > > > > > > > > >> > >
> >> > > > > > > > > > >> >
> >> > > > > > > > > > >>
> >> > > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > >
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
> https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/dev/table/functions/udfs/
> >> > > > > > > > > > >> > > > >> > [2]:
> >> > > > > > > > > > >> > >
> >> > > > > > > >
> >> > https://lists.apache.org/thread/qljwd40v5ntz6733cwcdr8s4z97b343b
> >> > > > > > > > > > >> > > > >> >
> >> > > > > > > > > > >> > > > >> > Thanks.
> >> > > > > > > > > > >> > > > >> > Aitozi.
> >> > > > > > > > > > >> > > > >> >
> >> > > > > > > > > > >> > > > >> >
> >> > > > > > > > > > >> > > > >> > Jing Ge <ji...@ververica.com.invalid>
> >> > > > 于2023年5月27日周六
> >> > > > > > > > > 06:40写道：
> >> > > > > > > > > > >> > > > >> >
> >> > > > > > > > > > >> > > > >> > > Hi Aitozi,
> >> > > > > > > > > > >> > > > >> > >
> >> > > > > > > > > > >> > > > >> > > Thanks for your proposal. I am not
> quite
> >> > sure
> >> > > > if
> >> > > > > I
> >> > > > > > > > > > understood
> >> > > > > > > > > > >> > your
> >> > > > > > > > > > >> > > > >> > thoughts
> >> > > > > > > > > > >> > > > >> > > correctly. You described a special
> case
> >> > > > > > > implementation
> >> > > > > > > > of
> >> > > > > > > > > > the
> >> > > > > > > > > > >> > > > >> > > AsyncTableFunction with on public API
> >> > > changes.
> >> > > > > > Would
> >> > > > > > > > you
> >> > > > > > > > > > >> please
> >> > > > > > > > > > >> > > > >> elaborate
> >> > > > > > > > > > >> > > > >> > > your purpose of writing a FLIP
> >> according to
> >> > > the
> >> > > > > > FLIP
> >> > > > > > > > > > >> > > > documentation[1]?
> >> > > > > > > > > > >> > > > >> > > Thanks!
> >> > > > > > > > > > >> > > > >> > >
> >> > > > > > > > > > >> > > > >> > > [1]
> >> > > > > > > > > > >> > > > >> > >
> >> > > > > > > > > > >> > > > >> > >
> >> > > > > > > > > > >> > > > >> >
> >> > > > > > > > > > >> > > > >>
> >> > > > > > > > > > >> > > >
> >> > > > > > > > > > >> > >
> >> > > > > > > > > > >> >
> >> > > > > > > > > > >>
> >> > > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > >
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
> https://cwiki.apache.org/confluence/display/FLINK/Flink+Improvement+Proposals
> >> > > > > > > > > > >> > > > >> > >
> >> > > > > > > > > > >> > > > >> > > Best regards,
> >> > > > > > > > > > >> > > > >> > > Jing
> >> > > > > > > > > > >> > > > >> > >
> >> > > > > > > > > > >> > > > >> > > On Wed, May 24, 2023 at 1:07 PM
> Aitozi <
> >> > > > > > > > > > gjying1314@gmail.com
> >> > > > > > > > > > >> >
> >> > > > > > > > > > >> > > > wrote:
> >> > > > > > > > > > >> > > > >> > >
> >> > > > > > > > > > >> > > > >> > > > May I ask for some feedback  :D
> >> > > > > > > > > > >> > > > >> > > >
> >> > > > > > > > > > >> > > > >> > > > Thanks,
> >> > > > > > > > > > >> > > > >> > > > Aitozi
> >> > > > > > > > > > >> > > > >> > > >
> >> > > > > > > > > > >> > > > >> > > > Aitozi <gj...@gmail.com>
> >> > > 于2023年5月23日周二
> >> > > > > > > 19:14写道：
> >> > > > > > > > > > >> > > > >> > > > >
> >> > > > > > > > > > >> > > > >> > > > > Just catch an user case report
> from
> >> > > Giannis
> >> > > > > > > Polyzos
> >> > > > > > > > > for
> >> > > > > > > > > > >> this
> >> > > > > > > > > > >> > > > >> usage:
> >> > > > > > > > > > >> > > > >> > > > >
> >> > > > > > > > > > >> > > > >> > > > >
> >> > > > > > > > > > >> > > >
> >> > > > > > > > >
> >> > > https://lists.apache.org/thread/qljwd40v5ntz6733cwcdr8s4z97b343b
> >> > > > > > > > > > >> > > > >> > > > >
> >> > > > > > > > > > >> > > > >> > > > > Aitozi <gj...@gmail.com>
> >> > > > 于2023年5月23日周二
> >> > > > > > > > 17:45写道：
> >> > > > > > > > > > >> > > > >> > > > > >
> >> > > > > > > > > > >> > > > >> > > > > > Hi guys,
> >> > > > > > > > > > >> > > > >> > > > > >     I want to bring up a
> >> discussion
> >> > > about
> >> > > > > > > adding
> >> > > > > > > > > > >> support
> >> > > > > > > > > > >> > of
> >> > > > > > > > > > >> > > > User
> >> > > > > > > > > > >> > > > >> > > > > > Defined AsyncTableFunction in
> >> Flink.
> >> > > > > > > > > > >> > > > >> > > > > > Currently, async table function
> >> are
> >> > > > special
> >> > > > > > > > > functions
> >> > > > > > > > > > >> for
> >> > > > > > > > > > >> > > > table
> >> > > > > > > > > > >> > > > >> > > source
> >> > > > > > > > > > >> > > > >> > > > > > to perform
> >> > > > > > > > > > >> > > > >> > > > > > async lookup. However, it's
> worth
> >> to
> >> > > > > support
> >> > > > > > > the
> >> > > > > > > > > user
> >> > > > > > > > > > >> > > defined
> >> > > > > > > > > > >> > > > >> async
> >> > > > > > > > > > >> > > > >> > > > > > table function.
> >> > > > > > > > > > >> > > > >> > > > > > Because, in this way, the end
> SQL
> >> > user
> >> > > > can
> >> > > > > > > > leverage
> >> > > > > > > > > > it
> >> > > > > > > > > > >> to
> >> > > > > > > > > > >> > > > >> perform
> >> > > > > > > > > > >> > > > >> > the
> >> > > > > > > > > > >> > > > >> > > > > > async operation
> >> > > > > > > > > > >> > > > >> > > > > > which is useful to maximum the
> >> system
> >> > > > > > > throughput
> >> > > > > > > > > > >> > especially
> >> > > > > > > > > > >> > > > for
> >> > > > > > > > > > >> > > > >> IO
> >> > > > > > > > > > >> > > > >> > > > > > bottleneck case.
> >> > > > > > > > > > >> > > > >> > > > > >
> >> > > > > > > > > > >> > > > >> > > > > > You can find some more detail in
> >> [1].
> >> > > > > > > > > > >> > > > >> > > > > >
> >> > > > > > > > > > >> > > > >> > > > > > Looking forward to feedback
> >> > > > > > > > > > >> > > > >> > > > > >
> >> > > > > > > > > > >> > > > >> > > > > >
> >> > > > > > > > > > >> > > > >> > > > > > [1]:
> >> > > > > > > > > > >> > > > >> > > >
> >> > > > > > > > > > >> > > > >> > >
> >> > > > > > > > > > >> > > > >> >
> >> > > > > > > > > > >> > > > >>
> >> > > > > > > > > > >> > > >
> >> > > > > > > > > > >> > >
> >> > > > > > > > > > >> >
> >> > > > > > > > > > >>
> >> > > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > >
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
> https://cwiki.apache.org/confluence/display/FLINK/%5BFLIP-313%5D+Add+support+of+User+Defined+AsyncTableFunction
> >> > > > > > > > > > >> > > > >> > > > > >
> >> > > > > > > > > > >> > > > >> > > > > > Thanks,
> >> > > > > > > > > > >> > > > >> > > > > > Aitozi.
> >> > > > > > > > > > >> > > > >> > > >
> >> > > > > > > > > > >> > > > >> > >
> >> > > > > > > > > > >> > > > >> >
> >> > > > > > > > > > >> > > > >>
> >> > > > > > > > > > >> > > > >
> >> > > > > > > > > > >> > > >
> >> > > > > > > > > > >> > >
> >> > > > > > > > > > >> >
> >> > > > > > > > > > >>
> >> > > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > >
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
> >
>

Re: [DISCUSS] FLIP-313 Add support of User Defined AsyncTableFunction

Posted by Aitozi <gj...@gmail.com>.

BTW, If there are no other more blocking issue / comments, I would like to
start a VOTE in another thread this wednesday 6.14

Thanks,
Aitozi.

Aitozi <gj...@gmail.com> 于2023年6月12日周一 23:34写道：

> Hi, Jing,
>     Thanks for your explanation. I get your point now.
>
> For the performance part, I think it's a good idea to run with returning a
> big table case, the memory consumption
> should be a point to be taken care about. Because in the ordered mode, the
> head element in buffer may affect the
> total memory consumption.
>
>
> Thanks,
> Aitozi.
>
>
>
> Jing Ge <ji...@ververica.com.invalid> 于2023年6月12日周一 20:28写道：
>
>> Hi Aitozi,
>>
>> Which key will be used for lookup is not an issue, only one row will be
>> required for each key in order to enrich it. True, it depends on the
>> implementation whether multiple rows or single row for each key will be
>> returned. However, for the lookup & enrichment scenario, one row/key is
>> recommended, otherwise, like I mentioned previously, enrichment won't
>> work.
>>
>> I am a little bit concerned about returning a big table for each key,
>> since
>> it will take the async call longer to return and need more memory. The
>> performance tests should cover this scenario. This is not a blocking issue
>> for this FLIP.
>>
>> Best regards,
>> Jing
>>
>> On Sat, Jun 10, 2023 at 4:11 AM Aitozi <gj...@gmail.com> wrote:
>>
>> > Hi Jing,
>> >     I means the join key is not necessary to be the primary key or
>> unique
>> > index of the database.
>> > In this situation, we may queried out multi rows for one join key. I
>> think
>> > that's why the
>> > LookupFunction#lookup will return a collection of RowData.
>> >
>> > BTW, I think the behavior of lookup join will not affect the semantic of
>> > the async udtf.
>> > We use the Async TableFunction here and the table function can collect
>> > multiple rows.
>> >
>> > Thanks,
>> > Atiozi.
>> >
>> >
>> >
>> > Jing Ge <ji...@ververica.com.invalid> 于2023年6月10日周六 00:15写道：
>> >
>> > > Hi Aitozi,
>> > >
>> > > The keyRow used in this case contains all keys[1].
>> > >
>> > > Best regards,
>> > > Jing
>> > >
>> > > [1]
>> > >
>> > >
>> >
>> https://github.com/apache/flink/blob/191ec6ca3943d7119f14837efe112e074d815c47/flink-table/flink-table-common/src/main/java/org/apache/flink/table/functions/LookupFunction.java#L49
>> > >
>> > >
>> > > On Fri, Jun 9, 2023 at 3:42 PM Aitozi <gj...@gmail.com> wrote:
>> > >
>> > > > Hi Jing,
>> > > >
>> > > >      The performance test is added to the FLIP.
>> > > >
>> > > >      As I know, The lookup join can return multi rows, it depends on
>> > > > whether  the join key
>> > > > is the primary key of the external database or not. The `lookup` [1]
>> > will
>> > > > return a collection of
>> > > > joined result, and each of them will be collected
>> > > >
>> > > >
>> > > > [1]:
>> > > >
>> > > >
>> > >
>> >
>> https://github.com/apache/flink/blob/191ec6ca3943d7119f14837efe112e074d815c47/flink-table/flink-table-common/src/main/java/org/apache/flink/table/functions/LookupFunction.java#L52
>> > > >
>> > > >
>> > > > Thanks,
>> > > > Aitozi.
>> > > >
>> > > > Jing Ge <ji...@ververica.com.invalid> 于2023年6月9日周五 17:05写道：
>> > > >
>> > > > > Hi Aitozi,
>> > > > >
>> > > > > Thanks for the feedback. Looking forward to the performance tests.
>> > > > >
>> > > > > Afaik, lookup returns one row for each key [1] [2]. Conceptually,
>> the
>> > > > > lookup function is used to enrich column(s) from the dimension
>> table.
>> > > If,
>> > > > > for the given key, there will be more than one row, there will be
>> no
>> > > way
>> > > > to
>> > > > > know which row will be used to enrich the key.
>> > > > >
>> > > > > [1]
>> > > > >
>> > > > >
>> > > >
>> > >
>> >
>> https://github.com/apache/flink/blob/191ec6ca3943d7119f14837efe112e074d815c47/flink-table/flink-table-common/src/main/java/org/apache/flink/table/functions/LookupFunction.java#L49
>> > > > > [2]
>> > > > >
>> > > > >
>> > > >
>> > >
>> >
>> https://github.com/apache/flink/blob/191ec6ca3943d7119f14837efe112e074d815c47/flink-table/flink-table-common/src/main/java/org/apache/flink/table/functions/TableFunction.java#L196
>> > > > >
>> > > > > Best regards,
>> > > > > Jing
>> > > > >
>> > > > > On Fri, Jun 9, 2023 at 5:18 AM Aitozi <gj...@gmail.com>
>> wrote:
>> > > > >
>> > > > > > Hi Jing
>> > > > > >     Thanks for your good questions. I have updated the example
>> to
>> > the
>> > > > > FLIP.
>> > > > > >
>> > > > > > > Only one row for each lookup
>> > > > > > lookup can also return multi rows, based on the query result.
>> [1]
>> > > > > >
>> > > > > > [1]:
>> > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> https://github.com/apache/flink/blob/191ec6ca3943d7119f14837efe112e074d815c47/flink-table/flink-table-common/src/main/java/org/apache/flink/table/functions/LookupFunction.java#L56
>> > > > > >
>> > > > > > > If we use async calls with lateral join, my gut feeling is
>> > > > > > that we might have many more async calls than lookup join. I am
>> not
>> > > > > really
>> > > > > > sure if we will be facing potential issues in this case or not.
>> > > > > >
>> > > > > > IMO, the work pattern is similar to the lookup function, for
>> each
>> > row
>> > > > > from
>> > > > > > the left table,
>> > > > > > it will evaluate the eval method once, so the async call numbers
>> > will
>> > > > not
>> > > > > > change.
>> > > > > > and the maximum calls in flight is limited by the Async
>> operators
>> > > > buffer
>> > > > > > capacity
>> > > > > > which will be controlled by the option.
>> > > > > >
>> > > > > > BTW, for the naming of these option, I updated the FLIP about
>> this
>> > > you
>> > > > > can
>> > > > > > refer to
>> > > > > > the section of "ConfigOption" and "Rejected Alternatives"
>> > > > > >
>> > > > > > In the end, for the performance evaluation, I'd like to do some
>> > tests
>> > > > and
>> > > > > > will update it to the FLIP doc
>> > > > > >
>> > > > > > Thanks,
>> > > > > > Aitozi.
>> > > > > >
>> > > > > >
>> > > > > > Jing Ge <ji...@ververica.com.invalid> 于2023年6月9日周五 07:23写道：
>> > > > > >
>> > > > > > > Hi Aitozi,
>> > > > > > >
>> > > > > > > Thanks for the clarification. The code example looks
>> > interesting. I
>> > > > > would
>> > > > > > > suggest adding them into the FLIP. The description with code
>> > > examples
>> > > > > > will
>> > > > > > > help readers understand the motivation and how to use it.
>> Afaiac,
>> > > it
>> > > > > is a
>> > > > > > > valid feature for Flink users.
>> > > > > > >
>> > > > > > > As we knew, lookup join is based on temporal join, i.e. FOR
>> > > > SYSTEM_TIME
>> > > > > > AS
>> > > > > > > OF which is also used in your code example. Temporal join
>> > performs
>> > > > the
>> > > > > > > lookup based on the processing time match. Only one row for
>> each
>> > > > > > > lookup(afaiu, I need to check the source code to double
>> confirm)
>> > > will
>> > > > > > > return for further enrichment. One the other hand, lateral
>> join
>> > > will
>> > > > > have
>> > > > > > > sub-queries correlated with every individual value of the
>> > reference
>> > > > > table
>> > > > > > > from the preceding part of the query and each sub query will
>> > return
>> > > > > > > multiple rows. If we use async calls with lateral join, my gut
>> > > > feeling
>> > > > > is
>> > > > > > > that we might have many more async calls than lookup join. I
>> am
>> > not
>> > > > > > really
>> > > > > > > sure if we will be facing potential issues in this case or
>> not.
>> > > > > Possible
>> > > > > > > issues I can think of now e.g. too many PRC calls, too many
>> async
>> > > > calls
>> > > > > > > processing, the sub query will return a table which might be
>> > (too)
>> > > > big,
>> > > > > > and
>> > > > > > > might cause performance issues. I would suggest preparing some
>> > use
>> > > > > cases
>> > > > > > > and running some performance tests to check it. These are my
>> > > concerns
>> > > > > > about
>> > > > > > > using async calls with lateral join and I'd like to share with
>> > you,
>> > > > > happy
>> > > > > > > to discuss with you and hear different opinions, hopefully the
>> > > > > > > discussion could help me understand it more deeply. Please
>> > correct
>> > > me
>> > > > > if
>> > > > > > I
>> > > > > > > am wrong.
>> > > > > > >
>> > > > > > > Best regards,
>> > > > > > > Jing
>> > > > > > >
>> > > > > > >
>> > > > > > > On Thu, Jun 8, 2023 at 7:22 AM Aitozi <gj...@gmail.com>
>> > > wrote:
>> > > > > > >
>> > > > > > > > Hi Mason,
>> > > > > > > >     Thanks for your input. I think if we support the user
>> > defined
>> > > > > async
>> > > > > > > > table function,
>> > > > > > > > user will be able to use it to hold a batch data then
>> handle it
>> > > at
>> > > > > one
>> > > > > > > time
>> > > > > > > > in the customized function.
>> > > > > > > >
>> > > > > > > > AsyncSink is meant for the sink operator. I have not figure
>> out
>> > > how
>> > > > > to
>> > > > > > > > integrate in this case.
>> > > > > > > >
>> > > > > > > > Thanks,
>> > > > > > > > Atiozi.
>> > > > > > > >
>> > > > > > > >
>> > > > > > > > Mason Chen <ma...@gmail.com> 于2023年6月8日周四 12:40写道：
>> > > > > > > >
>> > > > > > > > > Hi Aitozi,
>> > > > > > > > >
>> > > > > > > > > I think it makes sense to make it easier for SQL users to
>> > make
>> > > > > RPCs.
>> > > > > > Do
>> > > > > > > > you
>> > > > > > > > > think your proposal can extend to the ability to batch
>> data
>> > for
>> > > > the
>> > > > > > > RPC?
>> > > > > > > > > This is also another common strategy to increase
>> throughput.
>> > > > Also,
>> > > > > > have
>> > > > > > > > you
>> > > > > > > > > considered solving this a bit differently by leveraging
>> > Flink's
>> > > > > > > > AsyncSink?
>> > > > > > > > >
>> > > > > > > > > Best,
>> > > > > > > > > Mason
>> > > > > > > > >
>> > > > > > > > > On Mon, Jun 5, 2023 at 1:50 AM Aitozi <
>> gjying1314@gmail.com>
>> > > > > wrote:
>> > > > > > > > >
>> > > > > > > > > > One more thing for discussion:
>> > > > > > > > > >
>> > > > > > > > > > In our internal implementation, we reuse the option
>> > > > > > > > > > `table.exec.async-lookup.buffer-capacity` and
>> > > > > > > > > > `table.exec.async-lookup.timeout` to config
>> > > > > > > > > > the async udtf. Do you think we should add two extra
>> option
>> > > to
>> > > > > > > > > distinguish
>> > > > > > > > > > from the lookup option such as
>> > > > > > > > > >
>> > > > > > > > > > `table.exec.async-udtf.buffer-capacity`
>> > > > > > > > > > `table.exec.async-udtf.timeout`
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > > Best,
>> > > > > > > > > > Aitozi.
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > > Aitozi <gj...@gmail.com> 于2023年6月5日周一 12:20写道：
>> > > > > > > > > >
>> > > > > > > > > > > Hi Jing,
>> > > > > > > > > > >
>> > > > > > > > > > >     > what is the difference between the RPC call or
>> > query
>> > > > you
>> > > > > > > > > mentioned
>> > > > > > > > > > > and the lookup in a very
>> > > > > > > > > > > general way
>> > > > > > > > > > >
>> > > > > > > > > > > I think the RPC call or query service is quite
>> similar to
>> > > the
>> > > > > > > lookup
>> > > > > > > > > > join.
>> > > > > > > > > > > But lookup join should work
>> > > > > > > > > > > with `LookupTableSource`.
>> > > > > > > > > > >
>> > > > > > > > > > > Let's see how we can perform an async RPC call with
>> > lookup
>> > > > > join:
>> > > > > > > > > > >
>> > > > > > > > > > > (1) Implement an AsyncTableFunction with RPC call
>> logic.
>> > > > > > > > > > > (2) Implement a `LookupTableSource` connector run with
>> > the
>> > > > > async
>> > > > > > > udtf
>> > > > > > > > > > > defined in (1).
>> > > > > > > > > > > (3) Then define a DDL of this look up table in SQL
>> > > > > > > > > > >
>> > > > > > > > > > > CREATE TEMPORARY TABLE Customers (
>> > > > > > > > > > >   id INT,
>> > > > > > > > > > >   name STRING,
>> > > > > > > > > > >   country STRING,
>> > > > > > > > > > >   zip STRING
>> > > > > > > > > > > ) WITH (
>> > > > > > > > > > >   'connector' = 'custom'
>> > > > > > > > > > > );
>> > > > > > > > > > >
>> > > > > > > > > > > (4) Run with the query as below:
>> > > > > > > > > > >
>> > > > > > > > > > > SELECT o.order_id, o.total, c.country, c.zip
>> > > > > > > > > > > FROM Orders AS o
>> > > > > > > > > > >   JOIN Customers FOR SYSTEM_TIME AS OF o.proc_time AS
>> c
>> > > > > > > > > > >     ON o.customer_id = c.id;
>> > > > > > > > > > >
>> > > > > > > > > > > This example is from doc
>> > > > > > > > > > > <
>> > > > > > > > > >
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/dev/table/sql/queries/joins/#lookup-join
>> > > > > > > > > > >.You
>> > > > > > > > > > > can image the look up process as an async RPC call
>> > process.
>> > > > > > > > > > >
>> > > > > > > > > > > Let's see how we can perform an async RPC call with
>> > lateral
>> > > > > join:
>> > > > > > > > > > >
>> > > > > > > > > > > (1) Implement an AsyncTableFunction with RPC call
>> logic.
>> > > > > > > > > > > (2) Run query with
>> > > > > > > > > > >
>> > > > > > > > > > > Create function f1 as '...' ;
>> > > > > > > > > > >
>> > > > > > > > > > > SELECT o.order_id, o.total, c.country, c.zip FROM
>> Orders
>> > > > > lateral
>> > > > > > > > table
>> > > > > > > > > > > (f1(order_id)) as T(...);
>> > > > > > > > > > >
>> > > > > > > > > > > As you can see, the lateral join version is more
>> simple
>> > and
>> > > > > > > intuitive
>> > > > > > > > > to
>> > > > > > > > > > > users. Users do not have to wrap a
>> > > > > > > > > > > LookupTableSource for the purpose of using async udtf.
>> > > > > > > > > > >
>> > > > > > > > > > > In the end, We can also see the user defined async
>> table
>> > > > > function
>> > > > > > > is
>> > > > > > > > an
>> > > > > > > > > > > enhancement of the current lateral table join
>> > > > > > > > > > > which only supports sync lateral join now.
>> > > > > > > > > > >
>> > > > > > > > > > > Best,
>> > > > > > > > > > > Aitozi.
>> > > > > > > > > > >
>> > > > > > > > > > >
>> > > > > > > > > > > Jing Ge <ji...@ververica.com.invalid> 于2023年6月2日周五
>> > 19:37写道：
>> > > > > > > > > > >
>> > > > > > > > > > >> Hi Aitozi,
>> > > > > > > > > > >>
>> > > > > > > > > > >> Thanks for the update. Just out of curiosity, what is
>> > the
>> > > > > > > difference
>> > > > > > > > > > >> between the RPC call or query you mentioned and the
>> > lookup
>> > > > in
>> > > > > a
>> > > > > > > very
>> > > > > > > > > > >> general way? Since Lateral join is used in the FLIP.
>> Is
>> > > > there
>> > > > > > any
>> > > > > > > > > > special
>> > > > > > > > > > >> thought for that? Sorry for asking so many questions.
>> > The
>> > > > FLIP
>> > > > > > > > > contains
>> > > > > > > > > > >> limited information to understand the motivation.
>> > > > > > > > > > >>
>> > > > > > > > > > >> Best regards,
>> > > > > > > > > > >> Jing
>> > > > > > > > > > >>
>> > > > > > > > > > >> On Fri, Jun 2, 2023 at 3:48 AM Aitozi <
>> > > gjying1314@gmail.com
>> > > > >
>> > > > > > > wrote:
>> > > > > > > > > > >>
>> > > > > > > > > > >> > Hi Jing,
>> > > > > > > > > > >> >     I have updated the proposed changes to the
>> FLIP.
>> > > IMO,
>> > > > > > lookup
>> > > > > > > > has
>> > > > > > > > > > its
>> > > > > > > > > > >> > clear
>> > > > > > > > > > >> > async call requirement is due to its IO heavy
>> > operator.
>> > > In
>> > > > > our
>> > > > > > > > > usage,
>> > > > > > > > > > >> sql
>> > > > > > > > > > >> > users have
>> > > > > > > > > > >> > logic to do some RPC call or query the third-party
>> > > service
>> > > > > > which
>> > > > > > > > is
>> > > > > > > > > > >> also IO
>> > > > > > > > > > >> > intensive.
>> > > > > > > > > > >> > In these case, we'd like to leverage the async
>> > function
>> > > to
>> > > > > > > improve
>> > > > > > > > > the
>> > > > > > > > > > >> > throughput.
>> > > > > > > > > > >> >
>> > > > > > > > > > >> > Thanks,
>> > > > > > > > > > >> > Aitozi.
>> > > > > > > > > > >> >
>> > > > > > > > > > >> > Jing Ge <ji...@ververica.com.invalid> 于2023年6月1日周四
>> > > > 22:55写道：
>> > > > > > > > > > >> >
>> > > > > > > > > > >> > > Hi Aitozi,
>> > > > > > > > > > >> > >
>> > > > > > > > > > >> > > Sorry for the late reply. Would you like to
>> update
>> > the
>> > > > > > > proposed
>> > > > > > > > > > >> changes
>> > > > > > > > > > >> > > with more details into the FLIP too?
>> > > > > > > > > > >> > > I got your point. It looks like a rational idea.
>> > > > However,
>> > > > > > > since
>> > > > > > > > > > lookup
>> > > > > > > > > > >> > has
>> > > > > > > > > > >> > > its clear async call requirement, are there any
>> real
>> > > use
>> > > > > > cases
>> > > > > > > > > that
>> > > > > > > > > > >> > > need this change? This will help us understand
>> the
>> > > > > > motivation.
>> > > > > > > > > After
>> > > > > > > > > > >> all,
>> > > > > > > > > > >> > > lateral join and temporal lookup join[1] are
>> quite
>> > > > > > different.
>> > > > > > > > > > >> > >
>> > > > > > > > > > >> > > Best regards,
>> > > > > > > > > > >> > > Jing
>> > > > > > > > > > >> > >
>> > > > > > > > > > >> > >
>> > > > > > > > > > >> > > [1]
>> > > > > > > > > > >> > >
>> > > > > > > > > > >> > >
>> > > > > > > > > > >> >
>> > > > > > > > > > >>
>> > > > > > > > > >
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> https://github.com/apache/flink/blob/d90a72da2fd601ca4e2a46700e91ec5b348de2ad/flink-table/flink-table-common/src/main/java/org/apache/flink/table/functions/AsyncTableFunction.java#L54
>> > > > > > > > > > >> > >
>> > > > > > > > > > >> > > On Wed, May 31, 2023 at 8:53 AM Aitozi <
>> > > > > > gjying1314@gmail.com>
>> > > > > > > > > > wrote:
>> > > > > > > > > > >> > >
>> > > > > > > > > > >> > > > Hi Jing,
>> > > > > > > > > > >> > > >     What do you think about it? Can we move
>> > forward
>> > > > this
>> > > > > > > > > feature?
>> > > > > > > > > > >> > > >
>> > > > > > > > > > >> > > > Thanks,
>> > > > > > > > > > >> > > > Aitozi.
>> > > > > > > > > > >> > > >
>> > > > > > > > > > >> > > > Aitozi <gj...@gmail.com> 于2023年5月29日周一
>> > > 09:56写道：
>> > > > > > > > > > >> > > >
>> > > > > > > > > > >> > > > > Hi Jing,
>> > > > > > > > > > >> > > > >     > "Do you mean to support the
>> > > AyncTableFunction
>> > > > > > beyond
>> > > > > > > > the
>> > > > > > > > > > >> > > > > LookupTableSource?"
>> > > > > > > > > > >> > > > > Yes, I mean to support the AyncTableFunction
>> > > beyond
>> > > > > the
>> > > > > > > > > > >> > > > LookupTableSource.
>> > > > > > > > > > >> > > > >
>> > > > > > > > > > >> > > > > The "AsyncTableFunction" is the function with
>> > > > ability
>> > > > > to
>> > > > > > > be
>> > > > > > > > > > >> executed
>> > > > > > > > > > >> > > > async
>> > > > > > > > > > >> > > > > (with AsyncWaitOperator).
>> > > > > > > > > > >> > > > > The async lookup join is a one of usage of
>> this.
>> > > So,
>> > > > > we
>> > > > > > > > don't
>> > > > > > > > > > >> have to
>> > > > > > > > > > >> > > > bind
>> > > > > > > > > > >> > > > > the AyncTableFunction with LookupTableSource.
>> > > > > > > > > > >> > > > > If User-defined AsyncTableFunction is
>> supported,
>> > > > user
>> > > > > > can
>> > > > > > > > > > directly
>> > > > > > > > > > >> > use
>> > > > > > > > > > >> > > > > lateral table syntax to perform async
>> operation.
>> > > > > > > > > > >> > > > >
>> > > > > > > > > > >> > > > > > "It would be better if you could elaborate
>> the
>> > > > > > proposed
>> > > > > > > > > > changes
>> > > > > > > > > > >> wrt
>> > > > > > > > > > >> > > the
>> > > > > > > > > > >> > > > > CorrelatedCodeGenerator with more details"
>> > > > > > > > > > >> > > > >
>> > > > > > > > > > >> > > > > In the proposal, we use lateral table syntax
>> to
>> > > > > support
>> > > > > > > the
>> > > > > > > > > > async
>> > > > > > > > > > >> > table
>> > > > > > > > > > >> > > > > function. So the planner will also treat this
>> > > > > statement
>> > > > > > > to a
>> > > > > > > > > > >> > > > > CommonExecCorrelate node. So the runtime code
>> > > should
>> > > > > be
>> > > > > > > > > > generated
>> > > > > > > > > > >> in
>> > > > > > > > > > >> > > > > CorrelatedCodeGenerator.
>> > > > > > > > > > >> > > > > In CorrelatedCodeGenerator, we will know the
>> > > > > > > TableFunction's
>> > > > > > > > > > Kind
>> > > > > > > > > > >> of
>> > > > > > > > > > >> > > > > `FunctionKind.Table` or
>> > `FunctionKind.ASYNC_TABLE`
>> > > > > > > > > > >> > > > > For  `FunctionKind.ASYNC_TABLE` we can
>> generate
>> > a
>> > > > > > > > > > >> AsyncWaitOperator
>> > > > > > > > > > >> > to
>> > > > > > > > > > >> > > > > execute the async table function.
>> > > > > > > > > > >> > > > >
>> > > > > > > > > > >> > > > >
>> > > > > > > > > > >> > > > > Thanks,
>> > > > > > > > > > >> > > > > Aitozi.
>> > > > > > > > > > >> > > > >
>> > > > > > > > > > >> > > > >
>> > > > > > > > > > >> > > > > Jing Ge <ji...@ververica.com.invalid>
>> > > 于2023年5月29日周一
>> > > > > > > 03:22写道：
>> > > > > > > > > > >> > > > >
>> > > > > > > > > > >> > > > >> Hi Aitozi,
>> > > > > > > > > > >> > > > >>
>> > > > > > > > > > >> > > > >> Thanks for the clarification. The naming
>> > "Lookup"
>> > > > > might
>> > > > > > > > > suggest
>> > > > > > > > > > >> > using
>> > > > > > > > > > >> > > it
>> > > > > > > > > > >> > > > >> for table look up. But conceptually what the
>> > > eval()
>> > > > > > > method
>> > > > > > > > > will
>> > > > > > > > > > >> do
>> > > > > > > > > > >> > is
>> > > > > > > > > > >> > > to
>> > > > > > > > > > >> > > > >> get a collection of results(Row, RowData)
>> from
>> > > the
>> > > > > > given
>> > > > > > > > > keys.
>> > > > > > > > > > >> How
>> > > > > > > > > > >> > it
>> > > > > > > > > > >> > > > will
>> > > > > > > > > > >> > > > >> be done depends on the implementation, i.e.
>> you
>> > > can
>> > > > > > > > implement
>> > > > > > > > > > >> your
>> > > > > > > > > > >> > own
>> > > > > > > > > > >> > > > >> Source[1][2]. The example in the FLIP
>> should be
>> > > > able
>> > > > > to
>> > > > > > > be
>> > > > > > > > > > >> handled
>> > > > > > > > > > >> > in
>> > > > > > > > > > >> > > > this
>> > > > > > > > > > >> > > > >> way.
>> > > > > > > > > > >> > > > >>
>> > > > > > > > > > >> > > > >> Do you mean to support the AyncTableFunction
>> > > beyond
>> > > > > the
>> > > > > > > > > > >> > > > LookupTableSource?
>> > > > > > > > > > >> > > > >> It would be better if you could elaborate
>> the
>> > > > > proposed
>> > > > > > > > > changes
>> > > > > > > > > > >> wrt
>> > > > > > > > > > >> > the
>> > > > > > > > > > >> > > > >> CorrelatedCodeGenerator with more details.
>> > > Thanks!
>> > > > > > > > > > >> > > > >>
>> > > > > > > > > > >> > > > >> Best regards,
>> > > > > > > > > > >> > > > >> Jing
>> > > > > > > > > > >> > > > >>
>> > > > > > > > > > >> > > > >> [1]
>> > > > > > > > > > >> > > > >>
>> > > > > > > > > > >> > > > >>
>> > > > > > > > > > >> > > >
>> > > > > > > > > > >> > >
>> > > > > > > > > > >> >
>> > > > > > > > > > >>
>> > > > > > > > > >
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> https://github.com/apache/flink/blob/678370b18e1b6c4a23e5ce08f8efd05675a0cc17/flink-table/flink-table-common/src/main/java/org/apache/flink/table/connector/source/LookupTableSource.java#L64
>> > > > > > > > > > >> > > > >> [2]
>> > > > > > > > > > >> > > > >>
>> > > > > > > > > > >> > > > >>
>> > > > > > > > > > >> > > >
>> > > > > > > > > > >> > >
>> > > > > > > > > > >> >
>> > > > > > > > > > >>
>> > > > > > > > > >
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> https://github.com/apache/flink/blob/678370b18e1b6c4a23e5ce08f8efd05675a0cc17/flink-table/flink-table-common/src/main/java/org/apache/flink/table/connector/source/AsyncTableFunctionProvider.java#L49
>> > > > > > > > > > >> > > > >>
>> > > > > > > > > > >> > > > >> On Sat, May 27, 2023 at 9:48 AM Aitozi <
>> > > > > > > > gjying1314@gmail.com
>> > > > > > > > > >
>> > > > > > > > > > >> > wrote:
>> > > > > > > > > > >> > > > >>
>> > > > > > > > > > >> > > > >> > Hi Jing,
>> > > > > > > > > > >> > > > >> >     Thanks for your response. As stated in
>> > the
>> > > > > FLIP,
>> > > > > > > the
>> > > > > > > > > > >> purpose
>> > > > > > > > > > >> > of
>> > > > > > > > > > >> > > > this
>> > > > > > > > > > >> > > > >> > FLIP is meant to support
>> > > > > > > > > > >> > > > >> > user-defined async table function. As
>> > described
>> > > > in
>> > > > > > > flink
>> > > > > > > > > > >> document
>> > > > > > > > > > >> > > [1]
>> > > > > > > > > > >> > > > >> >
>> > > > > > > > > > >> > > > >> > Async table functions are special
>> functions
>> > for
>> > > > > table
>> > > > > > > > > sources
>> > > > > > > > > > >> that
>> > > > > > > > > > >> > > > >> perform
>> > > > > > > > > > >> > > > >> > > a lookup.
>> > > > > > > > > > >> > > > >> > >
>> > > > > > > > > > >> > > > >> >
>> > > > > > > > > > >> > > > >> > So end user can not directly define and
>> use
>> > > async
>> > > > > > table
>> > > > > > > > > > >> function
>> > > > > > > > > > >> > > now.
>> > > > > > > > > > >> > > > An
>> > > > > > > > > > >> > > > >> > user case is reported in [2]
>> > > > > > > > > > >> > > > >> >
>> > > > > > > > > > >> > > > >> > So, in conclusion, no new interface is
>> > > > introduced,
>> > > > > > but
>> > > > > > > we
>> > > > > > > > > > >> extend
>> > > > > > > > > > >> > the
>> > > > > > > > > > >> > > > >> > ability to support user-defined async
>> table
>> > > > > function.
>> > > > > > > > > > >> > > > >> >
>> > > > > > > > > > >> > > > >> > [1]:
>> > > > > > > > > > >> > > > >> >
>> > > > > > > > > > >> > > > >> >
>> > > > > > > > > > >> > > > >>
>> > > > > > > > > > >> > > >
>> > > > > > > > > > >> > >
>> > > > > > > > > > >> >
>> > > > > > > > > > >>
>> > > > > > > > > >
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/dev/table/functions/udfs/
>> > > > > > > > > > >> > > > >> > [2]:
>> > > > > > > > > > >> > >
>> > > > > > > >
>> > https://lists.apache.org/thread/qljwd40v5ntz6733cwcdr8s4z97b343b
>> > > > > > > > > > >> > > > >> >
>> > > > > > > > > > >> > > > >> > Thanks.
>> > > > > > > > > > >> > > > >> > Aitozi.
>> > > > > > > > > > >> > > > >> >
>> > > > > > > > > > >> > > > >> >
>> > > > > > > > > > >> > > > >> > Jing Ge <ji...@ververica.com.invalid>
>> > > > 于2023年5月27日周六
>> > > > > > > > > 06:40写道：
>> > > > > > > > > > >> > > > >> >
>> > > > > > > > > > >> > > > >> > > Hi Aitozi,
>> > > > > > > > > > >> > > > >> > >
>> > > > > > > > > > >> > > > >> > > Thanks for your proposal. I am not quite
>> > sure
>> > > > if
>> > > > > I
>> > > > > > > > > > understood
>> > > > > > > > > > >> > your
>> > > > > > > > > > >> > > > >> > thoughts
>> > > > > > > > > > >> > > > >> > > correctly. You described a special case
>> > > > > > > implementation
>> > > > > > > > of
>> > > > > > > > > > the
>> > > > > > > > > > >> > > > >> > > AsyncTableFunction with on public API
>> > > changes.
>> > > > > > Would
>> > > > > > > > you
>> > > > > > > > > > >> please
>> > > > > > > > > > >> > > > >> elaborate
>> > > > > > > > > > >> > > > >> > > your purpose of writing a FLIP
>> according to
>> > > the
>> > > > > > FLIP
>> > > > > > > > > > >> > > > documentation[1]?
>> > > > > > > > > > >> > > > >> > > Thanks!
>> > > > > > > > > > >> > > > >> > >
>> > > > > > > > > > >> > > > >> > > [1]
>> > > > > > > > > > >> > > > >> > >
>> > > > > > > > > > >> > > > >> > >
>> > > > > > > > > > >> > > > >> >
>> > > > > > > > > > >> > > > >>
>> > > > > > > > > > >> > > >
>> > > > > > > > > > >> > >
>> > > > > > > > > > >> >
>> > > > > > > > > > >>
>> > > > > > > > > >
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> https://cwiki.apache.org/confluence/display/FLINK/Flink+Improvement+Proposals
>> > > > > > > > > > >> > > > >> > >
>> > > > > > > > > > >> > > > >> > > Best regards,
>> > > > > > > > > > >> > > > >> > > Jing
>> > > > > > > > > > >> > > > >> > >
>> > > > > > > > > > >> > > > >> > > On Wed, May 24, 2023 at 1:07 PM Aitozi <
>> > > > > > > > > > gjying1314@gmail.com
>> > > > > > > > > > >> >
>> > > > > > > > > > >> > > > wrote:
>> > > > > > > > > > >> > > > >> > >
>> > > > > > > > > > >> > > > >> > > > May I ask for some feedback  :D
>> > > > > > > > > > >> > > > >> > > >
>> > > > > > > > > > >> > > > >> > > > Thanks,
>> > > > > > > > > > >> > > > >> > > > Aitozi
>> > > > > > > > > > >> > > > >> > > >
>> > > > > > > > > > >> > > > >> > > > Aitozi <gj...@gmail.com>
>> > > 于2023年5月23日周二
>> > > > > > > 19:14写道：
>> > > > > > > > > > >> > > > >> > > > >
>> > > > > > > > > > >> > > > >> > > > > Just catch an user case report from
>> > > Giannis
>> > > > > > > Polyzos
>> > > > > > > > > for
>> > > > > > > > > > >> this
>> > > > > > > > > > >> > > > >> usage:
>> > > > > > > > > > >> > > > >> > > > >
>> > > > > > > > > > >> > > > >> > > > >
>> > > > > > > > > > >> > > >
>> > > > > > > > >
>> > > https://lists.apache.org/thread/qljwd40v5ntz6733cwcdr8s4z97b343b
>> > > > > > > > > > >> > > > >> > > > >
>> > > > > > > > > > >> > > > >> > > > > Aitozi <gj...@gmail.com>
>> > > > 于2023年5月23日周二
>> > > > > > > > 17:45写道：
>> > > > > > > > > > >> > > > >> > > > > >
>> > > > > > > > > > >> > > > >> > > > > > Hi guys,
>> > > > > > > > > > >> > > > >> > > > > >     I want to bring up a
>> discussion
>> > > about
>> > > > > > > adding
>> > > > > > > > > > >> support
>> > > > > > > > > > >> > of
>> > > > > > > > > > >> > > > User
>> > > > > > > > > > >> > > > >> > > > > > Defined AsyncTableFunction in
>> Flink.
>> > > > > > > > > > >> > > > >> > > > > > Currently, async table function
>> are
>> > > > special
>> > > > > > > > > functions
>> > > > > > > > > > >> for
>> > > > > > > > > > >> > > > table
>> > > > > > > > > > >> > > > >> > > source
>> > > > > > > > > > >> > > > >> > > > > > to perform
>> > > > > > > > > > >> > > > >> > > > > > async lookup. However, it's worth
>> to
>> > > > > support
>> > > > > > > the
>> > > > > > > > > user
>> > > > > > > > > > >> > > defined
>> > > > > > > > > > >> > > > >> async
>> > > > > > > > > > >> > > > >> > > > > > table function.
>> > > > > > > > > > >> > > > >> > > > > > Because, in this way, the end SQL
>> > user
>> > > > can
>> > > > > > > > leverage
>> > > > > > > > > > it
>> > > > > > > > > > >> to
>> > > > > > > > > > >> > > > >> perform
>> > > > > > > > > > >> > > > >> > the
>> > > > > > > > > > >> > > > >> > > > > > async operation
>> > > > > > > > > > >> > > > >> > > > > > which is useful to maximum the
>> system
>> > > > > > > throughput
>> > > > > > > > > > >> > especially
>> > > > > > > > > > >> > > > for
>> > > > > > > > > > >> > > > >> IO
>> > > > > > > > > > >> > > > >> > > > > > bottleneck case.
>> > > > > > > > > > >> > > > >> > > > > >
>> > > > > > > > > > >> > > > >> > > > > > You can find some more detail in
>> [1].
>> > > > > > > > > > >> > > > >> > > > > >
>> > > > > > > > > > >> > > > >> > > > > > Looking forward to feedback
>> > > > > > > > > > >> > > > >> > > > > >
>> > > > > > > > > > >> > > > >> > > > > >
>> > > > > > > > > > >> > > > >> > > > > > [1]:
>> > > > > > > > > > >> > > > >> > > >
>> > > > > > > > > > >> > > > >> > >
>> > > > > > > > > > >> > > > >> >
>> > > > > > > > > > >> > > > >>
>> > > > > > > > > > >> > > >
>> > > > > > > > > > >> > >
>> > > > > > > > > > >> >
>> > > > > > > > > > >>
>> > > > > > > > > >
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> https://cwiki.apache.org/confluence/display/FLINK/%5BFLIP-313%5D+Add+support+of+User+Defined+AsyncTableFunction
>> > > > > > > > > > >> > > > >> > > > > >
>> > > > > > > > > > >> > > > >> > > > > > Thanks,
>> > > > > > > > > > >> > > > >> > > > > > Aitozi.
>> > > > > > > > > > >> > > > >> > > >
>> > > > > > > > > > >> > > > >> > >
>> > > > > > > > > > >> > > > >> >
>> > > > > > > > > > >> > > > >>
>> > > > > > > > > > >> > > > >
>> > > > > > > > > > >> > > >
>> > > > > > > > > > >> > >
>> > > > > > > > > > >> >
>> > > > > > > > > > >>
>> > > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>>
>

Re: [DISCUSS] FLIP-313 Add support of User Defined AsyncTableFunction

Posted by Aitozi <gj...@gmail.com>.

Hi, Jing,
    Thanks for your explanation. I get your point now.

For the performance part, I think it's a good idea to run with returning a
big table case, the memory consumption
should be a point to be taken care about. Because in the ordered mode, the
head element in buffer may affect the
total memory consumption.


Thanks,
Aitozi.



Jing Ge <ji...@ververica.com.invalid> 于2023年6月12日周一 20:28写道：

> Hi Aitozi,
>
> Which key will be used for lookup is not an issue, only one row will be
> required for each key in order to enrich it. True, it depends on the
> implementation whether multiple rows or single row for each key will be
> returned. However, for the lookup & enrichment scenario, one row/key is
> recommended, otherwise, like I mentioned previously, enrichment won't work.
>
> I am a little bit concerned about returning a big table for each key, since
> it will take the async call longer to return and need more memory. The
> performance tests should cover this scenario. This is not a blocking issue
> for this FLIP.
>
> Best regards,
> Jing
>
> On Sat, Jun 10, 2023 at 4:11 AM Aitozi <gj...@gmail.com> wrote:
>
> > Hi Jing,
> >     I means the join key is not necessary to be the primary key or unique
> > index of the database.
> > In this situation, we may queried out multi rows for one join key. I
> think
> > that's why the
> > LookupFunction#lookup will return a collection of RowData.
> >
> > BTW, I think the behavior of lookup join will not affect the semantic of
> > the async udtf.
> > We use the Async TableFunction here and the table function can collect
> > multiple rows.
> >
> > Thanks,
> > Atiozi.
> >
> >
> >
> > Jing Ge <ji...@ververica.com.invalid> 于2023年6月10日周六 00:15写道：
> >
> > > Hi Aitozi,
> > >
> > > The keyRow used in this case contains all keys[1].
> > >
> > > Best regards,
> > > Jing
> > >
> > > [1]
> > >
> > >
> >
> https://github.com/apache/flink/blob/191ec6ca3943d7119f14837efe112e074d815c47/flink-table/flink-table-common/src/main/java/org/apache/flink/table/functions/LookupFunction.java#L49
> > >
> > >
> > > On Fri, Jun 9, 2023 at 3:42 PM Aitozi <gj...@gmail.com> wrote:
> > >
> > > > Hi Jing,
> > > >
> > > >      The performance test is added to the FLIP.
> > > >
> > > >      As I know, The lookup join can return multi rows, it depends on
> > > > whether  the join key
> > > > is the primary key of the external database or not. The `lookup` [1]
> > will
> > > > return a collection of
> > > > joined result, and each of them will be collected
> > > >
> > > >
> > > > [1]:
> > > >
> > > >
> > >
> >
> https://github.com/apache/flink/blob/191ec6ca3943d7119f14837efe112e074d815c47/flink-table/flink-table-common/src/main/java/org/apache/flink/table/functions/LookupFunction.java#L52
> > > >
> > > >
> > > > Thanks,
> > > > Aitozi.
> > > >
> > > > Jing Ge <ji...@ververica.com.invalid> 于2023年6月9日周五 17:05写道：
> > > >
> > > > > Hi Aitozi,
> > > > >
> > > > > Thanks for the feedback. Looking forward to the performance tests.
> > > > >
> > > > > Afaik, lookup returns one row for each key [1] [2]. Conceptually,
> the
> > > > > lookup function is used to enrich column(s) from the dimension
> table.
> > > If,
> > > > > for the given key, there will be more than one row, there will be
> no
> > > way
> > > > to
> > > > > know which row will be used to enrich the key.
> > > > >
> > > > > [1]
> > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/flink/blob/191ec6ca3943d7119f14837efe112e074d815c47/flink-table/flink-table-common/src/main/java/org/apache/flink/table/functions/LookupFunction.java#L49
> > > > > [2]
> > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/flink/blob/191ec6ca3943d7119f14837efe112e074d815c47/flink-table/flink-table-common/src/main/java/org/apache/flink/table/functions/TableFunction.java#L196
> > > > >
> > > > > Best regards,
> > > > > Jing
> > > > >
> > > > > On Fri, Jun 9, 2023 at 5:18 AM Aitozi <gj...@gmail.com>
> wrote:
> > > > >
> > > > > > Hi Jing
> > > > > >     Thanks for your good questions. I have updated the example to
> > the
> > > > > FLIP.
> > > > > >
> > > > > > > Only one row for each lookup
> > > > > > lookup can also return multi rows, based on the query result. [1]
> > > > > >
> > > > > > [1]:
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/flink/blob/191ec6ca3943d7119f14837efe112e074d815c47/flink-table/flink-table-common/src/main/java/org/apache/flink/table/functions/LookupFunction.java#L56
> > > > > >
> > > > > > > If we use async calls with lateral join, my gut feeling is
> > > > > > that we might have many more async calls than lookup join. I am
> not
> > > > > really
> > > > > > sure if we will be facing potential issues in this case or not.
> > > > > >
> > > > > > IMO, the work pattern is similar to the lookup function, for each
> > row
> > > > > from
> > > > > > the left table,
> > > > > > it will evaluate the eval method once, so the async call numbers
> > will
> > > > not
> > > > > > change.
> > > > > > and the maximum calls in flight is limited by the Async operators
> > > > buffer
> > > > > > capacity
> > > > > > which will be controlled by the option.
> > > > > >
> > > > > > BTW, for the naming of these option, I updated the FLIP about
> this
> > > you
> > > > > can
> > > > > > refer to
> > > > > > the section of "ConfigOption" and "Rejected Alternatives"
> > > > > >
> > > > > > In the end, for the performance evaluation, I'd like to do some
> > tests
> > > > and
> > > > > > will update it to the FLIP doc
> > > > > >
> > > > > > Thanks,
> > > > > > Aitozi.
> > > > > >
> > > > > >
> > > > > > Jing Ge <ji...@ververica.com.invalid> 于2023年6月9日周五 07:23写道：
> > > > > >
> > > > > > > Hi Aitozi,
> > > > > > >
> > > > > > > Thanks for the clarification. The code example looks
> > interesting. I
> > > > > would
> > > > > > > suggest adding them into the FLIP. The description with code
> > > examples
> > > > > > will
> > > > > > > help readers understand the motivation and how to use it.
> Afaiac,
> > > it
> > > > > is a
> > > > > > > valid feature for Flink users.
> > > > > > >
> > > > > > > As we knew, lookup join is based on temporal join, i.e. FOR
> > > > SYSTEM_TIME
> > > > > > AS
> > > > > > > OF which is also used in your code example. Temporal join
> > performs
> > > > the
> > > > > > > lookup based on the processing time match. Only one row for
> each
> > > > > > > lookup(afaiu, I need to check the source code to double
> confirm)
> > > will
> > > > > > > return for further enrichment. One the other hand, lateral join
> > > will
> > > > > have
> > > > > > > sub-queries correlated with every individual value of the
> > reference
> > > > > table
> > > > > > > from the preceding part of the query and each sub query will
> > return
> > > > > > > multiple rows. If we use async calls with lateral join, my gut
> > > > feeling
> > > > > is
> > > > > > > that we might have many more async calls than lookup join. I am
> > not
> > > > > > really
> > > > > > > sure if we will be facing potential issues in this case or not.
> > > > > Possible
> > > > > > > issues I can think of now e.g. too many PRC calls, too many
> async
> > > > calls
> > > > > > > processing, the sub query will return a table which might be
> > (too)
> > > > big,
> > > > > > and
> > > > > > > might cause performance issues. I would suggest preparing some
> > use
> > > > > cases
> > > > > > > and running some performance tests to check it. These are my
> > > concerns
> > > > > > about
> > > > > > > using async calls with lateral join and I'd like to share with
> > you,
> > > > > happy
> > > > > > > to discuss with you and hear different opinions, hopefully the
> > > > > > > discussion could help me understand it more deeply. Please
> > correct
> > > me
> > > > > if
> > > > > > I
> > > > > > > am wrong.
> > > > > > >
> > > > > > > Best regards,
> > > > > > > Jing
> > > > > > >
> > > > > > >
> > > > > > > On Thu, Jun 8, 2023 at 7:22 AM Aitozi <gj...@gmail.com>
> > > wrote:
> > > > > > >
> > > > > > > > Hi Mason,
> > > > > > > >     Thanks for your input. I think if we support the user
> > defined
> > > > > async
> > > > > > > > table function,
> > > > > > > > user will be able to use it to hold a batch data then handle
> it
> > > at
> > > > > one
> > > > > > > time
> > > > > > > > in the customized function.
> > > > > > > >
> > > > > > > > AsyncSink is meant for the sink operator. I have not figure
> out
> > > how
> > > > > to
> > > > > > > > integrate in this case.
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > Atiozi.
> > > > > > > >
> > > > > > > >
> > > > > > > > Mason Chen <ma...@gmail.com> 于2023年6月8日周四 12:40写道：
> > > > > > > >
> > > > > > > > > Hi Aitozi,
> > > > > > > > >
> > > > > > > > > I think it makes sense to make it easier for SQL users to
> > make
> > > > > RPCs.
> > > > > > Do
> > > > > > > > you
> > > > > > > > > think your proposal can extend to the ability to batch data
> > for
> > > > the
> > > > > > > RPC?
> > > > > > > > > This is also another common strategy to increase
> throughput.
> > > > Also,
> > > > > > have
> > > > > > > > you
> > > > > > > > > considered solving this a bit differently by leveraging
> > Flink's
> > > > > > > > AsyncSink?
> > > > > > > > >
> > > > > > > > > Best,
> > > > > > > > > Mason
> > > > > > > > >
> > > > > > > > > On Mon, Jun 5, 2023 at 1:50 AM Aitozi <
> gjying1314@gmail.com>
> > > > > wrote:
> > > > > > > > >
> > > > > > > > > > One more thing for discussion:
> > > > > > > > > >
> > > > > > > > > > In our internal implementation, we reuse the option
> > > > > > > > > > `table.exec.async-lookup.buffer-capacity` and
> > > > > > > > > > `table.exec.async-lookup.timeout` to config
> > > > > > > > > > the async udtf. Do you think we should add two extra
> option
> > > to
> > > > > > > > > distinguish
> > > > > > > > > > from the lookup option such as
> > > > > > > > > >
> > > > > > > > > > `table.exec.async-udtf.buffer-capacity`
> > > > > > > > > > `table.exec.async-udtf.timeout`
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Best,
> > > > > > > > > > Aitozi.
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Aitozi <gj...@gmail.com> 于2023年6月5日周一 12:20写道：
> > > > > > > > > >
> > > > > > > > > > > Hi Jing,
> > > > > > > > > > >
> > > > > > > > > > >     > what is the difference between the RPC call or
> > query
> > > > you
> > > > > > > > > mentioned
> > > > > > > > > > > and the lookup in a very
> > > > > > > > > > > general way
> > > > > > > > > > >
> > > > > > > > > > > I think the RPC call or query service is quite similar
> to
> > > the
> > > > > > > lookup
> > > > > > > > > > join.
> > > > > > > > > > > But lookup join should work
> > > > > > > > > > > with `LookupTableSource`.
> > > > > > > > > > >
> > > > > > > > > > > Let's see how we can perform an async RPC call with
> > lookup
> > > > > join:
> > > > > > > > > > >
> > > > > > > > > > > (1) Implement an AsyncTableFunction with RPC call
> logic.
> > > > > > > > > > > (2) Implement a `LookupTableSource` connector run with
> > the
> > > > > async
> > > > > > > udtf
> > > > > > > > > > > defined in (1).
> > > > > > > > > > > (3) Then define a DDL of this look up table in SQL
> > > > > > > > > > >
> > > > > > > > > > > CREATE TEMPORARY TABLE Customers (
> > > > > > > > > > >   id INT,
> > > > > > > > > > >   name STRING,
> > > > > > > > > > >   country STRING,
> > > > > > > > > > >   zip STRING
> > > > > > > > > > > ) WITH (
> > > > > > > > > > >   'connector' = 'custom'
> > > > > > > > > > > );
> > > > > > > > > > >
> > > > > > > > > > > (4) Run with the query as below:
> > > > > > > > > > >
> > > > > > > > > > > SELECT o.order_id, o.total, c.country, c.zip
> > > > > > > > > > > FROM Orders AS o
> > > > > > > > > > >   JOIN Customers FOR SYSTEM_TIME AS OF o.proc_time AS c
> > > > > > > > > > >     ON o.customer_id = c.id;
> > > > > > > > > > >
> > > > > > > > > > > This example is from doc
> > > > > > > > > > > <
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/dev/table/sql/queries/joins/#lookup-join
> > > > > > > > > > >.You
> > > > > > > > > > > can image the look up process as an async RPC call
> > process.
> > > > > > > > > > >
> > > > > > > > > > > Let's see how we can perform an async RPC call with
> > lateral
> > > > > join:
> > > > > > > > > > >
> > > > > > > > > > > (1) Implement an AsyncTableFunction with RPC call
> logic.
> > > > > > > > > > > (2) Run query with
> > > > > > > > > > >
> > > > > > > > > > > Create function f1 as '...' ;
> > > > > > > > > > >
> > > > > > > > > > > SELECT o.order_id, o.total, c.country, c.zip FROM
> Orders
> > > > > lateral
> > > > > > > > table
> > > > > > > > > > > (f1(order_id)) as T(...);
> > > > > > > > > > >
> > > > > > > > > > > As you can see, the lateral join version is more simple
> > and
> > > > > > > intuitive
> > > > > > > > > to
> > > > > > > > > > > users. Users do not have to wrap a
> > > > > > > > > > > LookupTableSource for the purpose of using async udtf.
> > > > > > > > > > >
> > > > > > > > > > > In the end, We can also see the user defined async
> table
> > > > > function
> > > > > > > is
> > > > > > > > an
> > > > > > > > > > > enhancement of the current lateral table join
> > > > > > > > > > > which only supports sync lateral join now.
> > > > > > > > > > >
> > > > > > > > > > > Best,
> > > > > > > > > > > Aitozi.
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > Jing Ge <ji...@ververica.com.invalid> 于2023年6月2日周五
> > 19:37写道：
> > > > > > > > > > >
> > > > > > > > > > >> Hi Aitozi,
> > > > > > > > > > >>
> > > > > > > > > > >> Thanks for the update. Just out of curiosity, what is
> > the
> > > > > > > difference
> > > > > > > > > > >> between the RPC call or query you mentioned and the
> > lookup
> > > > in
> > > > > a
> > > > > > > very
> > > > > > > > > > >> general way? Since Lateral join is used in the FLIP.
> Is
> > > > there
> > > > > > any
> > > > > > > > > > special
> > > > > > > > > > >> thought for that? Sorry for asking so many questions.
> > The
> > > > FLIP
> > > > > > > > > contains
> > > > > > > > > > >> limited information to understand the motivation.
> > > > > > > > > > >>
> > > > > > > > > > >> Best regards,
> > > > > > > > > > >> Jing
> > > > > > > > > > >>
> > > > > > > > > > >> On Fri, Jun 2, 2023 at 3:48 AM Aitozi <
> > > gjying1314@gmail.com
> > > > >
> > > > > > > wrote:
> > > > > > > > > > >>
> > > > > > > > > > >> > Hi Jing,
> > > > > > > > > > >> >     I have updated the proposed changes to the FLIP.
> > > IMO,
> > > > > > lookup
> > > > > > > > has
> > > > > > > > > > its
> > > > > > > > > > >> > clear
> > > > > > > > > > >> > async call requirement is due to its IO heavy
> > operator.
> > > In
> > > > > our
> > > > > > > > > usage,
> > > > > > > > > > >> sql
> > > > > > > > > > >> > users have
> > > > > > > > > > >> > logic to do some RPC call or query the third-party
> > > service
> > > > > > which
> > > > > > > > is
> > > > > > > > > > >> also IO
> > > > > > > > > > >> > intensive.
> > > > > > > > > > >> > In these case, we'd like to leverage the async
> > function
> > > to
> > > > > > > improve
> > > > > > > > > the
> > > > > > > > > > >> > throughput.
> > > > > > > > > > >> >
> > > > > > > > > > >> > Thanks,
> > > > > > > > > > >> > Aitozi.
> > > > > > > > > > >> >
> > > > > > > > > > >> > Jing Ge <ji...@ververica.com.invalid> 于2023年6月1日周四
> > > > 22:55写道：
> > > > > > > > > > >> >
> > > > > > > > > > >> > > Hi Aitozi,
> > > > > > > > > > >> > >
> > > > > > > > > > >> > > Sorry for the late reply. Would you like to update
> > the
> > > > > > > proposed
> > > > > > > > > > >> changes
> > > > > > > > > > >> > > with more details into the FLIP too?
> > > > > > > > > > >> > > I got your point. It looks like a rational idea.
> > > > However,
> > > > > > > since
> > > > > > > > > > lookup
> > > > > > > > > > >> > has
> > > > > > > > > > >> > > its clear async call requirement, are there any
> real
> > > use
> > > > > > cases
> > > > > > > > > that
> > > > > > > > > > >> > > need this change? This will help us understand the
> > > > > > motivation.
> > > > > > > > > After
> > > > > > > > > > >> all,
> > > > > > > > > > >> > > lateral join and temporal lookup join[1] are quite
> > > > > > different.
> > > > > > > > > > >> > >
> > > > > > > > > > >> > > Best regards,
> > > > > > > > > > >> > > Jing
> > > > > > > > > > >> > >
> > > > > > > > > > >> > >
> > > > > > > > > > >> > > [1]
> > > > > > > > > > >> > >
> > > > > > > > > > >> > >
> > > > > > > > > > >> >
> > > > > > > > > > >>
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/flink/blob/d90a72da2fd601ca4e2a46700e91ec5b348de2ad/flink-table/flink-table-common/src/main/java/org/apache/flink/table/functions/AsyncTableFunction.java#L54
> > > > > > > > > > >> > >
> > > > > > > > > > >> > > On Wed, May 31, 2023 at 8:53 AM Aitozi <
> > > > > > gjying1314@gmail.com>
> > > > > > > > > > wrote:
> > > > > > > > > > >> > >
> > > > > > > > > > >> > > > Hi Jing,
> > > > > > > > > > >> > > >     What do you think about it? Can we move
> > forward
> > > > this
> > > > > > > > > feature?
> > > > > > > > > > >> > > >
> > > > > > > > > > >> > > > Thanks,
> > > > > > > > > > >> > > > Aitozi.
> > > > > > > > > > >> > > >
> > > > > > > > > > >> > > > Aitozi <gj...@gmail.com> 于2023年5月29日周一
> > > 09:56写道：
> > > > > > > > > > >> > > >
> > > > > > > > > > >> > > > > Hi Jing,
> > > > > > > > > > >> > > > >     > "Do you mean to support the
> > > AyncTableFunction
> > > > > > beyond
> > > > > > > > the
> > > > > > > > > > >> > > > > LookupTableSource?"
> > > > > > > > > > >> > > > > Yes, I mean to support the AyncTableFunction
> > > beyond
> > > > > the
> > > > > > > > > > >> > > > LookupTableSource.
> > > > > > > > > > >> > > > >
> > > > > > > > > > >> > > > > The "AsyncTableFunction" is the function with
> > > > ability
> > > > > to
> > > > > > > be
> > > > > > > > > > >> executed
> > > > > > > > > > >> > > > async
> > > > > > > > > > >> > > > > (with AsyncWaitOperator).
> > > > > > > > > > >> > > > > The async lookup join is a one of usage of
> this.
> > > So,
> > > > > we
> > > > > > > > don't
> > > > > > > > > > >> have to
> > > > > > > > > > >> > > > bind
> > > > > > > > > > >> > > > > the AyncTableFunction with LookupTableSource.
> > > > > > > > > > >> > > > > If User-defined AsyncTableFunction is
> supported,
> > > > user
> > > > > > can
> > > > > > > > > > directly
> > > > > > > > > > >> > use
> > > > > > > > > > >> > > > > lateral table syntax to perform async
> operation.
> > > > > > > > > > >> > > > >
> > > > > > > > > > >> > > > > > "It would be better if you could elaborate
> the
> > > > > > proposed
> > > > > > > > > > changes
> > > > > > > > > > >> wrt
> > > > > > > > > > >> > > the
> > > > > > > > > > >> > > > > CorrelatedCodeGenerator with more details"
> > > > > > > > > > >> > > > >
> > > > > > > > > > >> > > > > In the proposal, we use lateral table syntax
> to
> > > > > support
> > > > > > > the
> > > > > > > > > > async
> > > > > > > > > > >> > table
> > > > > > > > > > >> > > > > function. So the planner will also treat this
> > > > > statement
> > > > > > > to a
> > > > > > > > > > >> > > > > CommonExecCorrelate node. So the runtime code
> > > should
> > > > > be
> > > > > > > > > > generated
> > > > > > > > > > >> in
> > > > > > > > > > >> > > > > CorrelatedCodeGenerator.
> > > > > > > > > > >> > > > > In CorrelatedCodeGenerator, we will know the
> > > > > > > TableFunction's
> > > > > > > > > > Kind
> > > > > > > > > > >> of
> > > > > > > > > > >> > > > > `FunctionKind.Table` or
> > `FunctionKind.ASYNC_TABLE`
> > > > > > > > > > >> > > > > For  `FunctionKind.ASYNC_TABLE` we can
> generate
> > a
> > > > > > > > > > >> AsyncWaitOperator
> > > > > > > > > > >> > to
> > > > > > > > > > >> > > > > execute the async table function.
> > > > > > > > > > >> > > > >
> > > > > > > > > > >> > > > >
> > > > > > > > > > >> > > > > Thanks,
> > > > > > > > > > >> > > > > Aitozi.
> > > > > > > > > > >> > > > >
> > > > > > > > > > >> > > > >
> > > > > > > > > > >> > > > > Jing Ge <ji...@ververica.com.invalid>
> > > 于2023年5月29日周一
> > > > > > > 03:22写道：
> > > > > > > > > > >> > > > >
> > > > > > > > > > >> > > > >> Hi Aitozi,
> > > > > > > > > > >> > > > >>
> > > > > > > > > > >> > > > >> Thanks for the clarification. The naming
> > "Lookup"
> > > > > might
> > > > > > > > > suggest
> > > > > > > > > > >> > using
> > > > > > > > > > >> > > it
> > > > > > > > > > >> > > > >> for table look up. But conceptually what the
> > > eval()
> > > > > > > method
> > > > > > > > > will
> > > > > > > > > > >> do
> > > > > > > > > > >> > is
> > > > > > > > > > >> > > to
> > > > > > > > > > >> > > > >> get a collection of results(Row, RowData)
> from
> > > the
> > > > > > given
> > > > > > > > > keys.
> > > > > > > > > > >> How
> > > > > > > > > > >> > it
> > > > > > > > > > >> > > > will
> > > > > > > > > > >> > > > >> be done depends on the implementation, i.e.
> you
> > > can
> > > > > > > > implement
> > > > > > > > > > >> your
> > > > > > > > > > >> > own
> > > > > > > > > > >> > > > >> Source[1][2]. The example in the FLIP should
> be
> > > > able
> > > > > to
> > > > > > > be
> > > > > > > > > > >> handled
> > > > > > > > > > >> > in
> > > > > > > > > > >> > > > this
> > > > > > > > > > >> > > > >> way.
> > > > > > > > > > >> > > > >>
> > > > > > > > > > >> > > > >> Do you mean to support the AyncTableFunction
> > > beyond
> > > > > the
> > > > > > > > > > >> > > > LookupTableSource?
> > > > > > > > > > >> > > > >> It would be better if you could elaborate the
> > > > > proposed
> > > > > > > > > changes
> > > > > > > > > > >> wrt
> > > > > > > > > > >> > the
> > > > > > > > > > >> > > > >> CorrelatedCodeGenerator with more details.
> > > Thanks!
> > > > > > > > > > >> > > > >>
> > > > > > > > > > >> > > > >> Best regards,
> > > > > > > > > > >> > > > >> Jing
> > > > > > > > > > >> > > > >>
> > > > > > > > > > >> > > > >> [1]
> > > > > > > > > > >> > > > >>
> > > > > > > > > > >> > > > >>
> > > > > > > > > > >> > > >
> > > > > > > > > > >> > >
> > > > > > > > > > >> >
> > > > > > > > > > >>
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/flink/blob/678370b18e1b6c4a23e5ce08f8efd05675a0cc17/flink-table/flink-table-common/src/main/java/org/apache/flink/table/connector/source/LookupTableSource.java#L64
> > > > > > > > > > >> > > > >> [2]
> > > > > > > > > > >> > > > >>
> > > > > > > > > > >> > > > >>
> > > > > > > > > > >> > > >
> > > > > > > > > > >> > >
> > > > > > > > > > >> >
> > > > > > > > > > >>
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/flink/blob/678370b18e1b6c4a23e5ce08f8efd05675a0cc17/flink-table/flink-table-common/src/main/java/org/apache/flink/table/connector/source/AsyncTableFunctionProvider.java#L49
> > > > > > > > > > >> > > > >>
> > > > > > > > > > >> > > > >> On Sat, May 27, 2023 at 9:48 AM Aitozi <
> > > > > > > > gjying1314@gmail.com
> > > > > > > > > >
> > > > > > > > > > >> > wrote:
> > > > > > > > > > >> > > > >>
> > > > > > > > > > >> > > > >> > Hi Jing,
> > > > > > > > > > >> > > > >> >     Thanks for your response. As stated in
> > the
> > > > > FLIP,
> > > > > > > the
> > > > > > > > > > >> purpose
> > > > > > > > > > >> > of
> > > > > > > > > > >> > > > this
> > > > > > > > > > >> > > > >> > FLIP is meant to support
> > > > > > > > > > >> > > > >> > user-defined async table function. As
> > described
> > > > in
> > > > > > > flink
> > > > > > > > > > >> document
> > > > > > > > > > >> > > [1]
> > > > > > > > > > >> > > > >> >
> > > > > > > > > > >> > > > >> > Async table functions are special functions
> > for
> > > > > table
> > > > > > > > > sources
> > > > > > > > > > >> that
> > > > > > > > > > >> > > > >> perform
> > > > > > > > > > >> > > > >> > > a lookup.
> > > > > > > > > > >> > > > >> > >
> > > > > > > > > > >> > > > >> >
> > > > > > > > > > >> > > > >> > So end user can not directly define and use
> > > async
> > > > > > table
> > > > > > > > > > >> function
> > > > > > > > > > >> > > now.
> > > > > > > > > > >> > > > An
> > > > > > > > > > >> > > > >> > user case is reported in [2]
> > > > > > > > > > >> > > > >> >
> > > > > > > > > > >> > > > >> > So, in conclusion, no new interface is
> > > > introduced,
> > > > > > but
> > > > > > > we
> > > > > > > > > > >> extend
> > > > > > > > > > >> > the
> > > > > > > > > > >> > > > >> > ability to support user-defined async table
> > > > > function.
> > > > > > > > > > >> > > > >> >
> > > > > > > > > > >> > > > >> > [1]:
> > > > > > > > > > >> > > > >> >
> > > > > > > > > > >> > > > >> >
> > > > > > > > > > >> > > > >>
> > > > > > > > > > >> > > >
> > > > > > > > > > >> > >
> > > > > > > > > > >> >
> > > > > > > > > > >>
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/dev/table/functions/udfs/
> > > > > > > > > > >> > > > >> > [2]:
> > > > > > > > > > >> > >
> > > > > > > >
> > https://lists.apache.org/thread/qljwd40v5ntz6733cwcdr8s4z97b343b
> > > > > > > > > > >> > > > >> >
> > > > > > > > > > >> > > > >> > Thanks.
> > > > > > > > > > >> > > > >> > Aitozi.
> > > > > > > > > > >> > > > >> >
> > > > > > > > > > >> > > > >> >
> > > > > > > > > > >> > > > >> > Jing Ge <ji...@ververica.com.invalid>
> > > > 于2023年5月27日周六
> > > > > > > > > 06:40写道：
> > > > > > > > > > >> > > > >> >
> > > > > > > > > > >> > > > >> > > Hi Aitozi,
> > > > > > > > > > >> > > > >> > >
> > > > > > > > > > >> > > > >> > > Thanks for your proposal. I am not quite
> > sure
> > > > if
> > > > > I
> > > > > > > > > > understood
> > > > > > > > > > >> > your
> > > > > > > > > > >> > > > >> > thoughts
> > > > > > > > > > >> > > > >> > > correctly. You described a special case
> > > > > > > implementation
> > > > > > > > of
> > > > > > > > > > the
> > > > > > > > > > >> > > > >> > > AsyncTableFunction with on public API
> > > changes.
> > > > > > Would
> > > > > > > > you
> > > > > > > > > > >> please
> > > > > > > > > > >> > > > >> elaborate
> > > > > > > > > > >> > > > >> > > your purpose of writing a FLIP according
> to
> > > the
> > > > > > FLIP
> > > > > > > > > > >> > > > documentation[1]?
> > > > > > > > > > >> > > > >> > > Thanks!
> > > > > > > > > > >> > > > >> > >
> > > > > > > > > > >> > > > >> > > [1]
> > > > > > > > > > >> > > > >> > >
> > > > > > > > > > >> > > > >> > >
> > > > > > > > > > >> > > > >> >
> > > > > > > > > > >> > > > >>
> > > > > > > > > > >> > > >
> > > > > > > > > > >> > >
> > > > > > > > > > >> >
> > > > > > > > > > >>
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/Flink+Improvement+Proposals
> > > > > > > > > > >> > > > >> > >
> > > > > > > > > > >> > > > >> > > Best regards,
> > > > > > > > > > >> > > > >> > > Jing
> > > > > > > > > > >> > > > >> > >
> > > > > > > > > > >> > > > >> > > On Wed, May 24, 2023 at 1:07 PM Aitozi <
> > > > > > > > > > gjying1314@gmail.com
> > > > > > > > > > >> >
> > > > > > > > > > >> > > > wrote:
> > > > > > > > > > >> > > > >> > >
> > > > > > > > > > >> > > > >> > > > May I ask for some feedback  :D
> > > > > > > > > > >> > > > >> > > >
> > > > > > > > > > >> > > > >> > > > Thanks,
> > > > > > > > > > >> > > > >> > > > Aitozi
> > > > > > > > > > >> > > > >> > > >
> > > > > > > > > > >> > > > >> > > > Aitozi <gj...@gmail.com>
> > > 于2023年5月23日周二
> > > > > > > 19:14写道：
> > > > > > > > > > >> > > > >> > > > >
> > > > > > > > > > >> > > > >> > > > > Just catch an user case report from
> > > Giannis
> > > > > > > Polyzos
> > > > > > > > > for
> > > > > > > > > > >> this
> > > > > > > > > > >> > > > >> usage:
> > > > > > > > > > >> > > > >> > > > >
> > > > > > > > > > >> > > > >> > > > >
> > > > > > > > > > >> > > >
> > > > > > > > >
> > > https://lists.apache.org/thread/qljwd40v5ntz6733cwcdr8s4z97b343b
> > > > > > > > > > >> > > > >> > > > >
> > > > > > > > > > >> > > > >> > > > > Aitozi <gj...@gmail.com>
> > > > 于2023年5月23日周二
> > > > > > > > 17:45写道：
> > > > > > > > > > >> > > > >> > > > > >
> > > > > > > > > > >> > > > >> > > > > > Hi guys,
> > > > > > > > > > >> > > > >> > > > > >     I want to bring up a discussion
> > > about
> > > > > > > adding
> > > > > > > > > > >> support
> > > > > > > > > > >> > of
> > > > > > > > > > >> > > > User
> > > > > > > > > > >> > > > >> > > > > > Defined AsyncTableFunction in
> Flink.
> > > > > > > > > > >> > > > >> > > > > > Currently, async table function are
> > > > special
> > > > > > > > > functions
> > > > > > > > > > >> for
> > > > > > > > > > >> > > > table
> > > > > > > > > > >> > > > >> > > source
> > > > > > > > > > >> > > > >> > > > > > to perform
> > > > > > > > > > >> > > > >> > > > > > async lookup. However, it's worth
> to
> > > > > support
> > > > > > > the
> > > > > > > > > user
> > > > > > > > > > >> > > defined
> > > > > > > > > > >> > > > >> async
> > > > > > > > > > >> > > > >> > > > > > table function.
> > > > > > > > > > >> > > > >> > > > > > Because, in this way, the end SQL
> > user
> > > > can
> > > > > > > > leverage
> > > > > > > > > > it
> > > > > > > > > > >> to
> > > > > > > > > > >> > > > >> perform
> > > > > > > > > > >> > > > >> > the
> > > > > > > > > > >> > > > >> > > > > > async operation
> > > > > > > > > > >> > > > >> > > > > > which is useful to maximum the
> system
> > > > > > > throughput
> > > > > > > > > > >> > especially
> > > > > > > > > > >> > > > for
> > > > > > > > > > >> > > > >> IO
> > > > > > > > > > >> > > > >> > > > > > bottleneck case.
> > > > > > > > > > >> > > > >> > > > > >
> > > > > > > > > > >> > > > >> > > > > > You can find some more detail in
> [1].
> > > > > > > > > > >> > > > >> > > > > >
> > > > > > > > > > >> > > > >> > > > > > Looking forward to feedback
> > > > > > > > > > >> > > > >> > > > > >
> > > > > > > > > > >> > > > >> > > > > >
> > > > > > > > > > >> > > > >> > > > > > [1]:
> > > > > > > > > > >> > > > >> > > >
> > > > > > > > > > >> > > > >> > >
> > > > > > > > > > >> > > > >> >
> > > > > > > > > > >> > > > >>
> > > > > > > > > > >> > > >
> > > > > > > > > > >> > >
> > > > > > > > > > >> >
> > > > > > > > > > >>
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/%5BFLIP-313%5D+Add+support+of+User+Defined+AsyncTableFunction
> > > > > > > > > > >> > > > >> > > > > >
> > > > > > > > > > >> > > > >> > > > > > Thanks,
> > > > > > > > > > >> > > > >> > > > > > Aitozi.
> > > > > > > > > > >> > > > >> > > >
> > > > > > > > > > >> > > > >> > >
> > > > > > > > > > >> > > > >> >
> > > > > > > > > > >> > > > >>
> > > > > > > > > > >> > > > >
> > > > > > > > > > >> > > >
> > > > > > > > > > >> > >
> > > > > > > > > > >> >
> > > > > > > > > > >>
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] FLIP-313 Add support of User Defined AsyncTableFunction

Posted by Jing Ge <ji...@ververica.com.INVALID>.

Hi Aitozi,

Which key will be used for lookup is not an issue, only one row will be
required for each key in order to enrich it. True, it depends on the
implementation whether multiple rows or single row for each key will be
returned. However, for the lookup & enrichment scenario, one row/key is
recommended, otherwise, like I mentioned previously, enrichment won't work.

I am a little bit concerned about returning a big table for each key, since
it will take the async call longer to return and need more memory. The
performance tests should cover this scenario. This is not a blocking issue
for this FLIP.

Best regards,
Jing

On Sat, Jun 10, 2023 at 4:11 AM Aitozi <gj...@gmail.com> wrote:

> Hi Jing,
>     I means the join key is not necessary to be the primary key or unique
> index of the database.
> In this situation, we may queried out multi rows for one join key. I think
> that's why the
> LookupFunction#lookup will return a collection of RowData.
>
> BTW, I think the behavior of lookup join will not affect the semantic of
> the async udtf.
> We use the Async TableFunction here and the table function can collect
> multiple rows.
>
> Thanks,
> Atiozi.
>
>
>
> Jing Ge <ji...@ververica.com.invalid> 于2023年6月10日周六 00:15写道：
>
> > Hi Aitozi,
> >
> > The keyRow used in this case contains all keys[1].
> >
> > Best regards,
> > Jing
> >
> > [1]
> >
> >
> https://github.com/apache/flink/blob/191ec6ca3943d7119f14837efe112e074d815c47/flink-table/flink-table-common/src/main/java/org/apache/flink/table/functions/LookupFunction.java#L49
> >
> >
> > On Fri, Jun 9, 2023 at 3:42 PM Aitozi <gj...@gmail.com> wrote:
> >
> > > Hi Jing,
> > >
> > >      The performance test is added to the FLIP.
> > >
> > >      As I know, The lookup join can return multi rows, it depends on
> > > whether  the join key
> > > is the primary key of the external database or not. The `lookup` [1]
> will
> > > return a collection of
> > > joined result, and each of them will be collected
> > >
> > >
> > > [1]:
> > >
> > >
> >
> https://github.com/apache/flink/blob/191ec6ca3943d7119f14837efe112e074d815c47/flink-table/flink-table-common/src/main/java/org/apache/flink/table/functions/LookupFunction.java#L52
> > >
> > >
> > > Thanks,
> > > Aitozi.
> > >
> > > Jing Ge <ji...@ververica.com.invalid> 于2023年6月9日周五 17:05写道：
> > >
> > > > Hi Aitozi,
> > > >
> > > > Thanks for the feedback. Looking forward to the performance tests.
> > > >
> > > > Afaik, lookup returns one row for each key [1] [2]. Conceptually, the
> > > > lookup function is used to enrich column(s) from the dimension table.
> > If,
> > > > for the given key, there will be more than one row, there will be no
> > way
> > > to
> > > > know which row will be used to enrich the key.
> > > >
> > > > [1]
> > > >
> > > >
> > >
> >
> https://github.com/apache/flink/blob/191ec6ca3943d7119f14837efe112e074d815c47/flink-table/flink-table-common/src/main/java/org/apache/flink/table/functions/LookupFunction.java#L49
> > > > [2]
> > > >
> > > >
> > >
> >
> https://github.com/apache/flink/blob/191ec6ca3943d7119f14837efe112e074d815c47/flink-table/flink-table-common/src/main/java/org/apache/flink/table/functions/TableFunction.java#L196
> > > >
> > > > Best regards,
> > > > Jing
> > > >
> > > > On Fri, Jun 9, 2023 at 5:18 AM Aitozi <gj...@gmail.com> wrote:
> > > >
> > > > > Hi Jing
> > > > >     Thanks for your good questions. I have updated the example to
> the
> > > > FLIP.
> > > > >
> > > > > > Only one row for each lookup
> > > > > lookup can also return multi rows, based on the query result. [1]
> > > > >
> > > > > [1]:
> > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/flink/blob/191ec6ca3943d7119f14837efe112e074d815c47/flink-table/flink-table-common/src/main/java/org/apache/flink/table/functions/LookupFunction.java#L56
> > > > >
> > > > > > If we use async calls with lateral join, my gut feeling is
> > > > > that we might have many more async calls than lookup join. I am not
> > > > really
> > > > > sure if we will be facing potential issues in this case or not.
> > > > >
> > > > > IMO, the work pattern is similar to the lookup function, for each
> row
> > > > from
> > > > > the left table,
> > > > > it will evaluate the eval method once, so the async call numbers
> will
> > > not
> > > > > change.
> > > > > and the maximum calls in flight is limited by the Async operators
> > > buffer
> > > > > capacity
> > > > > which will be controlled by the option.
> > > > >
> > > > > BTW, for the naming of these option, I updated the FLIP about this
> > you
> > > > can
> > > > > refer to
> > > > > the section of "ConfigOption" and "Rejected Alternatives"
> > > > >
> > > > > In the end, for the performance evaluation, I'd like to do some
> tests
> > > and
> > > > > will update it to the FLIP doc
> > > > >
> > > > > Thanks,
> > > > > Aitozi.
> > > > >
> > > > >
> > > > > Jing Ge <ji...@ververica.com.invalid> 于2023年6月9日周五 07:23写道：
> > > > >
> > > > > > Hi Aitozi,
> > > > > >
> > > > > > Thanks for the clarification. The code example looks
> interesting. I
> > > > would
> > > > > > suggest adding them into the FLIP. The description with code
> > examples
> > > > > will
> > > > > > help readers understand the motivation and how to use it. Afaiac,
> > it
> > > > is a
> > > > > > valid feature for Flink users.
> > > > > >
> > > > > > As we knew, lookup join is based on temporal join, i.e. FOR
> > > SYSTEM_TIME
> > > > > AS
> > > > > > OF which is also used in your code example. Temporal join
> performs
> > > the
> > > > > > lookup based on the processing time match. Only one row for each
> > > > > > lookup(afaiu, I need to check the source code to double confirm)
> > will
> > > > > > return for further enrichment. One the other hand, lateral join
> > will
> > > > have
> > > > > > sub-queries correlated with every individual value of the
> reference
> > > > table
> > > > > > from the preceding part of the query and each sub query will
> return
> > > > > > multiple rows. If we use async calls with lateral join, my gut
> > > feeling
> > > > is
> > > > > > that we might have many more async calls than lookup join. I am
> not
> > > > > really
> > > > > > sure if we will be facing potential issues in this case or not.
> > > > Possible
> > > > > > issues I can think of now e.g. too many PRC calls, too many async
> > > calls
> > > > > > processing, the sub query will return a table which might be
> (too)
> > > big,
> > > > > and
> > > > > > might cause performance issues. I would suggest preparing some
> use
> > > > cases
> > > > > > and running some performance tests to check it. These are my
> > concerns
> > > > > about
> > > > > > using async calls with lateral join and I'd like to share with
> you,
> > > > happy
> > > > > > to discuss with you and hear different opinions, hopefully the
> > > > > > discussion could help me understand it more deeply. Please
> correct
> > me
> > > > if
> > > > > I
> > > > > > am wrong.
> > > > > >
> > > > > > Best regards,
> > > > > > Jing
> > > > > >
> > > > > >
> > > > > > On Thu, Jun 8, 2023 at 7:22 AM Aitozi <gj...@gmail.com>
> > wrote:
> > > > > >
> > > > > > > Hi Mason,
> > > > > > >     Thanks for your input. I think if we support the user
> defined
> > > > async
> > > > > > > table function,
> > > > > > > user will be able to use it to hold a batch data then handle it
> > at
> > > > one
> > > > > > time
> > > > > > > in the customized function.
> > > > > > >
> > > > > > > AsyncSink is meant for the sink operator. I have not figure out
> > how
> > > > to
> > > > > > > integrate in this case.
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Atiozi.
> > > > > > >
> > > > > > >
> > > > > > > Mason Chen <ma...@gmail.com> 于2023年6月8日周四 12:40写道：
> > > > > > >
> > > > > > > > Hi Aitozi,
> > > > > > > >
> > > > > > > > I think it makes sense to make it easier for SQL users to
> make
> > > > RPCs.
> > > > > Do
> > > > > > > you
> > > > > > > > think your proposal can extend to the ability to batch data
> for
> > > the
> > > > > > RPC?
> > > > > > > > This is also another common strategy to increase throughput.
> > > Also,
> > > > > have
> > > > > > > you
> > > > > > > > considered solving this a bit differently by leveraging
> Flink's
> > > > > > > AsyncSink?
> > > > > > > >
> > > > > > > > Best,
> > > > > > > > Mason
> > > > > > > >
> > > > > > > > On Mon, Jun 5, 2023 at 1:50 AM Aitozi <gj...@gmail.com>
> > > > wrote:
> > > > > > > >
> > > > > > > > > One more thing for discussion:
> > > > > > > > >
> > > > > > > > > In our internal implementation, we reuse the option
> > > > > > > > > `table.exec.async-lookup.buffer-capacity` and
> > > > > > > > > `table.exec.async-lookup.timeout` to config
> > > > > > > > > the async udtf. Do you think we should add two extra option
> > to
> > > > > > > > distinguish
> > > > > > > > > from the lookup option such as
> > > > > > > > >
> > > > > > > > > `table.exec.async-udtf.buffer-capacity`
> > > > > > > > > `table.exec.async-udtf.timeout`
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Best,
> > > > > > > > > Aitozi.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Aitozi <gj...@gmail.com> 于2023年6月5日周一 12:20写道：
> > > > > > > > >
> > > > > > > > > > Hi Jing,
> > > > > > > > > >
> > > > > > > > > >     > what is the difference between the RPC call or
> query
> > > you
> > > > > > > > mentioned
> > > > > > > > > > and the lookup in a very
> > > > > > > > > > general way
> > > > > > > > > >
> > > > > > > > > > I think the RPC call or query service is quite similar to
> > the
> > > > > > lookup
> > > > > > > > > join.
> > > > > > > > > > But lookup join should work
> > > > > > > > > > with `LookupTableSource`.
> > > > > > > > > >
> > > > > > > > > > Let's see how we can perform an async RPC call with
> lookup
> > > > join:
> > > > > > > > > >
> > > > > > > > > > (1) Implement an AsyncTableFunction with RPC call logic.
> > > > > > > > > > (2) Implement a `LookupTableSource` connector run with
> the
> > > > async
> > > > > > udtf
> > > > > > > > > > defined in (1).
> > > > > > > > > > (3) Then define a DDL of this look up table in SQL
> > > > > > > > > >
> > > > > > > > > > CREATE TEMPORARY TABLE Customers (
> > > > > > > > > >   id INT,
> > > > > > > > > >   name STRING,
> > > > > > > > > >   country STRING,
> > > > > > > > > >   zip STRING
> > > > > > > > > > ) WITH (
> > > > > > > > > >   'connector' = 'custom'
> > > > > > > > > > );
> > > > > > > > > >
> > > > > > > > > > (4) Run with the query as below:
> > > > > > > > > >
> > > > > > > > > > SELECT o.order_id, o.total, c.country, c.zip
> > > > > > > > > > FROM Orders AS o
> > > > > > > > > >   JOIN Customers FOR SYSTEM_TIME AS OF o.proc_time AS c
> > > > > > > > > >     ON o.customer_id = c.id;
> > > > > > > > > >
> > > > > > > > > > This example is from doc
> > > > > > > > > > <
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/dev/table/sql/queries/joins/#lookup-join
> > > > > > > > > >.You
> > > > > > > > > > can image the look up process as an async RPC call
> process.
> > > > > > > > > >
> > > > > > > > > > Let's see how we can perform an async RPC call with
> lateral
> > > > join:
> > > > > > > > > >
> > > > > > > > > > (1) Implement an AsyncTableFunction with RPC call logic.
> > > > > > > > > > (2) Run query with
> > > > > > > > > >
> > > > > > > > > > Create function f1 as '...' ;
> > > > > > > > > >
> > > > > > > > > > SELECT o.order_id, o.total, c.country, c.zip FROM Orders
> > > > lateral
> > > > > > > table
> > > > > > > > > > (f1(order_id)) as T(...);
> > > > > > > > > >
> > > > > > > > > > As you can see, the lateral join version is more simple
> and
> > > > > > intuitive
> > > > > > > > to
> > > > > > > > > > users. Users do not have to wrap a
> > > > > > > > > > LookupTableSource for the purpose of using async udtf.
> > > > > > > > > >
> > > > > > > > > > In the end, We can also see the user defined async table
> > > > function
> > > > > > is
> > > > > > > an
> > > > > > > > > > enhancement of the current lateral table join
> > > > > > > > > > which only supports sync lateral join now.
> > > > > > > > > >
> > > > > > > > > > Best,
> > > > > > > > > > Aitozi.
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Jing Ge <ji...@ververica.com.invalid> 于2023年6月2日周五
> 19:37写道：
> > > > > > > > > >
> > > > > > > > > >> Hi Aitozi,
> > > > > > > > > >>
> > > > > > > > > >> Thanks for the update. Just out of curiosity, what is
> the
> > > > > > difference
> > > > > > > > > >> between the RPC call or query you mentioned and the
> lookup
> > > in
> > > > a
> > > > > > very
> > > > > > > > > >> general way? Since Lateral join is used in the FLIP. Is
> > > there
> > > > > any
> > > > > > > > > special
> > > > > > > > > >> thought for that? Sorry for asking so many questions.
> The
> > > FLIP
> > > > > > > > contains
> > > > > > > > > >> limited information to understand the motivation.
> > > > > > > > > >>
> > > > > > > > > >> Best regards,
> > > > > > > > > >> Jing
> > > > > > > > > >>
> > > > > > > > > >> On Fri, Jun 2, 2023 at 3:48 AM Aitozi <
> > gjying1314@gmail.com
> > > >
> > > > > > wrote:
> > > > > > > > > >>
> > > > > > > > > >> > Hi Jing,
> > > > > > > > > >> >     I have updated the proposed changes to the FLIP.
> > IMO,
> > > > > lookup
> > > > > > > has
> > > > > > > > > its
> > > > > > > > > >> > clear
> > > > > > > > > >> > async call requirement is due to its IO heavy
> operator.
> > In
> > > > our
> > > > > > > > usage,
> > > > > > > > > >> sql
> > > > > > > > > >> > users have
> > > > > > > > > >> > logic to do some RPC call or query the third-party
> > service
> > > > > which
> > > > > > > is
> > > > > > > > > >> also IO
> > > > > > > > > >> > intensive.
> > > > > > > > > >> > In these case, we'd like to leverage the async
> function
> > to
> > > > > > improve
> > > > > > > > the
> > > > > > > > > >> > throughput.
> > > > > > > > > >> >
> > > > > > > > > >> > Thanks,
> > > > > > > > > >> > Aitozi.
> > > > > > > > > >> >
> > > > > > > > > >> > Jing Ge <ji...@ververica.com.invalid> 于2023年6月1日周四
> > > 22:55写道：
> > > > > > > > > >> >
> > > > > > > > > >> > > Hi Aitozi,
> > > > > > > > > >> > >
> > > > > > > > > >> > > Sorry for the late reply. Would you like to update
> the
> > > > > > proposed
> > > > > > > > > >> changes
> > > > > > > > > >> > > with more details into the FLIP too?
> > > > > > > > > >> > > I got your point. It looks like a rational idea.
> > > However,
> > > > > > since
> > > > > > > > > lookup
> > > > > > > > > >> > has
> > > > > > > > > >> > > its clear async call requirement, are there any real
> > use
> > > > > cases
> > > > > > > > that
> > > > > > > > > >> > > need this change? This will help us understand the
> > > > > motivation.
> > > > > > > > After
> > > > > > > > > >> all,
> > > > > > > > > >> > > lateral join and temporal lookup join[1] are quite
> > > > > different.
> > > > > > > > > >> > >
> > > > > > > > > >> > > Best regards,
> > > > > > > > > >> > > Jing
> > > > > > > > > >> > >
> > > > > > > > > >> > >
> > > > > > > > > >> > > [1]
> > > > > > > > > >> > >
> > > > > > > > > >> > >
> > > > > > > > > >> >
> > > > > > > > > >>
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/flink/blob/d90a72da2fd601ca4e2a46700e91ec5b348de2ad/flink-table/flink-table-common/src/main/java/org/apache/flink/table/functions/AsyncTableFunction.java#L54
> > > > > > > > > >> > >
> > > > > > > > > >> > > On Wed, May 31, 2023 at 8:53 AM Aitozi <
> > > > > gjying1314@gmail.com>
> > > > > > > > > wrote:
> > > > > > > > > >> > >
> > > > > > > > > >> > > > Hi Jing,
> > > > > > > > > >> > > >     What do you think about it? Can we move
> forward
> > > this
> > > > > > > > feature?
> > > > > > > > > >> > > >
> > > > > > > > > >> > > > Thanks,
> > > > > > > > > >> > > > Aitozi.
> > > > > > > > > >> > > >
> > > > > > > > > >> > > > Aitozi <gj...@gmail.com> 于2023年5月29日周一
> > 09:56写道：
> > > > > > > > > >> > > >
> > > > > > > > > >> > > > > Hi Jing,
> > > > > > > > > >> > > > >     > "Do you mean to support the
> > AyncTableFunction
> > > > > beyond
> > > > > > > the
> > > > > > > > > >> > > > > LookupTableSource?"
> > > > > > > > > >> > > > > Yes, I mean to support the AyncTableFunction
> > beyond
> > > > the
> > > > > > > > > >> > > > LookupTableSource.
> > > > > > > > > >> > > > >
> > > > > > > > > >> > > > > The "AsyncTableFunction" is the function with
> > > ability
> > > > to
> > > > > > be
> > > > > > > > > >> executed
> > > > > > > > > >> > > > async
> > > > > > > > > >> > > > > (with AsyncWaitOperator).
> > > > > > > > > >> > > > > The async lookup join is a one of usage of this.
> > So,
> > > > we
> > > > > > > don't
> > > > > > > > > >> have to
> > > > > > > > > >> > > > bind
> > > > > > > > > >> > > > > the AyncTableFunction with LookupTableSource.
> > > > > > > > > >> > > > > If User-defined AsyncTableFunction is supported,
> > > user
> > > > > can
> > > > > > > > > directly
> > > > > > > > > >> > use
> > > > > > > > > >> > > > > lateral table syntax to perform async operation.
> > > > > > > > > >> > > > >
> > > > > > > > > >> > > > > > "It would be better if you could elaborate the
> > > > > proposed
> > > > > > > > > changes
> > > > > > > > > >> wrt
> > > > > > > > > >> > > the
> > > > > > > > > >> > > > > CorrelatedCodeGenerator with more details"
> > > > > > > > > >> > > > >
> > > > > > > > > >> > > > > In the proposal, we use lateral table syntax to
> > > > support
> > > > > > the
> > > > > > > > > async
> > > > > > > > > >> > table
> > > > > > > > > >> > > > > function. So the planner will also treat this
> > > > statement
> > > > > > to a
> > > > > > > > > >> > > > > CommonExecCorrelate node. So the runtime code
> > should
> > > > be
> > > > > > > > > generated
> > > > > > > > > >> in
> > > > > > > > > >> > > > > CorrelatedCodeGenerator.
> > > > > > > > > >> > > > > In CorrelatedCodeGenerator, we will know the
> > > > > > TableFunction's
> > > > > > > > > Kind
> > > > > > > > > >> of
> > > > > > > > > >> > > > > `FunctionKind.Table` or
> `FunctionKind.ASYNC_TABLE`
> > > > > > > > > >> > > > > For  `FunctionKind.ASYNC_TABLE` we can generate
> a
> > > > > > > > > >> AsyncWaitOperator
> > > > > > > > > >> > to
> > > > > > > > > >> > > > > execute the async table function.
> > > > > > > > > >> > > > >
> > > > > > > > > >> > > > >
> > > > > > > > > >> > > > > Thanks,
> > > > > > > > > >> > > > > Aitozi.
> > > > > > > > > >> > > > >
> > > > > > > > > >> > > > >
> > > > > > > > > >> > > > > Jing Ge <ji...@ververica.com.invalid>
> > 于2023年5月29日周一
> > > > > > 03:22写道：
> > > > > > > > > >> > > > >
> > > > > > > > > >> > > > >> Hi Aitozi,
> > > > > > > > > >> > > > >>
> > > > > > > > > >> > > > >> Thanks for the clarification. The naming
> "Lookup"
> > > > might
> > > > > > > > suggest
> > > > > > > > > >> > using
> > > > > > > > > >> > > it
> > > > > > > > > >> > > > >> for table look up. But conceptually what the
> > eval()
> > > > > > method
> > > > > > > > will
> > > > > > > > > >> do
> > > > > > > > > >> > is
> > > > > > > > > >> > > to
> > > > > > > > > >> > > > >> get a collection of results(Row, RowData) from
> > the
> > > > > given
> > > > > > > > keys.
> > > > > > > > > >> How
> > > > > > > > > >> > it
> > > > > > > > > >> > > > will
> > > > > > > > > >> > > > >> be done depends on the implementation, i.e. you
> > can
> > > > > > > implement
> > > > > > > > > >> your
> > > > > > > > > >> > own
> > > > > > > > > >> > > > >> Source[1][2]. The example in the FLIP should be
> > > able
> > > > to
> > > > > > be
> > > > > > > > > >> handled
> > > > > > > > > >> > in
> > > > > > > > > >> > > > this
> > > > > > > > > >> > > > >> way.
> > > > > > > > > >> > > > >>
> > > > > > > > > >> > > > >> Do you mean to support the AyncTableFunction
> > beyond
> > > > the
> > > > > > > > > >> > > > LookupTableSource?
> > > > > > > > > >> > > > >> It would be better if you could elaborate the
> > > > proposed
> > > > > > > > changes
> > > > > > > > > >> wrt
> > > > > > > > > >> > the
> > > > > > > > > >> > > > >> CorrelatedCodeGenerator with more details.
> > Thanks!
> > > > > > > > > >> > > > >>
> > > > > > > > > >> > > > >> Best regards,
> > > > > > > > > >> > > > >> Jing
> > > > > > > > > >> > > > >>
> > > > > > > > > >> > > > >> [1]
> > > > > > > > > >> > > > >>
> > > > > > > > > >> > > > >>
> > > > > > > > > >> > > >
> > > > > > > > > >> > >
> > > > > > > > > >> >
> > > > > > > > > >>
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/flink/blob/678370b18e1b6c4a23e5ce08f8efd05675a0cc17/flink-table/flink-table-common/src/main/java/org/apache/flink/table/connector/source/LookupTableSource.java#L64
> > > > > > > > > >> > > > >> [2]
> > > > > > > > > >> > > > >>
> > > > > > > > > >> > > > >>
> > > > > > > > > >> > > >
> > > > > > > > > >> > >
> > > > > > > > > >> >
> > > > > > > > > >>
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/flink/blob/678370b18e1b6c4a23e5ce08f8efd05675a0cc17/flink-table/flink-table-common/src/main/java/org/apache/flink/table/connector/source/AsyncTableFunctionProvider.java#L49
> > > > > > > > > >> > > > >>
> > > > > > > > > >> > > > >> On Sat, May 27, 2023 at 9:48 AM Aitozi <
> > > > > > > gjying1314@gmail.com
> > > > > > > > >
> > > > > > > > > >> > wrote:
> > > > > > > > > >> > > > >>
> > > > > > > > > >> > > > >> > Hi Jing,
> > > > > > > > > >> > > > >> >     Thanks for your response. As stated in
> the
> > > > FLIP,
> > > > > > the
> > > > > > > > > >> purpose
> > > > > > > > > >> > of
> > > > > > > > > >> > > > this
> > > > > > > > > >> > > > >> > FLIP is meant to support
> > > > > > > > > >> > > > >> > user-defined async table function. As
> described
> > > in
> > > > > > flink
> > > > > > > > > >> document
> > > > > > > > > >> > > [1]
> > > > > > > > > >> > > > >> >
> > > > > > > > > >> > > > >> > Async table functions are special functions
> for
> > > > table
> > > > > > > > sources
> > > > > > > > > >> that
> > > > > > > > > >> > > > >> perform
> > > > > > > > > >> > > > >> > > a lookup.
> > > > > > > > > >> > > > >> > >
> > > > > > > > > >> > > > >> >
> > > > > > > > > >> > > > >> > So end user can not directly define and use
> > async
> > > > > table
> > > > > > > > > >> function
> > > > > > > > > >> > > now.
> > > > > > > > > >> > > > An
> > > > > > > > > >> > > > >> > user case is reported in [2]
> > > > > > > > > >> > > > >> >
> > > > > > > > > >> > > > >> > So, in conclusion, no new interface is
> > > introduced,
> > > > > but
> > > > > > we
> > > > > > > > > >> extend
> > > > > > > > > >> > the
> > > > > > > > > >> > > > >> > ability to support user-defined async table
> > > > function.
> > > > > > > > > >> > > > >> >
> > > > > > > > > >> > > > >> > [1]:
> > > > > > > > > >> > > > >> >
> > > > > > > > > >> > > > >> >
> > > > > > > > > >> > > > >>
> > > > > > > > > >> > > >
> > > > > > > > > >> > >
> > > > > > > > > >> >
> > > > > > > > > >>
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/dev/table/functions/udfs/
> > > > > > > > > >> > > > >> > [2]:
> > > > > > > > > >> > >
> > > > > > >
> https://lists.apache.org/thread/qljwd40v5ntz6733cwcdr8s4z97b343b
> > > > > > > > > >> > > > >> >
> > > > > > > > > >> > > > >> > Thanks.
> > > > > > > > > >> > > > >> > Aitozi.
> > > > > > > > > >> > > > >> >
> > > > > > > > > >> > > > >> >
> > > > > > > > > >> > > > >> > Jing Ge <ji...@ververica.com.invalid>
> > > 于2023年5月27日周六
> > > > > > > > 06:40写道：
> > > > > > > > > >> > > > >> >
> > > > > > > > > >> > > > >> > > Hi Aitozi,
> > > > > > > > > >> > > > >> > >
> > > > > > > > > >> > > > >> > > Thanks for your proposal. I am not quite
> sure
> > > if
> > > > I
> > > > > > > > > understood
> > > > > > > > > >> > your
> > > > > > > > > >> > > > >> > thoughts
> > > > > > > > > >> > > > >> > > correctly. You described a special case
> > > > > > implementation
> > > > > > > of
> > > > > > > > > the
> > > > > > > > > >> > > > >> > > AsyncTableFunction with on public API
> > changes.
> > > > > Would
> > > > > > > you
> > > > > > > > > >> please
> > > > > > > > > >> > > > >> elaborate
> > > > > > > > > >> > > > >> > > your purpose of writing a FLIP according to
> > the
> > > > > FLIP
> > > > > > > > > >> > > > documentation[1]?
> > > > > > > > > >> > > > >> > > Thanks!
> > > > > > > > > >> > > > >> > >
> > > > > > > > > >> > > > >> > > [1]
> > > > > > > > > >> > > > >> > >
> > > > > > > > > >> > > > >> > >
> > > > > > > > > >> > > > >> >
> > > > > > > > > >> > > > >>
> > > > > > > > > >> > > >
> > > > > > > > > >> > >
> > > > > > > > > >> >
> > > > > > > > > >>
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/Flink+Improvement+Proposals
> > > > > > > > > >> > > > >> > >
> > > > > > > > > >> > > > >> > > Best regards,
> > > > > > > > > >> > > > >> > > Jing
> > > > > > > > > >> > > > >> > >
> > > > > > > > > >> > > > >> > > On Wed, May 24, 2023 at 1:07 PM Aitozi <
> > > > > > > > > gjying1314@gmail.com
> > > > > > > > > >> >
> > > > > > > > > >> > > > wrote:
> > > > > > > > > >> > > > >> > >
> > > > > > > > > >> > > > >> > > > May I ask for some feedback  :D
> > > > > > > > > >> > > > >> > > >
> > > > > > > > > >> > > > >> > > > Thanks,
> > > > > > > > > >> > > > >> > > > Aitozi
> > > > > > > > > >> > > > >> > > >
> > > > > > > > > >> > > > >> > > > Aitozi <gj...@gmail.com>
> > 于2023年5月23日周二
> > > > > > 19:14写道：
> > > > > > > > > >> > > > >> > > > >
> > > > > > > > > >> > > > >> > > > > Just catch an user case report from
> > Giannis
> > > > > > Polyzos
> > > > > > > > for
> > > > > > > > > >> this
> > > > > > > > > >> > > > >> usage:
> > > > > > > > > >> > > > >> > > > >
> > > > > > > > > >> > > > >> > > > >
> > > > > > > > > >> > > >
> > > > > > > >
> > https://lists.apache.org/thread/qljwd40v5ntz6733cwcdr8s4z97b343b
> > > > > > > > > >> > > > >> > > > >
> > > > > > > > > >> > > > >> > > > > Aitozi <gj...@gmail.com>
> > > 于2023年5月23日周二
> > > > > > > 17:45写道：
> > > > > > > > > >> > > > >> > > > > >
> > > > > > > > > >> > > > >> > > > > > Hi guys,
> > > > > > > > > >> > > > >> > > > > >     I want to bring up a discussion
> > about
> > > > > > adding
> > > > > > > > > >> support
> > > > > > > > > >> > of
> > > > > > > > > >> > > > User
> > > > > > > > > >> > > > >> > > > > > Defined AsyncTableFunction in Flink.
> > > > > > > > > >> > > > >> > > > > > Currently, async table function are
> > > special
> > > > > > > > functions
> > > > > > > > > >> for
> > > > > > > > > >> > > > table
> > > > > > > > > >> > > > >> > > source
> > > > > > > > > >> > > > >> > > > > > to perform
> > > > > > > > > >> > > > >> > > > > > async lookup. However, it's worth to
> > > > support
> > > > > > the
> > > > > > > > user
> > > > > > > > > >> > > defined
> > > > > > > > > >> > > > >> async
> > > > > > > > > >> > > > >> > > > > > table function.
> > > > > > > > > >> > > > >> > > > > > Because, in this way, the end SQL
> user
> > > can
> > > > > > > leverage
> > > > > > > > > it
> > > > > > > > > >> to
> > > > > > > > > >> > > > >> perform
> > > > > > > > > >> > > > >> > the
> > > > > > > > > >> > > > >> > > > > > async operation
> > > > > > > > > >> > > > >> > > > > > which is useful to maximum the system
> > > > > > throughput
> > > > > > > > > >> > especially
> > > > > > > > > >> > > > for
> > > > > > > > > >> > > > >> IO
> > > > > > > > > >> > > > >> > > > > > bottleneck case.
> > > > > > > > > >> > > > >> > > > > >
> > > > > > > > > >> > > > >> > > > > > You can find some more detail in [1].
> > > > > > > > > >> > > > >> > > > > >
> > > > > > > > > >> > > > >> > > > > > Looking forward to feedback
> > > > > > > > > >> > > > >> > > > > >
> > > > > > > > > >> > > > >> > > > > >
> > > > > > > > > >> > > > >> > > > > > [1]:
> > > > > > > > > >> > > > >> > > >
> > > > > > > > > >> > > > >> > >
> > > > > > > > > >> > > > >> >
> > > > > > > > > >> > > > >>
> > > > > > > > > >> > > >
> > > > > > > > > >> > >
> > > > > > > > > >> >
> > > > > > > > > >>
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/%5BFLIP-313%5D+Add+support+of+User+Defined+AsyncTableFunction
> > > > > > > > > >> > > > >> > > > > >
> > > > > > > > > >> > > > >> > > > > > Thanks,
> > > > > > > > > >> > > > >> > > > > > Aitozi.
> > > > > > > > > >> > > > >> > > >
> > > > > > > > > >> > > > >> > >
> > > > > > > > > >> > > > >> >
> > > > > > > > > >> > > > >>
> > > > > > > > > >> > > > >
> > > > > > > > > >> > > >
> > > > > > > > > >> > >
> > > > > > > > > >> >
> > > > > > > > > >>
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] FLIP-313 Add support of User Defined AsyncTableFunction

Posted by Aitozi <gj...@gmail.com>.

Hi Jing,
    I means the join key is not necessary to be the primary key or unique
index of the database.
In this situation, we may queried out multi rows for one join key. I think
that's why the
LookupFunction#lookup will return a collection of RowData.

BTW, I think the behavior of lookup join will not affect the semantic of
the async udtf.
We use the Async TableFunction here and the table function can collect
multiple rows.

Thanks,
Atiozi.



Jing Ge <ji...@ververica.com.invalid> 于2023年6月10日周六 00:15写道：

> Hi Aitozi,
>
> The keyRow used in this case contains all keys[1].
>
> Best regards,
> Jing
>
> [1]
>
> https://github.com/apache/flink/blob/191ec6ca3943d7119f14837efe112e074d815c47/flink-table/flink-table-common/src/main/java/org/apache/flink/table/functions/LookupFunction.java#L49
>
>
> On Fri, Jun 9, 2023 at 3:42 PM Aitozi <gj...@gmail.com> wrote:
>
> > Hi Jing,
> >
> >      The performance test is added to the FLIP.
> >
> >      As I know, The lookup join can return multi rows, it depends on
> > whether  the join key
> > is the primary key of the external database or not. The `lookup` [1] will
> > return a collection of
> > joined result, and each of them will be collected
> >
> >
> > [1]:
> >
> >
> https://github.com/apache/flink/blob/191ec6ca3943d7119f14837efe112e074d815c47/flink-table/flink-table-common/src/main/java/org/apache/flink/table/functions/LookupFunction.java#L52
> >
> >
> > Thanks,
> > Aitozi.
> >
> > Jing Ge <ji...@ververica.com.invalid> 于2023年6月9日周五 17:05写道：
> >
> > > Hi Aitozi,
> > >
> > > Thanks for the feedback. Looking forward to the performance tests.
> > >
> > > Afaik, lookup returns one row for each key [1] [2]. Conceptually, the
> > > lookup function is used to enrich column(s) from the dimension table.
> If,
> > > for the given key, there will be more than one row, there will be no
> way
> > to
> > > know which row will be used to enrich the key.
> > >
> > > [1]
> > >
> > >
> >
> https://github.com/apache/flink/blob/191ec6ca3943d7119f14837efe112e074d815c47/flink-table/flink-table-common/src/main/java/org/apache/flink/table/functions/LookupFunction.java#L49
> > > [2]
> > >
> > >
> >
> https://github.com/apache/flink/blob/191ec6ca3943d7119f14837efe112e074d815c47/flink-table/flink-table-common/src/main/java/org/apache/flink/table/functions/TableFunction.java#L196
> > >
> > > Best regards,
> > > Jing
> > >
> > > On Fri, Jun 9, 2023 at 5:18 AM Aitozi <gj...@gmail.com> wrote:
> > >
> > > > Hi Jing
> > > >     Thanks for your good questions. I have updated the example to the
> > > FLIP.
> > > >
> > > > > Only one row for each lookup
> > > > lookup can also return multi rows, based on the query result. [1]
> > > >
> > > > [1]:
> > > >
> > > >
> > >
> >
> https://github.com/apache/flink/blob/191ec6ca3943d7119f14837efe112e074d815c47/flink-table/flink-table-common/src/main/java/org/apache/flink/table/functions/LookupFunction.java#L56
> > > >
> > > > > If we use async calls with lateral join, my gut feeling is
> > > > that we might have many more async calls than lookup join. I am not
> > > really
> > > > sure if we will be facing potential issues in this case or not.
> > > >
> > > > IMO, the work pattern is similar to the lookup function, for each row
> > > from
> > > > the left table,
> > > > it will evaluate the eval method once, so the async call numbers will
> > not
> > > > change.
> > > > and the maximum calls in flight is limited by the Async operators
> > buffer
> > > > capacity
> > > > which will be controlled by the option.
> > > >
> > > > BTW, for the naming of these option, I updated the FLIP about this
> you
> > > can
> > > > refer to
> > > > the section of "ConfigOption" and "Rejected Alternatives"
> > > >
> > > > In the end, for the performance evaluation, I'd like to do some tests
> > and
> > > > will update it to the FLIP doc
> > > >
> > > > Thanks,
> > > > Aitozi.
> > > >
> > > >
> > > > Jing Ge <ji...@ververica.com.invalid> 于2023年6月9日周五 07:23写道：
> > > >
> > > > > Hi Aitozi,
> > > > >
> > > > > Thanks for the clarification. The code example looks interesting. I
> > > would
> > > > > suggest adding them into the FLIP. The description with code
> examples
> > > > will
> > > > > help readers understand the motivation and how to use it. Afaiac,
> it
> > > is a
> > > > > valid feature for Flink users.
> > > > >
> > > > > As we knew, lookup join is based on temporal join, i.e. FOR
> > SYSTEM_TIME
> > > > AS
> > > > > OF which is also used in your code example. Temporal join performs
> > the
> > > > > lookup based on the processing time match. Only one row for each
> > > > > lookup(afaiu, I need to check the source code to double confirm)
> will
> > > > > return for further enrichment. One the other hand, lateral join
> will
> > > have
> > > > > sub-queries correlated with every individual value of the reference
> > > table
> > > > > from the preceding part of the query and each sub query will return
> > > > > multiple rows. If we use async calls with lateral join, my gut
> > feeling
> > > is
> > > > > that we might have many more async calls than lookup join. I am not
> > > > really
> > > > > sure if we will be facing potential issues in this case or not.
> > > Possible
> > > > > issues I can think of now e.g. too many PRC calls, too many async
> > calls
> > > > > processing, the sub query will return a table which might be (too)
> > big,
> > > > and
> > > > > might cause performance issues. I would suggest preparing some use
> > > cases
> > > > > and running some performance tests to check it. These are my
> concerns
> > > > about
> > > > > using async calls with lateral join and I'd like to share with you,
> > > happy
> > > > > to discuss with you and hear different opinions, hopefully the
> > > > > discussion could help me understand it more deeply. Please correct
> me
> > > if
> > > > I
> > > > > am wrong.
> > > > >
> > > > > Best regards,
> > > > > Jing
> > > > >
> > > > >
> > > > > On Thu, Jun 8, 2023 at 7:22 AM Aitozi <gj...@gmail.com>
> wrote:
> > > > >
> > > > > > Hi Mason,
> > > > > >     Thanks for your input. I think if we support the user defined
> > > async
> > > > > > table function,
> > > > > > user will be able to use it to hold a batch data then handle it
> at
> > > one
> > > > > time
> > > > > > in the customized function.
> > > > > >
> > > > > > AsyncSink is meant for the sink operator. I have not figure out
> how
> > > to
> > > > > > integrate in this case.
> > > > > >
> > > > > > Thanks,
> > > > > > Atiozi.
> > > > > >
> > > > > >
> > > > > > Mason Chen <ma...@gmail.com> 于2023年6月8日周四 12:40写道：
> > > > > >
> > > > > > > Hi Aitozi,
> > > > > > >
> > > > > > > I think it makes sense to make it easier for SQL users to make
> > > RPCs.
> > > > Do
> > > > > > you
> > > > > > > think your proposal can extend to the ability to batch data for
> > the
> > > > > RPC?
> > > > > > > This is also another common strategy to increase throughput.
> > Also,
> > > > have
> > > > > > you
> > > > > > > considered solving this a bit differently by leveraging Flink's
> > > > > > AsyncSink?
> > > > > > >
> > > > > > > Best,
> > > > > > > Mason
> > > > > > >
> > > > > > > On Mon, Jun 5, 2023 at 1:50 AM Aitozi <gj...@gmail.com>
> > > wrote:
> > > > > > >
> > > > > > > > One more thing for discussion:
> > > > > > > >
> > > > > > > > In our internal implementation, we reuse the option
> > > > > > > > `table.exec.async-lookup.buffer-capacity` and
> > > > > > > > `table.exec.async-lookup.timeout` to config
> > > > > > > > the async udtf. Do you think we should add two extra option
> to
> > > > > > > distinguish
> > > > > > > > from the lookup option such as
> > > > > > > >
> > > > > > > > `table.exec.async-udtf.buffer-capacity`
> > > > > > > > `table.exec.async-udtf.timeout`
> > > > > > > >
> > > > > > > >
> > > > > > > > Best,
> > > > > > > > Aitozi.
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > Aitozi <gj...@gmail.com> 于2023年6月5日周一 12:20写道：
> > > > > > > >
> > > > > > > > > Hi Jing,
> > > > > > > > >
> > > > > > > > >     > what is the difference between the RPC call or query
> > you
> > > > > > > mentioned
> > > > > > > > > and the lookup in a very
> > > > > > > > > general way
> > > > > > > > >
> > > > > > > > > I think the RPC call or query service is quite similar to
> the
> > > > > lookup
> > > > > > > > join.
> > > > > > > > > But lookup join should work
> > > > > > > > > with `LookupTableSource`.
> > > > > > > > >
> > > > > > > > > Let's see how we can perform an async RPC call with lookup
> > > join:
> > > > > > > > >
> > > > > > > > > (1) Implement an AsyncTableFunction with RPC call logic.
> > > > > > > > > (2) Implement a `LookupTableSource` connector run with the
> > > async
> > > > > udtf
> > > > > > > > > defined in (1).
> > > > > > > > > (3) Then define a DDL of this look up table in SQL
> > > > > > > > >
> > > > > > > > > CREATE TEMPORARY TABLE Customers (
> > > > > > > > >   id INT,
> > > > > > > > >   name STRING,
> > > > > > > > >   country STRING,
> > > > > > > > >   zip STRING
> > > > > > > > > ) WITH (
> > > > > > > > >   'connector' = 'custom'
> > > > > > > > > );
> > > > > > > > >
> > > > > > > > > (4) Run with the query as below:
> > > > > > > > >
> > > > > > > > > SELECT o.order_id, o.total, c.country, c.zip
> > > > > > > > > FROM Orders AS o
> > > > > > > > >   JOIN Customers FOR SYSTEM_TIME AS OF o.proc_time AS c
> > > > > > > > >     ON o.customer_id = c.id;
> > > > > > > > >
> > > > > > > > > This example is from doc
> > > > > > > > > <
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/dev/table/sql/queries/joins/#lookup-join
> > > > > > > > >.You
> > > > > > > > > can image the look up process as an async RPC call process.
> > > > > > > > >
> > > > > > > > > Let's see how we can perform an async RPC call with lateral
> > > join:
> > > > > > > > >
> > > > > > > > > (1) Implement an AsyncTableFunction with RPC call logic.
> > > > > > > > > (2) Run query with
> > > > > > > > >
> > > > > > > > > Create function f1 as '...' ;
> > > > > > > > >
> > > > > > > > > SELECT o.order_id, o.total, c.country, c.zip FROM Orders
> > > lateral
> > > > > > table
> > > > > > > > > (f1(order_id)) as T(...);
> > > > > > > > >
> > > > > > > > > As you can see, the lateral join version is more simple and
> > > > > intuitive
> > > > > > > to
> > > > > > > > > users. Users do not have to wrap a
> > > > > > > > > LookupTableSource for the purpose of using async udtf.
> > > > > > > > >
> > > > > > > > > In the end, We can also see the user defined async table
> > > function
> > > > > is
> > > > > > an
> > > > > > > > > enhancement of the current lateral table join
> > > > > > > > > which only supports sync lateral join now.
> > > > > > > > >
> > > > > > > > > Best,
> > > > > > > > > Aitozi.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Jing Ge <ji...@ververica.com.invalid> 于2023年6月2日周五 19:37写道：
> > > > > > > > >
> > > > > > > > >> Hi Aitozi,
> > > > > > > > >>
> > > > > > > > >> Thanks for the update. Just out of curiosity, what is the
> > > > > difference
> > > > > > > > >> between the RPC call or query you mentioned and the lookup
> > in
> > > a
> > > > > very
> > > > > > > > >> general way? Since Lateral join is used in the FLIP. Is
> > there
> > > > any
> > > > > > > > special
> > > > > > > > >> thought for that? Sorry for asking so many questions. The
> > FLIP
> > > > > > > contains
> > > > > > > > >> limited information to understand the motivation.
> > > > > > > > >>
> > > > > > > > >> Best regards,
> > > > > > > > >> Jing
> > > > > > > > >>
> > > > > > > > >> On Fri, Jun 2, 2023 at 3:48 AM Aitozi <
> gjying1314@gmail.com
> > >
> > > > > wrote:
> > > > > > > > >>
> > > > > > > > >> > Hi Jing,
> > > > > > > > >> >     I have updated the proposed changes to the FLIP.
> IMO,
> > > > lookup
> > > > > > has
> > > > > > > > its
> > > > > > > > >> > clear
> > > > > > > > >> > async call requirement is due to its IO heavy operator.
> In
> > > our
> > > > > > > usage,
> > > > > > > > >> sql
> > > > > > > > >> > users have
> > > > > > > > >> > logic to do some RPC call or query the third-party
> service
> > > > which
> > > > > > is
> > > > > > > > >> also IO
> > > > > > > > >> > intensive.
> > > > > > > > >> > In these case, we'd like to leverage the async function
> to
> > > > > improve
> > > > > > > the
> > > > > > > > >> > throughput.
> > > > > > > > >> >
> > > > > > > > >> > Thanks,
> > > > > > > > >> > Aitozi.
> > > > > > > > >> >
> > > > > > > > >> > Jing Ge <ji...@ververica.com.invalid> 于2023年6月1日周四
> > 22:55写道：
> > > > > > > > >> >
> > > > > > > > >> > > Hi Aitozi,
> > > > > > > > >> > >
> > > > > > > > >> > > Sorry for the late reply. Would you like to update the
> > > > > proposed
> > > > > > > > >> changes
> > > > > > > > >> > > with more details into the FLIP too?
> > > > > > > > >> > > I got your point. It looks like a rational idea.
> > However,
> > > > > since
> > > > > > > > lookup
> > > > > > > > >> > has
> > > > > > > > >> > > its clear async call requirement, are there any real
> use
> > > > cases
> > > > > > > that
> > > > > > > > >> > > need this change? This will help us understand the
> > > > motivation.
> > > > > > > After
> > > > > > > > >> all,
> > > > > > > > >> > > lateral join and temporal lookup join[1] are quite
> > > > different.
> > > > > > > > >> > >
> > > > > > > > >> > > Best regards,
> > > > > > > > >> > > Jing
> > > > > > > > >> > >
> > > > > > > > >> > >
> > > > > > > > >> > > [1]
> > > > > > > > >> > >
> > > > > > > > >> > >
> > > > > > > > >> >
> > > > > > > > >>
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/flink/blob/d90a72da2fd601ca4e2a46700e91ec5b348de2ad/flink-table/flink-table-common/src/main/java/org/apache/flink/table/functions/AsyncTableFunction.java#L54
> > > > > > > > >> > >
> > > > > > > > >> > > On Wed, May 31, 2023 at 8:53 AM Aitozi <
> > > > gjying1314@gmail.com>
> > > > > > > > wrote:
> > > > > > > > >> > >
> > > > > > > > >> > > > Hi Jing,
> > > > > > > > >> > > >     What do you think about it? Can we move forward
> > this
> > > > > > > feature?
> > > > > > > > >> > > >
> > > > > > > > >> > > > Thanks,
> > > > > > > > >> > > > Aitozi.
> > > > > > > > >> > > >
> > > > > > > > >> > > > Aitozi <gj...@gmail.com> 于2023年5月29日周一
> 09:56写道：
> > > > > > > > >> > > >
> > > > > > > > >> > > > > Hi Jing,
> > > > > > > > >> > > > >     > "Do you mean to support the
> AyncTableFunction
> > > > beyond
> > > > > > the
> > > > > > > > >> > > > > LookupTableSource?"
> > > > > > > > >> > > > > Yes, I mean to support the AyncTableFunction
> beyond
> > > the
> > > > > > > > >> > > > LookupTableSource.
> > > > > > > > >> > > > >
> > > > > > > > >> > > > > The "AsyncTableFunction" is the function with
> > ability
> > > to
> > > > > be
> > > > > > > > >> executed
> > > > > > > > >> > > > async
> > > > > > > > >> > > > > (with AsyncWaitOperator).
> > > > > > > > >> > > > > The async lookup join is a one of usage of this.
> So,
> > > we
> > > > > > don't
> > > > > > > > >> have to
> > > > > > > > >> > > > bind
> > > > > > > > >> > > > > the AyncTableFunction with LookupTableSource.
> > > > > > > > >> > > > > If User-defined AsyncTableFunction is supported,
> > user
> > > > can
> > > > > > > > directly
> > > > > > > > >> > use
> > > > > > > > >> > > > > lateral table syntax to perform async operation.
> > > > > > > > >> > > > >
> > > > > > > > >> > > > > > "It would be better if you could elaborate the
> > > > proposed
> > > > > > > > changes
> > > > > > > > >> wrt
> > > > > > > > >> > > the
> > > > > > > > >> > > > > CorrelatedCodeGenerator with more details"
> > > > > > > > >> > > > >
> > > > > > > > >> > > > > In the proposal, we use lateral table syntax to
> > > support
> > > > > the
> > > > > > > > async
> > > > > > > > >> > table
> > > > > > > > >> > > > > function. So the planner will also treat this
> > > statement
> > > > > to a
> > > > > > > > >> > > > > CommonExecCorrelate node. So the runtime code
> should
> > > be
> > > > > > > > generated
> > > > > > > > >> in
> > > > > > > > >> > > > > CorrelatedCodeGenerator.
> > > > > > > > >> > > > > In CorrelatedCodeGenerator, we will know the
> > > > > TableFunction's
> > > > > > > > Kind
> > > > > > > > >> of
> > > > > > > > >> > > > > `FunctionKind.Table` or `FunctionKind.ASYNC_TABLE`
> > > > > > > > >> > > > > For  `FunctionKind.ASYNC_TABLE` we can generate a
> > > > > > > > >> AsyncWaitOperator
> > > > > > > > >> > to
> > > > > > > > >> > > > > execute the async table function.
> > > > > > > > >> > > > >
> > > > > > > > >> > > > >
> > > > > > > > >> > > > > Thanks,
> > > > > > > > >> > > > > Aitozi.
> > > > > > > > >> > > > >
> > > > > > > > >> > > > >
> > > > > > > > >> > > > > Jing Ge <ji...@ververica.com.invalid>
> 于2023年5月29日周一
> > > > > 03:22写道：
> > > > > > > > >> > > > >
> > > > > > > > >> > > > >> Hi Aitozi,
> > > > > > > > >> > > > >>
> > > > > > > > >> > > > >> Thanks for the clarification. The naming "Lookup"
> > > might
> > > > > > > suggest
> > > > > > > > >> > using
> > > > > > > > >> > > it
> > > > > > > > >> > > > >> for table look up. But conceptually what the
> eval()
> > > > > method
> > > > > > > will
> > > > > > > > >> do
> > > > > > > > >> > is
> > > > > > > > >> > > to
> > > > > > > > >> > > > >> get a collection of results(Row, RowData) from
> the
> > > > given
> > > > > > > keys.
> > > > > > > > >> How
> > > > > > > > >> > it
> > > > > > > > >> > > > will
> > > > > > > > >> > > > >> be done depends on the implementation, i.e. you
> can
> > > > > > implement
> > > > > > > > >> your
> > > > > > > > >> > own
> > > > > > > > >> > > > >> Source[1][2]. The example in the FLIP should be
> > able
> > > to
> > > > > be
> > > > > > > > >> handled
> > > > > > > > >> > in
> > > > > > > > >> > > > this
> > > > > > > > >> > > > >> way.
> > > > > > > > >> > > > >>
> > > > > > > > >> > > > >> Do you mean to support the AyncTableFunction
> beyond
> > > the
> > > > > > > > >> > > > LookupTableSource?
> > > > > > > > >> > > > >> It would be better if you could elaborate the
> > > proposed
> > > > > > > changes
> > > > > > > > >> wrt
> > > > > > > > >> > the
> > > > > > > > >> > > > >> CorrelatedCodeGenerator with more details.
> Thanks!
> > > > > > > > >> > > > >>
> > > > > > > > >> > > > >> Best regards,
> > > > > > > > >> > > > >> Jing
> > > > > > > > >> > > > >>
> > > > > > > > >> > > > >> [1]
> > > > > > > > >> > > > >>
> > > > > > > > >> > > > >>
> > > > > > > > >> > > >
> > > > > > > > >> > >
> > > > > > > > >> >
> > > > > > > > >>
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/flink/blob/678370b18e1b6c4a23e5ce08f8efd05675a0cc17/flink-table/flink-table-common/src/main/java/org/apache/flink/table/connector/source/LookupTableSource.java#L64
> > > > > > > > >> > > > >> [2]
> > > > > > > > >> > > > >>
> > > > > > > > >> > > > >>
> > > > > > > > >> > > >
> > > > > > > > >> > >
> > > > > > > > >> >
> > > > > > > > >>
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/flink/blob/678370b18e1b6c4a23e5ce08f8efd05675a0cc17/flink-table/flink-table-common/src/main/java/org/apache/flink/table/connector/source/AsyncTableFunctionProvider.java#L49
> > > > > > > > >> > > > >>
> > > > > > > > >> > > > >> On Sat, May 27, 2023 at 9:48 AM Aitozi <
> > > > > > gjying1314@gmail.com
> > > > > > > >
> > > > > > > > >> > wrote:
> > > > > > > > >> > > > >>
> > > > > > > > >> > > > >> > Hi Jing,
> > > > > > > > >> > > > >> >     Thanks for your response. As stated in the
> > > FLIP,
> > > > > the
> > > > > > > > >> purpose
> > > > > > > > >> > of
> > > > > > > > >> > > > this
> > > > > > > > >> > > > >> > FLIP is meant to support
> > > > > > > > >> > > > >> > user-defined async table function. As described
> > in
> > > > > flink
> > > > > > > > >> document
> > > > > > > > >> > > [1]
> > > > > > > > >> > > > >> >
> > > > > > > > >> > > > >> > Async table functions are special functions for
> > > table
> > > > > > > sources
> > > > > > > > >> that
> > > > > > > > >> > > > >> perform
> > > > > > > > >> > > > >> > > a lookup.
> > > > > > > > >> > > > >> > >
> > > > > > > > >> > > > >> >
> > > > > > > > >> > > > >> > So end user can not directly define and use
> async
> > > > table
> > > > > > > > >> function
> > > > > > > > >> > > now.
> > > > > > > > >> > > > An
> > > > > > > > >> > > > >> > user case is reported in [2]
> > > > > > > > >> > > > >> >
> > > > > > > > >> > > > >> > So, in conclusion, no new interface is
> > introduced,
> > > > but
> > > > > we
> > > > > > > > >> extend
> > > > > > > > >> > the
> > > > > > > > >> > > > >> > ability to support user-defined async table
> > > function.
> > > > > > > > >> > > > >> >
> > > > > > > > >> > > > >> > [1]:
> > > > > > > > >> > > > >> >
> > > > > > > > >> > > > >> >
> > > > > > > > >> > > > >>
> > > > > > > > >> > > >
> > > > > > > > >> > >
> > > > > > > > >> >
> > > > > > > > >>
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/dev/table/functions/udfs/
> > > > > > > > >> > > > >> > [2]:
> > > > > > > > >> > >
> > > > > > https://lists.apache.org/thread/qljwd40v5ntz6733cwcdr8s4z97b343b
> > > > > > > > >> > > > >> >
> > > > > > > > >> > > > >> > Thanks.
> > > > > > > > >> > > > >> > Aitozi.
> > > > > > > > >> > > > >> >
> > > > > > > > >> > > > >> >
> > > > > > > > >> > > > >> > Jing Ge <ji...@ververica.com.invalid>
> > 于2023年5月27日周六
> > > > > > > 06:40写道：
> > > > > > > > >> > > > >> >
> > > > > > > > >> > > > >> > > Hi Aitozi,
> > > > > > > > >> > > > >> > >
> > > > > > > > >> > > > >> > > Thanks for your proposal. I am not quite sure
> > if
> > > I
> > > > > > > > understood
> > > > > > > > >> > your
> > > > > > > > >> > > > >> > thoughts
> > > > > > > > >> > > > >> > > correctly. You described a special case
> > > > > implementation
> > > > > > of
> > > > > > > > the
> > > > > > > > >> > > > >> > > AsyncTableFunction with on public API
> changes.
> > > > Would
> > > > > > you
> > > > > > > > >> please
> > > > > > > > >> > > > >> elaborate
> > > > > > > > >> > > > >> > > your purpose of writing a FLIP according to
> the
> > > > FLIP
> > > > > > > > >> > > > documentation[1]?
> > > > > > > > >> > > > >> > > Thanks!
> > > > > > > > >> > > > >> > >
> > > > > > > > >> > > > >> > > [1]
> > > > > > > > >> > > > >> > >
> > > > > > > > >> > > > >> > >
> > > > > > > > >> > > > >> >
> > > > > > > > >> > > > >>
> > > > > > > > >> > > >
> > > > > > > > >> > >
> > > > > > > > >> >
> > > > > > > > >>
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/Flink+Improvement+Proposals
> > > > > > > > >> > > > >> > >
> > > > > > > > >> > > > >> > > Best regards,
> > > > > > > > >> > > > >> > > Jing
> > > > > > > > >> > > > >> > >
> > > > > > > > >> > > > >> > > On Wed, May 24, 2023 at 1:07 PM Aitozi <
> > > > > > > > gjying1314@gmail.com
> > > > > > > > >> >
> > > > > > > > >> > > > wrote:
> > > > > > > > >> > > > >> > >
> > > > > > > > >> > > > >> > > > May I ask for some feedback  :D
> > > > > > > > >> > > > >> > > >
> > > > > > > > >> > > > >> > > > Thanks,
> > > > > > > > >> > > > >> > > > Aitozi
> > > > > > > > >> > > > >> > > >
> > > > > > > > >> > > > >> > > > Aitozi <gj...@gmail.com>
> 于2023年5月23日周二
> > > > > 19:14写道：
> > > > > > > > >> > > > >> > > > >
> > > > > > > > >> > > > >> > > > > Just catch an user case report from
> Giannis
> > > > > Polyzos
> > > > > > > for
> > > > > > > > >> this
> > > > > > > > >> > > > >> usage:
> > > > > > > > >> > > > >> > > > >
> > > > > > > > >> > > > >> > > > >
> > > > > > > > >> > > >
> > > > > > >
> https://lists.apache.org/thread/qljwd40v5ntz6733cwcdr8s4z97b343b
> > > > > > > > >> > > > >> > > > >
> > > > > > > > >> > > > >> > > > > Aitozi <gj...@gmail.com>
> > 于2023年5月23日周二
> > > > > > 17:45写道：
> > > > > > > > >> > > > >> > > > > >
> > > > > > > > >> > > > >> > > > > > Hi guys,
> > > > > > > > >> > > > >> > > > > >     I want to bring up a discussion
> about
> > > > > adding
> > > > > > > > >> support
> > > > > > > > >> > of
> > > > > > > > >> > > > User
> > > > > > > > >> > > > >> > > > > > Defined AsyncTableFunction in Flink.
> > > > > > > > >> > > > >> > > > > > Currently, async table function are
> > special
> > > > > > > functions
> > > > > > > > >> for
> > > > > > > > >> > > > table
> > > > > > > > >> > > > >> > > source
> > > > > > > > >> > > > >> > > > > > to perform
> > > > > > > > >> > > > >> > > > > > async lookup. However, it's worth to
> > > support
> > > > > the
> > > > > > > user
> > > > > > > > >> > > defined
> > > > > > > > >> > > > >> async
> > > > > > > > >> > > > >> > > > > > table function.
> > > > > > > > >> > > > >> > > > > > Because, in this way, the end SQL user
> > can
> > > > > > leverage
> > > > > > > > it
> > > > > > > > >> to
> > > > > > > > >> > > > >> perform
> > > > > > > > >> > > > >> > the
> > > > > > > > >> > > > >> > > > > > async operation
> > > > > > > > >> > > > >> > > > > > which is useful to maximum the system
> > > > > throughput
> > > > > > > > >> > especially
> > > > > > > > >> > > > for
> > > > > > > > >> > > > >> IO
> > > > > > > > >> > > > >> > > > > > bottleneck case.
> > > > > > > > >> > > > >> > > > > >
> > > > > > > > >> > > > >> > > > > > You can find some more detail in [1].
> > > > > > > > >> > > > >> > > > > >
> > > > > > > > >> > > > >> > > > > > Looking forward to feedback
> > > > > > > > >> > > > >> > > > > >
> > > > > > > > >> > > > >> > > > > >
> > > > > > > > >> > > > >> > > > > > [1]:
> > > > > > > > >> > > > >> > > >
> > > > > > > > >> > > > >> > >
> > > > > > > > >> > > > >> >
> > > > > > > > >> > > > >>
> > > > > > > > >> > > >
> > > > > > > > >> > >
> > > > > > > > >> >
> > > > > > > > >>
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/%5BFLIP-313%5D+Add+support+of+User+Defined+AsyncTableFunction
> > > > > > > > >> > > > >> > > > > >
> > > > > > > > >> > > > >> > > > > > Thanks,
> > > > > > > > >> > > > >> > > > > > Aitozi.
> > > > > > > > >> > > > >> > > >
> > > > > > > > >> > > > >> > >
> > > > > > > > >> > > > >> >
> > > > > > > > >> > > > >>
> > > > > > > > >> > > > >
> > > > > > > > >> > > >
> > > > > > > > >> > >
> > > > > > > > >> >
> > > > > > > > >>
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] FLIP-313 Add support of User Defined AsyncTableFunction

Posted by Jing Ge <ji...@ververica.com.INVALID>.

Hi Aitozi,

The keyRow used in this case contains all keys[1].

Best regards,
Jing

[1]
https://github.com/apache/flink/blob/191ec6ca3943d7119f14837efe112e074d815c47/flink-table/flink-table-common/src/main/java/org/apache/flink/table/functions/LookupFunction.java#L49


On Fri, Jun 9, 2023 at 3:42 PM Aitozi <gj...@gmail.com> wrote:

> Hi Jing,
>
>      The performance test is added to the FLIP.
>
>      As I know, The lookup join can return multi rows, it depends on
> whether  the join key
> is the primary key of the external database or not. The `lookup` [1] will
> return a collection of
> joined result, and each of them will be collected
>
>
> [1]:
>
> https://github.com/apache/flink/blob/191ec6ca3943d7119f14837efe112e074d815c47/flink-table/flink-table-common/src/main/java/org/apache/flink/table/functions/LookupFunction.java#L52
>
>
> Thanks,
> Aitozi.
>
> Jing Ge <ji...@ververica.com.invalid> 于2023年6月9日周五 17:05写道：
>
> > Hi Aitozi,
> >
> > Thanks for the feedback. Looking forward to the performance tests.
> >
> > Afaik, lookup returns one row for each key [1] [2]. Conceptually, the
> > lookup function is used to enrich column(s) from the dimension table. If,
> > for the given key, there will be more than one row, there will be no way
> to
> > know which row will be used to enrich the key.
> >
> > [1]
> >
> >
> https://github.com/apache/flink/blob/191ec6ca3943d7119f14837efe112e074d815c47/flink-table/flink-table-common/src/main/java/org/apache/flink/table/functions/LookupFunction.java#L49
> > [2]
> >
> >
> https://github.com/apache/flink/blob/191ec6ca3943d7119f14837efe112e074d815c47/flink-table/flink-table-common/src/main/java/org/apache/flink/table/functions/TableFunction.java#L196
> >
> > Best regards,
> > Jing
> >
> > On Fri, Jun 9, 2023 at 5:18 AM Aitozi <gj...@gmail.com> wrote:
> >
> > > Hi Jing
> > >     Thanks for your good questions. I have updated the example to the
> > FLIP.
> > >
> > > > Only one row for each lookup
> > > lookup can also return multi rows, based on the query result. [1]
> > >
> > > [1]:
> > >
> > >
> >
> https://github.com/apache/flink/blob/191ec6ca3943d7119f14837efe112e074d815c47/flink-table/flink-table-common/src/main/java/org/apache/flink/table/functions/LookupFunction.java#L56
> > >
> > > > If we use async calls with lateral join, my gut feeling is
> > > that we might have many more async calls than lookup join. I am not
> > really
> > > sure if we will be facing potential issues in this case or not.
> > >
> > > IMO, the work pattern is similar to the lookup function, for each row
> > from
> > > the left table,
> > > it will evaluate the eval method once, so the async call numbers will
> not
> > > change.
> > > and the maximum calls in flight is limited by the Async operators
> buffer
> > > capacity
> > > which will be controlled by the option.
> > >
> > > BTW, for the naming of these option, I updated the FLIP about this you
> > can
> > > refer to
> > > the section of "ConfigOption" and "Rejected Alternatives"
> > >
> > > In the end, for the performance evaluation, I'd like to do some tests
> and
> > > will update it to the FLIP doc
> > >
> > > Thanks,
> > > Aitozi.
> > >
> > >
> > > Jing Ge <ji...@ververica.com.invalid> 于2023年6月9日周五 07:23写道：
> > >
> > > > Hi Aitozi,
> > > >
> > > > Thanks for the clarification. The code example looks interesting. I
> > would
> > > > suggest adding them into the FLIP. The description with code examples
> > > will
> > > > help readers understand the motivation and how to use it. Afaiac, it
> > is a
> > > > valid feature for Flink users.
> > > >
> > > > As we knew, lookup join is based on temporal join, i.e. FOR
> SYSTEM_TIME
> > > AS
> > > > OF which is also used in your code example. Temporal join performs
> the
> > > > lookup based on the processing time match. Only one row for each
> > > > lookup(afaiu, I need to check the source code to double confirm) will
> > > > return for further enrichment. One the other hand, lateral join will
> > have
> > > > sub-queries correlated with every individual value of the reference
> > table
> > > > from the preceding part of the query and each sub query will return
> > > > multiple rows. If we use async calls with lateral join, my gut
> feeling
> > is
> > > > that we might have many more async calls than lookup join. I am not
> > > really
> > > > sure if we will be facing potential issues in this case or not.
> > Possible
> > > > issues I can think of now e.g. too many PRC calls, too many async
> calls
> > > > processing, the sub query will return a table which might be (too)
> big,
> > > and
> > > > might cause performance issues. I would suggest preparing some use
> > cases
> > > > and running some performance tests to check it. These are my concerns
> > > about
> > > > using async calls with lateral join and I'd like to share with you,
> > happy
> > > > to discuss with you and hear different opinions, hopefully the
> > > > discussion could help me understand it more deeply. Please correct me
> > if
> > > I
> > > > am wrong.
> > > >
> > > > Best regards,
> > > > Jing
> > > >
> > > >
> > > > On Thu, Jun 8, 2023 at 7:22 AM Aitozi <gj...@gmail.com> wrote:
> > > >
> > > > > Hi Mason,
> > > > >     Thanks for your input. I think if we support the user defined
> > async
> > > > > table function,
> > > > > user will be able to use it to hold a batch data then handle it at
> > one
> > > > time
> > > > > in the customized function.
> > > > >
> > > > > AsyncSink is meant for the sink operator. I have not figure out how
> > to
> > > > > integrate in this case.
> > > > >
> > > > > Thanks,
> > > > > Atiozi.
> > > > >
> > > > >
> > > > > Mason Chen <ma...@gmail.com> 于2023年6月8日周四 12:40写道：
> > > > >
> > > > > > Hi Aitozi,
> > > > > >
> > > > > > I think it makes sense to make it easier for SQL users to make
> > RPCs.
> > > Do
> > > > > you
> > > > > > think your proposal can extend to the ability to batch data for
> the
> > > > RPC?
> > > > > > This is also another common strategy to increase throughput.
> Also,
> > > have
> > > > > you
> > > > > > considered solving this a bit differently by leveraging Flink's
> > > > > AsyncSink?
> > > > > >
> > > > > > Best,
> > > > > > Mason
> > > > > >
> > > > > > On Mon, Jun 5, 2023 at 1:50 AM Aitozi <gj...@gmail.com>
> > wrote:
> > > > > >
> > > > > > > One more thing for discussion:
> > > > > > >
> > > > > > > In our internal implementation, we reuse the option
> > > > > > > `table.exec.async-lookup.buffer-capacity` and
> > > > > > > `table.exec.async-lookup.timeout` to config
> > > > > > > the async udtf. Do you think we should add two extra option to
> > > > > > distinguish
> > > > > > > from the lookup option such as
> > > > > > >
> > > > > > > `table.exec.async-udtf.buffer-capacity`
> > > > > > > `table.exec.async-udtf.timeout`
> > > > > > >
> > > > > > >
> > > > > > > Best,
> > > > > > > Aitozi.
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > Aitozi <gj...@gmail.com> 于2023年6月5日周一 12:20写道：
> > > > > > >
> > > > > > > > Hi Jing,
> > > > > > > >
> > > > > > > >     > what is the difference between the RPC call or query
> you
> > > > > > mentioned
> > > > > > > > and the lookup in a very
> > > > > > > > general way
> > > > > > > >
> > > > > > > > I think the RPC call or query service is quite similar to the
> > > > lookup
> > > > > > > join.
> > > > > > > > But lookup join should work
> > > > > > > > with `LookupTableSource`.
> > > > > > > >
> > > > > > > > Let's see how we can perform an async RPC call with lookup
> > join:
> > > > > > > >
> > > > > > > > (1) Implement an AsyncTableFunction with RPC call logic.
> > > > > > > > (2) Implement a `LookupTableSource` connector run with the
> > async
> > > > udtf
> > > > > > > > defined in (1).
> > > > > > > > (3) Then define a DDL of this look up table in SQL
> > > > > > > >
> > > > > > > > CREATE TEMPORARY TABLE Customers (
> > > > > > > >   id INT,
> > > > > > > >   name STRING,
> > > > > > > >   country STRING,
> > > > > > > >   zip STRING
> > > > > > > > ) WITH (
> > > > > > > >   'connector' = 'custom'
> > > > > > > > );
> > > > > > > >
> > > > > > > > (4) Run with the query as below:
> > > > > > > >
> > > > > > > > SELECT o.order_id, o.total, c.country, c.zip
> > > > > > > > FROM Orders AS o
> > > > > > > >   JOIN Customers FOR SYSTEM_TIME AS OF o.proc_time AS c
> > > > > > > >     ON o.customer_id = c.id;
> > > > > > > >
> > > > > > > > This example is from doc
> > > > > > > > <
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/dev/table/sql/queries/joins/#lookup-join
> > > > > > > >.You
> > > > > > > > can image the look up process as an async RPC call process.
> > > > > > > >
> > > > > > > > Let's see how we can perform an async RPC call with lateral
> > join:
> > > > > > > >
> > > > > > > > (1) Implement an AsyncTableFunction with RPC call logic.
> > > > > > > > (2) Run query with
> > > > > > > >
> > > > > > > > Create function f1 as '...' ;
> > > > > > > >
> > > > > > > > SELECT o.order_id, o.total, c.country, c.zip FROM Orders
> > lateral
> > > > > table
> > > > > > > > (f1(order_id)) as T(...);
> > > > > > > >
> > > > > > > > As you can see, the lateral join version is more simple and
> > > > intuitive
> > > > > > to
> > > > > > > > users. Users do not have to wrap a
> > > > > > > > LookupTableSource for the purpose of using async udtf.
> > > > > > > >
> > > > > > > > In the end, We can also see the user defined async table
> > function
> > > > is
> > > > > an
> > > > > > > > enhancement of the current lateral table join
> > > > > > > > which only supports sync lateral join now.
> > > > > > > >
> > > > > > > > Best,
> > > > > > > > Aitozi.
> > > > > > > >
> > > > > > > >
> > > > > > > > Jing Ge <ji...@ververica.com.invalid> 于2023年6月2日周五 19:37写道：
> > > > > > > >
> > > > > > > >> Hi Aitozi,
> > > > > > > >>
> > > > > > > >> Thanks for the update. Just out of curiosity, what is the
> > > > difference
> > > > > > > >> between the RPC call or query you mentioned and the lookup
> in
> > a
> > > > very
> > > > > > > >> general way? Since Lateral join is used in the FLIP. Is
> there
> > > any
> > > > > > > special
> > > > > > > >> thought for that? Sorry for asking so many questions. The
> FLIP
> > > > > > contains
> > > > > > > >> limited information to understand the motivation.
> > > > > > > >>
> > > > > > > >> Best regards,
> > > > > > > >> Jing
> > > > > > > >>
> > > > > > > >> On Fri, Jun 2, 2023 at 3:48 AM Aitozi <gjying1314@gmail.com
> >
> > > > wrote:
> > > > > > > >>
> > > > > > > >> > Hi Jing,
> > > > > > > >> >     I have updated the proposed changes to the FLIP. IMO,
> > > lookup
> > > > > has
> > > > > > > its
> > > > > > > >> > clear
> > > > > > > >> > async call requirement is due to its IO heavy operator. In
> > our
> > > > > > usage,
> > > > > > > >> sql
> > > > > > > >> > users have
> > > > > > > >> > logic to do some RPC call or query the third-party service
> > > which
> > > > > is
> > > > > > > >> also IO
> > > > > > > >> > intensive.
> > > > > > > >> > In these case, we'd like to leverage the async function to
> > > > improve
> > > > > > the
> > > > > > > >> > throughput.
> > > > > > > >> >
> > > > > > > >> > Thanks,
> > > > > > > >> > Aitozi.
> > > > > > > >> >
> > > > > > > >> > Jing Ge <ji...@ververica.com.invalid> 于2023年6月1日周四
> 22:55写道：
> > > > > > > >> >
> > > > > > > >> > > Hi Aitozi,
> > > > > > > >> > >
> > > > > > > >> > > Sorry for the late reply. Would you like to update the
> > > > proposed
> > > > > > > >> changes
> > > > > > > >> > > with more details into the FLIP too?
> > > > > > > >> > > I got your point. It looks like a rational idea.
> However,
> > > > since
> > > > > > > lookup
> > > > > > > >> > has
> > > > > > > >> > > its clear async call requirement, are there any real use
> > > cases
> > > > > > that
> > > > > > > >> > > need this change? This will help us understand the
> > > motivation.
> > > > > > After
> > > > > > > >> all,
> > > > > > > >> > > lateral join and temporal lookup join[1] are quite
> > > different.
> > > > > > > >> > >
> > > > > > > >> > > Best regards,
> > > > > > > >> > > Jing
> > > > > > > >> > >
> > > > > > > >> > >
> > > > > > > >> > > [1]
> > > > > > > >> > >
> > > > > > > >> > >
> > > > > > > >> >
> > > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/flink/blob/d90a72da2fd601ca4e2a46700e91ec5b348de2ad/flink-table/flink-table-common/src/main/java/org/apache/flink/table/functions/AsyncTableFunction.java#L54
> > > > > > > >> > >
> > > > > > > >> > > On Wed, May 31, 2023 at 8:53 AM Aitozi <
> > > gjying1314@gmail.com>
> > > > > > > wrote:
> > > > > > > >> > >
> > > > > > > >> > > > Hi Jing,
> > > > > > > >> > > >     What do you think about it? Can we move forward
> this
> > > > > > feature?
> > > > > > > >> > > >
> > > > > > > >> > > > Thanks,
> > > > > > > >> > > > Aitozi.
> > > > > > > >> > > >
> > > > > > > >> > > > Aitozi <gj...@gmail.com> 于2023年5月29日周一 09:56写道：
> > > > > > > >> > > >
> > > > > > > >> > > > > Hi Jing,
> > > > > > > >> > > > >     > "Do you mean to support the AyncTableFunction
> > > beyond
> > > > > the
> > > > > > > >> > > > > LookupTableSource?"
> > > > > > > >> > > > > Yes, I mean to support the AyncTableFunction beyond
> > the
> > > > > > > >> > > > LookupTableSource.
> > > > > > > >> > > > >
> > > > > > > >> > > > > The "AsyncTableFunction" is the function with
> ability
> > to
> > > > be
> > > > > > > >> executed
> > > > > > > >> > > > async
> > > > > > > >> > > > > (with AsyncWaitOperator).
> > > > > > > >> > > > > The async lookup join is a one of usage of this. So,
> > we
> > > > > don't
> > > > > > > >> have to
> > > > > > > >> > > > bind
> > > > > > > >> > > > > the AyncTableFunction with LookupTableSource.
> > > > > > > >> > > > > If User-defined AsyncTableFunction is supported,
> user
> > > can
> > > > > > > directly
> > > > > > > >> > use
> > > > > > > >> > > > > lateral table syntax to perform async operation.
> > > > > > > >> > > > >
> > > > > > > >> > > > > > "It would be better if you could elaborate the
> > > proposed
> > > > > > > changes
> > > > > > > >> wrt
> > > > > > > >> > > the
> > > > > > > >> > > > > CorrelatedCodeGenerator with more details"
> > > > > > > >> > > > >
> > > > > > > >> > > > > In the proposal, we use lateral table syntax to
> > support
> > > > the
> > > > > > > async
> > > > > > > >> > table
> > > > > > > >> > > > > function. So the planner will also treat this
> > statement
> > > > to a
> > > > > > > >> > > > > CommonExecCorrelate node. So the runtime code should
> > be
> > > > > > > generated
> > > > > > > >> in
> > > > > > > >> > > > > CorrelatedCodeGenerator.
> > > > > > > >> > > > > In CorrelatedCodeGenerator, we will know the
> > > > TableFunction's
> > > > > > > Kind
> > > > > > > >> of
> > > > > > > >> > > > > `FunctionKind.Table` or `FunctionKind.ASYNC_TABLE`
> > > > > > > >> > > > > For  `FunctionKind.ASYNC_TABLE` we can generate a
> > > > > > > >> AsyncWaitOperator
> > > > > > > >> > to
> > > > > > > >> > > > > execute the async table function.
> > > > > > > >> > > > >
> > > > > > > >> > > > >
> > > > > > > >> > > > > Thanks,
> > > > > > > >> > > > > Aitozi.
> > > > > > > >> > > > >
> > > > > > > >> > > > >
> > > > > > > >> > > > > Jing Ge <ji...@ververica.com.invalid> 于2023年5月29日周一
> > > > 03:22写道：
> > > > > > > >> > > > >
> > > > > > > >> > > > >> Hi Aitozi,
> > > > > > > >> > > > >>
> > > > > > > >> > > > >> Thanks for the clarification. The naming "Lookup"
> > might
> > > > > > suggest
> > > > > > > >> > using
> > > > > > > >> > > it
> > > > > > > >> > > > >> for table look up. But conceptually what the eval()
> > > > method
> > > > > > will
> > > > > > > >> do
> > > > > > > >> > is
> > > > > > > >> > > to
> > > > > > > >> > > > >> get a collection of results(Row, RowData) from the
> > > given
> > > > > > keys.
> > > > > > > >> How
> > > > > > > >> > it
> > > > > > > >> > > > will
> > > > > > > >> > > > >> be done depends on the implementation, i.e. you can
> > > > > implement
> > > > > > > >> your
> > > > > > > >> > own
> > > > > > > >> > > > >> Source[1][2]. The example in the FLIP should be
> able
> > to
> > > > be
> > > > > > > >> handled
> > > > > > > >> > in
> > > > > > > >> > > > this
> > > > > > > >> > > > >> way.
> > > > > > > >> > > > >>
> > > > > > > >> > > > >> Do you mean to support the AyncTableFunction beyond
> > the
> > > > > > > >> > > > LookupTableSource?
> > > > > > > >> > > > >> It would be better if you could elaborate the
> > proposed
> > > > > > changes
> > > > > > > >> wrt
> > > > > > > >> > the
> > > > > > > >> > > > >> CorrelatedCodeGenerator with more details. Thanks!
> > > > > > > >> > > > >>
> > > > > > > >> > > > >> Best regards,
> > > > > > > >> > > > >> Jing
> > > > > > > >> > > > >>
> > > > > > > >> > > > >> [1]
> > > > > > > >> > > > >>
> > > > > > > >> > > > >>
> > > > > > > >> > > >
> > > > > > > >> > >
> > > > > > > >> >
> > > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/flink/blob/678370b18e1b6c4a23e5ce08f8efd05675a0cc17/flink-table/flink-table-common/src/main/java/org/apache/flink/table/connector/source/LookupTableSource.java#L64
> > > > > > > >> > > > >> [2]
> > > > > > > >> > > > >>
> > > > > > > >> > > > >>
> > > > > > > >> > > >
> > > > > > > >> > >
> > > > > > > >> >
> > > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/flink/blob/678370b18e1b6c4a23e5ce08f8efd05675a0cc17/flink-table/flink-table-common/src/main/java/org/apache/flink/table/connector/source/AsyncTableFunctionProvider.java#L49
> > > > > > > >> > > > >>
> > > > > > > >> > > > >> On Sat, May 27, 2023 at 9:48 AM Aitozi <
> > > > > gjying1314@gmail.com
> > > > > > >
> > > > > > > >> > wrote:
> > > > > > > >> > > > >>
> > > > > > > >> > > > >> > Hi Jing,
> > > > > > > >> > > > >> >     Thanks for your response. As stated in the
> > FLIP,
> > > > the
> > > > > > > >> purpose
> > > > > > > >> > of
> > > > > > > >> > > > this
> > > > > > > >> > > > >> > FLIP is meant to support
> > > > > > > >> > > > >> > user-defined async table function. As described
> in
> > > > flink
> > > > > > > >> document
> > > > > > > >> > > [1]
> > > > > > > >> > > > >> >
> > > > > > > >> > > > >> > Async table functions are special functions for
> > table
> > > > > > sources
> > > > > > > >> that
> > > > > > > >> > > > >> perform
> > > > > > > >> > > > >> > > a lookup.
> > > > > > > >> > > > >> > >
> > > > > > > >> > > > >> >
> > > > > > > >> > > > >> > So end user can not directly define and use async
> > > table
> > > > > > > >> function
> > > > > > > >> > > now.
> > > > > > > >> > > > An
> > > > > > > >> > > > >> > user case is reported in [2]
> > > > > > > >> > > > >> >
> > > > > > > >> > > > >> > So, in conclusion, no new interface is
> introduced,
> > > but
> > > > we
> > > > > > > >> extend
> > > > > > > >> > the
> > > > > > > >> > > > >> > ability to support user-defined async table
> > function.
> > > > > > > >> > > > >> >
> > > > > > > >> > > > >> > [1]:
> > > > > > > >> > > > >> >
> > > > > > > >> > > > >> >
> > > > > > > >> > > > >>
> > > > > > > >> > > >
> > > > > > > >> > >
> > > > > > > >> >
> > > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/dev/table/functions/udfs/
> > > > > > > >> > > > >> > [2]:
> > > > > > > >> > >
> > > > > https://lists.apache.org/thread/qljwd40v5ntz6733cwcdr8s4z97b343b
> > > > > > > >> > > > >> >
> > > > > > > >> > > > >> > Thanks.
> > > > > > > >> > > > >> > Aitozi.
> > > > > > > >> > > > >> >
> > > > > > > >> > > > >> >
> > > > > > > >> > > > >> > Jing Ge <ji...@ververica.com.invalid>
> 于2023年5月27日周六
> > > > > > 06:40写道：
> > > > > > > >> > > > >> >
> > > > > > > >> > > > >> > > Hi Aitozi,
> > > > > > > >> > > > >> > >
> > > > > > > >> > > > >> > > Thanks for your proposal. I am not quite sure
> if
> > I
> > > > > > > understood
> > > > > > > >> > your
> > > > > > > >> > > > >> > thoughts
> > > > > > > >> > > > >> > > correctly. You described a special case
> > > > implementation
> > > > > of
> > > > > > > the
> > > > > > > >> > > > >> > > AsyncTableFunction with on public API changes.
> > > Would
> > > > > you
> > > > > > > >> please
> > > > > > > >> > > > >> elaborate
> > > > > > > >> > > > >> > > your purpose of writing a FLIP according to the
> > > FLIP
> > > > > > > >> > > > documentation[1]?
> > > > > > > >> > > > >> > > Thanks!
> > > > > > > >> > > > >> > >
> > > > > > > >> > > > >> > > [1]
> > > > > > > >> > > > >> > >
> > > > > > > >> > > > >> > >
> > > > > > > >> > > > >> >
> > > > > > > >> > > > >>
> > > > > > > >> > > >
> > > > > > > >> > >
> > > > > > > >> >
> > > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/Flink+Improvement+Proposals
> > > > > > > >> > > > >> > >
> > > > > > > >> > > > >> > > Best regards,
> > > > > > > >> > > > >> > > Jing
> > > > > > > >> > > > >> > >
> > > > > > > >> > > > >> > > On Wed, May 24, 2023 at 1:07 PM Aitozi <
> > > > > > > gjying1314@gmail.com
> > > > > > > >> >
> > > > > > > >> > > > wrote:
> > > > > > > >> > > > >> > >
> > > > > > > >> > > > >> > > > May I ask for some feedback  :D
> > > > > > > >> > > > >> > > >
> > > > > > > >> > > > >> > > > Thanks,
> > > > > > > >> > > > >> > > > Aitozi
> > > > > > > >> > > > >> > > >
> > > > > > > >> > > > >> > > > Aitozi <gj...@gmail.com> 于2023年5月23日周二
> > > > 19:14写道：
> > > > > > > >> > > > >> > > > >
> > > > > > > >> > > > >> > > > > Just catch an user case report from Giannis
> > > > Polyzos
> > > > > > for
> > > > > > > >> this
> > > > > > > >> > > > >> usage:
> > > > > > > >> > > > >> > > > >
> > > > > > > >> > > > >> > > > >
> > > > > > > >> > > >
> > > > > > https://lists.apache.org/thread/qljwd40v5ntz6733cwcdr8s4z97b343b
> > > > > > > >> > > > >> > > > >
> > > > > > > >> > > > >> > > > > Aitozi <gj...@gmail.com>
> 于2023年5月23日周二
> > > > > 17:45写道：
> > > > > > > >> > > > >> > > > > >
> > > > > > > >> > > > >> > > > > > Hi guys,
> > > > > > > >> > > > >> > > > > >     I want to bring up a discussion about
> > > > adding
> > > > > > > >> support
> > > > > > > >> > of
> > > > > > > >> > > > User
> > > > > > > >> > > > >> > > > > > Defined AsyncTableFunction in Flink.
> > > > > > > >> > > > >> > > > > > Currently, async table function are
> special
> > > > > > functions
> > > > > > > >> for
> > > > > > > >> > > > table
> > > > > > > >> > > > >> > > source
> > > > > > > >> > > > >> > > > > > to perform
> > > > > > > >> > > > >> > > > > > async lookup. However, it's worth to
> > support
> > > > the
> > > > > > user
> > > > > > > >> > > defined
> > > > > > > >> > > > >> async
> > > > > > > >> > > > >> > > > > > table function.
> > > > > > > >> > > > >> > > > > > Because, in this way, the end SQL user
> can
> > > > > leverage
> > > > > > > it
> > > > > > > >> to
> > > > > > > >> > > > >> perform
> > > > > > > >> > > > >> > the
> > > > > > > >> > > > >> > > > > > async operation
> > > > > > > >> > > > >> > > > > > which is useful to maximum the system
> > > > throughput
> > > > > > > >> > especially
> > > > > > > >> > > > for
> > > > > > > >> > > > >> IO
> > > > > > > >> > > > >> > > > > > bottleneck case.
> > > > > > > >> > > > >> > > > > >
> > > > > > > >> > > > >> > > > > > You can find some more detail in [1].
> > > > > > > >> > > > >> > > > > >
> > > > > > > >> > > > >> > > > > > Looking forward to feedback
> > > > > > > >> > > > >> > > > > >
> > > > > > > >> > > > >> > > > > >
> > > > > > > >> > > > >> > > > > > [1]:
> > > > > > > >> > > > >> > > >
> > > > > > > >> > > > >> > >
> > > > > > > >> > > > >> >
> > > > > > > >> > > > >>
> > > > > > > >> > > >
> > > > > > > >> > >
> > > > > > > >> >
> > > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/%5BFLIP-313%5D+Add+support+of+User+Defined+AsyncTableFunction
> > > > > > > >> > > > >> > > > > >
> > > > > > > >> > > > >> > > > > > Thanks,
> > > > > > > >> > > > >> > > > > > Aitozi.
> > > > > > > >> > > > >> > > >
> > > > > > > >> > > > >> > >
> > > > > > > >> > > > >> >
> > > > > > > >> > > > >>
> > > > > > > >> > > > >
> > > > > > > >> > > >
> > > > > > > >> > >
> > > > > > > >> >
> > > > > > > >>
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] FLIP-313 Add support of User Defined AsyncTableFunction

Posted by Aitozi <gj...@gmail.com>.

Hi Jing,

     The performance test is added to the FLIP.

     As I know, The lookup join can return multi rows, it depends on
whether  the join key
is the primary key of the external database or not. The `lookup` [1] will
return a collection of
joined result, and each of them will be collected


[1]:
https://github.com/apache/flink/blob/191ec6ca3943d7119f14837efe112e074d815c47/flink-table/flink-table-common/src/main/java/org/apache/flink/table/functions/LookupFunction.java#L52


Thanks,
Aitozi.

Jing Ge <ji...@ververica.com.invalid> 于2023年6月9日周五 17:05写道：

> Hi Aitozi,
>
> Thanks for the feedback. Looking forward to the performance tests.
>
> Afaik, lookup returns one row for each key [1] [2]. Conceptually, the
> lookup function is used to enrich column(s) from the dimension table. If,
> for the given key, there will be more than one row, there will be no way to
> know which row will be used to enrich the key.
>
> [1]
>
> https://github.com/apache/flink/blob/191ec6ca3943d7119f14837efe112e074d815c47/flink-table/flink-table-common/src/main/java/org/apache/flink/table/functions/LookupFunction.java#L49
> [2]
>
> https://github.com/apache/flink/blob/191ec6ca3943d7119f14837efe112e074d815c47/flink-table/flink-table-common/src/main/java/org/apache/flink/table/functions/TableFunction.java#L196
>
> Best regards,
> Jing
>
> On Fri, Jun 9, 2023 at 5:18 AM Aitozi <gj...@gmail.com> wrote:
>
> > Hi Jing
> >     Thanks for your good questions. I have updated the example to the
> FLIP.
> >
> > > Only one row for each lookup
> > lookup can also return multi rows, based on the query result. [1]
> >
> > [1]:
> >
> >
> https://github.com/apache/flink/blob/191ec6ca3943d7119f14837efe112e074d815c47/flink-table/flink-table-common/src/main/java/org/apache/flink/table/functions/LookupFunction.java#L56
> >
> > > If we use async calls with lateral join, my gut feeling is
> > that we might have many more async calls than lookup join. I am not
> really
> > sure if we will be facing potential issues in this case or not.
> >
> > IMO, the work pattern is similar to the lookup function, for each row
> from
> > the left table,
> > it will evaluate the eval method once, so the async call numbers will not
> > change.
> > and the maximum calls in flight is limited by the Async operators buffer
> > capacity
> > which will be controlled by the option.
> >
> > BTW, for the naming of these option, I updated the FLIP about this you
> can
> > refer to
> > the section of "ConfigOption" and "Rejected Alternatives"
> >
> > In the end, for the performance evaluation, I'd like to do some tests and
> > will update it to the FLIP doc
> >
> > Thanks,
> > Aitozi.
> >
> >
> > Jing Ge <ji...@ververica.com.invalid> 于2023年6月9日周五 07:23写道：
> >
> > > Hi Aitozi,
> > >
> > > Thanks for the clarification. The code example looks interesting. I
> would
> > > suggest adding them into the FLIP. The description with code examples
> > will
> > > help readers understand the motivation and how to use it. Afaiac, it
> is a
> > > valid feature for Flink users.
> > >
> > > As we knew, lookup join is based on temporal join, i.e. FOR SYSTEM_TIME
> > AS
> > > OF which is also used in your code example. Temporal join performs the
> > > lookup based on the processing time match. Only one row for each
> > > lookup(afaiu, I need to check the source code to double confirm) will
> > > return for further enrichment. One the other hand, lateral join will
> have
> > > sub-queries correlated with every individual value of the reference
> table
> > > from the preceding part of the query and each sub query will return
> > > multiple rows. If we use async calls with lateral join, my gut feeling
> is
> > > that we might have many more async calls than lookup join. I am not
> > really
> > > sure if we will be facing potential issues in this case or not.
> Possible
> > > issues I can think of now e.g. too many PRC calls, too many async calls
> > > processing, the sub query will return a table which might be (too) big,
> > and
> > > might cause performance issues. I would suggest preparing some use
> cases
> > > and running some performance tests to check it. These are my concerns
> > about
> > > using async calls with lateral join and I'd like to share with you,
> happy
> > > to discuss with you and hear different opinions, hopefully the
> > > discussion could help me understand it more deeply. Please correct me
> if
> > I
> > > am wrong.
> > >
> > > Best regards,
> > > Jing
> > >
> > >
> > > On Thu, Jun 8, 2023 at 7:22 AM Aitozi <gj...@gmail.com> wrote:
> > >
> > > > Hi Mason,
> > > >     Thanks for your input. I think if we support the user defined
> async
> > > > table function,
> > > > user will be able to use it to hold a batch data then handle it at
> one
> > > time
> > > > in the customized function.
> > > >
> > > > AsyncSink is meant for the sink operator. I have not figure out how
> to
> > > > integrate in this case.
> > > >
> > > > Thanks,
> > > > Atiozi.
> > > >
> > > >
> > > > Mason Chen <ma...@gmail.com> 于2023年6月8日周四 12:40写道：
> > > >
> > > > > Hi Aitozi,
> > > > >
> > > > > I think it makes sense to make it easier for SQL users to make
> RPCs.
> > Do
> > > > you
> > > > > think your proposal can extend to the ability to batch data for the
> > > RPC?
> > > > > This is also another common strategy to increase throughput. Also,
> > have
> > > > you
> > > > > considered solving this a bit differently by leveraging Flink's
> > > > AsyncSink?
> > > > >
> > > > > Best,
> > > > > Mason
> > > > >
> > > > > On Mon, Jun 5, 2023 at 1:50 AM Aitozi <gj...@gmail.com>
> wrote:
> > > > >
> > > > > > One more thing for discussion:
> > > > > >
> > > > > > In our internal implementation, we reuse the option
> > > > > > `table.exec.async-lookup.buffer-capacity` and
> > > > > > `table.exec.async-lookup.timeout` to config
> > > > > > the async udtf. Do you think we should add two extra option to
> > > > > distinguish
> > > > > > from the lookup option such as
> > > > > >
> > > > > > `table.exec.async-udtf.buffer-capacity`
> > > > > > `table.exec.async-udtf.timeout`
> > > > > >
> > > > > >
> > > > > > Best,
> > > > > > Aitozi.
> > > > > >
> > > > > >
> > > > > >
> > > > > > Aitozi <gj...@gmail.com> 于2023年6月5日周一 12:20写道：
> > > > > >
> > > > > > > Hi Jing,
> > > > > > >
> > > > > > >     > what is the difference between the RPC call or query you
> > > > > mentioned
> > > > > > > and the lookup in a very
> > > > > > > general way
> > > > > > >
> > > > > > > I think the RPC call or query service is quite similar to the
> > > lookup
> > > > > > join.
> > > > > > > But lookup join should work
> > > > > > > with `LookupTableSource`.
> > > > > > >
> > > > > > > Let's see how we can perform an async RPC call with lookup
> join:
> > > > > > >
> > > > > > > (1) Implement an AsyncTableFunction with RPC call logic.
> > > > > > > (2) Implement a `LookupTableSource` connector run with the
> async
> > > udtf
> > > > > > > defined in (1).
> > > > > > > (3) Then define a DDL of this look up table in SQL
> > > > > > >
> > > > > > > CREATE TEMPORARY TABLE Customers (
> > > > > > >   id INT,
> > > > > > >   name STRING,
> > > > > > >   country STRING,
> > > > > > >   zip STRING
> > > > > > > ) WITH (
> > > > > > >   'connector' = 'custom'
> > > > > > > );
> > > > > > >
> > > > > > > (4) Run with the query as below:
> > > > > > >
> > > > > > > SELECT o.order_id, o.total, c.country, c.zip
> > > > > > > FROM Orders AS o
> > > > > > >   JOIN Customers FOR SYSTEM_TIME AS OF o.proc_time AS c
> > > > > > >     ON o.customer_id = c.id;
> > > > > > >
> > > > > > > This example is from doc
> > > > > > > <
> > > > > >
> > > > >
> > > >
> > >
> >
> https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/dev/table/sql/queries/joins/#lookup-join
> > > > > > >.You
> > > > > > > can image the look up process as an async RPC call process.
> > > > > > >
> > > > > > > Let's see how we can perform an async RPC call with lateral
> join:
> > > > > > >
> > > > > > > (1) Implement an AsyncTableFunction with RPC call logic.
> > > > > > > (2) Run query with
> > > > > > >
> > > > > > > Create function f1 as '...' ;
> > > > > > >
> > > > > > > SELECT o.order_id, o.total, c.country, c.zip FROM Orders
> lateral
> > > > table
> > > > > > > (f1(order_id)) as T(...);
> > > > > > >
> > > > > > > As you can see, the lateral join version is more simple and
> > > intuitive
> > > > > to
> > > > > > > users. Users do not have to wrap a
> > > > > > > LookupTableSource for the purpose of using async udtf.
> > > > > > >
> > > > > > > In the end, We can also see the user defined async table
> function
> > > is
> > > > an
> > > > > > > enhancement of the current lateral table join
> > > > > > > which only supports sync lateral join now.
> > > > > > >
> > > > > > > Best,
> > > > > > > Aitozi.
> > > > > > >
> > > > > > >
> > > > > > > Jing Ge <ji...@ververica.com.invalid> 于2023年6月2日周五 19:37写道：
> > > > > > >
> > > > > > >> Hi Aitozi,
> > > > > > >>
> > > > > > >> Thanks for the update. Just out of curiosity, what is the
> > > difference
> > > > > > >> between the RPC call or query you mentioned and the lookup in
> a
> > > very
> > > > > > >> general way? Since Lateral join is used in the FLIP. Is there
> > any
> > > > > > special
> > > > > > >> thought for that? Sorry for asking so many questions. The FLIP
> > > > > contains
> > > > > > >> limited information to understand the motivation.
> > > > > > >>
> > > > > > >> Best regards,
> > > > > > >> Jing
> > > > > > >>
> > > > > > >> On Fri, Jun 2, 2023 at 3:48 AM Aitozi <gj...@gmail.com>
> > > wrote:
> > > > > > >>
> > > > > > >> > Hi Jing,
> > > > > > >> >     I have updated the proposed changes to the FLIP. IMO,
> > lookup
> > > > has
> > > > > > its
> > > > > > >> > clear
> > > > > > >> > async call requirement is due to its IO heavy operator. In
> our
> > > > > usage,
> > > > > > >> sql
> > > > > > >> > users have
> > > > > > >> > logic to do some RPC call or query the third-party service
> > which
> > > > is
> > > > > > >> also IO
> > > > > > >> > intensive.
> > > > > > >> > In these case, we'd like to leverage the async function to
> > > improve
> > > > > the
> > > > > > >> > throughput.
> > > > > > >> >
> > > > > > >> > Thanks,
> > > > > > >> > Aitozi.
> > > > > > >> >
> > > > > > >> > Jing Ge <ji...@ververica.com.invalid> 于2023年6月1日周四 22:55写道：
> > > > > > >> >
> > > > > > >> > > Hi Aitozi,
> > > > > > >> > >
> > > > > > >> > > Sorry for the late reply. Would you like to update the
> > > proposed
> > > > > > >> changes
> > > > > > >> > > with more details into the FLIP too?
> > > > > > >> > > I got your point. It looks like a rational idea. However,
> > > since
> > > > > > lookup
> > > > > > >> > has
> > > > > > >> > > its clear async call requirement, are there any real use
> > cases
> > > > > that
> > > > > > >> > > need this change? This will help us understand the
> > motivation.
> > > > > After
> > > > > > >> all,
> > > > > > >> > > lateral join and temporal lookup join[1] are quite
> > different.
> > > > > > >> > >
> > > > > > >> > > Best regards,
> > > > > > >> > > Jing
> > > > > > >> > >
> > > > > > >> > >
> > > > > > >> > > [1]
> > > > > > >> > >
> > > > > > >> > >
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/flink/blob/d90a72da2fd601ca4e2a46700e91ec5b348de2ad/flink-table/flink-table-common/src/main/java/org/apache/flink/table/functions/AsyncTableFunction.java#L54
> > > > > > >> > >
> > > > > > >> > > On Wed, May 31, 2023 at 8:53 AM Aitozi <
> > gjying1314@gmail.com>
> > > > > > wrote:
> > > > > > >> > >
> > > > > > >> > > > Hi Jing,
> > > > > > >> > > >     What do you think about it? Can we move forward this
> > > > > feature?
> > > > > > >> > > >
> > > > > > >> > > > Thanks,
> > > > > > >> > > > Aitozi.
> > > > > > >> > > >
> > > > > > >> > > > Aitozi <gj...@gmail.com> 于2023年5月29日周一 09:56写道：
> > > > > > >> > > >
> > > > > > >> > > > > Hi Jing,
> > > > > > >> > > > >     > "Do you mean to support the AyncTableFunction
> > beyond
> > > > the
> > > > > > >> > > > > LookupTableSource?"
> > > > > > >> > > > > Yes, I mean to support the AyncTableFunction beyond
> the
> > > > > > >> > > > LookupTableSource.
> > > > > > >> > > > >
> > > > > > >> > > > > The "AsyncTableFunction" is the function with ability
> to
> > > be
> > > > > > >> executed
> > > > > > >> > > > async
> > > > > > >> > > > > (with AsyncWaitOperator).
> > > > > > >> > > > > The async lookup join is a one of usage of this. So,
> we
> > > > don't
> > > > > > >> have to
> > > > > > >> > > > bind
> > > > > > >> > > > > the AyncTableFunction with LookupTableSource.
> > > > > > >> > > > > If User-defined AsyncTableFunction is supported, user
> > can
> > > > > > directly
> > > > > > >> > use
> > > > > > >> > > > > lateral table syntax to perform async operation.
> > > > > > >> > > > >
> > > > > > >> > > > > > "It would be better if you could elaborate the
> > proposed
> > > > > > changes
> > > > > > >> wrt
> > > > > > >> > > the
> > > > > > >> > > > > CorrelatedCodeGenerator with more details"
> > > > > > >> > > > >
> > > > > > >> > > > > In the proposal, we use lateral table syntax to
> support
> > > the
> > > > > > async
> > > > > > >> > table
> > > > > > >> > > > > function. So the planner will also treat this
> statement
> > > to a
> > > > > > >> > > > > CommonExecCorrelate node. So the runtime code should
> be
> > > > > > generated
> > > > > > >> in
> > > > > > >> > > > > CorrelatedCodeGenerator.
> > > > > > >> > > > > In CorrelatedCodeGenerator, we will know the
> > > TableFunction's
> > > > > > Kind
> > > > > > >> of
> > > > > > >> > > > > `FunctionKind.Table` or `FunctionKind.ASYNC_TABLE`
> > > > > > >> > > > > For  `FunctionKind.ASYNC_TABLE` we can generate a
> > > > > > >> AsyncWaitOperator
> > > > > > >> > to
> > > > > > >> > > > > execute the async table function.
> > > > > > >> > > > >
> > > > > > >> > > > >
> > > > > > >> > > > > Thanks,
> > > > > > >> > > > > Aitozi.
> > > > > > >> > > > >
> > > > > > >> > > > >
> > > > > > >> > > > > Jing Ge <ji...@ververica.com.invalid> 于2023年5月29日周一
> > > 03:22写道：
> > > > > > >> > > > >
> > > > > > >> > > > >> Hi Aitozi,
> > > > > > >> > > > >>
> > > > > > >> > > > >> Thanks for the clarification. The naming "Lookup"
> might
> > > > > suggest
> > > > > > >> > using
> > > > > > >> > > it
> > > > > > >> > > > >> for table look up. But conceptually what the eval()
> > > method
> > > > > will
> > > > > > >> do
> > > > > > >> > is
> > > > > > >> > > to
> > > > > > >> > > > >> get a collection of results(Row, RowData) from the
> > given
> > > > > keys.
> > > > > > >> How
> > > > > > >> > it
> > > > > > >> > > > will
> > > > > > >> > > > >> be done depends on the implementation, i.e. you can
> > > > implement
> > > > > > >> your
> > > > > > >> > own
> > > > > > >> > > > >> Source[1][2]. The example in the FLIP should be able
> to
> > > be
> > > > > > >> handled
> > > > > > >> > in
> > > > > > >> > > > this
> > > > > > >> > > > >> way.
> > > > > > >> > > > >>
> > > > > > >> > > > >> Do you mean to support the AyncTableFunction beyond
> the
> > > > > > >> > > > LookupTableSource?
> > > > > > >> > > > >> It would be better if you could elaborate the
> proposed
> > > > > changes
> > > > > > >> wrt
> > > > > > >> > the
> > > > > > >> > > > >> CorrelatedCodeGenerator with more details. Thanks!
> > > > > > >> > > > >>
> > > > > > >> > > > >> Best regards,
> > > > > > >> > > > >> Jing
> > > > > > >> > > > >>
> > > > > > >> > > > >> [1]
> > > > > > >> > > > >>
> > > > > > >> > > > >>
> > > > > > >> > > >
> > > > > > >> > >
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/flink/blob/678370b18e1b6c4a23e5ce08f8efd05675a0cc17/flink-table/flink-table-common/src/main/java/org/apache/flink/table/connector/source/LookupTableSource.java#L64
> > > > > > >> > > > >> [2]
> > > > > > >> > > > >>
> > > > > > >> > > > >>
> > > > > > >> > > >
> > > > > > >> > >
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/flink/blob/678370b18e1b6c4a23e5ce08f8efd05675a0cc17/flink-table/flink-table-common/src/main/java/org/apache/flink/table/connector/source/AsyncTableFunctionProvider.java#L49
> > > > > > >> > > > >>
> > > > > > >> > > > >> On Sat, May 27, 2023 at 9:48 AM Aitozi <
> > > > gjying1314@gmail.com
> > > > > >
> > > > > > >> > wrote:
> > > > > > >> > > > >>
> > > > > > >> > > > >> > Hi Jing,
> > > > > > >> > > > >> >     Thanks for your response. As stated in the
> FLIP,
> > > the
> > > > > > >> purpose
> > > > > > >> > of
> > > > > > >> > > > this
> > > > > > >> > > > >> > FLIP is meant to support
> > > > > > >> > > > >> > user-defined async table function. As described in
> > > flink
> > > > > > >> document
> > > > > > >> > > [1]
> > > > > > >> > > > >> >
> > > > > > >> > > > >> > Async table functions are special functions for
> table
> > > > > sources
> > > > > > >> that
> > > > > > >> > > > >> perform
> > > > > > >> > > > >> > > a lookup.
> > > > > > >> > > > >> > >
> > > > > > >> > > > >> >
> > > > > > >> > > > >> > So end user can not directly define and use async
> > table
> > > > > > >> function
> > > > > > >> > > now.
> > > > > > >> > > > An
> > > > > > >> > > > >> > user case is reported in [2]
> > > > > > >> > > > >> >
> > > > > > >> > > > >> > So, in conclusion, no new interface is introduced,
> > but
> > > we
> > > > > > >> extend
> > > > > > >> > the
> > > > > > >> > > > >> > ability to support user-defined async table
> function.
> > > > > > >> > > > >> >
> > > > > > >> > > > >> > [1]:
> > > > > > >> > > > >> >
> > > > > > >> > > > >> >
> > > > > > >> > > > >>
> > > > > > >> > > >
> > > > > > >> > >
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/dev/table/functions/udfs/
> > > > > > >> > > > >> > [2]:
> > > > > > >> > >
> > > > https://lists.apache.org/thread/qljwd40v5ntz6733cwcdr8s4z97b343b
> > > > > > >> > > > >> >
> > > > > > >> > > > >> > Thanks.
> > > > > > >> > > > >> > Aitozi.
> > > > > > >> > > > >> >
> > > > > > >> > > > >> >
> > > > > > >> > > > >> > Jing Ge <ji...@ververica.com.invalid> 于2023年5月27日周六
> > > > > 06:40写道：
> > > > > > >> > > > >> >
> > > > > > >> > > > >> > > Hi Aitozi,
> > > > > > >> > > > >> > >
> > > > > > >> > > > >> > > Thanks for your proposal. I am not quite sure if
> I
> > > > > > understood
> > > > > > >> > your
> > > > > > >> > > > >> > thoughts
> > > > > > >> > > > >> > > correctly. You described a special case
> > > implementation
> > > > of
> > > > > > the
> > > > > > >> > > > >> > > AsyncTableFunction with on public API changes.
> > Would
> > > > you
> > > > > > >> please
> > > > > > >> > > > >> elaborate
> > > > > > >> > > > >> > > your purpose of writing a FLIP according to the
> > FLIP
> > > > > > >> > > > documentation[1]?
> > > > > > >> > > > >> > > Thanks!
> > > > > > >> > > > >> > >
> > > > > > >> > > > >> > > [1]
> > > > > > >> > > > >> > >
> > > > > > >> > > > >> > >
> > > > > > >> > > > >> >
> > > > > > >> > > > >>
> > > > > > >> > > >
> > > > > > >> > >
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/Flink+Improvement+Proposals
> > > > > > >> > > > >> > >
> > > > > > >> > > > >> > > Best regards,
> > > > > > >> > > > >> > > Jing
> > > > > > >> > > > >> > >
> > > > > > >> > > > >> > > On Wed, May 24, 2023 at 1:07 PM Aitozi <
> > > > > > gjying1314@gmail.com
> > > > > > >> >
> > > > > > >> > > > wrote:
> > > > > > >> > > > >> > >
> > > > > > >> > > > >> > > > May I ask for some feedback  :D
> > > > > > >> > > > >> > > >
> > > > > > >> > > > >> > > > Thanks,
> > > > > > >> > > > >> > > > Aitozi
> > > > > > >> > > > >> > > >
> > > > > > >> > > > >> > > > Aitozi <gj...@gmail.com> 于2023年5月23日周二
> > > 19:14写道：
> > > > > > >> > > > >> > > > >
> > > > > > >> > > > >> > > > > Just catch an user case report from Giannis
> > > Polyzos
> > > > > for
> > > > > > >> this
> > > > > > >> > > > >> usage:
> > > > > > >> > > > >> > > > >
> > > > > > >> > > > >> > > > >
> > > > > > >> > > >
> > > > > https://lists.apache.org/thread/qljwd40v5ntz6733cwcdr8s4z97b343b
> > > > > > >> > > > >> > > > >
> > > > > > >> > > > >> > > > > Aitozi <gj...@gmail.com> 于2023年5月23日周二
> > > > 17:45写道：
> > > > > > >> > > > >> > > > > >
> > > > > > >> > > > >> > > > > > Hi guys,
> > > > > > >> > > > >> > > > > >     I want to bring up a discussion about
> > > adding
> > > > > > >> support
> > > > > > >> > of
> > > > > > >> > > > User
> > > > > > >> > > > >> > > > > > Defined AsyncTableFunction in Flink.
> > > > > > >> > > > >> > > > > > Currently, async table function are special
> > > > > functions
> > > > > > >> for
> > > > > > >> > > > table
> > > > > > >> > > > >> > > source
> > > > > > >> > > > >> > > > > > to perform
> > > > > > >> > > > >> > > > > > async lookup. However, it's worth to
> support
> > > the
> > > > > user
> > > > > > >> > > defined
> > > > > > >> > > > >> async
> > > > > > >> > > > >> > > > > > table function.
> > > > > > >> > > > >> > > > > > Because, in this way, the end SQL user can
> > > > leverage
> > > > > > it
> > > > > > >> to
> > > > > > >> > > > >> perform
> > > > > > >> > > > >> > the
> > > > > > >> > > > >> > > > > > async operation
> > > > > > >> > > > >> > > > > > which is useful to maximum the system
> > > throughput
> > > > > > >> > especially
> > > > > > >> > > > for
> > > > > > >> > > > >> IO
> > > > > > >> > > > >> > > > > > bottleneck case.
> > > > > > >> > > > >> > > > > >
> > > > > > >> > > > >> > > > > > You can find some more detail in [1].
> > > > > > >> > > > >> > > > > >
> > > > > > >> > > > >> > > > > > Looking forward to feedback
> > > > > > >> > > > >> > > > > >
> > > > > > >> > > > >> > > > > >
> > > > > > >> > > > >> > > > > > [1]:
> > > > > > >> > > > >> > > >
> > > > > > >> > > > >> > >
> > > > > > >> > > > >> >
> > > > > > >> > > > >>
> > > > > > >> > > >
> > > > > > >> > >
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/%5BFLIP-313%5D+Add+support+of+User+Defined+AsyncTableFunction
> > > > > > >> > > > >> > > > > >
> > > > > > >> > > > >> > > > > > Thanks,
> > > > > > >> > > > >> > > > > > Aitozi.
> > > > > > >> > > > >> > > >
> > > > > > >> > > > >> > >
> > > > > > >> > > > >> >
> > > > > > >> > > > >>
> > > > > > >> > > > >
> > > > > > >> > > >
> > > > > > >> > >
> > > > > > >> >
> > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] FLIP-313 Add support of User Defined AsyncTableFunction

Posted by Jing Ge <ji...@ververica.com.INVALID>.

Hi Aitozi,

Thanks for the feedback. Looking forward to the performance tests.

Afaik, lookup returns one row for each key [1] [2]. Conceptually, the
lookup function is used to enrich column(s) from the dimension table. If,
for the given key, there will be more than one row, there will be no way to
know which row will be used to enrich the key.

[1]
https://github.com/apache/flink/blob/191ec6ca3943d7119f14837efe112e074d815c47/flink-table/flink-table-common/src/main/java/org/apache/flink/table/functions/LookupFunction.java#L49
[2]
https://github.com/apache/flink/blob/191ec6ca3943d7119f14837efe112e074d815c47/flink-table/flink-table-common/src/main/java/org/apache/flink/table/functions/TableFunction.java#L196

Best regards,
Jing

On Fri, Jun 9, 2023 at 5:18 AM Aitozi <gj...@gmail.com> wrote:

> Hi Jing
>     Thanks for your good questions. I have updated the example to the FLIP.
>
> > Only one row for each lookup
> lookup can also return multi rows, based on the query result. [1]
>
> [1]:
>
> https://github.com/apache/flink/blob/191ec6ca3943d7119f14837efe112e074d815c47/flink-table/flink-table-common/src/main/java/org/apache/flink/table/functions/LookupFunction.java#L56
>
> > If we use async calls with lateral join, my gut feeling is
> that we might have many more async calls than lookup join. I am not really
> sure if we will be facing potential issues in this case or not.
>
> IMO, the work pattern is similar to the lookup function, for each row from
> the left table,
> it will evaluate the eval method once, so the async call numbers will not
> change.
> and the maximum calls in flight is limited by the Async operators buffer
> capacity
> which will be controlled by the option.
>
> BTW, for the naming of these option, I updated the FLIP about this you can
> refer to
> the section of "ConfigOption" and "Rejected Alternatives"
>
> In the end, for the performance evaluation, I'd like to do some tests and
> will update it to the FLIP doc
>
> Thanks,
> Aitozi.
>
>
> Jing Ge <ji...@ververica.com.invalid> 于2023年6月9日周五 07:23写道：
>
> > Hi Aitozi,
> >
> > Thanks for the clarification. The code example looks interesting. I would
> > suggest adding them into the FLIP. The description with code examples
> will
> > help readers understand the motivation and how to use it. Afaiac, it is a
> > valid feature for Flink users.
> >
> > As we knew, lookup join is based on temporal join, i.e. FOR SYSTEM_TIME
> AS
> > OF which is also used in your code example. Temporal join performs the
> > lookup based on the processing time match. Only one row for each
> > lookup(afaiu, I need to check the source code to double confirm) will
> > return for further enrichment. One the other hand, lateral join will have
> > sub-queries correlated with every individual value of the reference table
> > from the preceding part of the query and each sub query will return
> > multiple rows. If we use async calls with lateral join, my gut feeling is
> > that we might have many more async calls than lookup join. I am not
> really
> > sure if we will be facing potential issues in this case or not. Possible
> > issues I can think of now e.g. too many PRC calls, too many async calls
> > processing, the sub query will return a table which might be (too) big,
> and
> > might cause performance issues. I would suggest preparing some use cases
> > and running some performance tests to check it. These are my concerns
> about
> > using async calls with lateral join and I'd like to share with you, happy
> > to discuss with you and hear different opinions, hopefully the
> > discussion could help me understand it more deeply. Please correct me if
> I
> > am wrong.
> >
> > Best regards,
> > Jing
> >
> >
> > On Thu, Jun 8, 2023 at 7:22 AM Aitozi <gj...@gmail.com> wrote:
> >
> > > Hi Mason,
> > >     Thanks for your input. I think if we support the user defined async
> > > table function,
> > > user will be able to use it to hold a batch data then handle it at one
> > time
> > > in the customized function.
> > >
> > > AsyncSink is meant for the sink operator. I have not figure out how to
> > > integrate in this case.
> > >
> > > Thanks,
> > > Atiozi.
> > >
> > >
> > > Mason Chen <ma...@gmail.com> 于2023年6月8日周四 12:40写道：
> > >
> > > > Hi Aitozi,
> > > >
> > > > I think it makes sense to make it easier for SQL users to make RPCs.
> Do
> > > you
> > > > think your proposal can extend to the ability to batch data for the
> > RPC?
> > > > This is also another common strategy to increase throughput. Also,
> have
> > > you
> > > > considered solving this a bit differently by leveraging Flink's
> > > AsyncSink?
> > > >
> > > > Best,
> > > > Mason
> > > >
> > > > On Mon, Jun 5, 2023 at 1:50 AM Aitozi <gj...@gmail.com> wrote:
> > > >
> > > > > One more thing for discussion:
> > > > >
> > > > > In our internal implementation, we reuse the option
> > > > > `table.exec.async-lookup.buffer-capacity` and
> > > > > `table.exec.async-lookup.timeout` to config
> > > > > the async udtf. Do you think we should add two extra option to
> > > > distinguish
> > > > > from the lookup option such as
> > > > >
> > > > > `table.exec.async-udtf.buffer-capacity`
> > > > > `table.exec.async-udtf.timeout`
> > > > >
> > > > >
> > > > > Best,
> > > > > Aitozi.
> > > > >
> > > > >
> > > > >
> > > > > Aitozi <gj...@gmail.com> 于2023年6月5日周一 12:20写道：
> > > > >
> > > > > > Hi Jing,
> > > > > >
> > > > > >     > what is the difference between the RPC call or query you
> > > > mentioned
> > > > > > and the lookup in a very
> > > > > > general way
> > > > > >
> > > > > > I think the RPC call or query service is quite similar to the
> > lookup
> > > > > join.
> > > > > > But lookup join should work
> > > > > > with `LookupTableSource`.
> > > > > >
> > > > > > Let's see how we can perform an async RPC call with lookup join:
> > > > > >
> > > > > > (1) Implement an AsyncTableFunction with RPC call logic.
> > > > > > (2) Implement a `LookupTableSource` connector run with the async
> > udtf
> > > > > > defined in (1).
> > > > > > (3) Then define a DDL of this look up table in SQL
> > > > > >
> > > > > > CREATE TEMPORARY TABLE Customers (
> > > > > >   id INT,
> > > > > >   name STRING,
> > > > > >   country STRING,
> > > > > >   zip STRING
> > > > > > ) WITH (
> > > > > >   'connector' = 'custom'
> > > > > > );
> > > > > >
> > > > > > (4) Run with the query as below:
> > > > > >
> > > > > > SELECT o.order_id, o.total, c.country, c.zip
> > > > > > FROM Orders AS o
> > > > > >   JOIN Customers FOR SYSTEM_TIME AS OF o.proc_time AS c
> > > > > >     ON o.customer_id = c.id;
> > > > > >
> > > > > > This example is from doc
> > > > > > <
> > > > >
> > > >
> > >
> >
> https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/dev/table/sql/queries/joins/#lookup-join
> > > > > >.You
> > > > > > can image the look up process as an async RPC call process.
> > > > > >
> > > > > > Let's see how we can perform an async RPC call with lateral join:
> > > > > >
> > > > > > (1) Implement an AsyncTableFunction with RPC call logic.
> > > > > > (2) Run query with
> > > > > >
> > > > > > Create function f1 as '...' ;
> > > > > >
> > > > > > SELECT o.order_id, o.total, c.country, c.zip FROM Orders  lateral
> > > table
> > > > > > (f1(order_id)) as T(...);
> > > > > >
> > > > > > As you can see, the lateral join version is more simple and
> > intuitive
> > > > to
> > > > > > users. Users do not have to wrap a
> > > > > > LookupTableSource for the purpose of using async udtf.
> > > > > >
> > > > > > In the end, We can also see the user defined async table function
> > is
> > > an
> > > > > > enhancement of the current lateral table join
> > > > > > which only supports sync lateral join now.
> > > > > >
> > > > > > Best,
> > > > > > Aitozi.
> > > > > >
> > > > > >
> > > > > > Jing Ge <ji...@ververica.com.invalid> 于2023年6月2日周五 19:37写道：
> > > > > >
> > > > > >> Hi Aitozi,
> > > > > >>
> > > > > >> Thanks for the update. Just out of curiosity, what is the
> > difference
> > > > > >> between the RPC call or query you mentioned and the lookup in a
> > very
> > > > > >> general way? Since Lateral join is used in the FLIP. Is there
> any
> > > > > special
> > > > > >> thought for that? Sorry for asking so many questions. The FLIP
> > > > contains
> > > > > >> limited information to understand the motivation.
> > > > > >>
> > > > > >> Best regards,
> > > > > >> Jing
> > > > > >>
> > > > > >> On Fri, Jun 2, 2023 at 3:48 AM Aitozi <gj...@gmail.com>
> > wrote:
> > > > > >>
> > > > > >> > Hi Jing,
> > > > > >> >     I have updated the proposed changes to the FLIP. IMO,
> lookup
> > > has
> > > > > its
> > > > > >> > clear
> > > > > >> > async call requirement is due to its IO heavy operator. In our
> > > > usage,
> > > > > >> sql
> > > > > >> > users have
> > > > > >> > logic to do some RPC call or query the third-party service
> which
> > > is
> > > > > >> also IO
> > > > > >> > intensive.
> > > > > >> > In these case, we'd like to leverage the async function to
> > improve
> > > > the
> > > > > >> > throughput.
> > > > > >> >
> > > > > >> > Thanks,
> > > > > >> > Aitozi.
> > > > > >> >
> > > > > >> > Jing Ge <ji...@ververica.com.invalid> 于2023年6月1日周四 22:55写道：
> > > > > >> >
> > > > > >> > > Hi Aitozi,
> > > > > >> > >
> > > > > >> > > Sorry for the late reply. Would you like to update the
> > proposed
> > > > > >> changes
> > > > > >> > > with more details into the FLIP too?
> > > > > >> > > I got your point. It looks like a rational idea. However,
> > since
> > > > > lookup
> > > > > >> > has
> > > > > >> > > its clear async call requirement, are there any real use
> cases
> > > > that
> > > > > >> > > need this change? This will help us understand the
> motivation.
> > > > After
> > > > > >> all,
> > > > > >> > > lateral join and temporal lookup join[1] are quite
> different.
> > > > > >> > >
> > > > > >> > > Best regards,
> > > > > >> > > Jing
> > > > > >> > >
> > > > > >> > >
> > > > > >> > > [1]
> > > > > >> > >
> > > > > >> > >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
> https://github.com/apache/flink/blob/d90a72da2fd601ca4e2a46700e91ec5b348de2ad/flink-table/flink-table-common/src/main/java/org/apache/flink/table/functions/AsyncTableFunction.java#L54
> > > > > >> > >
> > > > > >> > > On Wed, May 31, 2023 at 8:53 AM Aitozi <
> gjying1314@gmail.com>
> > > > > wrote:
> > > > > >> > >
> > > > > >> > > > Hi Jing,
> > > > > >> > > >     What do you think about it? Can we move forward this
> > > > feature?
> > > > > >> > > >
> > > > > >> > > > Thanks,
> > > > > >> > > > Aitozi.
> > > > > >> > > >
> > > > > >> > > > Aitozi <gj...@gmail.com> 于2023年5月29日周一 09:56写道：
> > > > > >> > > >
> > > > > >> > > > > Hi Jing,
> > > > > >> > > > >     > "Do you mean to support the AyncTableFunction
> beyond
> > > the
> > > > > >> > > > > LookupTableSource?"
> > > > > >> > > > > Yes, I mean to support the AyncTableFunction beyond the
> > > > > >> > > > LookupTableSource.
> > > > > >> > > > >
> > > > > >> > > > > The "AsyncTableFunction" is the function with ability to
> > be
> > > > > >> executed
> > > > > >> > > > async
> > > > > >> > > > > (with AsyncWaitOperator).
> > > > > >> > > > > The async lookup join is a one of usage of this. So, we
> > > don't
> > > > > >> have to
> > > > > >> > > > bind
> > > > > >> > > > > the AyncTableFunction with LookupTableSource.
> > > > > >> > > > > If User-defined AsyncTableFunction is supported, user
> can
> > > > > directly
> > > > > >> > use
> > > > > >> > > > > lateral table syntax to perform async operation.
> > > > > >> > > > >
> > > > > >> > > > > > "It would be better if you could elaborate the
> proposed
> > > > > changes
> > > > > >> wrt
> > > > > >> > > the
> > > > > >> > > > > CorrelatedCodeGenerator with more details"
> > > > > >> > > > >
> > > > > >> > > > > In the proposal, we use lateral table syntax to support
> > the
> > > > > async
> > > > > >> > table
> > > > > >> > > > > function. So the planner will also treat this statement
> > to a
> > > > > >> > > > > CommonExecCorrelate node. So the runtime code should be
> > > > > generated
> > > > > >> in
> > > > > >> > > > > CorrelatedCodeGenerator.
> > > > > >> > > > > In CorrelatedCodeGenerator, we will know the
> > TableFunction's
> > > > > Kind
> > > > > >> of
> > > > > >> > > > > `FunctionKind.Table` or `FunctionKind.ASYNC_TABLE`
> > > > > >> > > > > For  `FunctionKind.ASYNC_TABLE` we can generate a
> > > > > >> AsyncWaitOperator
> > > > > >> > to
> > > > > >> > > > > execute the async table function.
> > > > > >> > > > >
> > > > > >> > > > >
> > > > > >> > > > > Thanks,
> > > > > >> > > > > Aitozi.
> > > > > >> > > > >
> > > > > >> > > > >
> > > > > >> > > > > Jing Ge <ji...@ververica.com.invalid> 于2023年5月29日周一
> > 03:22写道：
> > > > > >> > > > >
> > > > > >> > > > >> Hi Aitozi,
> > > > > >> > > > >>
> > > > > >> > > > >> Thanks for the clarification. The naming "Lookup" might
> > > > suggest
> > > > > >> > using
> > > > > >> > > it
> > > > > >> > > > >> for table look up. But conceptually what the eval()
> > method
> > > > will
> > > > > >> do
> > > > > >> > is
> > > > > >> > > to
> > > > > >> > > > >> get a collection of results(Row, RowData) from the
> given
> > > > keys.
> > > > > >> How
> > > > > >> > it
> > > > > >> > > > will
> > > > > >> > > > >> be done depends on the implementation, i.e. you can
> > > implement
> > > > > >> your
> > > > > >> > own
> > > > > >> > > > >> Source[1][2]. The example in the FLIP should be able to
> > be
> > > > > >> handled
> > > > > >> > in
> > > > > >> > > > this
> > > > > >> > > > >> way.
> > > > > >> > > > >>
> > > > > >> > > > >> Do you mean to support the AyncTableFunction beyond the
> > > > > >> > > > LookupTableSource?
> > > > > >> > > > >> It would be better if you could elaborate the proposed
> > > > changes
> > > > > >> wrt
> > > > > >> > the
> > > > > >> > > > >> CorrelatedCodeGenerator with more details. Thanks!
> > > > > >> > > > >>
> > > > > >> > > > >> Best regards,
> > > > > >> > > > >> Jing
> > > > > >> > > > >>
> > > > > >> > > > >> [1]
> > > > > >> > > > >>
> > > > > >> > > > >>
> > > > > >> > > >
> > > > > >> > >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
> https://github.com/apache/flink/blob/678370b18e1b6c4a23e5ce08f8efd05675a0cc17/flink-table/flink-table-common/src/main/java/org/apache/flink/table/connector/source/LookupTableSource.java#L64
> > > > > >> > > > >> [2]
> > > > > >> > > > >>
> > > > > >> > > > >>
> > > > > >> > > >
> > > > > >> > >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
> https://github.com/apache/flink/blob/678370b18e1b6c4a23e5ce08f8efd05675a0cc17/flink-table/flink-table-common/src/main/java/org/apache/flink/table/connector/source/AsyncTableFunctionProvider.java#L49
> > > > > >> > > > >>
> > > > > >> > > > >> On Sat, May 27, 2023 at 9:48 AM Aitozi <
> > > gjying1314@gmail.com
> > > > >
> > > > > >> > wrote:
> > > > > >> > > > >>
> > > > > >> > > > >> > Hi Jing,
> > > > > >> > > > >> >     Thanks for your response. As stated in the FLIP,
> > the
> > > > > >> purpose
> > > > > >> > of
> > > > > >> > > > this
> > > > > >> > > > >> > FLIP is meant to support
> > > > > >> > > > >> > user-defined async table function. As described in
> > flink
> > > > > >> document
> > > > > >> > > [1]
> > > > > >> > > > >> >
> > > > > >> > > > >> > Async table functions are special functions for table
> > > > sources
> > > > > >> that
> > > > > >> > > > >> perform
> > > > > >> > > > >> > > a lookup.
> > > > > >> > > > >> > >
> > > > > >> > > > >> >
> > > > > >> > > > >> > So end user can not directly define and use async
> table
> > > > > >> function
> > > > > >> > > now.
> > > > > >> > > > An
> > > > > >> > > > >> > user case is reported in [2]
> > > > > >> > > > >> >
> > > > > >> > > > >> > So, in conclusion, no new interface is introduced,
> but
> > we
> > > > > >> extend
> > > > > >> > the
> > > > > >> > > > >> > ability to support user-defined async table function.
> > > > > >> > > > >> >
> > > > > >> > > > >> > [1]:
> > > > > >> > > > >> >
> > > > > >> > > > >> >
> > > > > >> > > > >>
> > > > > >> > > >
> > > > > >> > >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
> https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/dev/table/functions/udfs/
> > > > > >> > > > >> > [2]:
> > > > > >> > >
> > > https://lists.apache.org/thread/qljwd40v5ntz6733cwcdr8s4z97b343b
> > > > > >> > > > >> >
> > > > > >> > > > >> > Thanks.
> > > > > >> > > > >> > Aitozi.
> > > > > >> > > > >> >
> > > > > >> > > > >> >
> > > > > >> > > > >> > Jing Ge <ji...@ververica.com.invalid> 于2023年5月27日周六
> > > > 06:40写道：
> > > > > >> > > > >> >
> > > > > >> > > > >> > > Hi Aitozi,
> > > > > >> > > > >> > >
> > > > > >> > > > >> > > Thanks for your proposal. I am not quite sure if I
> > > > > understood
> > > > > >> > your
> > > > > >> > > > >> > thoughts
> > > > > >> > > > >> > > correctly. You described a special case
> > implementation
> > > of
> > > > > the
> > > > > >> > > > >> > > AsyncTableFunction with on public API changes.
> Would
> > > you
> > > > > >> please
> > > > > >> > > > >> elaborate
> > > > > >> > > > >> > > your purpose of writing a FLIP according to the
> FLIP
> > > > > >> > > > documentation[1]?
> > > > > >> > > > >> > > Thanks!
> > > > > >> > > > >> > >
> > > > > >> > > > >> > > [1]
> > > > > >> > > > >> > >
> > > > > >> > > > >> > >
> > > > > >> > > > >> >
> > > > > >> > > > >>
> > > > > >> > > >
> > > > > >> > >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/Flink+Improvement+Proposals
> > > > > >> > > > >> > >
> > > > > >> > > > >> > > Best regards,
> > > > > >> > > > >> > > Jing
> > > > > >> > > > >> > >
> > > > > >> > > > >> > > On Wed, May 24, 2023 at 1:07 PM Aitozi <
> > > > > gjying1314@gmail.com
> > > > > >> >
> > > > > >> > > > wrote:
> > > > > >> > > > >> > >
> > > > > >> > > > >> > > > May I ask for some feedback  :D
> > > > > >> > > > >> > > >
> > > > > >> > > > >> > > > Thanks,
> > > > > >> > > > >> > > > Aitozi
> > > > > >> > > > >> > > >
> > > > > >> > > > >> > > > Aitozi <gj...@gmail.com> 于2023年5月23日周二
> > 19:14写道：
> > > > > >> > > > >> > > > >
> > > > > >> > > > >> > > > > Just catch an user case report from Giannis
> > Polyzos
> > > > for
> > > > > >> this
> > > > > >> > > > >> usage:
> > > > > >> > > > >> > > > >
> > > > > >> > > > >> > > > >
> > > > > >> > > >
> > > > https://lists.apache.org/thread/qljwd40v5ntz6733cwcdr8s4z97b343b
> > > > > >> > > > >> > > > >
> > > > > >> > > > >> > > > > Aitozi <gj...@gmail.com> 于2023年5月23日周二
> > > 17:45写道：
> > > > > >> > > > >> > > > > >
> > > > > >> > > > >> > > > > > Hi guys,
> > > > > >> > > > >> > > > > >     I want to bring up a discussion about
> > adding
> > > > > >> support
> > > > > >> > of
> > > > > >> > > > User
> > > > > >> > > > >> > > > > > Defined AsyncTableFunction in Flink.
> > > > > >> > > > >> > > > > > Currently, async table function are special
> > > > functions
> > > > > >> for
> > > > > >> > > > table
> > > > > >> > > > >> > > source
> > > > > >> > > > >> > > > > > to perform
> > > > > >> > > > >> > > > > > async lookup. However, it's worth to support
> > the
> > > > user
> > > > > >> > > defined
> > > > > >> > > > >> async
> > > > > >> > > > >> > > > > > table function.
> > > > > >> > > > >> > > > > > Because, in this way, the end SQL user can
> > > leverage
> > > > > it
> > > > > >> to
> > > > > >> > > > >> perform
> > > > > >> > > > >> > the
> > > > > >> > > > >> > > > > > async operation
> > > > > >> > > > >> > > > > > which is useful to maximum the system
> > throughput
> > > > > >> > especially
> > > > > >> > > > for
> > > > > >> > > > >> IO
> > > > > >> > > > >> > > > > > bottleneck case.
> > > > > >> > > > >> > > > > >
> > > > > >> > > > >> > > > > > You can find some more detail in [1].
> > > > > >> > > > >> > > > > >
> > > > > >> > > > >> > > > > > Looking forward to feedback
> > > > > >> > > > >> > > > > >
> > > > > >> > > > >> > > > > >
> > > > > >> > > > >> > > > > > [1]:
> > > > > >> > > > >> > > >
> > > > > >> > > > >> > >
> > > > > >> > > > >> >
> > > > > >> > > > >>
> > > > > >> > > >
> > > > > >> > >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/%5BFLIP-313%5D+Add+support+of+User+Defined+AsyncTableFunction
> > > > > >> > > > >> > > > > >
> > > > > >> > > > >> > > > > > Thanks,
> > > > > >> > > > >> > > > > > Aitozi.
> > > > > >> > > > >> > > >
> > > > > >> > > > >> > >
> > > > > >> > > > >> >
> > > > > >> > > > >>
> > > > > >> > > > >
> > > > > >> > > >
> > > > > >> > >
> > > > > >> >
> > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] FLIP-313 Add support of User Defined AsyncTableFunction

Posted by Aitozi <gj...@gmail.com>.

Hi Feng,
    Thanks for your good question, It's very attractive if we can support
run the original
UDTF asynchronously without introducing new UDTFs.

But I think it's not easy, because the original UDTFs are executed one
instance per parallelism
So there is no thread-safe problem to user. But for the asynchronously
version, usually means
the main process is running in a dedicated thread pool. So if we run the
originally UDTFs in
asynchronously way, it may bring thread-safe problem.


@Jing Ge <ji...@ververica.com> Also add the "Performance" section in FLIP.

Thanks,
Aitozi.

Feng Jin <ji...@gmail.com> 于2023年6月9日周五 13:16写道：

> hi, Aitozi
>
> Thank you for your proposal.
>
> In our production environment, we often encounter efficiency issues with
> user-defined functions (UDFs), which can lead to slower processing speeds.
> I believe that this FLIP will make it easier for UDFs to be executed more
> efficiently.
>
>
> I have a small question:
>
> Is it possible for us to execute the original UDTF asynchronously without
> introducing new UDTFs?
> Of course, this is just my personal idea and I am not sure if it is a
> feasible solution.
>
>
> Best,
> Feng
>
>
>
> On Fri, Jun 9, 2023 at 11:18 AM Aitozi <gj...@gmail.com> wrote:
>
> > Hi Jing
> >     Thanks for your good questions. I have updated the example to the
> FLIP.
> >
> > > Only one row for each lookup
> > lookup can also return multi rows, based on the query result. [1]
> >
> > [1]:
> >
> >
> https://github.com/apache/flink/blob/191ec6ca3943d7119f14837efe112e074d815c47/flink-table/flink-table-common/src/main/java/org/apache/flink/table/functions/LookupFunction.java#L56
> >
> > > If we use async calls with lateral join, my gut feeling is
> > that we might have many more async calls than lookup join. I am not
> really
> > sure if we will be facing potential issues in this case or not.
> >
> > IMO, the work pattern is similar to the lookup function, for each row
> from
> > the left table,
> > it will evaluate the eval method once, so the async call numbers will not
> > change.
> > and the maximum calls in flight is limited by the Async operators buffer
> > capacity
> > which will be controlled by the option.
> >
> > BTW, for the naming of these option, I updated the FLIP about this you
> can
> > refer to
> > the section of "ConfigOption" and "Rejected Alternatives"
> >
> > In the end, for the performance evaluation, I'd like to do some tests and
> > will update it to the FLIP doc
> >
> > Thanks,
> > Aitozi.
> >
> >
> > Jing Ge <ji...@ververica.com.invalid> 于2023年6月9日周五 07:23写道：
> >
> > > Hi Aitozi,
> > >
> > > Thanks for the clarification. The code example looks interesting. I
> would
> > > suggest adding them into the FLIP. The description with code examples
> > will
> > > help readers understand the motivation and how to use it. Afaiac, it
> is a
> > > valid feature for Flink users.
> > >
> > > As we knew, lookup join is based on temporal join, i.e. FOR SYSTEM_TIME
> > AS
> > > OF which is also used in your code example. Temporal join performs the
> > > lookup based on the processing time match. Only one row for each
> > > lookup(afaiu, I need to check the source code to double confirm) will
> > > return for further enrichment. One the other hand, lateral join will
> have
> > > sub-queries correlated with every individual value of the reference
> table
> > > from the preceding part of the query and each sub query will return
> > > multiple rows. If we use async calls with lateral join, my gut feeling
> is
> > > that we might have many more async calls than lookup join. I am not
> > really
> > > sure if we will be facing potential issues in this case or not.
> Possible
> > > issues I can think of now e.g. too many PRC calls, too many async calls
> > > processing, the sub query will return a table which might be (too) big,
> > and
> > > might cause performance issues. I would suggest preparing some use
> cases
> > > and running some performance tests to check it. These are my concerns
> > about
> > > using async calls with lateral join and I'd like to share with you,
> happy
> > > to discuss with you and hear different opinions, hopefully the
> > > discussion could help me understand it more deeply. Please correct me
> if
> > I
> > > am wrong.
> > >
> > > Best regards,
> > > Jing
> > >
> > >
> > > On Thu, Jun 8, 2023 at 7:22 AM Aitozi <gj...@gmail.com> wrote:
> > >
> > > > Hi Mason,
> > > >     Thanks for your input. I think if we support the user defined
> async
> > > > table function,
> > > > user will be able to use it to hold a batch data then handle it at
> one
> > > time
> > > > in the customized function.
> > > >
> > > > AsyncSink is meant for the sink operator. I have not figure out how
> to
> > > > integrate in this case.
> > > >
> > > > Thanks,
> > > > Atiozi.
> > > >
> > > >
> > > > Mason Chen <ma...@gmail.com> 于2023年6月8日周四 12:40写道：
> > > >
> > > > > Hi Aitozi,
> > > > >
> > > > > I think it makes sense to make it easier for SQL users to make
> RPCs.
> > Do
> > > > you
> > > > > think your proposal can extend to the ability to batch data for the
> > > RPC?
> > > > > This is also another common strategy to increase throughput. Also,
> > have
> > > > you
> > > > > considered solving this a bit differently by leveraging Flink's
> > > > AsyncSink?
> > > > >
> > > > > Best,
> > > > > Mason
> > > > >
> > > > > On Mon, Jun 5, 2023 at 1:50 AM Aitozi <gj...@gmail.com>
> wrote:
> > > > >
> > > > > > One more thing for discussion:
> > > > > >
> > > > > > In our internal implementation, we reuse the option
> > > > > > `table.exec.async-lookup.buffer-capacity` and
> > > > > > `table.exec.async-lookup.timeout` to config
> > > > > > the async udtf. Do you think we should add two extra option to
> > > > > distinguish
> > > > > > from the lookup option such as
> > > > > >
> > > > > > `table.exec.async-udtf.buffer-capacity`
> > > > > > `table.exec.async-udtf.timeout`
> > > > > >
> > > > > >
> > > > > > Best,
> > > > > > Aitozi.
> > > > > >
> > > > > >
> > > > > >
> > > > > > Aitozi <gj...@gmail.com> 于2023年6月5日周一 12:20写道：
> > > > > >
> > > > > > > Hi Jing,
> > > > > > >
> > > > > > >     > what is the difference between the RPC call or query you
> > > > > mentioned
> > > > > > > and the lookup in a very
> > > > > > > general way
> > > > > > >
> > > > > > > I think the RPC call or query service is quite similar to the
> > > lookup
> > > > > > join.
> > > > > > > But lookup join should work
> > > > > > > with `LookupTableSource`.
> > > > > > >
> > > > > > > Let's see how we can perform an async RPC call with lookup
> join:
> > > > > > >
> > > > > > > (1) Implement an AsyncTableFunction with RPC call logic.
> > > > > > > (2) Implement a `LookupTableSource` connector run with the
> async
> > > udtf
> > > > > > > defined in (1).
> > > > > > > (3) Then define a DDL of this look up table in SQL
> > > > > > >
> > > > > > > CREATE TEMPORARY TABLE Customers (
> > > > > > >   id INT,
> > > > > > >   name STRING,
> > > > > > >   country STRING,
> > > > > > >   zip STRING
> > > > > > > ) WITH (
> > > > > > >   'connector' = 'custom'
> > > > > > > );
> > > > > > >
> > > > > > > (4) Run with the query as below:
> > > > > > >
> > > > > > > SELECT o.order_id, o.total, c.country, c.zip
> > > > > > > FROM Orders AS o
> > > > > > >   JOIN Customers FOR SYSTEM_TIME AS OF o.proc_time AS c
> > > > > > >     ON o.customer_id = c.id;
> > > > > > >
> > > > > > > This example is from doc
> > > > > > > <
> > > > > >
> > > > >
> > > >
> > >
> >
> https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/dev/table/sql/queries/joins/#lookup-join
> > > > > > >.You
> > > > > > > can image the look up process as an async RPC call process.
> > > > > > >
> > > > > > > Let's see how we can perform an async RPC call with lateral
> join:
> > > > > > >
> > > > > > > (1) Implement an AsyncTableFunction with RPC call logic.
> > > > > > > (2) Run query with
> > > > > > >
> > > > > > > Create function f1 as '...' ;
> > > > > > >
> > > > > > > SELECT o.order_id, o.total, c.country, c.zip FROM Orders
> lateral
> > > > table
> > > > > > > (f1(order_id)) as T(...);
> > > > > > >
> > > > > > > As you can see, the lateral join version is more simple and
> > > intuitive
> > > > > to
> > > > > > > users. Users do not have to wrap a
> > > > > > > LookupTableSource for the purpose of using async udtf.
> > > > > > >
> > > > > > > In the end, We can also see the user defined async table
> function
> > > is
> > > > an
> > > > > > > enhancement of the current lateral table join
> > > > > > > which only supports sync lateral join now.
> > > > > > >
> > > > > > > Best,
> > > > > > > Aitozi.
> > > > > > >
> > > > > > >
> > > > > > > Jing Ge <ji...@ververica.com.invalid> 于2023年6月2日周五 19:37写道：
> > > > > > >
> > > > > > >> Hi Aitozi,
> > > > > > >>
> > > > > > >> Thanks for the update. Just out of curiosity, what is the
> > > difference
> > > > > > >> between the RPC call or query you mentioned and the lookup in
> a
> > > very
> > > > > > >> general way? Since Lateral join is used in the FLIP. Is there
> > any
> > > > > > special
> > > > > > >> thought for that? Sorry for asking so many questions. The FLIP
> > > > > contains
> > > > > > >> limited information to understand the motivation.
> > > > > > >>
> > > > > > >> Best regards,
> > > > > > >> Jing
> > > > > > >>
> > > > > > >> On Fri, Jun 2, 2023 at 3:48 AM Aitozi <gj...@gmail.com>
> > > wrote:
> > > > > > >>
> > > > > > >> > Hi Jing,
> > > > > > >> >     I have updated the proposed changes to the FLIP. IMO,
> > lookup
> > > > has
> > > > > > its
> > > > > > >> > clear
> > > > > > >> > async call requirement is due to its IO heavy operator. In
> our
> > > > > usage,
> > > > > > >> sql
> > > > > > >> > users have
> > > > > > >> > logic to do some RPC call or query the third-party service
> > which
> > > > is
> > > > > > >> also IO
> > > > > > >> > intensive.
> > > > > > >> > In these case, we'd like to leverage the async function to
> > > improve
> > > > > the
> > > > > > >> > throughput.
> > > > > > >> >
> > > > > > >> > Thanks,
> > > > > > >> > Aitozi.
> > > > > > >> >
> > > > > > >> > Jing Ge <ji...@ververica.com.invalid> 于2023年6月1日周四 22:55写道：
> > > > > > >> >
> > > > > > >> > > Hi Aitozi,
> > > > > > >> > >
> > > > > > >> > > Sorry for the late reply. Would you like to update the
> > > proposed
> > > > > > >> changes
> > > > > > >> > > with more details into the FLIP too?
> > > > > > >> > > I got your point. It looks like a rational idea. However,
> > > since
> > > > > > lookup
> > > > > > >> > has
> > > > > > >> > > its clear async call requirement, are there any real use
> > cases
> > > > > that
> > > > > > >> > > need this change? This will help us understand the
> > motivation.
> > > > > After
> > > > > > >> all,
> > > > > > >> > > lateral join and temporal lookup join[1] are quite
> > different.
> > > > > > >> > >
> > > > > > >> > > Best regards,
> > > > > > >> > > Jing
> > > > > > >> > >
> > > > > > >> > >
> > > > > > >> > > [1]
> > > > > > >> > >
> > > > > > >> > >
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/flink/blob/d90a72da2fd601ca4e2a46700e91ec5b348de2ad/flink-table/flink-table-common/src/main/java/org/apache/flink/table/functions/AsyncTableFunction.java#L54
> > > > > > >> > >
> > > > > > >> > > On Wed, May 31, 2023 at 8:53 AM Aitozi <
> > gjying1314@gmail.com>
> > > > > > wrote:
> > > > > > >> > >
> > > > > > >> > > > Hi Jing,
> > > > > > >> > > >     What do you think about it? Can we move forward this
> > > > > feature?
> > > > > > >> > > >
> > > > > > >> > > > Thanks,
> > > > > > >> > > > Aitozi.
> > > > > > >> > > >
> > > > > > >> > > > Aitozi <gj...@gmail.com> 于2023年5月29日周一 09:56写道：
> > > > > > >> > > >
> > > > > > >> > > > > Hi Jing,
> > > > > > >> > > > >     > "Do you mean to support the AyncTableFunction
> > beyond
> > > > the
> > > > > > >> > > > > LookupTableSource?"
> > > > > > >> > > > > Yes, I mean to support the AyncTableFunction beyond
> the
> > > > > > >> > > > LookupTableSource.
> > > > > > >> > > > >
> > > > > > >> > > > > The "AsyncTableFunction" is the function with ability
> to
> > > be
> > > > > > >> executed
> > > > > > >> > > > async
> > > > > > >> > > > > (with AsyncWaitOperator).
> > > > > > >> > > > > The async lookup join is a one of usage of this. So,
> we
> > > > don't
> > > > > > >> have to
> > > > > > >> > > > bind
> > > > > > >> > > > > the AyncTableFunction with LookupTableSource.
> > > > > > >> > > > > If User-defined AsyncTableFunction is supported, user
> > can
> > > > > > directly
> > > > > > >> > use
> > > > > > >> > > > > lateral table syntax to perform async operation.
> > > > > > >> > > > >
> > > > > > >> > > > > > "It would be better if you could elaborate the
> > proposed
> > > > > > changes
> > > > > > >> wrt
> > > > > > >> > > the
> > > > > > >> > > > > CorrelatedCodeGenerator with more details"
> > > > > > >> > > > >
> > > > > > >> > > > > In the proposal, we use lateral table syntax to
> support
> > > the
> > > > > > async
> > > > > > >> > table
> > > > > > >> > > > > function. So the planner will also treat this
> statement
> > > to a
> > > > > > >> > > > > CommonExecCorrelate node. So the runtime code should
> be
> > > > > > generated
> > > > > > >> in
> > > > > > >> > > > > CorrelatedCodeGenerator.
> > > > > > >> > > > > In CorrelatedCodeGenerator, we will know the
> > > TableFunction's
> > > > > > Kind
> > > > > > >> of
> > > > > > >> > > > > `FunctionKind.Table` or `FunctionKind.ASYNC_TABLE`
> > > > > > >> > > > > For  `FunctionKind.ASYNC_TABLE` we can generate a
> > > > > > >> AsyncWaitOperator
> > > > > > >> > to
> > > > > > >> > > > > execute the async table function.
> > > > > > >> > > > >
> > > > > > >> > > > >
> > > > > > >> > > > > Thanks,
> > > > > > >> > > > > Aitozi.
> > > > > > >> > > > >
> > > > > > >> > > > >
> > > > > > >> > > > > Jing Ge <ji...@ververica.com.invalid> 于2023年5月29日周一
> > > 03:22写道：
> > > > > > >> > > > >
> > > > > > >> > > > >> Hi Aitozi,
> > > > > > >> > > > >>
> > > > > > >> > > > >> Thanks for the clarification. The naming "Lookup"
> might
> > > > > suggest
> > > > > > >> > using
> > > > > > >> > > it
> > > > > > >> > > > >> for table look up. But conceptually what the eval()
> > > method
> > > > > will
> > > > > > >> do
> > > > > > >> > is
> > > > > > >> > > to
> > > > > > >> > > > >> get a collection of results(Row, RowData) from the
> > given
> > > > > keys.
> > > > > > >> How
> > > > > > >> > it
> > > > > > >> > > > will
> > > > > > >> > > > >> be done depends on the implementation, i.e. you can
> > > > implement
> > > > > > >> your
> > > > > > >> > own
> > > > > > >> > > > >> Source[1][2]. The example in the FLIP should be able
> to
> > > be
> > > > > > >> handled
> > > > > > >> > in
> > > > > > >> > > > this
> > > > > > >> > > > >> way.
> > > > > > >> > > > >>
> > > > > > >> > > > >> Do you mean to support the AyncTableFunction beyond
> the
> > > > > > >> > > > LookupTableSource?
> > > > > > >> > > > >> It would be better if you could elaborate the
> proposed
> > > > > changes
> > > > > > >> wrt
> > > > > > >> > the
> > > > > > >> > > > >> CorrelatedCodeGenerator with more details. Thanks!
> > > > > > >> > > > >>
> > > > > > >> > > > >> Best regards,
> > > > > > >> > > > >> Jing
> > > > > > >> > > > >>
> > > > > > >> > > > >> [1]
> > > > > > >> > > > >>
> > > > > > >> > > > >>
> > > > > > >> > > >
> > > > > > >> > >
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/flink/blob/678370b18e1b6c4a23e5ce08f8efd05675a0cc17/flink-table/flink-table-common/src/main/java/org/apache/flink/table/connector/source/LookupTableSource.java#L64
> > > > > > >> > > > >> [2]
> > > > > > >> > > > >>
> > > > > > >> > > > >>
> > > > > > >> > > >
> > > > > > >> > >
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/flink/blob/678370b18e1b6c4a23e5ce08f8efd05675a0cc17/flink-table/flink-table-common/src/main/java/org/apache/flink/table/connector/source/AsyncTableFunctionProvider.java#L49
> > > > > > >> > > > >>
> > > > > > >> > > > >> On Sat, May 27, 2023 at 9:48 AM Aitozi <
> > > > gjying1314@gmail.com
> > > > > >
> > > > > > >> > wrote:
> > > > > > >> > > > >>
> > > > > > >> > > > >> > Hi Jing,
> > > > > > >> > > > >> >     Thanks for your response. As stated in the
> FLIP,
> > > the
> > > > > > >> purpose
> > > > > > >> > of
> > > > > > >> > > > this
> > > > > > >> > > > >> > FLIP is meant to support
> > > > > > >> > > > >> > user-defined async table function. As described in
> > > flink
> > > > > > >> document
> > > > > > >> > > [1]
> > > > > > >> > > > >> >
> > > > > > >> > > > >> > Async table functions are special functions for
> table
> > > > > sources
> > > > > > >> that
> > > > > > >> > > > >> perform
> > > > > > >> > > > >> > > a lookup.
> > > > > > >> > > > >> > >
> > > > > > >> > > > >> >
> > > > > > >> > > > >> > So end user can not directly define and use async
> > table
> > > > > > >> function
> > > > > > >> > > now.
> > > > > > >> > > > An
> > > > > > >> > > > >> > user case is reported in [2]
> > > > > > >> > > > >> >
> > > > > > >> > > > >> > So, in conclusion, no new interface is introduced,
> > but
> > > we
> > > > > > >> extend
> > > > > > >> > the
> > > > > > >> > > > >> > ability to support user-defined async table
> function.
> > > > > > >> > > > >> >
> > > > > > >> > > > >> > [1]:
> > > > > > >> > > > >> >
> > > > > > >> > > > >> >
> > > > > > >> > > > >>
> > > > > > >> > > >
> > > > > > >> > >
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/dev/table/functions/udfs/
> > > > > > >> > > > >> > [2]:
> > > > > > >> > >
> > > > https://lists.apache.org/thread/qljwd40v5ntz6733cwcdr8s4z97b343b
> > > > > > >> > > > >> >
> > > > > > >> > > > >> > Thanks.
> > > > > > >> > > > >> > Aitozi.
> > > > > > >> > > > >> >
> > > > > > >> > > > >> >
> > > > > > >> > > > >> > Jing Ge <ji...@ververica.com.invalid> 于2023年5月27日周六
> > > > > 06:40写道：
> > > > > > >> > > > >> >
> > > > > > >> > > > >> > > Hi Aitozi,
> > > > > > >> > > > >> > >
> > > > > > >> > > > >> > > Thanks for your proposal. I am not quite sure if
> I
> > > > > > understood
> > > > > > >> > your
> > > > > > >> > > > >> > thoughts
> > > > > > >> > > > >> > > correctly. You described a special case
> > > implementation
> > > > of
> > > > > > the
> > > > > > >> > > > >> > > AsyncTableFunction with on public API changes.
> > Would
> > > > you
> > > > > > >> please
> > > > > > >> > > > >> elaborate
> > > > > > >> > > > >> > > your purpose of writing a FLIP according to the
> > FLIP
> > > > > > >> > > > documentation[1]?
> > > > > > >> > > > >> > > Thanks!
> > > > > > >> > > > >> > >
> > > > > > >> > > > >> > > [1]
> > > > > > >> > > > >> > >
> > > > > > >> > > > >> > >
> > > > > > >> > > > >> >
> > > > > > >> > > > >>
> > > > > > >> > > >
> > > > > > >> > >
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/Flink+Improvement+Proposals
> > > > > > >> > > > >> > >
> > > > > > >> > > > >> > > Best regards,
> > > > > > >> > > > >> > > Jing
> > > > > > >> > > > >> > >
> > > > > > >> > > > >> > > On Wed, May 24, 2023 at 1:07 PM Aitozi <
> > > > > > gjying1314@gmail.com
> > > > > > >> >
> > > > > > >> > > > wrote:
> > > > > > >> > > > >> > >
> > > > > > >> > > > >> > > > May I ask for some feedback  :D
> > > > > > >> > > > >> > > >
> > > > > > >> > > > >> > > > Thanks,
> > > > > > >> > > > >> > > > Aitozi
> > > > > > >> > > > >> > > >
> > > > > > >> > > > >> > > > Aitozi <gj...@gmail.com> 于2023年5月23日周二
> > > 19:14写道：
> > > > > > >> > > > >> > > > >
> > > > > > >> > > > >> > > > > Just catch an user case report from Giannis
> > > Polyzos
> > > > > for
> > > > > > >> this
> > > > > > >> > > > >> usage:
> > > > > > >> > > > >> > > > >
> > > > > > >> > > > >> > > > >
> > > > > > >> > > >
> > > > > https://lists.apache.org/thread/qljwd40v5ntz6733cwcdr8s4z97b343b
> > > > > > >> > > > >> > > > >
> > > > > > >> > > > >> > > > > Aitozi <gj...@gmail.com> 于2023年5月23日周二
> > > > 17:45写道：
> > > > > > >> > > > >> > > > > >
> > > > > > >> > > > >> > > > > > Hi guys,
> > > > > > >> > > > >> > > > > >     I want to bring up a discussion about
> > > adding
> > > > > > >> support
> > > > > > >> > of
> > > > > > >> > > > User
> > > > > > >> > > > >> > > > > > Defined AsyncTableFunction in Flink.
> > > > > > >> > > > >> > > > > > Currently, async table function are special
> > > > > functions
> > > > > > >> for
> > > > > > >> > > > table
> > > > > > >> > > > >> > > source
> > > > > > >> > > > >> > > > > > to perform
> > > > > > >> > > > >> > > > > > async lookup. However, it's worth to
> support
> > > the
> > > > > user
> > > > > > >> > > defined
> > > > > > >> > > > >> async
> > > > > > >> > > > >> > > > > > table function.
> > > > > > >> > > > >> > > > > > Because, in this way, the end SQL user can
> > > > leverage
> > > > > > it
> > > > > > >> to
> > > > > > >> > > > >> perform
> > > > > > >> > > > >> > the
> > > > > > >> > > > >> > > > > > async operation
> > > > > > >> > > > >> > > > > > which is useful to maximum the system
> > > throughput
> > > > > > >> > especially
> > > > > > >> > > > for
> > > > > > >> > > > >> IO
> > > > > > >> > > > >> > > > > > bottleneck case.
> > > > > > >> > > > >> > > > > >
> > > > > > >> > > > >> > > > > > You can find some more detail in [1].
> > > > > > >> > > > >> > > > > >
> > > > > > >> > > > >> > > > > > Looking forward to feedback
> > > > > > >> > > > >> > > > > >
> > > > > > >> > > > >> > > > > >
> > > > > > >> > > > >> > > > > > [1]:
> > > > > > >> > > > >> > > >
> > > > > > >> > > > >> > >
> > > > > > >> > > > >> >
> > > > > > >> > > > >>
> > > > > > >> > > >
> > > > > > >> > >
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/%5BFLIP-313%5D+Add+support+of+User+Defined+AsyncTableFunction
> > > > > > >> > > > >> > > > > >
> > > > > > >> > > > >> > > > > > Thanks,
> > > > > > >> > > > >> > > > > > Aitozi.
> > > > > > >> > > > >> > > >
> > > > > > >> > > > >> > >
> > > > > > >> > > > >> >
> > > > > > >> > > > >>
> > > > > > >> > > > >
> > > > > > >> > > >
> > > > > > >> > >
> > > > > > >> >
> > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] FLIP-313 Add support of User Defined AsyncTableFunction

Posted by Feng Jin <ji...@gmail.com>.

hi, Aitozi

Thank you for your proposal.

In our production environment, we often encounter efficiency issues with
user-defined functions (UDFs), which can lead to slower processing speeds.
I believe that this FLIP will make it easier for UDFs to be executed more
efficiently.


I have a small question:

Is it possible for us to execute the original UDTF asynchronously without
introducing new UDTFs?
Of course, this is just my personal idea and I am not sure if it is a
feasible solution.


Best,
Feng



On Fri, Jun 9, 2023 at 11:18 AM Aitozi <gj...@gmail.com> wrote:

> Hi Jing
>     Thanks for your good questions. I have updated the example to the FLIP.
>
> > Only one row for each lookup
> lookup can also return multi rows, based on the query result. [1]
>
> [1]:
>
> https://github.com/apache/flink/blob/191ec6ca3943d7119f14837efe112e074d815c47/flink-table/flink-table-common/src/main/java/org/apache/flink/table/functions/LookupFunction.java#L56
>
> > If we use async calls with lateral join, my gut feeling is
> that we might have many more async calls than lookup join. I am not really
> sure if we will be facing potential issues in this case or not.
>
> IMO, the work pattern is similar to the lookup function, for each row from
> the left table,
> it will evaluate the eval method once, so the async call numbers will not
> change.
> and the maximum calls in flight is limited by the Async operators buffer
> capacity
> which will be controlled by the option.
>
> BTW, for the naming of these option, I updated the FLIP about this you can
> refer to
> the section of "ConfigOption" and "Rejected Alternatives"
>
> In the end, for the performance evaluation, I'd like to do some tests and
> will update it to the FLIP doc
>
> Thanks,
> Aitozi.
>
>
> Jing Ge <ji...@ververica.com.invalid> 于2023年6月9日周五 07:23写道：
>
> > Hi Aitozi,
> >
> > Thanks for the clarification. The code example looks interesting. I would
> > suggest adding them into the FLIP. The description with code examples
> will
> > help readers understand the motivation and how to use it. Afaiac, it is a
> > valid feature for Flink users.
> >
> > As we knew, lookup join is based on temporal join, i.e. FOR SYSTEM_TIME
> AS
> > OF which is also used in your code example. Temporal join performs the
> > lookup based on the processing time match. Only one row for each
> > lookup(afaiu, I need to check the source code to double confirm) will
> > return for further enrichment. One the other hand, lateral join will have
> > sub-queries correlated with every individual value of the reference table
> > from the preceding part of the query and each sub query will return
> > multiple rows. If we use async calls with lateral join, my gut feeling is
> > that we might have many more async calls than lookup join. I am not
> really
> > sure if we will be facing potential issues in this case or not. Possible
> > issues I can think of now e.g. too many PRC calls, too many async calls
> > processing, the sub query will return a table which might be (too) big,
> and
> > might cause performance issues. I would suggest preparing some use cases
> > and running some performance tests to check it. These are my concerns
> about
> > using async calls with lateral join and I'd like to share with you, happy
> > to discuss with you and hear different opinions, hopefully the
> > discussion could help me understand it more deeply. Please correct me if
> I
> > am wrong.
> >
> > Best regards,
> > Jing
> >
> >
> > On Thu, Jun 8, 2023 at 7:22 AM Aitozi <gj...@gmail.com> wrote:
> >
> > > Hi Mason,
> > >     Thanks for your input. I think if we support the user defined async
> > > table function,
> > > user will be able to use it to hold a batch data then handle it at one
> > time
> > > in the customized function.
> > >
> > > AsyncSink is meant for the sink operator. I have not figure out how to
> > > integrate in this case.
> > >
> > > Thanks,
> > > Atiozi.
> > >
> > >
> > > Mason Chen <ma...@gmail.com> 于2023年6月8日周四 12:40写道：
> > >
> > > > Hi Aitozi,
> > > >
> > > > I think it makes sense to make it easier for SQL users to make RPCs.
> Do
> > > you
> > > > think your proposal can extend to the ability to batch data for the
> > RPC?
> > > > This is also another common strategy to increase throughput. Also,
> have
> > > you
> > > > considered solving this a bit differently by leveraging Flink's
> > > AsyncSink?
> > > >
> > > > Best,
> > > > Mason
> > > >
> > > > On Mon, Jun 5, 2023 at 1:50 AM Aitozi <gj...@gmail.com> wrote:
> > > >
> > > > > One more thing for discussion:
> > > > >
> > > > > In our internal implementation, we reuse the option
> > > > > `table.exec.async-lookup.buffer-capacity` and
> > > > > `table.exec.async-lookup.timeout` to config
> > > > > the async udtf. Do you think we should add two extra option to
> > > > distinguish
> > > > > from the lookup option such as
> > > > >
> > > > > `table.exec.async-udtf.buffer-capacity`
> > > > > `table.exec.async-udtf.timeout`
> > > > >
> > > > >
> > > > > Best,
> > > > > Aitozi.
> > > > >
> > > > >
> > > > >
> > > > > Aitozi <gj...@gmail.com> 于2023年6月5日周一 12:20写道：
> > > > >
> > > > > > Hi Jing,
> > > > > >
> > > > > >     > what is the difference between the RPC call or query you
> > > > mentioned
> > > > > > and the lookup in a very
> > > > > > general way
> > > > > >
> > > > > > I think the RPC call or query service is quite similar to the
> > lookup
> > > > > join.
> > > > > > But lookup join should work
> > > > > > with `LookupTableSource`.
> > > > > >
> > > > > > Let's see how we can perform an async RPC call with lookup join:
> > > > > >
> > > > > > (1) Implement an AsyncTableFunction with RPC call logic.
> > > > > > (2) Implement a `LookupTableSource` connector run with the async
> > udtf
> > > > > > defined in (1).
> > > > > > (3) Then define a DDL of this look up table in SQL
> > > > > >
> > > > > > CREATE TEMPORARY TABLE Customers (
> > > > > >   id INT,
> > > > > >   name STRING,
> > > > > >   country STRING,
> > > > > >   zip STRING
> > > > > > ) WITH (
> > > > > >   'connector' = 'custom'
> > > > > > );
> > > > > >
> > > > > > (4) Run with the query as below:
> > > > > >
> > > > > > SELECT o.order_id, o.total, c.country, c.zip
> > > > > > FROM Orders AS o
> > > > > >   JOIN Customers FOR SYSTEM_TIME AS OF o.proc_time AS c
> > > > > >     ON o.customer_id = c.id;
> > > > > >
> > > > > > This example is from doc
> > > > > > <
> > > > >
> > > >
> > >
> >
> https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/dev/table/sql/queries/joins/#lookup-join
> > > > > >.You
> > > > > > can image the look up process as an async RPC call process.
> > > > > >
> > > > > > Let's see how we can perform an async RPC call with lateral join:
> > > > > >
> > > > > > (1) Implement an AsyncTableFunction with RPC call logic.
> > > > > > (2) Run query with
> > > > > >
> > > > > > Create function f1 as '...' ;
> > > > > >
> > > > > > SELECT o.order_id, o.total, c.country, c.zip FROM Orders  lateral
> > > table
> > > > > > (f1(order_id)) as T(...);
> > > > > >
> > > > > > As you can see, the lateral join version is more simple and
> > intuitive
> > > > to
> > > > > > users. Users do not have to wrap a
> > > > > > LookupTableSource for the purpose of using async udtf.
> > > > > >
> > > > > > In the end, We can also see the user defined async table function
> > is
> > > an
> > > > > > enhancement of the current lateral table join
> > > > > > which only supports sync lateral join now.
> > > > > >
> > > > > > Best,
> > > > > > Aitozi.
> > > > > >
> > > > > >
> > > > > > Jing Ge <ji...@ververica.com.invalid> 于2023年6月2日周五 19:37写道：
> > > > > >
> > > > > >> Hi Aitozi,
> > > > > >>
> > > > > >> Thanks for the update. Just out of curiosity, what is the
> > difference
> > > > > >> between the RPC call or query you mentioned and the lookup in a
> > very
> > > > > >> general way? Since Lateral join is used in the FLIP. Is there
> any
> > > > > special
> > > > > >> thought for that? Sorry for asking so many questions. The FLIP
> > > > contains
> > > > > >> limited information to understand the motivation.
> > > > > >>
> > > > > >> Best regards,
> > > > > >> Jing
> > > > > >>
> > > > > >> On Fri, Jun 2, 2023 at 3:48 AM Aitozi <gj...@gmail.com>
> > wrote:
> > > > > >>
> > > > > >> > Hi Jing,
> > > > > >> >     I have updated the proposed changes to the FLIP. IMO,
> lookup
> > > has
> > > > > its
> > > > > >> > clear
> > > > > >> > async call requirement is due to its IO heavy operator. In our
> > > > usage,
> > > > > >> sql
> > > > > >> > users have
> > > > > >> > logic to do some RPC call or query the third-party service
> which
> > > is
> > > > > >> also IO
> > > > > >> > intensive.
> > > > > >> > In these case, we'd like to leverage the async function to
> > improve
> > > > the
> > > > > >> > throughput.
> > > > > >> >
> > > > > >> > Thanks,
> > > > > >> > Aitozi.
> > > > > >> >
> > > > > >> > Jing Ge <ji...@ververica.com.invalid> 于2023年6月1日周四 22:55写道：
> > > > > >> >
> > > > > >> > > Hi Aitozi,
> > > > > >> > >
> > > > > >> > > Sorry for the late reply. Would you like to update the
> > proposed
> > > > > >> changes
> > > > > >> > > with more details into the FLIP too?
> > > > > >> > > I got your point. It looks like a rational idea. However,
> > since
> > > > > lookup
> > > > > >> > has
> > > > > >> > > its clear async call requirement, are there any real use
> cases
> > > > that
> > > > > >> > > need this change? This will help us understand the
> motivation.
> > > > After
> > > > > >> all,
> > > > > >> > > lateral join and temporal lookup join[1] are quite
> different.
> > > > > >> > >
> > > > > >> > > Best regards,
> > > > > >> > > Jing
> > > > > >> > >
> > > > > >> > >
> > > > > >> > > [1]
> > > > > >> > >
> > > > > >> > >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
> https://github.com/apache/flink/blob/d90a72da2fd601ca4e2a46700e91ec5b348de2ad/flink-table/flink-table-common/src/main/java/org/apache/flink/table/functions/AsyncTableFunction.java#L54
> > > > > >> > >
> > > > > >> > > On Wed, May 31, 2023 at 8:53 AM Aitozi <
> gjying1314@gmail.com>
> > > > > wrote:
> > > > > >> > >
> > > > > >> > > > Hi Jing,
> > > > > >> > > >     What do you think about it? Can we move forward this
> > > > feature?
> > > > > >> > > >
> > > > > >> > > > Thanks,
> > > > > >> > > > Aitozi.
> > > > > >> > > >
> > > > > >> > > > Aitozi <gj...@gmail.com> 于2023年5月29日周一 09:56写道：
> > > > > >> > > >
> > > > > >> > > > > Hi Jing,
> > > > > >> > > > >     > "Do you mean to support the AyncTableFunction
> beyond
> > > the
> > > > > >> > > > > LookupTableSource?"
> > > > > >> > > > > Yes, I mean to support the AyncTableFunction beyond the
> > > > > >> > > > LookupTableSource.
> > > > > >> > > > >
> > > > > >> > > > > The "AsyncTableFunction" is the function with ability to
> > be
> > > > > >> executed
> > > > > >> > > > async
> > > > > >> > > > > (with AsyncWaitOperator).
> > > > > >> > > > > The async lookup join is a one of usage of this. So, we
> > > don't
> > > > > >> have to
> > > > > >> > > > bind
> > > > > >> > > > > the AyncTableFunction with LookupTableSource.
> > > > > >> > > > > If User-defined AsyncTableFunction is supported, user
> can
> > > > > directly
> > > > > >> > use
> > > > > >> > > > > lateral table syntax to perform async operation.
> > > > > >> > > > >
> > > > > >> > > > > > "It would be better if you could elaborate the
> proposed
> > > > > changes
> > > > > >> wrt
> > > > > >> > > the
> > > > > >> > > > > CorrelatedCodeGenerator with more details"
> > > > > >> > > > >
> > > > > >> > > > > In the proposal, we use lateral table syntax to support
> > the
> > > > > async
> > > > > >> > table
> > > > > >> > > > > function. So the planner will also treat this statement
> > to a
> > > > > >> > > > > CommonExecCorrelate node. So the runtime code should be
> > > > > generated
> > > > > >> in
> > > > > >> > > > > CorrelatedCodeGenerator.
> > > > > >> > > > > In CorrelatedCodeGenerator, we will know the
> > TableFunction's
> > > > > Kind
> > > > > >> of
> > > > > >> > > > > `FunctionKind.Table` or `FunctionKind.ASYNC_TABLE`
> > > > > >> > > > > For  `FunctionKind.ASYNC_TABLE` we can generate a
> > > > > >> AsyncWaitOperator
> > > > > >> > to
> > > > > >> > > > > execute the async table function.
> > > > > >> > > > >
> > > > > >> > > > >
> > > > > >> > > > > Thanks,
> > > > > >> > > > > Aitozi.
> > > > > >> > > > >
> > > > > >> > > > >
> > > > > >> > > > > Jing Ge <ji...@ververica.com.invalid> 于2023年5月29日周一
> > 03:22写道：
> > > > > >> > > > >
> > > > > >> > > > >> Hi Aitozi,
> > > > > >> > > > >>
> > > > > >> > > > >> Thanks for the clarification. The naming "Lookup" might
> > > > suggest
> > > > > >> > using
> > > > > >> > > it
> > > > > >> > > > >> for table look up. But conceptually what the eval()
> > method
> > > > will
> > > > > >> do
> > > > > >> > is
> > > > > >> > > to
> > > > > >> > > > >> get a collection of results(Row, RowData) from the
> given
> > > > keys.
> > > > > >> How
> > > > > >> > it
> > > > > >> > > > will
> > > > > >> > > > >> be done depends on the implementation, i.e. you can
> > > implement
> > > > > >> your
> > > > > >> > own
> > > > > >> > > > >> Source[1][2]. The example in the FLIP should be able to
> > be
> > > > > >> handled
> > > > > >> > in
> > > > > >> > > > this
> > > > > >> > > > >> way.
> > > > > >> > > > >>
> > > > > >> > > > >> Do you mean to support the AyncTableFunction beyond the
> > > > > >> > > > LookupTableSource?
> > > > > >> > > > >> It would be better if you could elaborate the proposed
> > > > changes
> > > > > >> wrt
> > > > > >> > the
> > > > > >> > > > >> CorrelatedCodeGenerator with more details. Thanks!
> > > > > >> > > > >>
> > > > > >> > > > >> Best regards,
> > > > > >> > > > >> Jing
> > > > > >> > > > >>
> > > > > >> > > > >> [1]
> > > > > >> > > > >>
> > > > > >> > > > >>
> > > > > >> > > >
> > > > > >> > >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
> https://github.com/apache/flink/blob/678370b18e1b6c4a23e5ce08f8efd05675a0cc17/flink-table/flink-table-common/src/main/java/org/apache/flink/table/connector/source/LookupTableSource.java#L64
> > > > > >> > > > >> [2]
> > > > > >> > > > >>
> > > > > >> > > > >>
> > > > > >> > > >
> > > > > >> > >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
> https://github.com/apache/flink/blob/678370b18e1b6c4a23e5ce08f8efd05675a0cc17/flink-table/flink-table-common/src/main/java/org/apache/flink/table/connector/source/AsyncTableFunctionProvider.java#L49
> > > > > >> > > > >>
> > > > > >> > > > >> On Sat, May 27, 2023 at 9:48 AM Aitozi <
> > > gjying1314@gmail.com
> > > > >
> > > > > >> > wrote:
> > > > > >> > > > >>
> > > > > >> > > > >> > Hi Jing,
> > > > > >> > > > >> >     Thanks for your response. As stated in the FLIP,
> > the
> > > > > >> purpose
> > > > > >> > of
> > > > > >> > > > this
> > > > > >> > > > >> > FLIP is meant to support
> > > > > >> > > > >> > user-defined async table function. As described in
> > flink
> > > > > >> document
> > > > > >> > > [1]
> > > > > >> > > > >> >
> > > > > >> > > > >> > Async table functions are special functions for table
> > > > sources
> > > > > >> that
> > > > > >> > > > >> perform
> > > > > >> > > > >> > > a lookup.
> > > > > >> > > > >> > >
> > > > > >> > > > >> >
> > > > > >> > > > >> > So end user can not directly define and use async
> table
> > > > > >> function
> > > > > >> > > now.
> > > > > >> > > > An
> > > > > >> > > > >> > user case is reported in [2]
> > > > > >> > > > >> >
> > > > > >> > > > >> > So, in conclusion, no new interface is introduced,
> but
> > we
> > > > > >> extend
> > > > > >> > the
> > > > > >> > > > >> > ability to support user-defined async table function.
> > > > > >> > > > >> >
> > > > > >> > > > >> > [1]:
> > > > > >> > > > >> >
> > > > > >> > > > >> >
> > > > > >> > > > >>
> > > > > >> > > >
> > > > > >> > >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
> https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/dev/table/functions/udfs/
> > > > > >> > > > >> > [2]:
> > > > > >> > >
> > > https://lists.apache.org/thread/qljwd40v5ntz6733cwcdr8s4z97b343b
> > > > > >> > > > >> >
> > > > > >> > > > >> > Thanks.
> > > > > >> > > > >> > Aitozi.
> > > > > >> > > > >> >
> > > > > >> > > > >> >
> > > > > >> > > > >> > Jing Ge <ji...@ververica.com.invalid> 于2023年5月27日周六
> > > > 06:40写道：
> > > > > >> > > > >> >
> > > > > >> > > > >> > > Hi Aitozi,
> > > > > >> > > > >> > >
> > > > > >> > > > >> > > Thanks for your proposal. I am not quite sure if I
> > > > > understood
> > > > > >> > your
> > > > > >> > > > >> > thoughts
> > > > > >> > > > >> > > correctly. You described a special case
> > implementation
> > > of
> > > > > the
> > > > > >> > > > >> > > AsyncTableFunction with on public API changes.
> Would
> > > you
> > > > > >> please
> > > > > >> > > > >> elaborate
> > > > > >> > > > >> > > your purpose of writing a FLIP according to the
> FLIP
> > > > > >> > > > documentation[1]?
> > > > > >> > > > >> > > Thanks!
> > > > > >> > > > >> > >
> > > > > >> > > > >> > > [1]
> > > > > >> > > > >> > >
> > > > > >> > > > >> > >
> > > > > >> > > > >> >
> > > > > >> > > > >>
> > > > > >> > > >
> > > > > >> > >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/Flink+Improvement+Proposals
> > > > > >> > > > >> > >
> > > > > >> > > > >> > > Best regards,
> > > > > >> > > > >> > > Jing
> > > > > >> > > > >> > >
> > > > > >> > > > >> > > On Wed, May 24, 2023 at 1:07 PM Aitozi <
> > > > > gjying1314@gmail.com
> > > > > >> >
> > > > > >> > > > wrote:
> > > > > >> > > > >> > >
> > > > > >> > > > >> > > > May I ask for some feedback  :D
> > > > > >> > > > >> > > >
> > > > > >> > > > >> > > > Thanks,
> > > > > >> > > > >> > > > Aitozi
> > > > > >> > > > >> > > >
> > > > > >> > > > >> > > > Aitozi <gj...@gmail.com> 于2023年5月23日周二
> > 19:14写道：
> > > > > >> > > > >> > > > >
> > > > > >> > > > >> > > > > Just catch an user case report from Giannis
> > Polyzos
> > > > for
> > > > > >> this
> > > > > >> > > > >> usage:
> > > > > >> > > > >> > > > >
> > > > > >> > > > >> > > > >
> > > > > >> > > >
> > > > https://lists.apache.org/thread/qljwd40v5ntz6733cwcdr8s4z97b343b
> > > > > >> > > > >> > > > >
> > > > > >> > > > >> > > > > Aitozi <gj...@gmail.com> 于2023年5月23日周二
> > > 17:45写道：
> > > > > >> > > > >> > > > > >
> > > > > >> > > > >> > > > > > Hi guys,
> > > > > >> > > > >> > > > > >     I want to bring up a discussion about
> > adding
> > > > > >> support
> > > > > >> > of
> > > > > >> > > > User
> > > > > >> > > > >> > > > > > Defined AsyncTableFunction in Flink.
> > > > > >> > > > >> > > > > > Currently, async table function are special
> > > > functions
> > > > > >> for
> > > > > >> > > > table
> > > > > >> > > > >> > > source
> > > > > >> > > > >> > > > > > to perform
> > > > > >> > > > >> > > > > > async lookup. However, it's worth to support
> > the
> > > > user
> > > > > >> > > defined
> > > > > >> > > > >> async
> > > > > >> > > > >> > > > > > table function.
> > > > > >> > > > >> > > > > > Because, in this way, the end SQL user can
> > > leverage
> > > > > it
> > > > > >> to
> > > > > >> > > > >> perform
> > > > > >> > > > >> > the
> > > > > >> > > > >> > > > > > async operation
> > > > > >> > > > >> > > > > > which is useful to maximum the system
> > throughput
> > > > > >> > especially
> > > > > >> > > > for
> > > > > >> > > > >> IO
> > > > > >> > > > >> > > > > > bottleneck case.
> > > > > >> > > > >> > > > > >
> > > > > >> > > > >> > > > > > You can find some more detail in [1].
> > > > > >> > > > >> > > > > >
> > > > > >> > > > >> > > > > > Looking forward to feedback
> > > > > >> > > > >> > > > > >
> > > > > >> > > > >> > > > > >
> > > > > >> > > > >> > > > > > [1]:
> > > > > >> > > > >> > > >
> > > > > >> > > > >> > >
> > > > > >> > > > >> >
> > > > > >> > > > >>
> > > > > >> > > >
> > > > > >> > >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/%5BFLIP-313%5D+Add+support+of+User+Defined+AsyncTableFunction
> > > > > >> > > > >> > > > > >
> > > > > >> > > > >> > > > > > Thanks,
> > > > > >> > > > >> > > > > > Aitozi.
> > > > > >> > > > >> > > >
> > > > > >> > > > >> > >
> > > > > >> > > > >> >
> > > > > >> > > > >>
> > > > > >> > > > >
> > > > > >> > > >
> > > > > >> > >
> > > > > >> >
> > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] FLIP-313 Add support of User Defined AsyncTableFunction

Posted by Aitozi <gj...@gmail.com>.

Hi Jing
    Thanks for your good questions. I have updated the example to the FLIP.

> Only one row for each lookup
lookup can also return multi rows, based on the query result. [1]

[1]:
https://github.com/apache/flink/blob/191ec6ca3943d7119f14837efe112e074d815c47/flink-table/flink-table-common/src/main/java/org/apache/flink/table/functions/LookupFunction.java#L56

> If we use async calls with lateral join, my gut feeling is
that we might have many more async calls than lookup join. I am not really
sure if we will be facing potential issues in this case or not.

IMO, the work pattern is similar to the lookup function, for each row from
the left table,
it will evaluate the eval method once, so the async call numbers will not
change.
and the maximum calls in flight is limited by the Async operators buffer
capacity
which will be controlled by the option.

BTW, for the naming of these option, I updated the FLIP about this you can
refer to
the section of "ConfigOption" and "Rejected Alternatives"

In the end, for the performance evaluation, I'd like to do some tests and
will update it to the FLIP doc

Thanks,
Aitozi.


Jing Ge <ji...@ververica.com.invalid> 于2023年6月9日周五 07:23写道：

> Hi Aitozi,
>
> Thanks for the clarification. The code example looks interesting. I would
> suggest adding them into the FLIP. The description with code examples will
> help readers understand the motivation and how to use it. Afaiac, it is a
> valid feature for Flink users.
>
> As we knew, lookup join is based on temporal join, i.e. FOR SYSTEM_TIME AS
> OF which is also used in your code example. Temporal join performs the
> lookup based on the processing time match. Only one row for each
> lookup(afaiu, I need to check the source code to double confirm) will
> return for further enrichment. One the other hand, lateral join will have
> sub-queries correlated with every individual value of the reference table
> from the preceding part of the query and each sub query will return
> multiple rows. If we use async calls with lateral join, my gut feeling is
> that we might have many more async calls than lookup join. I am not really
> sure if we will be facing potential issues in this case or not. Possible
> issues I can think of now e.g. too many PRC calls, too many async calls
> processing, the sub query will return a table which might be (too) big, and
> might cause performance issues. I would suggest preparing some use cases
> and running some performance tests to check it. These are my concerns about
> using async calls with lateral join and I'd like to share with you, happy
> to discuss with you and hear different opinions, hopefully the
> discussion could help me understand it more deeply. Please correct me if I
> am wrong.
>
> Best regards,
> Jing
>
>
> On Thu, Jun 8, 2023 at 7:22 AM Aitozi <gj...@gmail.com> wrote:
>
> > Hi Mason,
> >     Thanks for your input. I think if we support the user defined async
> > table function,
> > user will be able to use it to hold a batch data then handle it at one
> time
> > in the customized function.
> >
> > AsyncSink is meant for the sink operator. I have not figure out how to
> > integrate in this case.
> >
> > Thanks,
> > Atiozi.
> >
> >
> > Mason Chen <ma...@gmail.com> 于2023年6月8日周四 12:40写道：
> >
> > > Hi Aitozi,
> > >
> > > I think it makes sense to make it easier for SQL users to make RPCs. Do
> > you
> > > think your proposal can extend to the ability to batch data for the
> RPC?
> > > This is also another common strategy to increase throughput. Also, have
> > you
> > > considered solving this a bit differently by leveraging Flink's
> > AsyncSink?
> > >
> > > Best,
> > > Mason
> > >
> > > On Mon, Jun 5, 2023 at 1:50 AM Aitozi <gj...@gmail.com> wrote:
> > >
> > > > One more thing for discussion:
> > > >
> > > > In our internal implementation, we reuse the option
> > > > `table.exec.async-lookup.buffer-capacity` and
> > > > `table.exec.async-lookup.timeout` to config
> > > > the async udtf. Do you think we should add two extra option to
> > > distinguish
> > > > from the lookup option such as
> > > >
> > > > `table.exec.async-udtf.buffer-capacity`
> > > > `table.exec.async-udtf.timeout`
> > > >
> > > >
> > > > Best,
> > > > Aitozi.
> > > >
> > > >
> > > >
> > > > Aitozi <gj...@gmail.com> 于2023年6月5日周一 12:20写道：
> > > >
> > > > > Hi Jing,
> > > > >
> > > > >     > what is the difference between the RPC call or query you
> > > mentioned
> > > > > and the lookup in a very
> > > > > general way
> > > > >
> > > > > I think the RPC call or query service is quite similar to the
> lookup
> > > > join.
> > > > > But lookup join should work
> > > > > with `LookupTableSource`.
> > > > >
> > > > > Let's see how we can perform an async RPC call with lookup join:
> > > > >
> > > > > (1) Implement an AsyncTableFunction with RPC call logic.
> > > > > (2) Implement a `LookupTableSource` connector run with the async
> udtf
> > > > > defined in (1).
> > > > > (3) Then define a DDL of this look up table in SQL
> > > > >
> > > > > CREATE TEMPORARY TABLE Customers (
> > > > >   id INT,
> > > > >   name STRING,
> > > > >   country STRING,
> > > > >   zip STRING
> > > > > ) WITH (
> > > > >   'connector' = 'custom'
> > > > > );
> > > > >
> > > > > (4) Run with the query as below:
> > > > >
> > > > > SELECT o.order_id, o.total, c.country, c.zip
> > > > > FROM Orders AS o
> > > > >   JOIN Customers FOR SYSTEM_TIME AS OF o.proc_time AS c
> > > > >     ON o.customer_id = c.id;
> > > > >
> > > > > This example is from doc
> > > > > <
> > > >
> > >
> >
> https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/dev/table/sql/queries/joins/#lookup-join
> > > > >.You
> > > > > can image the look up process as an async RPC call process.
> > > > >
> > > > > Let's see how we can perform an async RPC call with lateral join:
> > > > >
> > > > > (1) Implement an AsyncTableFunction with RPC call logic.
> > > > > (2) Run query with
> > > > >
> > > > > Create function f1 as '...' ;
> > > > >
> > > > > SELECT o.order_id, o.total, c.country, c.zip FROM Orders  lateral
> > table
> > > > > (f1(order_id)) as T(...);
> > > > >
> > > > > As you can see, the lateral join version is more simple and
> intuitive
> > > to
> > > > > users. Users do not have to wrap a
> > > > > LookupTableSource for the purpose of using async udtf.
> > > > >
> > > > > In the end, We can also see the user defined async table function
> is
> > an
> > > > > enhancement of the current lateral table join
> > > > > which only supports sync lateral join now.
> > > > >
> > > > > Best,
> > > > > Aitozi.
> > > > >
> > > > >
> > > > > Jing Ge <ji...@ververica.com.invalid> 于2023年6月2日周五 19:37写道：
> > > > >
> > > > >> Hi Aitozi,
> > > > >>
> > > > >> Thanks for the update. Just out of curiosity, what is the
> difference
> > > > >> between the RPC call or query you mentioned and the lookup in a
> very
> > > > >> general way? Since Lateral join is used in the FLIP. Is there any
> > > > special
> > > > >> thought for that? Sorry for asking so many questions. The FLIP
> > > contains
> > > > >> limited information to understand the motivation.
> > > > >>
> > > > >> Best regards,
> > > > >> Jing
> > > > >>
> > > > >> On Fri, Jun 2, 2023 at 3:48 AM Aitozi <gj...@gmail.com>
> wrote:
> > > > >>
> > > > >> > Hi Jing,
> > > > >> >     I have updated the proposed changes to the FLIP. IMO, lookup
> > has
> > > > its
> > > > >> > clear
> > > > >> > async call requirement is due to its IO heavy operator. In our
> > > usage,
> > > > >> sql
> > > > >> > users have
> > > > >> > logic to do some RPC call or query the third-party service which
> > is
> > > > >> also IO
> > > > >> > intensive.
> > > > >> > In these case, we'd like to leverage the async function to
> improve
> > > the
> > > > >> > throughput.
> > > > >> >
> > > > >> > Thanks,
> > > > >> > Aitozi.
> > > > >> >
> > > > >> > Jing Ge <ji...@ververica.com.invalid> 于2023年6月1日周四 22:55写道：
> > > > >> >
> > > > >> > > Hi Aitozi,
> > > > >> > >
> > > > >> > > Sorry for the late reply. Would you like to update the
> proposed
> > > > >> changes
> > > > >> > > with more details into the FLIP too?
> > > > >> > > I got your point. It looks like a rational idea. However,
> since
> > > > lookup
> > > > >> > has
> > > > >> > > its clear async call requirement, are there any real use cases
> > > that
> > > > >> > > need this change? This will help us understand the motivation.
> > > After
> > > > >> all,
> > > > >> > > lateral join and temporal lookup join[1] are quite different.
> > > > >> > >
> > > > >> > > Best regards,
> > > > >> > > Jing
> > > > >> > >
> > > > >> > >
> > > > >> > > [1]
> > > > >> > >
> > > > >> > >
> > > > >> >
> > > > >>
> > > >
> > >
> >
> https://github.com/apache/flink/blob/d90a72da2fd601ca4e2a46700e91ec5b348de2ad/flink-table/flink-table-common/src/main/java/org/apache/flink/table/functions/AsyncTableFunction.java#L54
> > > > >> > >
> > > > >> > > On Wed, May 31, 2023 at 8:53 AM Aitozi <gj...@gmail.com>
> > > > wrote:
> > > > >> > >
> > > > >> > > > Hi Jing,
> > > > >> > > >     What do you think about it? Can we move forward this
> > > feature?
> > > > >> > > >
> > > > >> > > > Thanks,
> > > > >> > > > Aitozi.
> > > > >> > > >
> > > > >> > > > Aitozi <gj...@gmail.com> 于2023年5月29日周一 09:56写道：
> > > > >> > > >
> > > > >> > > > > Hi Jing,
> > > > >> > > > >     > "Do you mean to support the AyncTableFunction beyond
> > the
> > > > >> > > > > LookupTableSource?"
> > > > >> > > > > Yes, I mean to support the AyncTableFunction beyond the
> > > > >> > > > LookupTableSource.
> > > > >> > > > >
> > > > >> > > > > The "AsyncTableFunction" is the function with ability to
> be
> > > > >> executed
> > > > >> > > > async
> > > > >> > > > > (with AsyncWaitOperator).
> > > > >> > > > > The async lookup join is a one of usage of this. So, we
> > don't
> > > > >> have to
> > > > >> > > > bind
> > > > >> > > > > the AyncTableFunction with LookupTableSource.
> > > > >> > > > > If User-defined AsyncTableFunction is supported, user can
> > > > directly
> > > > >> > use
> > > > >> > > > > lateral table syntax to perform async operation.
> > > > >> > > > >
> > > > >> > > > > > "It would be better if you could elaborate the proposed
> > > > changes
> > > > >> wrt
> > > > >> > > the
> > > > >> > > > > CorrelatedCodeGenerator with more details"
> > > > >> > > > >
> > > > >> > > > > In the proposal, we use lateral table syntax to support
> the
> > > > async
> > > > >> > table
> > > > >> > > > > function. So the planner will also treat this statement
> to a
> > > > >> > > > > CommonExecCorrelate node. So the runtime code should be
> > > > generated
> > > > >> in
> > > > >> > > > > CorrelatedCodeGenerator.
> > > > >> > > > > In CorrelatedCodeGenerator, we will know the
> TableFunction's
> > > > Kind
> > > > >> of
> > > > >> > > > > `FunctionKind.Table` or `FunctionKind.ASYNC_TABLE`
> > > > >> > > > > For  `FunctionKind.ASYNC_TABLE` we can generate a
> > > > >> AsyncWaitOperator
> > > > >> > to
> > > > >> > > > > execute the async table function.
> > > > >> > > > >
> > > > >> > > > >
> > > > >> > > > > Thanks,
> > > > >> > > > > Aitozi.
> > > > >> > > > >
> > > > >> > > > >
> > > > >> > > > > Jing Ge <ji...@ververica.com.invalid> 于2023年5月29日周一
> 03:22写道：
> > > > >> > > > >
> > > > >> > > > >> Hi Aitozi,
> > > > >> > > > >>
> > > > >> > > > >> Thanks for the clarification. The naming "Lookup" might
> > > suggest
> > > > >> > using
> > > > >> > > it
> > > > >> > > > >> for table look up. But conceptually what the eval()
> method
> > > will
> > > > >> do
> > > > >> > is
> > > > >> > > to
> > > > >> > > > >> get a collection of results(Row, RowData) from the given
> > > keys.
> > > > >> How
> > > > >> > it
> > > > >> > > > will
> > > > >> > > > >> be done depends on the implementation, i.e. you can
> > implement
> > > > >> your
> > > > >> > own
> > > > >> > > > >> Source[1][2]. The example in the FLIP should be able to
> be
> > > > >> handled
> > > > >> > in
> > > > >> > > > this
> > > > >> > > > >> way.
> > > > >> > > > >>
> > > > >> > > > >> Do you mean to support the AyncTableFunction beyond the
> > > > >> > > > LookupTableSource?
> > > > >> > > > >> It would be better if you could elaborate the proposed
> > > changes
> > > > >> wrt
> > > > >> > the
> > > > >> > > > >> CorrelatedCodeGenerator with more details. Thanks!
> > > > >> > > > >>
> > > > >> > > > >> Best regards,
> > > > >> > > > >> Jing
> > > > >> > > > >>
> > > > >> > > > >> [1]
> > > > >> > > > >>
> > > > >> > > > >>
> > > > >> > > >
> > > > >> > >
> > > > >> >
> > > > >>
> > > >
> > >
> >
> https://github.com/apache/flink/blob/678370b18e1b6c4a23e5ce08f8efd05675a0cc17/flink-table/flink-table-common/src/main/java/org/apache/flink/table/connector/source/LookupTableSource.java#L64
> > > > >> > > > >> [2]
> > > > >> > > > >>
> > > > >> > > > >>
> > > > >> > > >
> > > > >> > >
> > > > >> >
> > > > >>
> > > >
> > >
> >
> https://github.com/apache/flink/blob/678370b18e1b6c4a23e5ce08f8efd05675a0cc17/flink-table/flink-table-common/src/main/java/org/apache/flink/table/connector/source/AsyncTableFunctionProvider.java#L49
> > > > >> > > > >>
> > > > >> > > > >> On Sat, May 27, 2023 at 9:48 AM Aitozi <
> > gjying1314@gmail.com
> > > >
> > > > >> > wrote:
> > > > >> > > > >>
> > > > >> > > > >> > Hi Jing,
> > > > >> > > > >> >     Thanks for your response. As stated in the FLIP,
> the
> > > > >> purpose
> > > > >> > of
> > > > >> > > > this
> > > > >> > > > >> > FLIP is meant to support
> > > > >> > > > >> > user-defined async table function. As described in
> flink
> > > > >> document
> > > > >> > > [1]
> > > > >> > > > >> >
> > > > >> > > > >> > Async table functions are special functions for table
> > > sources
> > > > >> that
> > > > >> > > > >> perform
> > > > >> > > > >> > > a lookup.
> > > > >> > > > >> > >
> > > > >> > > > >> >
> > > > >> > > > >> > So end user can not directly define and use async table
> > > > >> function
> > > > >> > > now.
> > > > >> > > > An
> > > > >> > > > >> > user case is reported in [2]
> > > > >> > > > >> >
> > > > >> > > > >> > So, in conclusion, no new interface is introduced, but
> we
> > > > >> extend
> > > > >> > the
> > > > >> > > > >> > ability to support user-defined async table function.
> > > > >> > > > >> >
> > > > >> > > > >> > [1]:
> > > > >> > > > >> >
> > > > >> > > > >> >
> > > > >> > > > >>
> > > > >> > > >
> > > > >> > >
> > > > >> >
> > > > >>
> > > >
> > >
> >
> https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/dev/table/functions/udfs/
> > > > >> > > > >> > [2]:
> > > > >> > >
> > https://lists.apache.org/thread/qljwd40v5ntz6733cwcdr8s4z97b343b
> > > > >> > > > >> >
> > > > >> > > > >> > Thanks.
> > > > >> > > > >> > Aitozi.
> > > > >> > > > >> >
> > > > >> > > > >> >
> > > > >> > > > >> > Jing Ge <ji...@ververica.com.invalid> 于2023年5月27日周六
> > > 06:40写道：
> > > > >> > > > >> >
> > > > >> > > > >> > > Hi Aitozi,
> > > > >> > > > >> > >
> > > > >> > > > >> > > Thanks for your proposal. I am not quite sure if I
> > > > understood
> > > > >> > your
> > > > >> > > > >> > thoughts
> > > > >> > > > >> > > correctly. You described a special case
> implementation
> > of
> > > > the
> > > > >> > > > >> > > AsyncTableFunction with on public API changes. Would
> > you
> > > > >> please
> > > > >> > > > >> elaborate
> > > > >> > > > >> > > your purpose of writing a FLIP according to the FLIP
> > > > >> > > > documentation[1]?
> > > > >> > > > >> > > Thanks!
> > > > >> > > > >> > >
> > > > >> > > > >> > > [1]
> > > > >> > > > >> > >
> > > > >> > > > >> > >
> > > > >> > > > >> >
> > > > >> > > > >>
> > > > >> > > >
> > > > >> > >
> > > > >> >
> > > > >>
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/Flink+Improvement+Proposals
> > > > >> > > > >> > >
> > > > >> > > > >> > > Best regards,
> > > > >> > > > >> > > Jing
> > > > >> > > > >> > >
> > > > >> > > > >> > > On Wed, May 24, 2023 at 1:07 PM Aitozi <
> > > > gjying1314@gmail.com
> > > > >> >
> > > > >> > > > wrote:
> > > > >> > > > >> > >
> > > > >> > > > >> > > > May I ask for some feedback  :D
> > > > >> > > > >> > > >
> > > > >> > > > >> > > > Thanks,
> > > > >> > > > >> > > > Aitozi
> > > > >> > > > >> > > >
> > > > >> > > > >> > > > Aitozi <gj...@gmail.com> 于2023年5月23日周二
> 19:14写道：
> > > > >> > > > >> > > > >
> > > > >> > > > >> > > > > Just catch an user case report from Giannis
> Polyzos
> > > for
> > > > >> this
> > > > >> > > > >> usage:
> > > > >> > > > >> > > > >
> > > > >> > > > >> > > > >
> > > > >> > > >
> > > https://lists.apache.org/thread/qljwd40v5ntz6733cwcdr8s4z97b343b
> > > > >> > > > >> > > > >
> > > > >> > > > >> > > > > Aitozi <gj...@gmail.com> 于2023年5月23日周二
> > 17:45写道：
> > > > >> > > > >> > > > > >
> > > > >> > > > >> > > > > > Hi guys,
> > > > >> > > > >> > > > > >     I want to bring up a discussion about
> adding
> > > > >> support
> > > > >> > of
> > > > >> > > > User
> > > > >> > > > >> > > > > > Defined AsyncTableFunction in Flink.
> > > > >> > > > >> > > > > > Currently, async table function are special
> > > functions
> > > > >> for
> > > > >> > > > table
> > > > >> > > > >> > > source
> > > > >> > > > >> > > > > > to perform
> > > > >> > > > >> > > > > > async lookup. However, it's worth to support
> the
> > > user
> > > > >> > > defined
> > > > >> > > > >> async
> > > > >> > > > >> > > > > > table function.
> > > > >> > > > >> > > > > > Because, in this way, the end SQL user can
> > leverage
> > > > it
> > > > >> to
> > > > >> > > > >> perform
> > > > >> > > > >> > the
> > > > >> > > > >> > > > > > async operation
> > > > >> > > > >> > > > > > which is useful to maximum the system
> throughput
> > > > >> > especially
> > > > >> > > > for
> > > > >> > > > >> IO
> > > > >> > > > >> > > > > > bottleneck case.
> > > > >> > > > >> > > > > >
> > > > >> > > > >> > > > > > You can find some more detail in [1].
> > > > >> > > > >> > > > > >
> > > > >> > > > >> > > > > > Looking forward to feedback
> > > > >> > > > >> > > > > >
> > > > >> > > > >> > > > > >
> > > > >> > > > >> > > > > > [1]:
> > > > >> > > > >> > > >
> > > > >> > > > >> > >
> > > > >> > > > >> >
> > > > >> > > > >>
> > > > >> > > >
> > > > >> > >
> > > > >> >
> > > > >>
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/%5BFLIP-313%5D+Add+support+of+User+Defined+AsyncTableFunction
> > > > >> > > > >> > > > > >
> > > > >> > > > >> > > > > > Thanks,
> > > > >> > > > >> > > > > > Aitozi.
> > > > >> > > > >> > > >
> > > > >> > > > >> > >
> > > > >> > > > >> >
> > > > >> > > > >>
> > > > >> > > > >
> > > > >> > > >
> > > > >> > >
> > > > >> >
> > > > >>
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] FLIP-313 Add support of User Defined AsyncTableFunction

Posted by Jing Ge <ji...@ververica.com.INVALID>.

Hi Aitozi,

Thanks for the clarification. The code example looks interesting. I would
suggest adding them into the FLIP. The description with code examples will
help readers understand the motivation and how to use it. Afaiac, it is a
valid feature for Flink users.

As we knew, lookup join is based on temporal join, i.e. FOR SYSTEM_TIME AS
OF which is also used in your code example. Temporal join performs the
lookup based on the processing time match. Only one row for each
lookup(afaiu, I need to check the source code to double confirm) will
return for further enrichment. One the other hand, lateral join will have
sub-queries correlated with every individual value of the reference table
from the preceding part of the query and each sub query will return
multiple rows. If we use async calls with lateral join, my gut feeling is
that we might have many more async calls than lookup join. I am not really
sure if we will be facing potential issues in this case or not. Possible
issues I can think of now e.g. too many PRC calls, too many async calls
processing, the sub query will return a table which might be (too) big, and
might cause performance issues. I would suggest preparing some use cases
and running some performance tests to check it. These are my concerns about
using async calls with lateral join and I'd like to share with you, happy
to discuss with you and hear different opinions, hopefully the
discussion could help me understand it more deeply. Please correct me if I
am wrong.

Best regards,
Jing


On Thu, Jun 8, 2023 at 7:22 AM Aitozi <gj...@gmail.com> wrote:

> Hi Mason,
>     Thanks for your input. I think if we support the user defined async
> table function,
> user will be able to use it to hold a batch data then handle it at one time
> in the customized function.
>
> AsyncSink is meant for the sink operator. I have not figure out how to
> integrate in this case.
>
> Thanks,
> Atiozi.
>
>
> Mason Chen <ma...@gmail.com> 于2023年6月8日周四 12:40写道：
>
> > Hi Aitozi,
> >
> > I think it makes sense to make it easier for SQL users to make RPCs. Do
> you
> > think your proposal can extend to the ability to batch data for the RPC?
> > This is also another common strategy to increase throughput. Also, have
> you
> > considered solving this a bit differently by leveraging Flink's
> AsyncSink?
> >
> > Best,
> > Mason
> >
> > On Mon, Jun 5, 2023 at 1:50 AM Aitozi <gj...@gmail.com> wrote:
> >
> > > One more thing for discussion:
> > >
> > > In our internal implementation, we reuse the option
> > > `table.exec.async-lookup.buffer-capacity` and
> > > `table.exec.async-lookup.timeout` to config
> > > the async udtf. Do you think we should add two extra option to
> > distinguish
> > > from the lookup option such as
> > >
> > > `table.exec.async-udtf.buffer-capacity`
> > > `table.exec.async-udtf.timeout`
> > >
> > >
> > > Best,
> > > Aitozi.
> > >
> > >
> > >
> > > Aitozi <gj...@gmail.com> 于2023年6月5日周一 12:20写道：
> > >
> > > > Hi Jing,
> > > >
> > > >     > what is the difference between the RPC call or query you
> > mentioned
> > > > and the lookup in a very
> > > > general way
> > > >
> > > > I think the RPC call or query service is quite similar to the lookup
> > > join.
> > > > But lookup join should work
> > > > with `LookupTableSource`.
> > > >
> > > > Let's see how we can perform an async RPC call with lookup join:
> > > >
> > > > (1) Implement an AsyncTableFunction with RPC call logic.
> > > > (2) Implement a `LookupTableSource` connector run with the async udtf
> > > > defined in (1).
> > > > (3) Then define a DDL of this look up table in SQL
> > > >
> > > > CREATE TEMPORARY TABLE Customers (
> > > >   id INT,
> > > >   name STRING,
> > > >   country STRING,
> > > >   zip STRING
> > > > ) WITH (
> > > >   'connector' = 'custom'
> > > > );
> > > >
> > > > (4) Run with the query as below:
> > > >
> > > > SELECT o.order_id, o.total, c.country, c.zip
> > > > FROM Orders AS o
> > > >   JOIN Customers FOR SYSTEM_TIME AS OF o.proc_time AS c
> > > >     ON o.customer_id = c.id;
> > > >
> > > > This example is from doc
> > > > <
> > >
> >
> https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/dev/table/sql/queries/joins/#lookup-join
> > > >.You
> > > > can image the look up process as an async RPC call process.
> > > >
> > > > Let's see how we can perform an async RPC call with lateral join:
> > > >
> > > > (1) Implement an AsyncTableFunction with RPC call logic.
> > > > (2) Run query with
> > > >
> > > > Create function f1 as '...' ;
> > > >
> > > > SELECT o.order_id, o.total, c.country, c.zip FROM Orders  lateral
> table
> > > > (f1(order_id)) as T(...);
> > > >
> > > > As you can see, the lateral join version is more simple and intuitive
> > to
> > > > users. Users do not have to wrap a
> > > > LookupTableSource for the purpose of using async udtf.
> > > >
> > > > In the end, We can also see the user defined async table function is
> an
> > > > enhancement of the current lateral table join
> > > > which only supports sync lateral join now.
> > > >
> > > > Best,
> > > > Aitozi.
> > > >
> > > >
> > > > Jing Ge <ji...@ververica.com.invalid> 于2023年6月2日周五 19:37写道：
> > > >
> > > >> Hi Aitozi,
> > > >>
> > > >> Thanks for the update. Just out of curiosity, what is the difference
> > > >> between the RPC call or query you mentioned and the lookup in a very
> > > >> general way? Since Lateral join is used in the FLIP. Is there any
> > > special
> > > >> thought for that? Sorry for asking so many questions. The FLIP
> > contains
> > > >> limited information to understand the motivation.
> > > >>
> > > >> Best regards,
> > > >> Jing
> > > >>
> > > >> On Fri, Jun 2, 2023 at 3:48 AM Aitozi <gj...@gmail.com> wrote:
> > > >>
> > > >> > Hi Jing,
> > > >> >     I have updated the proposed changes to the FLIP. IMO, lookup
> has
> > > its
> > > >> > clear
> > > >> > async call requirement is due to its IO heavy operator. In our
> > usage,
> > > >> sql
> > > >> > users have
> > > >> > logic to do some RPC call or query the third-party service which
> is
> > > >> also IO
> > > >> > intensive.
> > > >> > In these case, we'd like to leverage the async function to improve
> > the
> > > >> > throughput.
> > > >> >
> > > >> > Thanks,
> > > >> > Aitozi.
> > > >> >
> > > >> > Jing Ge <ji...@ververica.com.invalid> 于2023年6月1日周四 22:55写道：
> > > >> >
> > > >> > > Hi Aitozi,
> > > >> > >
> > > >> > > Sorry for the late reply. Would you like to update the proposed
> > > >> changes
> > > >> > > with more details into the FLIP too?
> > > >> > > I got your point. It looks like a rational idea. However, since
> > > lookup
> > > >> > has
> > > >> > > its clear async call requirement, are there any real use cases
> > that
> > > >> > > need this change? This will help us understand the motivation.
> > After
> > > >> all,
> > > >> > > lateral join and temporal lookup join[1] are quite different.
> > > >> > >
> > > >> > > Best regards,
> > > >> > > Jing
> > > >> > >
> > > >> > >
> > > >> > > [1]
> > > >> > >
> > > >> > >
> > > >> >
> > > >>
> > >
> >
> https://github.com/apache/flink/blob/d90a72da2fd601ca4e2a46700e91ec5b348de2ad/flink-table/flink-table-common/src/main/java/org/apache/flink/table/functions/AsyncTableFunction.java#L54
> > > >> > >
> > > >> > > On Wed, May 31, 2023 at 8:53 AM Aitozi <gj...@gmail.com>
> > > wrote:
> > > >> > >
> > > >> > > > Hi Jing,
> > > >> > > >     What do you think about it? Can we move forward this
> > feature?
> > > >> > > >
> > > >> > > > Thanks,
> > > >> > > > Aitozi.
> > > >> > > >
> > > >> > > > Aitozi <gj...@gmail.com> 于2023年5月29日周一 09:56写道：
> > > >> > > >
> > > >> > > > > Hi Jing,
> > > >> > > > >     > "Do you mean to support the AyncTableFunction beyond
> the
> > > >> > > > > LookupTableSource?"
> > > >> > > > > Yes, I mean to support the AyncTableFunction beyond the
> > > >> > > > LookupTableSource.
> > > >> > > > >
> > > >> > > > > The "AsyncTableFunction" is the function with ability to be
> > > >> executed
> > > >> > > > async
> > > >> > > > > (with AsyncWaitOperator).
> > > >> > > > > The async lookup join is a one of usage of this. So, we
> don't
> > > >> have to
> > > >> > > > bind
> > > >> > > > > the AyncTableFunction with LookupTableSource.
> > > >> > > > > If User-defined AsyncTableFunction is supported, user can
> > > directly
> > > >> > use
> > > >> > > > > lateral table syntax to perform async operation.
> > > >> > > > >
> > > >> > > > > > "It would be better if you could elaborate the proposed
> > > changes
> > > >> wrt
> > > >> > > the
> > > >> > > > > CorrelatedCodeGenerator with more details"
> > > >> > > > >
> > > >> > > > > In the proposal, we use lateral table syntax to support the
> > > async
> > > >> > table
> > > >> > > > > function. So the planner will also treat this statement to a
> > > >> > > > > CommonExecCorrelate node. So the runtime code should be
> > > generated
> > > >> in
> > > >> > > > > CorrelatedCodeGenerator.
> > > >> > > > > In CorrelatedCodeGenerator, we will know the TableFunction's
> > > Kind
> > > >> of
> > > >> > > > > `FunctionKind.Table` or `FunctionKind.ASYNC_TABLE`
> > > >> > > > > For  `FunctionKind.ASYNC_TABLE` we can generate a
> > > >> AsyncWaitOperator
> > > >> > to
> > > >> > > > > execute the async table function.
> > > >> > > > >
> > > >> > > > >
> > > >> > > > > Thanks,
> > > >> > > > > Aitozi.
> > > >> > > > >
> > > >> > > > >
> > > >> > > > > Jing Ge <ji...@ververica.com.invalid> 于2023年5月29日周一 03:22写道：
> > > >> > > > >
> > > >> > > > >> Hi Aitozi,
> > > >> > > > >>
> > > >> > > > >> Thanks for the clarification. The naming "Lookup" might
> > suggest
> > > >> > using
> > > >> > > it
> > > >> > > > >> for table look up. But conceptually what the eval() method
> > will
> > > >> do
> > > >> > is
> > > >> > > to
> > > >> > > > >> get a collection of results(Row, RowData) from the given
> > keys.
> > > >> How
> > > >> > it
> > > >> > > > will
> > > >> > > > >> be done depends on the implementation, i.e. you can
> implement
> > > >> your
> > > >> > own
> > > >> > > > >> Source[1][2]. The example in the FLIP should be able to be
> > > >> handled
> > > >> > in
> > > >> > > > this
> > > >> > > > >> way.
> > > >> > > > >>
> > > >> > > > >> Do you mean to support the AyncTableFunction beyond the
> > > >> > > > LookupTableSource?
> > > >> > > > >> It would be better if you could elaborate the proposed
> > changes
> > > >> wrt
> > > >> > the
> > > >> > > > >> CorrelatedCodeGenerator with more details. Thanks!
> > > >> > > > >>
> > > >> > > > >> Best regards,
> > > >> > > > >> Jing
> > > >> > > > >>
> > > >> > > > >> [1]
> > > >> > > > >>
> > > >> > > > >>
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > >
> >
> https://github.com/apache/flink/blob/678370b18e1b6c4a23e5ce08f8efd05675a0cc17/flink-table/flink-table-common/src/main/java/org/apache/flink/table/connector/source/LookupTableSource.java#L64
> > > >> > > > >> [2]
> > > >> > > > >>
> > > >> > > > >>
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > >
> >
> https://github.com/apache/flink/blob/678370b18e1b6c4a23e5ce08f8efd05675a0cc17/flink-table/flink-table-common/src/main/java/org/apache/flink/table/connector/source/AsyncTableFunctionProvider.java#L49
> > > >> > > > >>
> > > >> > > > >> On Sat, May 27, 2023 at 9:48 AM Aitozi <
> gjying1314@gmail.com
> > >
> > > >> > wrote:
> > > >> > > > >>
> > > >> > > > >> > Hi Jing,
> > > >> > > > >> >     Thanks for your response. As stated in the FLIP, the
> > > >> purpose
> > > >> > of
> > > >> > > > this
> > > >> > > > >> > FLIP is meant to support
> > > >> > > > >> > user-defined async table function. As described in flink
> > > >> document
> > > >> > > [1]
> > > >> > > > >> >
> > > >> > > > >> > Async table functions are special functions for table
> > sources
> > > >> that
> > > >> > > > >> perform
> > > >> > > > >> > > a lookup.
> > > >> > > > >> > >
> > > >> > > > >> >
> > > >> > > > >> > So end user can not directly define and use async table
> > > >> function
> > > >> > > now.
> > > >> > > > An
> > > >> > > > >> > user case is reported in [2]
> > > >> > > > >> >
> > > >> > > > >> > So, in conclusion, no new interface is introduced, but we
> > > >> extend
> > > >> > the
> > > >> > > > >> > ability to support user-defined async table function.
> > > >> > > > >> >
> > > >> > > > >> > [1]:
> > > >> > > > >> >
> > > >> > > > >> >
> > > >> > > > >>
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > >
> >
> https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/dev/table/functions/udfs/
> > > >> > > > >> > [2]:
> > > >> > >
> https://lists.apache.org/thread/qljwd40v5ntz6733cwcdr8s4z97b343b
> > > >> > > > >> >
> > > >> > > > >> > Thanks.
> > > >> > > > >> > Aitozi.
> > > >> > > > >> >
> > > >> > > > >> >
> > > >> > > > >> > Jing Ge <ji...@ververica.com.invalid> 于2023年5月27日周六
> > 06:40写道：
> > > >> > > > >> >
> > > >> > > > >> > > Hi Aitozi,
> > > >> > > > >> > >
> > > >> > > > >> > > Thanks for your proposal. I am not quite sure if I
> > > understood
> > > >> > your
> > > >> > > > >> > thoughts
> > > >> > > > >> > > correctly. You described a special case implementation
> of
> > > the
> > > >> > > > >> > > AsyncTableFunction with on public API changes. Would
> you
> > > >> please
> > > >> > > > >> elaborate
> > > >> > > > >> > > your purpose of writing a FLIP according to the FLIP
> > > >> > > > documentation[1]?
> > > >> > > > >> > > Thanks!
> > > >> > > > >> > >
> > > >> > > > >> > > [1]
> > > >> > > > >> > >
> > > >> > > > >> > >
> > > >> > > > >> >
> > > >> > > > >>
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/Flink+Improvement+Proposals
> > > >> > > > >> > >
> > > >> > > > >> > > Best regards,
> > > >> > > > >> > > Jing
> > > >> > > > >> > >
> > > >> > > > >> > > On Wed, May 24, 2023 at 1:07 PM Aitozi <
> > > gjying1314@gmail.com
> > > >> >
> > > >> > > > wrote:
> > > >> > > > >> > >
> > > >> > > > >> > > > May I ask for some feedback  :D
> > > >> > > > >> > > >
> > > >> > > > >> > > > Thanks,
> > > >> > > > >> > > > Aitozi
> > > >> > > > >> > > >
> > > >> > > > >> > > > Aitozi <gj...@gmail.com> 于2023年5月23日周二 19:14写道：
> > > >> > > > >> > > > >
> > > >> > > > >> > > > > Just catch an user case report from Giannis Polyzos
> > for
> > > >> this
> > > >> > > > >> usage:
> > > >> > > > >> > > > >
> > > >> > > > >> > > > >
> > > >> > > >
> > https://lists.apache.org/thread/qljwd40v5ntz6733cwcdr8s4z97b343b
> > > >> > > > >> > > > >
> > > >> > > > >> > > > > Aitozi <gj...@gmail.com> 于2023年5月23日周二
> 17:45写道：
> > > >> > > > >> > > > > >
> > > >> > > > >> > > > > > Hi guys,
> > > >> > > > >> > > > > >     I want to bring up a discussion about adding
> > > >> support
> > > >> > of
> > > >> > > > User
> > > >> > > > >> > > > > > Defined AsyncTableFunction in Flink.
> > > >> > > > >> > > > > > Currently, async table function are special
> > functions
> > > >> for
> > > >> > > > table
> > > >> > > > >> > > source
> > > >> > > > >> > > > > > to perform
> > > >> > > > >> > > > > > async lookup. However, it's worth to support the
> > user
> > > >> > > defined
> > > >> > > > >> async
> > > >> > > > >> > > > > > table function.
> > > >> > > > >> > > > > > Because, in this way, the end SQL user can
> leverage
> > > it
> > > >> to
> > > >> > > > >> perform
> > > >> > > > >> > the
> > > >> > > > >> > > > > > async operation
> > > >> > > > >> > > > > > which is useful to maximum the system throughput
> > > >> > especially
> > > >> > > > for
> > > >> > > > >> IO
> > > >> > > > >> > > > > > bottleneck case.
> > > >> > > > >> > > > > >
> > > >> > > > >> > > > > > You can find some more detail in [1].
> > > >> > > > >> > > > > >
> > > >> > > > >> > > > > > Looking forward to feedback
> > > >> > > > >> > > > > >
> > > >> > > > >> > > > > >
> > > >> > > > >> > > > > > [1]:
> > > >> > > > >> > > >
> > > >> > > > >> > >
> > > >> > > > >> >
> > > >> > > > >>
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/%5BFLIP-313%5D+Add+support+of+User+Defined+AsyncTableFunction
> > > >> > > > >> > > > > >
> > > >> > > > >> > > > > > Thanks,
> > > >> > > > >> > > > > > Aitozi.
> > > >> > > > >> > > >
> > > >> > > > >> > >
> > > >> > > > >> >
> > > >> > > > >>
> > > >> > > > >
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > > >
> > >
> >
>

Re: [DISCUSS] FLIP-313 Add support of User Defined AsyncTableFunction

Posted by Aitozi <gj...@gmail.com>.

Hi Mason,
    Thanks for your input. I think if we support the user defined async
table function,
user will be able to use it to hold a batch data then handle it at one time
in the customized function.

AsyncSink is meant for the sink operator. I have not figure out how to
integrate in this case.

Thanks,
Atiozi.


Mason Chen <ma...@gmail.com> 于2023年6月8日周四 12:40写道：

> Hi Aitozi,
>
> I think it makes sense to make it easier for SQL users to make RPCs. Do you
> think your proposal can extend to the ability to batch data for the RPC?
> This is also another common strategy to increase throughput. Also, have you
> considered solving this a bit differently by leveraging Flink's AsyncSink?
>
> Best,
> Mason
>
> On Mon, Jun 5, 2023 at 1:50 AM Aitozi <gj...@gmail.com> wrote:
>
> > One more thing for discussion:
> >
> > In our internal implementation, we reuse the option
> > `table.exec.async-lookup.buffer-capacity` and
> > `table.exec.async-lookup.timeout` to config
> > the async udtf. Do you think we should add two extra option to
> distinguish
> > from the lookup option such as
> >
> > `table.exec.async-udtf.buffer-capacity`
> > `table.exec.async-udtf.timeout`
> >
> >
> > Best,
> > Aitozi.
> >
> >
> >
> > Aitozi <gj...@gmail.com> 于2023年6月5日周一 12:20写道：
> >
> > > Hi Jing,
> > >
> > >     > what is the difference between the RPC call or query you
> mentioned
> > > and the lookup in a very
> > > general way
> > >
> > > I think the RPC call or query service is quite similar to the lookup
> > join.
> > > But lookup join should work
> > > with `LookupTableSource`.
> > >
> > > Let's see how we can perform an async RPC call with lookup join:
> > >
> > > (1) Implement an AsyncTableFunction with RPC call logic.
> > > (2) Implement a `LookupTableSource` connector run with the async udtf
> > > defined in (1).
> > > (3) Then define a DDL of this look up table in SQL
> > >
> > > CREATE TEMPORARY TABLE Customers (
> > >   id INT,
> > >   name STRING,
> > >   country STRING,
> > >   zip STRING
> > > ) WITH (
> > >   'connector' = 'custom'
> > > );
> > >
> > > (4) Run with the query as below:
> > >
> > > SELECT o.order_id, o.total, c.country, c.zip
> > > FROM Orders AS o
> > >   JOIN Customers FOR SYSTEM_TIME AS OF o.proc_time AS c
> > >     ON o.customer_id = c.id;
> > >
> > > This example is from doc
> > > <
> >
> https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/dev/table/sql/queries/joins/#lookup-join
> > >.You
> > > can image the look up process as an async RPC call process.
> > >
> > > Let's see how we can perform an async RPC call with lateral join:
> > >
> > > (1) Implement an AsyncTableFunction with RPC call logic.
> > > (2) Run query with
> > >
> > > Create function f1 as '...' ;
> > >
> > > SELECT o.order_id, o.total, c.country, c.zip FROM Orders  lateral table
> > > (f1(order_id)) as T(...);
> > >
> > > As you can see, the lateral join version is more simple and intuitive
> to
> > > users. Users do not have to wrap a
> > > LookupTableSource for the purpose of using async udtf.
> > >
> > > In the end, We can also see the user defined async table function is an
> > > enhancement of the current lateral table join
> > > which only supports sync lateral join now.
> > >
> > > Best,
> > > Aitozi.
> > >
> > >
> > > Jing Ge <ji...@ververica.com.invalid> 于2023年6月2日周五 19:37写道：
> > >
> > >> Hi Aitozi,
> > >>
> > >> Thanks for the update. Just out of curiosity, what is the difference
> > >> between the RPC call or query you mentioned and the lookup in a very
> > >> general way? Since Lateral join is used in the FLIP. Is there any
> > special
> > >> thought for that? Sorry for asking so many questions. The FLIP
> contains
> > >> limited information to understand the motivation.
> > >>
> > >> Best regards,
> > >> Jing
> > >>
> > >> On Fri, Jun 2, 2023 at 3:48 AM Aitozi <gj...@gmail.com> wrote:
> > >>
> > >> > Hi Jing,
> > >> >     I have updated the proposed changes to the FLIP. IMO, lookup has
> > its
> > >> > clear
> > >> > async call requirement is due to its IO heavy operator. In our
> usage,
> > >> sql
> > >> > users have
> > >> > logic to do some RPC call or query the third-party service which is
> > >> also IO
> > >> > intensive.
> > >> > In these case, we'd like to leverage the async function to improve
> the
> > >> > throughput.
> > >> >
> > >> > Thanks,
> > >> > Aitozi.
> > >> >
> > >> > Jing Ge <ji...@ververica.com.invalid> 于2023年6月1日周四 22:55写道：
> > >> >
> > >> > > Hi Aitozi,
> > >> > >
> > >> > > Sorry for the late reply. Would you like to update the proposed
> > >> changes
> > >> > > with more details into the FLIP too?
> > >> > > I got your point. It looks like a rational idea. However, since
> > lookup
> > >> > has
> > >> > > its clear async call requirement, are there any real use cases
> that
> > >> > > need this change? This will help us understand the motivation.
> After
> > >> all,
> > >> > > lateral join and temporal lookup join[1] are quite different.
> > >> > >
> > >> > > Best regards,
> > >> > > Jing
> > >> > >
> > >> > >
> > >> > > [1]
> > >> > >
> > >> > >
> > >> >
> > >>
> >
> https://github.com/apache/flink/blob/d90a72da2fd601ca4e2a46700e91ec5b348de2ad/flink-table/flink-table-common/src/main/java/org/apache/flink/table/functions/AsyncTableFunction.java#L54
> > >> > >
> > >> > > On Wed, May 31, 2023 at 8:53 AM Aitozi <gj...@gmail.com>
> > wrote:
> > >> > >
> > >> > > > Hi Jing,
> > >> > > >     What do you think about it? Can we move forward this
> feature?
> > >> > > >
> > >> > > > Thanks,
> > >> > > > Aitozi.
> > >> > > >
> > >> > > > Aitozi <gj...@gmail.com> 于2023年5月29日周一 09:56写道：
> > >> > > >
> > >> > > > > Hi Jing,
> > >> > > > >     > "Do you mean to support the AyncTableFunction beyond the
> > >> > > > > LookupTableSource?"
> > >> > > > > Yes, I mean to support the AyncTableFunction beyond the
> > >> > > > LookupTableSource.
> > >> > > > >
> > >> > > > > The "AsyncTableFunction" is the function with ability to be
> > >> executed
> > >> > > > async
> > >> > > > > (with AsyncWaitOperator).
> > >> > > > > The async lookup join is a one of usage of this. So, we don't
> > >> have to
> > >> > > > bind
> > >> > > > > the AyncTableFunction with LookupTableSource.
> > >> > > > > If User-defined AsyncTableFunction is supported, user can
> > directly
> > >> > use
> > >> > > > > lateral table syntax to perform async operation.
> > >> > > > >
> > >> > > > > > "It would be better if you could elaborate the proposed
> > changes
> > >> wrt
> > >> > > the
> > >> > > > > CorrelatedCodeGenerator with more details"
> > >> > > > >
> > >> > > > > In the proposal, we use lateral table syntax to support the
> > async
> > >> > table
> > >> > > > > function. So the planner will also treat this statement to a
> > >> > > > > CommonExecCorrelate node. So the runtime code should be
> > generated
> > >> in
> > >> > > > > CorrelatedCodeGenerator.
> > >> > > > > In CorrelatedCodeGenerator, we will know the TableFunction's
> > Kind
> > >> of
> > >> > > > > `FunctionKind.Table` or `FunctionKind.ASYNC_TABLE`
> > >> > > > > For  `FunctionKind.ASYNC_TABLE` we can generate a
> > >> AsyncWaitOperator
> > >> > to
> > >> > > > > execute the async table function.
> > >> > > > >
> > >> > > > >
> > >> > > > > Thanks,
> > >> > > > > Aitozi.
> > >> > > > >
> > >> > > > >
> > >> > > > > Jing Ge <ji...@ververica.com.invalid> 于2023年5月29日周一 03:22写道：
> > >> > > > >
> > >> > > > >> Hi Aitozi,
> > >> > > > >>
> > >> > > > >> Thanks for the clarification. The naming "Lookup" might
> suggest
> > >> > using
> > >> > > it
> > >> > > > >> for table look up. But conceptually what the eval() method
> will
> > >> do
> > >> > is
> > >> > > to
> > >> > > > >> get a collection of results(Row, RowData) from the given
> keys.
> > >> How
> > >> > it
> > >> > > > will
> > >> > > > >> be done depends on the implementation, i.e. you can implement
> > >> your
> > >> > own
> > >> > > > >> Source[1][2]. The example in the FLIP should be able to be
> > >> handled
> > >> > in
> > >> > > > this
> > >> > > > >> way.
> > >> > > > >>
> > >> > > > >> Do you mean to support the AyncTableFunction beyond the
> > >> > > > LookupTableSource?
> > >> > > > >> It would be better if you could elaborate the proposed
> changes
> > >> wrt
> > >> > the
> > >> > > > >> CorrelatedCodeGenerator with more details. Thanks!
> > >> > > > >>
> > >> > > > >> Best regards,
> > >> > > > >> Jing
> > >> > > > >>
> > >> > > > >> [1]
> > >> > > > >>
> > >> > > > >>
> > >> > > >
> > >> > >
> > >> >
> > >>
> >
> https://github.com/apache/flink/blob/678370b18e1b6c4a23e5ce08f8efd05675a0cc17/flink-table/flink-table-common/src/main/java/org/apache/flink/table/connector/source/LookupTableSource.java#L64
> > >> > > > >> [2]
> > >> > > > >>
> > >> > > > >>
> > >> > > >
> > >> > >
> > >> >
> > >>
> >
> https://github.com/apache/flink/blob/678370b18e1b6c4a23e5ce08f8efd05675a0cc17/flink-table/flink-table-common/src/main/java/org/apache/flink/table/connector/source/AsyncTableFunctionProvider.java#L49
> > >> > > > >>
> > >> > > > >> On Sat, May 27, 2023 at 9:48 AM Aitozi <gjying1314@gmail.com
> >
> > >> > wrote:
> > >> > > > >>
> > >> > > > >> > Hi Jing,
> > >> > > > >> >     Thanks for your response. As stated in the FLIP, the
> > >> purpose
> > >> > of
> > >> > > > this
> > >> > > > >> > FLIP is meant to support
> > >> > > > >> > user-defined async table function. As described in flink
> > >> document
> > >> > > [1]
> > >> > > > >> >
> > >> > > > >> > Async table functions are special functions for table
> sources
> > >> that
> > >> > > > >> perform
> > >> > > > >> > > a lookup.
> > >> > > > >> > >
> > >> > > > >> >
> > >> > > > >> > So end user can not directly define and use async table
> > >> function
> > >> > > now.
> > >> > > > An
> > >> > > > >> > user case is reported in [2]
> > >> > > > >> >
> > >> > > > >> > So, in conclusion, no new interface is introduced, but we
> > >> extend
> > >> > the
> > >> > > > >> > ability to support user-defined async table function.
> > >> > > > >> >
> > >> > > > >> > [1]:
> > >> > > > >> >
> > >> > > > >> >
> > >> > > > >>
> > >> > > >
> > >> > >
> > >> >
> > >>
> >
> https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/dev/table/functions/udfs/
> > >> > > > >> > [2]:
> > >> > > https://lists.apache.org/thread/qljwd40v5ntz6733cwcdr8s4z97b343b
> > >> > > > >> >
> > >> > > > >> > Thanks.
> > >> > > > >> > Aitozi.
> > >> > > > >> >
> > >> > > > >> >
> > >> > > > >> > Jing Ge <ji...@ververica.com.invalid> 于2023年5月27日周六
> 06:40写道：
> > >> > > > >> >
> > >> > > > >> > > Hi Aitozi,
> > >> > > > >> > >
> > >> > > > >> > > Thanks for your proposal. I am not quite sure if I
> > understood
> > >> > your
> > >> > > > >> > thoughts
> > >> > > > >> > > correctly. You described a special case implementation of
> > the
> > >> > > > >> > > AsyncTableFunction with on public API changes. Would you
> > >> please
> > >> > > > >> elaborate
> > >> > > > >> > > your purpose of writing a FLIP according to the FLIP
> > >> > > > documentation[1]?
> > >> > > > >> > > Thanks!
> > >> > > > >> > >
> > >> > > > >> > > [1]
> > >> > > > >> > >
> > >> > > > >> > >
> > >> > > > >> >
> > >> > > > >>
> > >> > > >
> > >> > >
> > >> >
> > >>
> >
> https://cwiki.apache.org/confluence/display/FLINK/Flink+Improvement+Proposals
> > >> > > > >> > >
> > >> > > > >> > > Best regards,
> > >> > > > >> > > Jing
> > >> > > > >> > >
> > >> > > > >> > > On Wed, May 24, 2023 at 1:07 PM Aitozi <
> > gjying1314@gmail.com
> > >> >
> > >> > > > wrote:
> > >> > > > >> > >
> > >> > > > >> > > > May I ask for some feedback  :D
> > >> > > > >> > > >
> > >> > > > >> > > > Thanks,
> > >> > > > >> > > > Aitozi
> > >> > > > >> > > >
> > >> > > > >> > > > Aitozi <gj...@gmail.com> 于2023年5月23日周二 19:14写道：
> > >> > > > >> > > > >
> > >> > > > >> > > > > Just catch an user case report from Giannis Polyzos
> for
> > >> this
> > >> > > > >> usage:
> > >> > > > >> > > > >
> > >> > > > >> > > > >
> > >> > > >
> https://lists.apache.org/thread/qljwd40v5ntz6733cwcdr8s4z97b343b
> > >> > > > >> > > > >
> > >> > > > >> > > > > Aitozi <gj...@gmail.com> 于2023年5月23日周二 17:45写道：
> > >> > > > >> > > > > >
> > >> > > > >> > > > > > Hi guys,
> > >> > > > >> > > > > >     I want to bring up a discussion about adding
> > >> support
> > >> > of
> > >> > > > User
> > >> > > > >> > > > > > Defined AsyncTableFunction in Flink.
> > >> > > > >> > > > > > Currently, async table function are special
> functions
> > >> for
> > >> > > > table
> > >> > > > >> > > source
> > >> > > > >> > > > > > to perform
> > >> > > > >> > > > > > async lookup. However, it's worth to support the
> user
> > >> > > defined
> > >> > > > >> async
> > >> > > > >> > > > > > table function.
> > >> > > > >> > > > > > Because, in this way, the end SQL user can leverage
> > it
> > >> to
> > >> > > > >> perform
> > >> > > > >> > the
> > >> > > > >> > > > > > async operation
> > >> > > > >> > > > > > which is useful to maximum the system throughput
> > >> > especially
> > >> > > > for
> > >> > > > >> IO
> > >> > > > >> > > > > > bottleneck case.
> > >> > > > >> > > > > >
> > >> > > > >> > > > > > You can find some more detail in [1].
> > >> > > > >> > > > > >
> > >> > > > >> > > > > > Looking forward to feedback
> > >> > > > >> > > > > >
> > >> > > > >> > > > > >
> > >> > > > >> > > > > > [1]:
> > >> > > > >> > > >
> > >> > > > >> > >
> > >> > > > >> >
> > >> > > > >>
> > >> > > >
> > >> > >
> > >> >
> > >>
> >
> https://cwiki.apache.org/confluence/display/FLINK/%5BFLIP-313%5D+Add+support+of+User+Defined+AsyncTableFunction
> > >> > > > >> > > > > >
> > >> > > > >> > > > > > Thanks,
> > >> > > > >> > > > > > Aitozi.
> > >> > > > >> > > >
> > >> > > > >> > >
> > >> > > > >> >
> > >> > > > >>
> > >> > > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> > >
> >
>

Re: [DISCUSS] FLIP-313 Add support of User Defined AsyncTableFunction

Posted by Mason Chen <ma...@gmail.com>.

Hi Aitozi,

I think it makes sense to make it easier for SQL users to make RPCs. Do you
think your proposal can extend to the ability to batch data for the RPC?
This is also another common strategy to increase throughput. Also, have you
considered solving this a bit differently by leveraging Flink's AsyncSink?

Best,
Mason

On Mon, Jun 5, 2023 at 1:50 AM Aitozi <gj...@gmail.com> wrote:

> One more thing for discussion:
>
> In our internal implementation, we reuse the option
> `table.exec.async-lookup.buffer-capacity` and
> `table.exec.async-lookup.timeout` to config
> the async udtf. Do you think we should add two extra option to distinguish
> from the lookup option such as
>
> `table.exec.async-udtf.buffer-capacity`
> `table.exec.async-udtf.timeout`
>
>
> Best,
> Aitozi.
>
>
>
> Aitozi <gj...@gmail.com> 于2023年6月5日周一 12:20写道：
>
> > Hi Jing,
> >
> >     > what is the difference between the RPC call or query you mentioned
> > and the lookup in a very
> > general way
> >
> > I think the RPC call or query service is quite similar to the lookup
> join.
> > But lookup join should work
> > with `LookupTableSource`.
> >
> > Let's see how we can perform an async RPC call with lookup join:
> >
> > (1) Implement an AsyncTableFunction with RPC call logic.
> > (2) Implement a `LookupTableSource` connector run with the async udtf
> > defined in (1).
> > (3) Then define a DDL of this look up table in SQL
> >
> > CREATE TEMPORARY TABLE Customers (
> >   id INT,
> >   name STRING,
> >   country STRING,
> >   zip STRING
> > ) WITH (
> >   'connector' = 'custom'
> > );
> >
> > (4) Run with the query as below:
> >
> > SELECT o.order_id, o.total, c.country, c.zip
> > FROM Orders AS o
> >   JOIN Customers FOR SYSTEM_TIME AS OF o.proc_time AS c
> >     ON o.customer_id = c.id;
> >
> > This example is from doc
> > <
> https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/dev/table/sql/queries/joins/#lookup-join
> >.You
> > can image the look up process as an async RPC call process.
> >
> > Let's see how we can perform an async RPC call with lateral join:
> >
> > (1) Implement an AsyncTableFunction with RPC call logic.
> > (2) Run query with
> >
> > Create function f1 as '...' ;
> >
> > SELECT o.order_id, o.total, c.country, c.zip FROM Orders  lateral table
> > (f1(order_id)) as T(...);
> >
> > As you can see, the lateral join version is more simple and intuitive to
> > users. Users do not have to wrap a
> > LookupTableSource for the purpose of using async udtf.
> >
> > In the end, We can also see the user defined async table function is an
> > enhancement of the current lateral table join
> > which only supports sync lateral join now.
> >
> > Best,
> > Aitozi.
> >
> >
> > Jing Ge <ji...@ververica.com.invalid> 于2023年6月2日周五 19:37写道：
> >
> >> Hi Aitozi,
> >>
> >> Thanks for the update. Just out of curiosity, what is the difference
> >> between the RPC call or query you mentioned and the lookup in a very
> >> general way? Since Lateral join is used in the FLIP. Is there any
> special
> >> thought for that? Sorry for asking so many questions. The FLIP contains
> >> limited information to understand the motivation.
> >>
> >> Best regards,
> >> Jing
> >>
> >> On Fri, Jun 2, 2023 at 3:48 AM Aitozi <gj...@gmail.com> wrote:
> >>
> >> > Hi Jing,
> >> >     I have updated the proposed changes to the FLIP. IMO, lookup has
> its
> >> > clear
> >> > async call requirement is due to its IO heavy operator. In our usage,
> >> sql
> >> > users have
> >> > logic to do some RPC call or query the third-party service which is
> >> also IO
> >> > intensive.
> >> > In these case, we'd like to leverage the async function to improve the
> >> > throughput.
> >> >
> >> > Thanks,
> >> > Aitozi.
> >> >
> >> > Jing Ge <ji...@ververica.com.invalid> 于2023年6月1日周四 22:55写道：
> >> >
> >> > > Hi Aitozi,
> >> > >
> >> > > Sorry for the late reply. Would you like to update the proposed
> >> changes
> >> > > with more details into the FLIP too?
> >> > > I got your point. It looks like a rational idea. However, since
> lookup
> >> > has
> >> > > its clear async call requirement, are there any real use cases that
> >> > > need this change? This will help us understand the motivation. After
> >> all,
> >> > > lateral join and temporal lookup join[1] are quite different.
> >> > >
> >> > > Best regards,
> >> > > Jing
> >> > >
> >> > >
> >> > > [1]
> >> > >
> >> > >
> >> >
> >>
> https://github.com/apache/flink/blob/d90a72da2fd601ca4e2a46700e91ec5b348de2ad/flink-table/flink-table-common/src/main/java/org/apache/flink/table/functions/AsyncTableFunction.java#L54
> >> > >
> >> > > On Wed, May 31, 2023 at 8:53 AM Aitozi <gj...@gmail.com>
> wrote:
> >> > >
> >> > > > Hi Jing,
> >> > > >     What do you think about it? Can we move forward this feature?
> >> > > >
> >> > > > Thanks,
> >> > > > Aitozi.
> >> > > >
> >> > > > Aitozi <gj...@gmail.com> 于2023年5月29日周一 09:56写道：
> >> > > >
> >> > > > > Hi Jing,
> >> > > > >     > "Do you mean to support the AyncTableFunction beyond the
> >> > > > > LookupTableSource?"
> >> > > > > Yes, I mean to support the AyncTableFunction beyond the
> >> > > > LookupTableSource.
> >> > > > >
> >> > > > > The "AsyncTableFunction" is the function with ability to be
> >> executed
> >> > > > async
> >> > > > > (with AsyncWaitOperator).
> >> > > > > The async lookup join is a one of usage of this. So, we don't
> >> have to
> >> > > > bind
> >> > > > > the AyncTableFunction with LookupTableSource.
> >> > > > > If User-defined AsyncTableFunction is supported, user can
> directly
> >> > use
> >> > > > > lateral table syntax to perform async operation.
> >> > > > >
> >> > > > > > "It would be better if you could elaborate the proposed
> changes
> >> wrt
> >> > > the
> >> > > > > CorrelatedCodeGenerator with more details"
> >> > > > >
> >> > > > > In the proposal, we use lateral table syntax to support the
> async
> >> > table
> >> > > > > function. So the planner will also treat this statement to a
> >> > > > > CommonExecCorrelate node. So the runtime code should be
> generated
> >> in
> >> > > > > CorrelatedCodeGenerator.
> >> > > > > In CorrelatedCodeGenerator, we will know the TableFunction's
> Kind
> >> of
> >> > > > > `FunctionKind.Table` or `FunctionKind.ASYNC_TABLE`
> >> > > > > For  `FunctionKind.ASYNC_TABLE` we can generate a
> >> AsyncWaitOperator
> >> > to
> >> > > > > execute the async table function.
> >> > > > >
> >> > > > >
> >> > > > > Thanks,
> >> > > > > Aitozi.
> >> > > > >
> >> > > > >
> >> > > > > Jing Ge <ji...@ververica.com.invalid> 于2023年5月29日周一 03:22写道：
> >> > > > >
> >> > > > >> Hi Aitozi,
> >> > > > >>
> >> > > > >> Thanks for the clarification. The naming "Lookup" might suggest
> >> > using
> >> > > it
> >> > > > >> for table look up. But conceptually what the eval() method will
> >> do
> >> > is
> >> > > to
> >> > > > >> get a collection of results(Row, RowData) from the given keys.
> >> How
> >> > it
> >> > > > will
> >> > > > >> be done depends on the implementation, i.e. you can implement
> >> your
> >> > own
> >> > > > >> Source[1][2]. The example in the FLIP should be able to be
> >> handled
> >> > in
> >> > > > this
> >> > > > >> way.
> >> > > > >>
> >> > > > >> Do you mean to support the AyncTableFunction beyond the
> >> > > > LookupTableSource?
> >> > > > >> It would be better if you could elaborate the proposed changes
> >> wrt
> >> > the
> >> > > > >> CorrelatedCodeGenerator with more details. Thanks!
> >> > > > >>
> >> > > > >> Best regards,
> >> > > > >> Jing
> >> > > > >>
> >> > > > >> [1]
> >> > > > >>
> >> > > > >>
> >> > > >
> >> > >
> >> >
> >>
> https://github.com/apache/flink/blob/678370b18e1b6c4a23e5ce08f8efd05675a0cc17/flink-table/flink-table-common/src/main/java/org/apache/flink/table/connector/source/LookupTableSource.java#L64
> >> > > > >> [2]
> >> > > > >>
> >> > > > >>
> >> > > >
> >> > >
> >> >
> >>
> https://github.com/apache/flink/blob/678370b18e1b6c4a23e5ce08f8efd05675a0cc17/flink-table/flink-table-common/src/main/java/org/apache/flink/table/connector/source/AsyncTableFunctionProvider.java#L49
> >> > > > >>
> >> > > > >> On Sat, May 27, 2023 at 9:48 AM Aitozi <gj...@gmail.com>
> >> > wrote:
> >> > > > >>
> >> > > > >> > Hi Jing,
> >> > > > >> >     Thanks for your response. As stated in the FLIP, the
> >> purpose
> >> > of
> >> > > > this
> >> > > > >> > FLIP is meant to support
> >> > > > >> > user-defined async table function. As described in flink
> >> document
> >> > > [1]
> >> > > > >> >
> >> > > > >> > Async table functions are special functions for table sources
> >> that
> >> > > > >> perform
> >> > > > >> > > a lookup.
> >> > > > >> > >
> >> > > > >> >
> >> > > > >> > So end user can not directly define and use async table
> >> function
> >> > > now.
> >> > > > An
> >> > > > >> > user case is reported in [2]
> >> > > > >> >
> >> > > > >> > So, in conclusion, no new interface is introduced, but we
> >> extend
> >> > the
> >> > > > >> > ability to support user-defined async table function.
> >> > > > >> >
> >> > > > >> > [1]:
> >> > > > >> >
> >> > > > >> >
> >> > > > >>
> >> > > >
> >> > >
> >> >
> >>
> https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/dev/table/functions/udfs/
> >> > > > >> > [2]:
> >> > > https://lists.apache.org/thread/qljwd40v5ntz6733cwcdr8s4z97b343b
> >> > > > >> >
> >> > > > >> > Thanks.
> >> > > > >> > Aitozi.
> >> > > > >> >
> >> > > > >> >
> >> > > > >> > Jing Ge <ji...@ververica.com.invalid> 于2023年5月27日周六 06:40写道：
> >> > > > >> >
> >> > > > >> > > Hi Aitozi,
> >> > > > >> > >
> >> > > > >> > > Thanks for your proposal. I am not quite sure if I
> understood
> >> > your
> >> > > > >> > thoughts
> >> > > > >> > > correctly. You described a special case implementation of
> the
> >> > > > >> > > AsyncTableFunction with on public API changes. Would you
> >> please
> >> > > > >> elaborate
> >> > > > >> > > your purpose of writing a FLIP according to the FLIP
> >> > > > documentation[1]?
> >> > > > >> > > Thanks!
> >> > > > >> > >
> >> > > > >> > > [1]
> >> > > > >> > >
> >> > > > >> > >
> >> > > > >> >
> >> > > > >>
> >> > > >
> >> > >
> >> >
> >>
> https://cwiki.apache.org/confluence/display/FLINK/Flink+Improvement+Proposals
> >> > > > >> > >
> >> > > > >> > > Best regards,
> >> > > > >> > > Jing
> >> > > > >> > >
> >> > > > >> > > On Wed, May 24, 2023 at 1:07 PM Aitozi <
> gjying1314@gmail.com
> >> >
> >> > > > wrote:
> >> > > > >> > >
> >> > > > >> > > > May I ask for some feedback  :D
> >> > > > >> > > >
> >> > > > >> > > > Thanks,
> >> > > > >> > > > Aitozi
> >> > > > >> > > >
> >> > > > >> > > > Aitozi <gj...@gmail.com> 于2023年5月23日周二 19:14写道：
> >> > > > >> > > > >
> >> > > > >> > > > > Just catch an user case report from Giannis Polyzos for
> >> this
> >> > > > >> usage:
> >> > > > >> > > > >
> >> > > > >> > > > >
> >> > > > https://lists.apache.org/thread/qljwd40v5ntz6733cwcdr8s4z97b343b
> >> > > > >> > > > >
> >> > > > >> > > > > Aitozi <gj...@gmail.com> 于2023年5月23日周二 17:45写道：
> >> > > > >> > > > > >
> >> > > > >> > > > > > Hi guys,
> >> > > > >> > > > > >     I want to bring up a discussion about adding
> >> support
> >> > of
> >> > > > User
> >> > > > >> > > > > > Defined AsyncTableFunction in Flink.
> >> > > > >> > > > > > Currently, async table function are special functions
> >> for
> >> > > > table
> >> > > > >> > > source
> >> > > > >> > > > > > to perform
> >> > > > >> > > > > > async lookup. However, it's worth to support the user
> >> > > defined
> >> > > > >> async
> >> > > > >> > > > > > table function.
> >> > > > >> > > > > > Because, in this way, the end SQL user can leverage
> it
> >> to
> >> > > > >> perform
> >> > > > >> > the
> >> > > > >> > > > > > async operation
> >> > > > >> > > > > > which is useful to maximum the system throughput
> >> > especially
> >> > > > for
> >> > > > >> IO
> >> > > > >> > > > > > bottleneck case.
> >> > > > >> > > > > >
> >> > > > >> > > > > > You can find some more detail in [1].
> >> > > > >> > > > > >
> >> > > > >> > > > > > Looking forward to feedback
> >> > > > >> > > > > >
> >> > > > >> > > > > >
> >> > > > >> > > > > > [1]:
> >> > > > >> > > >
> >> > > > >> > >
> >> > > > >> >
> >> > > > >>
> >> > > >
> >> > >
> >> >
> >>
> https://cwiki.apache.org/confluence/display/FLINK/%5BFLIP-313%5D+Add+support+of+User+Defined+AsyncTableFunction
> >> > > > >> > > > > >
> >> > > > >> > > > > > Thanks,
> >> > > > >> > > > > > Aitozi.
> >> > > > >> > > >
> >> > > > >> > >
> >> > > > >> >
> >> > > > >>
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
> >
>

Re: [DISCUSS] FLIP-313 Add support of User Defined AsyncTableFunction

Posted by Aitozi <gj...@gmail.com>.

One more thing for discussion:

In our internal implementation, we reuse the option
`table.exec.async-lookup.buffer-capacity` and
`table.exec.async-lookup.timeout` to config
the async udtf. Do you think we should add two extra option to distinguish
from the lookup option such as

`table.exec.async-udtf.buffer-capacity`
`table.exec.async-udtf.timeout`


Best,
Aitozi.



Aitozi <gj...@gmail.com> 于2023年6月5日周一 12:20写道：

> Hi Jing,
>
>     > what is the difference between the RPC call or query you mentioned
> and the lookup in a very
> general way
>
> I think the RPC call or query service is quite similar to the lookup join.
> But lookup join should work
> with `LookupTableSource`.
>
> Let's see how we can perform an async RPC call with lookup join:
>
> (1) Implement an AsyncTableFunction with RPC call logic.
> (2) Implement a `LookupTableSource` connector run with the async udtf
> defined in (1).
> (3) Then define a DDL of this look up table in SQL
>
> CREATE TEMPORARY TABLE Customers (
>   id INT,
>   name STRING,
>   country STRING,
>   zip STRING
> ) WITH (
>   'connector' = 'custom'
> );
>
> (4) Run with the query as below:
>
> SELECT o.order_id, o.total, c.country, c.zip
> FROM Orders AS o
>   JOIN Customers FOR SYSTEM_TIME AS OF o.proc_time AS c
>     ON o.customer_id = c.id;
>
> This example is from doc
> <https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/dev/table/sql/queries/joins/#lookup-join>.You
> can image the look up process as an async RPC call process.
>
> Let's see how we can perform an async RPC call with lateral join:
>
> (1) Implement an AsyncTableFunction with RPC call logic.
> (2) Run query with
>
> Create function f1 as '...' ;
>
> SELECT o.order_id, o.total, c.country, c.zip FROM Orders  lateral table
> (f1(order_id)) as T(...);
>
> As you can see, the lateral join version is more simple and intuitive to
> users. Users do not have to wrap a
> LookupTableSource for the purpose of using async udtf.
>
> In the end, We can also see the user defined async table function is an
> enhancement of the current lateral table join
> which only supports sync lateral join now.
>
> Best,
> Aitozi.
>
>
> Jing Ge <ji...@ververica.com.invalid> 于2023年6月2日周五 19:37写道：
>
>> Hi Aitozi,
>>
>> Thanks for the update. Just out of curiosity, what is the difference
>> between the RPC call or query you mentioned and the lookup in a very
>> general way? Since Lateral join is used in the FLIP. Is there any special
>> thought for that? Sorry for asking so many questions. The FLIP contains
>> limited information to understand the motivation.
>>
>> Best regards,
>> Jing
>>
>> On Fri, Jun 2, 2023 at 3:48 AM Aitozi <gj...@gmail.com> wrote:
>>
>> > Hi Jing,
>> >     I have updated the proposed changes to the FLIP. IMO, lookup has its
>> > clear
>> > async call requirement is due to its IO heavy operator. In our usage,
>> sql
>> > users have
>> > logic to do some RPC call or query the third-party service which is
>> also IO
>> > intensive.
>> > In these case, we'd like to leverage the async function to improve the
>> > throughput.
>> >
>> > Thanks,
>> > Aitozi.
>> >
>> > Jing Ge <ji...@ververica.com.invalid> 于2023年6月1日周四 22:55写道：
>> >
>> > > Hi Aitozi,
>> > >
>> > > Sorry for the late reply. Would you like to update the proposed
>> changes
>> > > with more details into the FLIP too?
>> > > I got your point. It looks like a rational idea. However, since lookup
>> > has
>> > > its clear async call requirement, are there any real use cases that
>> > > need this change? This will help us understand the motivation. After
>> all,
>> > > lateral join and temporal lookup join[1] are quite different.
>> > >
>> > > Best regards,
>> > > Jing
>> > >
>> > >
>> > > [1]
>> > >
>> > >
>> >
>> https://github.com/apache/flink/blob/d90a72da2fd601ca4e2a46700e91ec5b348de2ad/flink-table/flink-table-common/src/main/java/org/apache/flink/table/functions/AsyncTableFunction.java#L54
>> > >
>> > > On Wed, May 31, 2023 at 8:53 AM Aitozi <gj...@gmail.com> wrote:
>> > >
>> > > > Hi Jing,
>> > > >     What do you think about it? Can we move forward this feature?
>> > > >
>> > > > Thanks,
>> > > > Aitozi.
>> > > >
>> > > > Aitozi <gj...@gmail.com> 于2023年5月29日周一 09:56写道：
>> > > >
>> > > > > Hi Jing,
>> > > > >     > "Do you mean to support the AyncTableFunction beyond the
>> > > > > LookupTableSource?"
>> > > > > Yes, I mean to support the AyncTableFunction beyond the
>> > > > LookupTableSource.
>> > > > >
>> > > > > The "AsyncTableFunction" is the function with ability to be
>> executed
>> > > > async
>> > > > > (with AsyncWaitOperator).
>> > > > > The async lookup join is a one of usage of this. So, we don't
>> have to
>> > > > bind
>> > > > > the AyncTableFunction with LookupTableSource.
>> > > > > If User-defined AsyncTableFunction is supported, user can directly
>> > use
>> > > > > lateral table syntax to perform async operation.
>> > > > >
>> > > > > > "It would be better if you could elaborate the proposed changes
>> wrt
>> > > the
>> > > > > CorrelatedCodeGenerator with more details"
>> > > > >
>> > > > > In the proposal, we use lateral table syntax to support the async
>> > table
>> > > > > function. So the planner will also treat this statement to a
>> > > > > CommonExecCorrelate node. So the runtime code should be generated
>> in
>> > > > > CorrelatedCodeGenerator.
>> > > > > In CorrelatedCodeGenerator, we will know the TableFunction's Kind
>> of
>> > > > > `FunctionKind.Table` or `FunctionKind.ASYNC_TABLE`
>> > > > > For  `FunctionKind.ASYNC_TABLE` we can generate a
>> AsyncWaitOperator
>> > to
>> > > > > execute the async table function.
>> > > > >
>> > > > >
>> > > > > Thanks,
>> > > > > Aitozi.
>> > > > >
>> > > > >
>> > > > > Jing Ge <ji...@ververica.com.invalid> 于2023年5月29日周一 03:22写道：
>> > > > >
>> > > > >> Hi Aitozi,
>> > > > >>
>> > > > >> Thanks for the clarification. The naming "Lookup" might suggest
>> > using
>> > > it
>> > > > >> for table look up. But conceptually what the eval() method will
>> do
>> > is
>> > > to
>> > > > >> get a collection of results(Row, RowData) from the given keys.
>> How
>> > it
>> > > > will
>> > > > >> be done depends on the implementation, i.e. you can implement
>> your
>> > own
>> > > > >> Source[1][2]. The example in the FLIP should be able to be
>> handled
>> > in
>> > > > this
>> > > > >> way.
>> > > > >>
>> > > > >> Do you mean to support the AyncTableFunction beyond the
>> > > > LookupTableSource?
>> > > > >> It would be better if you could elaborate the proposed changes
>> wrt
>> > the
>> > > > >> CorrelatedCodeGenerator with more details. Thanks!
>> > > > >>
>> > > > >> Best regards,
>> > > > >> Jing
>> > > > >>
>> > > > >> [1]
>> > > > >>
>> > > > >>
>> > > >
>> > >
>> >
>> https://github.com/apache/flink/blob/678370b18e1b6c4a23e5ce08f8efd05675a0cc17/flink-table/flink-table-common/src/main/java/org/apache/flink/table/connector/source/LookupTableSource.java#L64
>> > > > >> [2]
>> > > > >>
>> > > > >>
>> > > >
>> > >
>> >
>> https://github.com/apache/flink/blob/678370b18e1b6c4a23e5ce08f8efd05675a0cc17/flink-table/flink-table-common/src/main/java/org/apache/flink/table/connector/source/AsyncTableFunctionProvider.java#L49
>> > > > >>
>> > > > >> On Sat, May 27, 2023 at 9:48 AM Aitozi <gj...@gmail.com>
>> > wrote:
>> > > > >>
>> > > > >> > Hi Jing,
>> > > > >> >     Thanks for your response. As stated in the FLIP, the
>> purpose
>> > of
>> > > > this
>> > > > >> > FLIP is meant to support
>> > > > >> > user-defined async table function. As described in flink
>> document
>> > > [1]
>> > > > >> >
>> > > > >> > Async table functions are special functions for table sources
>> that
>> > > > >> perform
>> > > > >> > > a lookup.
>> > > > >> > >
>> > > > >> >
>> > > > >> > So end user can not directly define and use async table
>> function
>> > > now.
>> > > > An
>> > > > >> > user case is reported in [2]
>> > > > >> >
>> > > > >> > So, in conclusion, no new interface is introduced, but we
>> extend
>> > the
>> > > > >> > ability to support user-defined async table function.
>> > > > >> >
>> > > > >> > [1]:
>> > > > >> >
>> > > > >> >
>> > > > >>
>> > > >
>> > >
>> >
>> https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/dev/table/functions/udfs/
>> > > > >> > [2]:
>> > > https://lists.apache.org/thread/qljwd40v5ntz6733cwcdr8s4z97b343b
>> > > > >> >
>> > > > >> > Thanks.
>> > > > >> > Aitozi.
>> > > > >> >
>> > > > >> >
>> > > > >> > Jing Ge <ji...@ververica.com.invalid> 于2023年5月27日周六 06:40写道：
>> > > > >> >
>> > > > >> > > Hi Aitozi,
>> > > > >> > >
>> > > > >> > > Thanks for your proposal. I am not quite sure if I understood
>> > your
>> > > > >> > thoughts
>> > > > >> > > correctly. You described a special case implementation of the
>> > > > >> > > AsyncTableFunction with on public API changes. Would you
>> please
>> > > > >> elaborate
>> > > > >> > > your purpose of writing a FLIP according to the FLIP
>> > > > documentation[1]?
>> > > > >> > > Thanks!
>> > > > >> > >
>> > > > >> > > [1]
>> > > > >> > >
>> > > > >> > >
>> > > > >> >
>> > > > >>
>> > > >
>> > >
>> >
>> https://cwiki.apache.org/confluence/display/FLINK/Flink+Improvement+Proposals
>> > > > >> > >
>> > > > >> > > Best regards,
>> > > > >> > > Jing
>> > > > >> > >
>> > > > >> > > On Wed, May 24, 2023 at 1:07 PM Aitozi <gjying1314@gmail.com
>> >
>> > > > wrote:
>> > > > >> > >
>> > > > >> > > > May I ask for some feedback  :D
>> > > > >> > > >
>> > > > >> > > > Thanks,
>> > > > >> > > > Aitozi
>> > > > >> > > >
>> > > > >> > > > Aitozi <gj...@gmail.com> 于2023年5月23日周二 19:14写道：
>> > > > >> > > > >
>> > > > >> > > > > Just catch an user case report from Giannis Polyzos for
>> this
>> > > > >> usage:
>> > > > >> > > > >
>> > > > >> > > > >
>> > > > https://lists.apache.org/thread/qljwd40v5ntz6733cwcdr8s4z97b343b
>> > > > >> > > > >
>> > > > >> > > > > Aitozi <gj...@gmail.com> 于2023年5月23日周二 17:45写道：
>> > > > >> > > > > >
>> > > > >> > > > > > Hi guys,
>> > > > >> > > > > >     I want to bring up a discussion about adding
>> support
>> > of
>> > > > User
>> > > > >> > > > > > Defined AsyncTableFunction in Flink.
>> > > > >> > > > > > Currently, async table function are special functions
>> for
>> > > > table
>> > > > >> > > source
>> > > > >> > > > > > to perform
>> > > > >> > > > > > async lookup. However, it's worth to support the user
>> > > defined
>> > > > >> async
>> > > > >> > > > > > table function.
>> > > > >> > > > > > Because, in this way, the end SQL user can leverage it
>> to
>> > > > >> perform
>> > > > >> > the
>> > > > >> > > > > > async operation
>> > > > >> > > > > > which is useful to maximum the system throughput
>> > especially
>> > > > for
>> > > > >> IO
>> > > > >> > > > > > bottleneck case.
>> > > > >> > > > > >
>> > > > >> > > > > > You can find some more detail in [1].
>> > > > >> > > > > >
>> > > > >> > > > > > Looking forward to feedback
>> > > > >> > > > > >
>> > > > >> > > > > >
>> > > > >> > > > > > [1]:
>> > > > >> > > >
>> > > > >> > >
>> > > > >> >
>> > > > >>
>> > > >
>> > >
>> >
>> https://cwiki.apache.org/confluence/display/FLINK/%5BFLIP-313%5D+Add+support+of+User+Defined+AsyncTableFunction
>> > > > >> > > > > >
>> > > > >> > > > > > Thanks,
>> > > > >> > > > > > Aitozi.
>> > > > >> > > >
>> > > > >> > >
>> > > > >> >
>> > > > >>
>> > > > >
>> > > >
>> > >
>> >
>>
>

Re: [DISCUSS] FLIP-313 Add support of User Defined AsyncTableFunction

Posted by Aitozi <gj...@gmail.com>.

Hi Jing,

    > what is the difference between the RPC call or query you mentioned
and the lookup in a very
general way

I think the RPC call or query service is quite similar to the lookup join.
But lookup join should work
with `LookupTableSource`.

Let's see how we can perform an async RPC call with lookup join:

(1) Implement an AsyncTableFunction with RPC call logic.
(2) Implement a `LookupTableSource` connector run with the async udtf
defined in (1).
(3) Then define a DDL of this look up table in SQL

CREATE TEMPORARY TABLE Customers (
  id INT,
  name STRING,
  country STRING,
  zip STRING
) WITH (
  'connector' = 'custom'
);

(4) Run with the query as below:

SELECT o.order_id, o.total, c.country, c.zip
FROM Orders AS o
  JOIN Customers FOR SYSTEM_TIME AS OF o.proc_time AS c
    ON o.customer_id = c.id;

This example is from doc
<https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/dev/table/sql/queries/joins/#lookup-join>.You
can image the look up process as an async RPC call process.

Let's see how we can perform an async RPC call with lateral join:

(1) Implement an AsyncTableFunction with RPC call logic.
(2) Run query with

Create function f1 as '...' ;

SELECT o.order_id, o.total, c.country, c.zip FROM Orders  lateral table
(f1(order_id)) as T(...);

As you can see, the lateral join version is more simple and intuitive to
users. Users do not have to wrap a
LookupTableSource for the purpose of using async udtf.

In the end, We can also see the user defined async table function is an
enhancement of the current lateral table join
which only supports sync lateral join now.

Best,
Aitozi.


Jing Ge <ji...@ververica.com.invalid> 于2023年6月2日周五 19:37写道：

> Hi Aitozi,
>
> Thanks for the update. Just out of curiosity, what is the difference
> between the RPC call or query you mentioned and the lookup in a very
> general way? Since Lateral join is used in the FLIP. Is there any special
> thought for that? Sorry for asking so many questions. The FLIP contains
> limited information to understand the motivation.
>
> Best regards,
> Jing
>
> On Fri, Jun 2, 2023 at 3:48 AM Aitozi <gj...@gmail.com> wrote:
>
> > Hi Jing,
> >     I have updated the proposed changes to the FLIP. IMO, lookup has its
> > clear
> > async call requirement is due to its IO heavy operator. In our usage, sql
> > users have
> > logic to do some RPC call or query the third-party service which is also
> IO
> > intensive.
> > In these case, we'd like to leverage the async function to improve the
> > throughput.
> >
> > Thanks,
> > Aitozi.
> >
> > Jing Ge <ji...@ververica.com.invalid> 于2023年6月1日周四 22:55写道：
> >
> > > Hi Aitozi,
> > >
> > > Sorry for the late reply. Would you like to update the proposed changes
> > > with more details into the FLIP too?
> > > I got your point. It looks like a rational idea. However, since lookup
> > has
> > > its clear async call requirement, are there any real use cases that
> > > need this change? This will help us understand the motivation. After
> all,
> > > lateral join and temporal lookup join[1] are quite different.
> > >
> > > Best regards,
> > > Jing
> > >
> > >
> > > [1]
> > >
> > >
> >
> https://github.com/apache/flink/blob/d90a72da2fd601ca4e2a46700e91ec5b348de2ad/flink-table/flink-table-common/src/main/java/org/apache/flink/table/functions/AsyncTableFunction.java#L54
> > >
> > > On Wed, May 31, 2023 at 8:53 AM Aitozi <gj...@gmail.com> wrote:
> > >
> > > > Hi Jing,
> > > >     What do you think about it? Can we move forward this feature?
> > > >
> > > > Thanks,
> > > > Aitozi.
> > > >
> > > > Aitozi <gj...@gmail.com> 于2023年5月29日周一 09:56写道：
> > > >
> > > > > Hi Jing,
> > > > >     > "Do you mean to support the AyncTableFunction beyond the
> > > > > LookupTableSource?"
> > > > > Yes, I mean to support the AyncTableFunction beyond the
> > > > LookupTableSource.
> > > > >
> > > > > The "AsyncTableFunction" is the function with ability to be
> executed
> > > > async
> > > > > (with AsyncWaitOperator).
> > > > > The async lookup join is a one of usage of this. So, we don't have
> to
> > > > bind
> > > > > the AyncTableFunction with LookupTableSource.
> > > > > If User-defined AsyncTableFunction is supported, user can directly
> > use
> > > > > lateral table syntax to perform async operation.
> > > > >
> > > > > > "It would be better if you could elaborate the proposed changes
> wrt
> > > the
> > > > > CorrelatedCodeGenerator with more details"
> > > > >
> > > > > In the proposal, we use lateral table syntax to support the async
> > table
> > > > > function. So the planner will also treat this statement to a
> > > > > CommonExecCorrelate node. So the runtime code should be generated
> in
> > > > > CorrelatedCodeGenerator.
> > > > > In CorrelatedCodeGenerator, we will know the TableFunction's Kind
> of
> > > > > `FunctionKind.Table` or `FunctionKind.ASYNC_TABLE`
> > > > > For  `FunctionKind.ASYNC_TABLE` we can generate a AsyncWaitOperator
> > to
> > > > > execute the async table function.
> > > > >
> > > > >
> > > > > Thanks,
> > > > > Aitozi.
> > > > >
> > > > >
> > > > > Jing Ge <ji...@ververica.com.invalid> 于2023年5月29日周一 03:22写道：
> > > > >
> > > > >> Hi Aitozi,
> > > > >>
> > > > >> Thanks for the clarification. The naming "Lookup" might suggest
> > using
> > > it
> > > > >> for table look up. But conceptually what the eval() method will do
> > is
> > > to
> > > > >> get a collection of results(Row, RowData) from the given keys. How
> > it
> > > > will
> > > > >> be done depends on the implementation, i.e. you can implement your
> > own
> > > > >> Source[1][2]. The example in the FLIP should be able to be handled
> > in
> > > > this
> > > > >> way.
> > > > >>
> > > > >> Do you mean to support the AyncTableFunction beyond the
> > > > LookupTableSource?
> > > > >> It would be better if you could elaborate the proposed changes wrt
> > the
> > > > >> CorrelatedCodeGenerator with more details. Thanks!
> > > > >>
> > > > >> Best regards,
> > > > >> Jing
> > > > >>
> > > > >> [1]
> > > > >>
> > > > >>
> > > >
> > >
> >
> https://github.com/apache/flink/blob/678370b18e1b6c4a23e5ce08f8efd05675a0cc17/flink-table/flink-table-common/src/main/java/org/apache/flink/table/connector/source/LookupTableSource.java#L64
> > > > >> [2]
> > > > >>
> > > > >>
> > > >
> > >
> >
> https://github.com/apache/flink/blob/678370b18e1b6c4a23e5ce08f8efd05675a0cc17/flink-table/flink-table-common/src/main/java/org/apache/flink/table/connector/source/AsyncTableFunctionProvider.java#L49
> > > > >>
> > > > >> On Sat, May 27, 2023 at 9:48 AM Aitozi <gj...@gmail.com>
> > wrote:
> > > > >>
> > > > >> > Hi Jing,
> > > > >> >     Thanks for your response. As stated in the FLIP, the purpose
> > of
> > > > this
> > > > >> > FLIP is meant to support
> > > > >> > user-defined async table function. As described in flink
> document
> > > [1]
> > > > >> >
> > > > >> > Async table functions are special functions for table sources
> that
> > > > >> perform
> > > > >> > > a lookup.
> > > > >> > >
> > > > >> >
> > > > >> > So end user can not directly define and use async table function
> > > now.
> > > > An
> > > > >> > user case is reported in [2]
> > > > >> >
> > > > >> > So, in conclusion, no new interface is introduced, but we extend
> > the
> > > > >> > ability to support user-defined async table function.
> > > > >> >
> > > > >> > [1]:
> > > > >> >
> > > > >> >
> > > > >>
> > > >
> > >
> >
> https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/dev/table/functions/udfs/
> > > > >> > [2]:
> > > https://lists.apache.org/thread/qljwd40v5ntz6733cwcdr8s4z97b343b
> > > > >> >
> > > > >> > Thanks.
> > > > >> > Aitozi.
> > > > >> >
> > > > >> >
> > > > >> > Jing Ge <ji...@ververica.com.invalid> 于2023年5月27日周六 06:40写道：
> > > > >> >
> > > > >> > > Hi Aitozi,
> > > > >> > >
> > > > >> > > Thanks for your proposal. I am not quite sure if I understood
> > your
> > > > >> > thoughts
> > > > >> > > correctly. You described a special case implementation of the
> > > > >> > > AsyncTableFunction with on public API changes. Would you
> please
> > > > >> elaborate
> > > > >> > > your purpose of writing a FLIP according to the FLIP
> > > > documentation[1]?
> > > > >> > > Thanks!
> > > > >> > >
> > > > >> > > [1]
> > > > >> > >
> > > > >> > >
> > > > >> >
> > > > >>
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/Flink+Improvement+Proposals
> > > > >> > >
> > > > >> > > Best regards,
> > > > >> > > Jing
> > > > >> > >
> > > > >> > > On Wed, May 24, 2023 at 1:07 PM Aitozi <gj...@gmail.com>
> > > > wrote:
> > > > >> > >
> > > > >> > > > May I ask for some feedback  :D
> > > > >> > > >
> > > > >> > > > Thanks,
> > > > >> > > > Aitozi
> > > > >> > > >
> > > > >> > > > Aitozi <gj...@gmail.com> 于2023年5月23日周二 19:14写道：
> > > > >> > > > >
> > > > >> > > > > Just catch an user case report from Giannis Polyzos for
> this
> > > > >> usage:
> > > > >> > > > >
> > > > >> > > > >
> > > > https://lists.apache.org/thread/qljwd40v5ntz6733cwcdr8s4z97b343b
> > > > >> > > > >
> > > > >> > > > > Aitozi <gj...@gmail.com> 于2023年5月23日周二 17:45写道：
> > > > >> > > > > >
> > > > >> > > > > > Hi guys,
> > > > >> > > > > >     I want to bring up a discussion about adding support
> > of
> > > > User
> > > > >> > > > > > Defined AsyncTableFunction in Flink.
> > > > >> > > > > > Currently, async table function are special functions
> for
> > > > table
> > > > >> > > source
> > > > >> > > > > > to perform
> > > > >> > > > > > async lookup. However, it's worth to support the user
> > > defined
> > > > >> async
> > > > >> > > > > > table function.
> > > > >> > > > > > Because, in this way, the end SQL user can leverage it
> to
> > > > >> perform
> > > > >> > the
> > > > >> > > > > > async operation
> > > > >> > > > > > which is useful to maximum the system throughput
> > especially
> > > > for
> > > > >> IO
> > > > >> > > > > > bottleneck case.
> > > > >> > > > > >
> > > > >> > > > > > You can find some more detail in [1].
> > > > >> > > > > >
> > > > >> > > > > > Looking forward to feedback
> > > > >> > > > > >
> > > > >> > > > > >
> > > > >> > > > > > [1]:
> > > > >> > > >
> > > > >> > >
> > > > >> >
> > > > >>
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/%5BFLIP-313%5D+Add+support+of+User+Defined+AsyncTableFunction
> > > > >> > > > > >
> > > > >> > > > > > Thanks,
> > > > >> > > > > > Aitozi.
> > > > >> > > >
> > > > >> > >
> > > > >> >
> > > > >>
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] FLIP-313 Add support of User Defined AsyncTableFunction

Posted by Jing Ge <ji...@ververica.com.INVALID>.

Hi Aitozi,

Thanks for the update. Just out of curiosity, what is the difference
between the RPC call or query you mentioned and the lookup in a very
general way? Since Lateral join is used in the FLIP. Is there any special
thought for that? Sorry for asking so many questions. The FLIP contains
limited information to understand the motivation.

Best regards,
Jing

On Fri, Jun 2, 2023 at 3:48 AM Aitozi <gj...@gmail.com> wrote:

> Hi Jing,
>     I have updated the proposed changes to the FLIP. IMO, lookup has its
> clear
> async call requirement is due to its IO heavy operator. In our usage, sql
> users have
> logic to do some RPC call or query the third-party service which is also IO
> intensive.
> In these case, we'd like to leverage the async function to improve the
> throughput.
>
> Thanks,
> Aitozi.
>
> Jing Ge <ji...@ververica.com.invalid> 于2023年6月1日周四 22:55写道：
>
> > Hi Aitozi,
> >
> > Sorry for the late reply. Would you like to update the proposed changes
> > with more details into the FLIP too?
> > I got your point. It looks like a rational idea. However, since lookup
> has
> > its clear async call requirement, are there any real use cases that
> > need this change? This will help us understand the motivation. After all,
> > lateral join and temporal lookup join[1] are quite different.
> >
> > Best regards,
> > Jing
> >
> >
> > [1]
> >
> >
> https://github.com/apache/flink/blob/d90a72da2fd601ca4e2a46700e91ec5b348de2ad/flink-table/flink-table-common/src/main/java/org/apache/flink/table/functions/AsyncTableFunction.java#L54
> >
> > On Wed, May 31, 2023 at 8:53 AM Aitozi <gj...@gmail.com> wrote:
> >
> > > Hi Jing,
> > >     What do you think about it? Can we move forward this feature?
> > >
> > > Thanks,
> > > Aitozi.
> > >
> > > Aitozi <gj...@gmail.com> 于2023年5月29日周一 09:56写道：
> > >
> > > > Hi Jing,
> > > >     > "Do you mean to support the AyncTableFunction beyond the
> > > > LookupTableSource?"
> > > > Yes, I mean to support the AyncTableFunction beyond the
> > > LookupTableSource.
> > > >
> > > > The "AsyncTableFunction" is the function with ability to be executed
> > > async
> > > > (with AsyncWaitOperator).
> > > > The async lookup join is a one of usage of this. So, we don't have to
> > > bind
> > > > the AyncTableFunction with LookupTableSource.
> > > > If User-defined AsyncTableFunction is supported, user can directly
> use
> > > > lateral table syntax to perform async operation.
> > > >
> > > > > "It would be better if you could elaborate the proposed changes wrt
> > the
> > > > CorrelatedCodeGenerator with more details"
> > > >
> > > > In the proposal, we use lateral table syntax to support the async
> table
> > > > function. So the planner will also treat this statement to a
> > > > CommonExecCorrelate node. So the runtime code should be generated in
> > > > CorrelatedCodeGenerator.
> > > > In CorrelatedCodeGenerator, we will know the TableFunction's Kind of
> > > > `FunctionKind.Table` or `FunctionKind.ASYNC_TABLE`
> > > > For  `FunctionKind.ASYNC_TABLE` we can generate a AsyncWaitOperator
> to
> > > > execute the async table function.
> > > >
> > > >
> > > > Thanks,
> > > > Aitozi.
> > > >
> > > >
> > > > Jing Ge <ji...@ververica.com.invalid> 于2023年5月29日周一 03:22写道：
> > > >
> > > >> Hi Aitozi,
> > > >>
> > > >> Thanks for the clarification. The naming "Lookup" might suggest
> using
> > it
> > > >> for table look up. But conceptually what the eval() method will do
> is
> > to
> > > >> get a collection of results(Row, RowData) from the given keys. How
> it
> > > will
> > > >> be done depends on the implementation, i.e. you can implement your
> own
> > > >> Source[1][2]. The example in the FLIP should be able to be handled
> in
> > > this
> > > >> way.
> > > >>
> > > >> Do you mean to support the AyncTableFunction beyond the
> > > LookupTableSource?
> > > >> It would be better if you could elaborate the proposed changes wrt
> the
> > > >> CorrelatedCodeGenerator with more details. Thanks!
> > > >>
> > > >> Best regards,
> > > >> Jing
> > > >>
> > > >> [1]
> > > >>
> > > >>
> > >
> >
> https://github.com/apache/flink/blob/678370b18e1b6c4a23e5ce08f8efd05675a0cc17/flink-table/flink-table-common/src/main/java/org/apache/flink/table/connector/source/LookupTableSource.java#L64
> > > >> [2]
> > > >>
> > > >>
> > >
> >
> https://github.com/apache/flink/blob/678370b18e1b6c4a23e5ce08f8efd05675a0cc17/flink-table/flink-table-common/src/main/java/org/apache/flink/table/connector/source/AsyncTableFunctionProvider.java#L49
> > > >>
> > > >> On Sat, May 27, 2023 at 9:48 AM Aitozi <gj...@gmail.com>
> wrote:
> > > >>
> > > >> > Hi Jing,
> > > >> >     Thanks for your response. As stated in the FLIP, the purpose
> of
> > > this
> > > >> > FLIP is meant to support
> > > >> > user-defined async table function. As described in flink document
> > [1]
> > > >> >
> > > >> > Async table functions are special functions for table sources that
> > > >> perform
> > > >> > > a lookup.
> > > >> > >
> > > >> >
> > > >> > So end user can not directly define and use async table function
> > now.
> > > An
> > > >> > user case is reported in [2]
> > > >> >
> > > >> > So, in conclusion, no new interface is introduced, but we extend
> the
> > > >> > ability to support user-defined async table function.
> > > >> >
> > > >> > [1]:
> > > >> >
> > > >> >
> > > >>
> > >
> >
> https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/dev/table/functions/udfs/
> > > >> > [2]:
> > https://lists.apache.org/thread/qljwd40v5ntz6733cwcdr8s4z97b343b
> > > >> >
> > > >> > Thanks.
> > > >> > Aitozi.
> > > >> >
> > > >> >
> > > >> > Jing Ge <ji...@ververica.com.invalid> 于2023年5月27日周六 06:40写道：
> > > >> >
> > > >> > > Hi Aitozi,
> > > >> > >
> > > >> > > Thanks for your proposal. I am not quite sure if I understood
> your
> > > >> > thoughts
> > > >> > > correctly. You described a special case implementation of the
> > > >> > > AsyncTableFunction with on public API changes. Would you please
> > > >> elaborate
> > > >> > > your purpose of writing a FLIP according to the FLIP
> > > documentation[1]?
> > > >> > > Thanks!
> > > >> > >
> > > >> > > [1]
> > > >> > >
> > > >> > >
> > > >> >
> > > >>
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/Flink+Improvement+Proposals
> > > >> > >
> > > >> > > Best regards,
> > > >> > > Jing
> > > >> > >
> > > >> > > On Wed, May 24, 2023 at 1:07 PM Aitozi <gj...@gmail.com>
> > > wrote:
> > > >> > >
> > > >> > > > May I ask for some feedback  :D
> > > >> > > >
> > > >> > > > Thanks,
> > > >> > > > Aitozi
> > > >> > > >
> > > >> > > > Aitozi <gj...@gmail.com> 于2023年5月23日周二 19:14写道：
> > > >> > > > >
> > > >> > > > > Just catch an user case report from Giannis Polyzos for this
> > > >> usage:
> > > >> > > > >
> > > >> > > > >
> > > https://lists.apache.org/thread/qljwd40v5ntz6733cwcdr8s4z97b343b
> > > >> > > > >
> > > >> > > > > Aitozi <gj...@gmail.com> 于2023年5月23日周二 17:45写道：
> > > >> > > > > >
> > > >> > > > > > Hi guys,
> > > >> > > > > >     I want to bring up a discussion about adding support
> of
> > > User
> > > >> > > > > > Defined AsyncTableFunction in Flink.
> > > >> > > > > > Currently, async table function are special functions for
> > > table
> > > >> > > source
> > > >> > > > > > to perform
> > > >> > > > > > async lookup. However, it's worth to support the user
> > defined
> > > >> async
> > > >> > > > > > table function.
> > > >> > > > > > Because, in this way, the end SQL user can leverage it to
> > > >> perform
> > > >> > the
> > > >> > > > > > async operation
> > > >> > > > > > which is useful to maximum the system throughput
> especially
> > > for
> > > >> IO
> > > >> > > > > > bottleneck case.
> > > >> > > > > >
> > > >> > > > > > You can find some more detail in [1].
> > > >> > > > > >
> > > >> > > > > > Looking forward to feedback
> > > >> > > > > >
> > > >> > > > > >
> > > >> > > > > > [1]:
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/%5BFLIP-313%5D+Add+support+of+User+Defined+AsyncTableFunction
> > > >> > > > > >
> > > >> > > > > > Thanks,
> > > >> > > > > > Aitozi.
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > > >
> > >
> >
>

Re: [DISCUSS] FLIP-313 Add support of User Defined AsyncTableFunction

Posted by Aitozi <gj...@gmail.com>.

Hi Jing,
    I have updated the proposed changes to the FLIP. IMO, lookup has its
clear
async call requirement is due to its IO heavy operator. In our usage, sql
users have
logic to do some RPC call or query the third-party service which is also IO
intensive.
In these case, we'd like to leverage the async function to improve the
throughput.

Thanks,
Aitozi.

Jing Ge <ji...@ververica.com.invalid> 于2023年6月1日周四 22:55写道：

> Hi Aitozi,
>
> Sorry for the late reply. Would you like to update the proposed changes
> with more details into the FLIP too?
> I got your point. It looks like a rational idea. However, since lookup has
> its clear async call requirement, are there any real use cases that
> need this change? This will help us understand the motivation. After all,
> lateral join and temporal lookup join[1] are quite different.
>
> Best regards,
> Jing
>
>
> [1]
>
> https://github.com/apache/flink/blob/d90a72da2fd601ca4e2a46700e91ec5b348de2ad/flink-table/flink-table-common/src/main/java/org/apache/flink/table/functions/AsyncTableFunction.java#L54
>
> On Wed, May 31, 2023 at 8:53 AM Aitozi <gj...@gmail.com> wrote:
>
> > Hi Jing,
> >     What do you think about it? Can we move forward this feature?
> >
> > Thanks,
> > Aitozi.
> >
> > Aitozi <gj...@gmail.com> 于2023年5月29日周一 09:56写道：
> >
> > > Hi Jing,
> > >     > "Do you mean to support the AyncTableFunction beyond the
> > > LookupTableSource?"
> > > Yes, I mean to support the AyncTableFunction beyond the
> > LookupTableSource.
> > >
> > > The "AsyncTableFunction" is the function with ability to be executed
> > async
> > > (with AsyncWaitOperator).
> > > The async lookup join is a one of usage of this. So, we don't have to
> > bind
> > > the AyncTableFunction with LookupTableSource.
> > > If User-defined AsyncTableFunction is supported, user can directly use
> > > lateral table syntax to perform async operation.
> > >
> > > > "It would be better if you could elaborate the proposed changes wrt
> the
> > > CorrelatedCodeGenerator with more details"
> > >
> > > In the proposal, we use lateral table syntax to support the async table
> > > function. So the planner will also treat this statement to a
> > > CommonExecCorrelate node. So the runtime code should be generated in
> > > CorrelatedCodeGenerator.
> > > In CorrelatedCodeGenerator, we will know the TableFunction's Kind of
> > > `FunctionKind.Table` or `FunctionKind.ASYNC_TABLE`
> > > For  `FunctionKind.ASYNC_TABLE` we can generate a AsyncWaitOperator to
> > > execute the async table function.
> > >
> > >
> > > Thanks,
> > > Aitozi.
> > >
> > >
> > > Jing Ge <ji...@ververica.com.invalid> 于2023年5月29日周一 03:22写道：
> > >
> > >> Hi Aitozi,
> > >>
> > >> Thanks for the clarification. The naming "Lookup" might suggest using
> it
> > >> for table look up. But conceptually what the eval() method will do is
> to
> > >> get a collection of results(Row, RowData) from the given keys. How it
> > will
> > >> be done depends on the implementation, i.e. you can implement your own
> > >> Source[1][2]. The example in the FLIP should be able to be handled in
> > this
> > >> way.
> > >>
> > >> Do you mean to support the AyncTableFunction beyond the
> > LookupTableSource?
> > >> It would be better if you could elaborate the proposed changes wrt the
> > >> CorrelatedCodeGenerator with more details. Thanks!
> > >>
> > >> Best regards,
> > >> Jing
> > >>
> > >> [1]
> > >>
> > >>
> >
> https://github.com/apache/flink/blob/678370b18e1b6c4a23e5ce08f8efd05675a0cc17/flink-table/flink-table-common/src/main/java/org/apache/flink/table/connector/source/LookupTableSource.java#L64
> > >> [2]
> > >>
> > >>
> >
> https://github.com/apache/flink/blob/678370b18e1b6c4a23e5ce08f8efd05675a0cc17/flink-table/flink-table-common/src/main/java/org/apache/flink/table/connector/source/AsyncTableFunctionProvider.java#L49
> > >>
> > >> On Sat, May 27, 2023 at 9:48 AM Aitozi <gj...@gmail.com> wrote:
> > >>
> > >> > Hi Jing,
> > >> >     Thanks for your response. As stated in the FLIP, the purpose of
> > this
> > >> > FLIP is meant to support
> > >> > user-defined async table function. As described in flink document
> [1]
> > >> >
> > >> > Async table functions are special functions for table sources that
> > >> perform
> > >> > > a lookup.
> > >> > >
> > >> >
> > >> > So end user can not directly define and use async table function
> now.
> > An
> > >> > user case is reported in [2]
> > >> >
> > >> > So, in conclusion, no new interface is introduced, but we extend the
> > >> > ability to support user-defined async table function.
> > >> >
> > >> > [1]:
> > >> >
> > >> >
> > >>
> >
> https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/dev/table/functions/udfs/
> > >> > [2]:
> https://lists.apache.org/thread/qljwd40v5ntz6733cwcdr8s4z97b343b
> > >> >
> > >> > Thanks.
> > >> > Aitozi.
> > >> >
> > >> >
> > >> > Jing Ge <ji...@ververica.com.invalid> 于2023年5月27日周六 06:40写道：
> > >> >
> > >> > > Hi Aitozi,
> > >> > >
> > >> > > Thanks for your proposal. I am not quite sure if I understood your
> > >> > thoughts
> > >> > > correctly. You described a special case implementation of the
> > >> > > AsyncTableFunction with on public API changes. Would you please
> > >> elaborate
> > >> > > your purpose of writing a FLIP according to the FLIP
> > documentation[1]?
> > >> > > Thanks!
> > >> > >
> > >> > > [1]
> > >> > >
> > >> > >
> > >> >
> > >>
> >
> https://cwiki.apache.org/confluence/display/FLINK/Flink+Improvement+Proposals
> > >> > >
> > >> > > Best regards,
> > >> > > Jing
> > >> > >
> > >> > > On Wed, May 24, 2023 at 1:07 PM Aitozi <gj...@gmail.com>
> > wrote:
> > >> > >
> > >> > > > May I ask for some feedback  :D
> > >> > > >
> > >> > > > Thanks,
> > >> > > > Aitozi
> > >> > > >
> > >> > > > Aitozi <gj...@gmail.com> 于2023年5月23日周二 19:14写道：
> > >> > > > >
> > >> > > > > Just catch an user case report from Giannis Polyzos for this
> > >> usage:
> > >> > > > >
> > >> > > > >
> > https://lists.apache.org/thread/qljwd40v5ntz6733cwcdr8s4z97b343b
> > >> > > > >
> > >> > > > > Aitozi <gj...@gmail.com> 于2023年5月23日周二 17:45写道：
> > >> > > > > >
> > >> > > > > > Hi guys,
> > >> > > > > >     I want to bring up a discussion about adding support of
> > User
> > >> > > > > > Defined AsyncTableFunction in Flink.
> > >> > > > > > Currently, async table function are special functions for
> > table
> > >> > > source
> > >> > > > > > to perform
> > >> > > > > > async lookup. However, it's worth to support the user
> defined
> > >> async
> > >> > > > > > table function.
> > >> > > > > > Because, in this way, the end SQL user can leverage it to
> > >> perform
> > >> > the
> > >> > > > > > async operation
> > >> > > > > > which is useful to maximum the system throughput especially
> > for
> > >> IO
> > >> > > > > > bottleneck case.
> > >> > > > > >
> > >> > > > > > You can find some more detail in [1].
> > >> > > > > >
> > >> > > > > > Looking forward to feedback
> > >> > > > > >
> > >> > > > > >
> > >> > > > > > [1]:
> > >> > > >
> > >> > >
> > >> >
> > >>
> >
> https://cwiki.apache.org/confluence/display/FLINK/%5BFLIP-313%5D+Add+support+of+User+Defined+AsyncTableFunction
> > >> > > > > >
> > >> > > > > > Thanks,
> > >> > > > > > Aitozi.
> > >> > > >
> > >> > >
> > >> >
> > >>
> > >
> >
>

Re: [DISCUSS] FLIP-313 Add support of User Defined AsyncTableFunction

Posted by Jing Ge <ji...@ververica.com.INVALID>.

Hi Aitozi,

Sorry for the late reply. Would you like to update the proposed changes
with more details into the FLIP too?
I got your point. It looks like a rational idea. However, since lookup has
its clear async call requirement, are there any real use cases that
need this change? This will help us understand the motivation. After all,
lateral join and temporal lookup join[1] are quite different.

Best regards,
Jing


[1]
https://github.com/apache/flink/blob/d90a72da2fd601ca4e2a46700e91ec5b348de2ad/flink-table/flink-table-common/src/main/java/org/apache/flink/table/functions/AsyncTableFunction.java#L54

On Wed, May 31, 2023 at 8:53 AM Aitozi <gj...@gmail.com> wrote:

> Hi Jing,
>     What do you think about it? Can we move forward this feature?
>
> Thanks,
> Aitozi.
>
> Aitozi <gj...@gmail.com> 于2023年5月29日周一 09:56写道：
>
> > Hi Jing,
> >     > "Do you mean to support the AyncTableFunction beyond the
> > LookupTableSource?"
> > Yes, I mean to support the AyncTableFunction beyond the
> LookupTableSource.
> >
> > The "AsyncTableFunction" is the function with ability to be executed
> async
> > (with AsyncWaitOperator).
> > The async lookup join is a one of usage of this. So, we don't have to
> bind
> > the AyncTableFunction with LookupTableSource.
> > If User-defined AsyncTableFunction is supported, user can directly use
> > lateral table syntax to perform async operation.
> >
> > > "It would be better if you could elaborate the proposed changes wrt the
> > CorrelatedCodeGenerator with more details"
> >
> > In the proposal, we use lateral table syntax to support the async table
> > function. So the planner will also treat this statement to a
> > CommonExecCorrelate node. So the runtime code should be generated in
> > CorrelatedCodeGenerator.
> > In CorrelatedCodeGenerator, we will know the TableFunction's Kind of
> > `FunctionKind.Table` or `FunctionKind.ASYNC_TABLE`
> > For  `FunctionKind.ASYNC_TABLE` we can generate a AsyncWaitOperator to
> > execute the async table function.
> >
> >
> > Thanks,
> > Aitozi.
> >
> >
> > Jing Ge <ji...@ververica.com.invalid> 于2023年5月29日周一 03:22写道：
> >
> >> Hi Aitozi,
> >>
> >> Thanks for the clarification. The naming "Lookup" might suggest using it
> >> for table look up. But conceptually what the eval() method will do is to
> >> get a collection of results(Row, RowData) from the given keys. How it
> will
> >> be done depends on the implementation, i.e. you can implement your own
> >> Source[1][2]. The example in the FLIP should be able to be handled in
> this
> >> way.
> >>
> >> Do you mean to support the AyncTableFunction beyond the
> LookupTableSource?
> >> It would be better if you could elaborate the proposed changes wrt the
> >> CorrelatedCodeGenerator with more details. Thanks!
> >>
> >> Best regards,
> >> Jing
> >>
> >> [1]
> >>
> >>
> https://github.com/apache/flink/blob/678370b18e1b6c4a23e5ce08f8efd05675a0cc17/flink-table/flink-table-common/src/main/java/org/apache/flink/table/connector/source/LookupTableSource.java#L64
> >> [2]
> >>
> >>
> https://github.com/apache/flink/blob/678370b18e1b6c4a23e5ce08f8efd05675a0cc17/flink-table/flink-table-common/src/main/java/org/apache/flink/table/connector/source/AsyncTableFunctionProvider.java#L49
> >>
> >> On Sat, May 27, 2023 at 9:48 AM Aitozi <gj...@gmail.com> wrote:
> >>
> >> > Hi Jing,
> >> >     Thanks for your response. As stated in the FLIP, the purpose of
> this
> >> > FLIP is meant to support
> >> > user-defined async table function. As described in flink document [1]
> >> >
> >> > Async table functions are special functions for table sources that
> >> perform
> >> > > a lookup.
> >> > >
> >> >
> >> > So end user can not directly define and use async table function now.
> An
> >> > user case is reported in [2]
> >> >
> >> > So, in conclusion, no new interface is introduced, but we extend the
> >> > ability to support user-defined async table function.
> >> >
> >> > [1]:
> >> >
> >> >
> >>
> https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/dev/table/functions/udfs/
> >> > [2]: https://lists.apache.org/thread/qljwd40v5ntz6733cwcdr8s4z97b343b
> >> >
> >> > Thanks.
> >> > Aitozi.
> >> >
> >> >
> >> > Jing Ge <ji...@ververica.com.invalid> 于2023年5月27日周六 06:40写道：
> >> >
> >> > > Hi Aitozi,
> >> > >
> >> > > Thanks for your proposal. I am not quite sure if I understood your
> >> > thoughts
> >> > > correctly. You described a special case implementation of the
> >> > > AsyncTableFunction with on public API changes. Would you please
> >> elaborate
> >> > > your purpose of writing a FLIP according to the FLIP
> documentation[1]?
> >> > > Thanks!
> >> > >
> >> > > [1]
> >> > >
> >> > >
> >> >
> >>
> https://cwiki.apache.org/confluence/display/FLINK/Flink+Improvement+Proposals
> >> > >
> >> > > Best regards,
> >> > > Jing
> >> > >
> >> > > On Wed, May 24, 2023 at 1:07 PM Aitozi <gj...@gmail.com>
> wrote:
> >> > >
> >> > > > May I ask for some feedback  :D
> >> > > >
> >> > > > Thanks,
> >> > > > Aitozi
> >> > > >
> >> > > > Aitozi <gj...@gmail.com> 于2023年5月23日周二 19:14写道：
> >> > > > >
> >> > > > > Just catch an user case report from Giannis Polyzos for this
> >> usage:
> >> > > > >
> >> > > > >
> https://lists.apache.org/thread/qljwd40v5ntz6733cwcdr8s4z97b343b
> >> > > > >
> >> > > > > Aitozi <gj...@gmail.com> 于2023年5月23日周二 17:45写道：
> >> > > > > >
> >> > > > > > Hi guys,
> >> > > > > >     I want to bring up a discussion about adding support of
> User
> >> > > > > > Defined AsyncTableFunction in Flink.
> >> > > > > > Currently, async table function are special functions for
> table
> >> > > source
> >> > > > > > to perform
> >> > > > > > async lookup. However, it's worth to support the user defined
> >> async
> >> > > > > > table function.
> >> > > > > > Because, in this way, the end SQL user can leverage it to
> >> perform
> >> > the
> >> > > > > > async operation
> >> > > > > > which is useful to maximum the system throughput especially
> for
> >> IO
> >> > > > > > bottleneck case.
> >> > > > > >
> >> > > > > > You can find some more detail in [1].
> >> > > > > >
> >> > > > > > Looking forward to feedback
> >> > > > > >
> >> > > > > >
> >> > > > > > [1]:
> >> > > >
> >> > >
> >> >
> >>
> https://cwiki.apache.org/confluence/display/FLINK/%5BFLIP-313%5D+Add+support+of+User+Defined+AsyncTableFunction
> >> > > > > >
> >> > > > > > Thanks,
> >> > > > > > Aitozi.
> >> > > >
> >> > >
> >> >
> >>
> >
>

Re: [DISCUSS] FLIP-313 Add support of User Defined AsyncTableFunction

Posted by Aitozi <gj...@gmail.com>.

Hi Jing,
    What do you think about it? Can we move forward this feature?

Thanks,
Aitozi.

Aitozi <gj...@gmail.com> 于2023年5月29日周一 09:56写道：

> Hi Jing,
>     > "Do you mean to support the AyncTableFunction beyond the
> LookupTableSource?"
> Yes, I mean to support the AyncTableFunction beyond the LookupTableSource.
>
> The "AsyncTableFunction" is the function with ability to be executed async
> (with AsyncWaitOperator).
> The async lookup join is a one of usage of this. So, we don't have to bind
> the AyncTableFunction with LookupTableSource.
> If User-defined AsyncTableFunction is supported, user can directly use
> lateral table syntax to perform async operation.
>
> > "It would be better if you could elaborate the proposed changes wrt the
> CorrelatedCodeGenerator with more details"
>
> In the proposal, we use lateral table syntax to support the async table
> function. So the planner will also treat this statement to a
> CommonExecCorrelate node. So the runtime code should be generated in
> CorrelatedCodeGenerator.
> In CorrelatedCodeGenerator, we will know the TableFunction's Kind of
> `FunctionKind.Table` or `FunctionKind.ASYNC_TABLE`
> For  `FunctionKind.ASYNC_TABLE` we can generate a AsyncWaitOperator to
> execute the async table function.
>
>
> Thanks,
> Aitozi.
>
>
> Jing Ge <ji...@ververica.com.invalid> 于2023年5月29日周一 03:22写道：
>
>> Hi Aitozi,
>>
>> Thanks for the clarification. The naming "Lookup" might suggest using it
>> for table look up. But conceptually what the eval() method will do is to
>> get a collection of results(Row, RowData) from the given keys. How it will
>> be done depends on the implementation, i.e. you can implement your own
>> Source[1][2]. The example in the FLIP should be able to be handled in this
>> way.
>>
>> Do you mean to support the AyncTableFunction beyond the LookupTableSource?
>> It would be better if you could elaborate the proposed changes wrt the
>> CorrelatedCodeGenerator with more details. Thanks!
>>
>> Best regards,
>> Jing
>>
>> [1]
>>
>> https://github.com/apache/flink/blob/678370b18e1b6c4a23e5ce08f8efd05675a0cc17/flink-table/flink-table-common/src/main/java/org/apache/flink/table/connector/source/LookupTableSource.java#L64
>> [2]
>>
>> https://github.com/apache/flink/blob/678370b18e1b6c4a23e5ce08f8efd05675a0cc17/flink-table/flink-table-common/src/main/java/org/apache/flink/table/connector/source/AsyncTableFunctionProvider.java#L49
>>
>> On Sat, May 27, 2023 at 9:48 AM Aitozi <gj...@gmail.com> wrote:
>>
>> > Hi Jing,
>> >     Thanks for your response. As stated in the FLIP, the purpose of this
>> > FLIP is meant to support
>> > user-defined async table function. As described in flink document [1]
>> >
>> > Async table functions are special functions for table sources that
>> perform
>> > > a lookup.
>> > >
>> >
>> > So end user can not directly define and use async table function now. An
>> > user case is reported in [2]
>> >
>> > So, in conclusion, no new interface is introduced, but we extend the
>> > ability to support user-defined async table function.
>> >
>> > [1]:
>> >
>> >
>> https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/dev/table/functions/udfs/
>> > [2]: https://lists.apache.org/thread/qljwd40v5ntz6733cwcdr8s4z97b343b
>> >
>> > Thanks.
>> > Aitozi.
>> >
>> >
>> > Jing Ge <ji...@ververica.com.invalid> 于2023年5月27日周六 06:40写道：
>> >
>> > > Hi Aitozi,
>> > >
>> > > Thanks for your proposal. I am not quite sure if I understood your
>> > thoughts
>> > > correctly. You described a special case implementation of the
>> > > AsyncTableFunction with on public API changes. Would you please
>> elaborate
>> > > your purpose of writing a FLIP according to the FLIP documentation[1]?
>> > > Thanks!
>> > >
>> > > [1]
>> > >
>> > >
>> >
>> https://cwiki.apache.org/confluence/display/FLINK/Flink+Improvement+Proposals
>> > >
>> > > Best regards,
>> > > Jing
>> > >
>> > > On Wed, May 24, 2023 at 1:07 PM Aitozi <gj...@gmail.com> wrote:
>> > >
>> > > > May I ask for some feedback  :D
>> > > >
>> > > > Thanks,
>> > > > Aitozi
>> > > >
>> > > > Aitozi <gj...@gmail.com> 于2023年5月23日周二 19:14写道：
>> > > > >
>> > > > > Just catch an user case report from Giannis Polyzos for this
>> usage:
>> > > > >
>> > > > > https://lists.apache.org/thread/qljwd40v5ntz6733cwcdr8s4z97b343b
>> > > > >
>> > > > > Aitozi <gj...@gmail.com> 于2023年5月23日周二 17:45写道：
>> > > > > >
>> > > > > > Hi guys,
>> > > > > >     I want to bring up a discussion about adding support of User
>> > > > > > Defined AsyncTableFunction in Flink.
>> > > > > > Currently, async table function are special functions for table
>> > > source
>> > > > > > to perform
>> > > > > > async lookup. However, it's worth to support the user defined
>> async
>> > > > > > table function.
>> > > > > > Because, in this way, the end SQL user can leverage it to
>> perform
>> > the
>> > > > > > async operation
>> > > > > > which is useful to maximum the system throughput especially for
>> IO
>> > > > > > bottleneck case.
>> > > > > >
>> > > > > > You can find some more detail in [1].
>> > > > > >
>> > > > > > Looking forward to feedback
>> > > > > >
>> > > > > >
>> > > > > > [1]:
>> > > >
>> > >
>> >
>> https://cwiki.apache.org/confluence/display/FLINK/%5BFLIP-313%5D+Add+support+of+User+Defined+AsyncTableFunction
>> > > > > >
>> > > > > > Thanks,
>> > > > > > Aitozi.
>> > > >
>> > >
>> >
>>
>

Re: [DISCUSS] FLIP-313 Add support of User Defined AsyncTableFunction

Posted by Aitozi <gj...@gmail.com>.

Hi Jing,
    > "Do you mean to support the AyncTableFunction beyond the
LookupTableSource?"
Yes, I mean to support the AyncTableFunction beyond the LookupTableSource.

The "AsyncTableFunction" is the function with ability to be executed async
(with AsyncWaitOperator).
The async lookup join is a one of usage of this. So, we don't have to bind
the AyncTableFunction with LookupTableSource.
If User-defined AsyncTableFunction is supported, user can directly use
lateral table syntax to perform async operation.

> "It would be better if you could elaborate the proposed changes wrt the
CorrelatedCodeGenerator with more details"

In the proposal, we use lateral table syntax to support the async table
function. So the planner will also treat this statement to a
CommonExecCorrelate node. So the runtime code should be generated in
CorrelatedCodeGenerator.
In CorrelatedCodeGenerator, we will know the TableFunction's Kind of
`FunctionKind.Table` or `FunctionKind.ASYNC_TABLE`
For  `FunctionKind.ASYNC_TABLE` we can generate a AsyncWaitOperator to
execute the async table function.


Thanks,
Aitozi.


Jing Ge <ji...@ververica.com.invalid> 于2023年5月29日周一 03:22写道：

> Hi Aitozi,
>
> Thanks for the clarification. The naming "Lookup" might suggest using it
> for table look up. But conceptually what the eval() method will do is to
> get a collection of results(Row, RowData) from the given keys. How it will
> be done depends on the implementation, i.e. you can implement your own
> Source[1][2]. The example in the FLIP should be able to be handled in this
> way.
>
> Do you mean to support the AyncTableFunction beyond the LookupTableSource?
> It would be better if you could elaborate the proposed changes wrt the
> CorrelatedCodeGenerator with more details. Thanks!
>
> Best regards,
> Jing
>
> [1]
>
> https://github.com/apache/flink/blob/678370b18e1b6c4a23e5ce08f8efd05675a0cc17/flink-table/flink-table-common/src/main/java/org/apache/flink/table/connector/source/LookupTableSource.java#L64
> [2]
>
> https://github.com/apache/flink/blob/678370b18e1b6c4a23e5ce08f8efd05675a0cc17/flink-table/flink-table-common/src/main/java/org/apache/flink/table/connector/source/AsyncTableFunctionProvider.java#L49
>
> On Sat, May 27, 2023 at 9:48 AM Aitozi <gj...@gmail.com> wrote:
>
> > Hi Jing,
> >     Thanks for your response. As stated in the FLIP, the purpose of this
> > FLIP is meant to support
> > user-defined async table function. As described in flink document [1]
> >
> > Async table functions are special functions for table sources that
> perform
> > > a lookup.
> > >
> >
> > So end user can not directly define and use async table function now. An
> > user case is reported in [2]
> >
> > So, in conclusion, no new interface is introduced, but we extend the
> > ability to support user-defined async table function.
> >
> > [1]:
> >
> >
> https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/dev/table/functions/udfs/
> > [2]: https://lists.apache.org/thread/qljwd40v5ntz6733cwcdr8s4z97b343b
> >
> > Thanks.
> > Aitozi.
> >
> >
> > Jing Ge <ji...@ververica.com.invalid> 于2023年5月27日周六 06:40写道：
> >
> > > Hi Aitozi,
> > >
> > > Thanks for your proposal. I am not quite sure if I understood your
> > thoughts
> > > correctly. You described a special case implementation of the
> > > AsyncTableFunction with on public API changes. Would you please
> elaborate
> > > your purpose of writing a FLIP according to the FLIP documentation[1]?
> > > Thanks!
> > >
> > > [1]
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/Flink+Improvement+Proposals
> > >
> > > Best regards,
> > > Jing
> > >
> > > On Wed, May 24, 2023 at 1:07 PM Aitozi <gj...@gmail.com> wrote:
> > >
> > > > May I ask for some feedback  :D
> > > >
> > > > Thanks,
> > > > Aitozi
> > > >
> > > > Aitozi <gj...@gmail.com> 于2023年5月23日周二 19:14写道：
> > > > >
> > > > > Just catch an user case report from Giannis Polyzos for this usage:
> > > > >
> > > > > https://lists.apache.org/thread/qljwd40v5ntz6733cwcdr8s4z97b343b
> > > > >
> > > > > Aitozi <gj...@gmail.com> 于2023年5月23日周二 17:45写道：
> > > > > >
> > > > > > Hi guys,
> > > > > >     I want to bring up a discussion about adding support of User
> > > > > > Defined AsyncTableFunction in Flink.
> > > > > > Currently, async table function are special functions for table
> > > source
> > > > > > to perform
> > > > > > async lookup. However, it's worth to support the user defined
> async
> > > > > > table function.
> > > > > > Because, in this way, the end SQL user can leverage it to perform
> > the
> > > > > > async operation
> > > > > > which is useful to maximum the system throughput especially for
> IO
> > > > > > bottleneck case.
> > > > > >
> > > > > > You can find some more detail in [1].
> > > > > >
> > > > > > Looking forward to feedback
> > > > > >
> > > > > >
> > > > > > [1]:
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/%5BFLIP-313%5D+Add+support+of+User+Defined+AsyncTableFunction
> > > > > >
> > > > > > Thanks,
> > > > > > Aitozi.
> > > >
> > >
> >
>

Re: [DISCUSS] FLIP-313 Add support of User Defined AsyncTableFunction

Posted by Jing Ge <ji...@ververica.com.INVALID>.

Hi Aitozi,

Thanks for the clarification. The naming "Lookup" might suggest using it
for table look up. But conceptually what the eval() method will do is to
get a collection of results(Row, RowData) from the given keys. How it will
be done depends on the implementation, i.e. you can implement your own
Source[1][2]. The example in the FLIP should be able to be handled in this
way.

Do you mean to support the AyncTableFunction beyond the LookupTableSource?
It would be better if you could elaborate the proposed changes wrt the
CorrelatedCodeGenerator with more details. Thanks!

Best regards,
Jing

[1]
https://github.com/apache/flink/blob/678370b18e1b6c4a23e5ce08f8efd05675a0cc17/flink-table/flink-table-common/src/main/java/org/apache/flink/table/connector/source/LookupTableSource.java#L64
[2]
https://github.com/apache/flink/blob/678370b18e1b6c4a23e5ce08f8efd05675a0cc17/flink-table/flink-table-common/src/main/java/org/apache/flink/table/connector/source/AsyncTableFunctionProvider.java#L49

On Sat, May 27, 2023 at 9:48 AM Aitozi <gj...@gmail.com> wrote:

> Hi Jing,
>     Thanks for your response. As stated in the FLIP, the purpose of this
> FLIP is meant to support
> user-defined async table function. As described in flink document [1]
>
> Async table functions are special functions for table sources that perform
> > a lookup.
> >
>
> So end user can not directly define and use async table function now. An
> user case is reported in [2]
>
> So, in conclusion, no new interface is introduced, but we extend the
> ability to support user-defined async table function.
>
> [1]:
>
> https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/dev/table/functions/udfs/
> [2]: https://lists.apache.org/thread/qljwd40v5ntz6733cwcdr8s4z97b343b
>
> Thanks.
> Aitozi.
>
>
> Jing Ge <ji...@ververica.com.invalid> 于2023年5月27日周六 06:40写道：
>
> > Hi Aitozi,
> >
> > Thanks for your proposal. I am not quite sure if I understood your
> thoughts
> > correctly. You described a special case implementation of the
> > AsyncTableFunction with on public API changes. Would you please elaborate
> > your purpose of writing a FLIP according to the FLIP documentation[1]?
> > Thanks!
> >
> > [1]
> >
> >
> https://cwiki.apache.org/confluence/display/FLINK/Flink+Improvement+Proposals
> >
> > Best regards,
> > Jing
> >
> > On Wed, May 24, 2023 at 1:07 PM Aitozi <gj...@gmail.com> wrote:
> >
> > > May I ask for some feedback  :D
> > >
> > > Thanks,
> > > Aitozi
> > >
> > > Aitozi <gj...@gmail.com> 于2023年5月23日周二 19:14写道：
> > > >
> > > > Just catch an user case report from Giannis Polyzos for this usage:
> > > >
> > > > https://lists.apache.org/thread/qljwd40v5ntz6733cwcdr8s4z97b343b
> > > >
> > > > Aitozi <gj...@gmail.com> 于2023年5月23日周二 17:45写道：
> > > > >
> > > > > Hi guys,
> > > > >     I want to bring up a discussion about adding support of User
> > > > > Defined AsyncTableFunction in Flink.
> > > > > Currently, async table function are special functions for table
> > source
> > > > > to perform
> > > > > async lookup. However, it's worth to support the user defined async
> > > > > table function.
> > > > > Because, in this way, the end SQL user can leverage it to perform
> the
> > > > > async operation
> > > > > which is useful to maximum the system throughput especially for IO
> > > > > bottleneck case.
> > > > >
> > > > > You can find some more detail in [1].
> > > > >
> > > > > Looking forward to feedback
> > > > >
> > > > >
> > > > > [1]:
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/%5BFLIP-313%5D+Add+support+of+User+Defined+AsyncTableFunction
> > > > >
> > > > > Thanks,
> > > > > Aitozi.
> > >
> >
>

Re: [DISCUSS] FLIP-313 Add support of User Defined AsyncTableFunction

Posted by Aitozi <gj...@gmail.com>.

Hi Jing,
    Thanks for your response. As stated in the FLIP, the purpose of this
FLIP is meant to support
user-defined async table function. As described in flink document [1]

Async table functions are special functions for table sources that perform
> a lookup.
>

So end user can not directly define and use async table function now. An
user case is reported in [2]

So, in conclusion, no new interface is introduced, but we extend the
ability to support user-defined async table function.

[1]:
https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/dev/table/functions/udfs/
[2]: https://lists.apache.org/thread/qljwd40v5ntz6733cwcdr8s4z97b343b

Thanks.
Aitozi.


Jing Ge <ji...@ververica.com.invalid> 于2023年5月27日周六 06:40写道：

> Hi Aitozi,
>
> Thanks for your proposal. I am not quite sure if I understood your thoughts
> correctly. You described a special case implementation of the
> AsyncTableFunction with on public API changes. Would you please elaborate
> your purpose of writing a FLIP according to the FLIP documentation[1]?
> Thanks!
>
> [1]
>
> https://cwiki.apache.org/confluence/display/FLINK/Flink+Improvement+Proposals
>
> Best regards,
> Jing
>
> On Wed, May 24, 2023 at 1:07 PM Aitozi <gj...@gmail.com> wrote:
>
> > May I ask for some feedback  :D
> >
> > Thanks,
> > Aitozi
> >
> > Aitozi <gj...@gmail.com> 于2023年5月23日周二 19:14写道：
> > >
> > > Just catch an user case report from Giannis Polyzos for this usage:
> > >
> > > https://lists.apache.org/thread/qljwd40v5ntz6733cwcdr8s4z97b343b
> > >
> > > Aitozi <gj...@gmail.com> 于2023年5月23日周二 17:45写道：
> > > >
> > > > Hi guys,
> > > >     I want to bring up a discussion about adding support of User
> > > > Defined AsyncTableFunction in Flink.
> > > > Currently, async table function are special functions for table
> source
> > > > to perform
> > > > async lookup. However, it's worth to support the user defined async
> > > > table function.
> > > > Because, in this way, the end SQL user can leverage it to perform the
> > > > async operation
> > > > which is useful to maximum the system throughput especially for IO
> > > > bottleneck case.
> > > >
> > > > You can find some more detail in [1].
> > > >
> > > > Looking forward to feedback
> > > >
> > > >
> > > > [1]:
> >
> https://cwiki.apache.org/confluence/display/FLINK/%5BFLIP-313%5D+Add+support+of+User+Defined+AsyncTableFunction
> > > >
> > > > Thanks,
> > > > Aitozi.
> >
>

Re: [DISCUSS] FLIP-313 Add support of User Defined AsyncTableFunction

Posted by Jing Ge <ji...@ververica.com.INVALID>.

Hi Aitozi,

Thanks for your proposal. I am not quite sure if I understood your thoughts
correctly. You described a special case implementation of the
AsyncTableFunction with on public API changes. Would you please elaborate
your purpose of writing a FLIP according to the FLIP documentation[1]?
Thanks!

[1]
https://cwiki.apache.org/confluence/display/FLINK/Flink+Improvement+Proposals

Best regards,
Jing

On Wed, May 24, 2023 at 1:07 PM Aitozi <gj...@gmail.com> wrote:

> May I ask for some feedback  :D
>
> Thanks,
> Aitozi
>
> Aitozi <gj...@gmail.com> 于2023年5月23日周二 19:14写道：
> >
> > Just catch an user case report from Giannis Polyzos for this usage:
> >
> > https://lists.apache.org/thread/qljwd40v5ntz6733cwcdr8s4z97b343b
> >
> > Aitozi <gj...@gmail.com> 于2023年5月23日周二 17:45写道：
> > >
> > > Hi guys,
> > >     I want to bring up a discussion about adding support of User
> > > Defined AsyncTableFunction in Flink.
> > > Currently, async table function are special functions for table source
> > > to perform
> > > async lookup. However, it's worth to support the user defined async
> > > table function.
> > > Because, in this way, the end SQL user can leverage it to perform the
> > > async operation
> > > which is useful to maximum the system throughput especially for IO
> > > bottleneck case.
> > >
> > > You can find some more detail in [1].
> > >
> > > Looking forward to feedback
> > >
> > >
> > > [1]:
> https://cwiki.apache.org/confluence/display/FLINK/%5BFLIP-313%5D+Add+support+of+User+Defined+AsyncTableFunction
> > >
> > > Thanks,
> > > Aitozi.
>

Re: [DISCUSS] FLIP-313 Add support of User Defined AsyncTableFunction

Posted by Aitozi <gj...@gmail.com>.

May I ask for some feedback  :D

Thanks,
Aitozi

Aitozi <gj...@gmail.com> 于2023年5月23日周二 19:14写道：
>
> Just catch an user case report from Giannis Polyzos for this usage:
>
> https://lists.apache.org/thread/qljwd40v5ntz6733cwcdr8s4z97b343b
>
> Aitozi <gj...@gmail.com> 于2023年5月23日周二 17:45写道：
> >
> > Hi guys,
> >     I want to bring up a discussion about adding support of User
> > Defined AsyncTableFunction in Flink.
> > Currently, async table function are special functions for table source
> > to perform
> > async lookup. However, it's worth to support the user defined async
> > table function.
> > Because, in this way, the end SQL user can leverage it to perform the
> > async operation
> > which is useful to maximum the system throughput especially for IO
> > bottleneck case.
> >
> > You can find some more detail in [1].
> >
> > Looking forward to feedback
> >
> >
> > [1]: https://cwiki.apache.org/confluence/display/FLINK/%5BFLIP-313%5D+Add+support+of+User+Defined+AsyncTableFunction
> >
> > Thanks,
> > Aitozi.

Re: [DISCUSS] FLIP-313 Add support of User Defined AsyncTableFunction

Posted by Aitozi <gj...@gmail.com>.

Just catch an user case report from Giannis Polyzos for this usage:

https://lists.apache.org/thread/qljwd40v5ntz6733cwcdr8s4z97b343b

Aitozi <gj...@gmail.com> 于2023年5月23日周二 17:45写道：
>
> Hi guys,
>     I want to bring up a discussion about adding support of User
> Defined AsyncTableFunction in Flink.
> Currently, async table function are special functions for table source
> to perform
> async lookup. However, it's worth to support the user defined async
> table function.
> Because, in this way, the end SQL user can leverage it to perform the
> async operation
> which is useful to maximum the system throughput especially for IO
> bottleneck case.
>
> You can find some more detail in [1].
>
> Looking forward to feedback
>
>
> [1]: https://cwiki.apache.org/confluence/display/FLINK/%5BFLIP-313%5D+Add+support+of+User+Defined+AsyncTableFunction
>
> Thanks,
> Aitozi.