You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Spurthi Chaganti <sp...@gmail.com> on 2020/08/18 18:10:48 UTC
Async IO with SQL API
Hello folks,
We are using flink 1.9 SQL API and we are NOT using blink planner. Our
platform users express their flink jobs as SQL queries. We currently have a
use case of asynchronously lookup data from third parties for every event
we read from kafka stream and populate additional fields which we use later
in our group bys.
We were trying to prototype the best way we can asynchronously enrich our
events and two options we could think of:
1. UDTF extending AsyncTableFunction and users can use these functions
as part of SQL query. But it seems like AsyncTableFunction cannot be used
as UDTF currently.
2. LookupableTableSource to access data in Temporal table but its only
supported in blink.
Can you please suggest any other options I can try out?
Thanks,
-spurthi
Re: Async IO with SQL API
Posted by Jark Wu <im...@gmail.com>.
Hi,
Sorry for the late reply.
AFAIK, it's impossible to do Async IO on pure Table API / SQL in 1.9
old planner.
A doable way is convert the Table into DataStream and apply AsyncFunction
on it.
Best,
Jark
On Thu, 20 Aug 2020 at 00:35, Spurthi Chaganti <sp...@gmail.com>
wrote:
> Thank you Till for your response.
>
> We haven't completely evaluated migration to blink planner as it was not
> prioritized, certainly a task we would look into if we arent able to get a
> solution using SQL api.
>
> On Wed, Aug 19, 2020 at 3:23 AM Till Rohrmann <tr...@apache.org>
> wrote:
>
>> Hi Spurthi,
>>
>> thanks for reaching out to the Flink community. Have you tried using the
>> Blink planner where these features are available?
>>
>> Pulling in Jark and Timo who worked on this feature and who might be able
>> to explain to you the rationale behind making the LookupableTableSource and
>> AsyncTableFunction a Blink planner feature only.
>>
>> Cheers,
>> Till
>>
>> On Tue, Aug 18, 2020 at 8:11 PM Spurthi Chaganti <
>> spurthi.chaganti@gmail.com> wrote:
>>
>>> Hello folks,
>>>
>>> We are using flink 1.9 SQL API and we are NOT using blink planner.
>>> Our platform users express their flink jobs as SQL queries. We currently
>>> have a use case of asynchronously lookup data from third parties for every
>>> event we read from kafka stream and populate additional fields which we use
>>> later in our group bys.
>>>
>>> We were trying to prototype the best way we can asynchronously enrich
>>> our events and two options we could think of:
>>> 1. UDTF extending AsyncTableFunction and users can use these
>>> functions as part of SQL query. But it seems like AsyncTableFunction cannot
>>> be used as UDTF currently.
>>> 2. LookupableTableSource to access data in Temporal table but its
>>> only supported in blink.
>>>
>>> Can you please suggest any other options I can try out?
>>>
>>> Thanks,
>>> -spurthi
>>>
>>
>
> --
> -spurthi
>
Re: Async IO with SQL API
Posted by Spurthi Chaganti <sp...@gmail.com>.
Thank you Till for your response.
We haven't completely evaluated migration to blink planner as it was not
prioritized, certainly a task we would look into if we arent able to get a
solution using SQL api.
On Wed, Aug 19, 2020 at 3:23 AM Till Rohrmann <tr...@apache.org> wrote:
> Hi Spurthi,
>
> thanks for reaching out to the Flink community. Have you tried using the
> Blink planner where these features are available?
>
> Pulling in Jark and Timo who worked on this feature and who might be able
> to explain to you the rationale behind making the LookupableTableSource and
> AsyncTableFunction a Blink planner feature only.
>
> Cheers,
> Till
>
> On Tue, Aug 18, 2020 at 8:11 PM Spurthi Chaganti <
> spurthi.chaganti@gmail.com> wrote:
>
>> Hello folks,
>>
>> We are using flink 1.9 SQL API and we are NOT using blink planner.
>> Our platform users express their flink jobs as SQL queries. We currently
>> have a use case of asynchronously lookup data from third parties for every
>> event we read from kafka stream and populate additional fields which we use
>> later in our group bys.
>>
>> We were trying to prototype the best way we can asynchronously enrich our
>> events and two options we could think of:
>> 1. UDTF extending AsyncTableFunction and users can use these
>> functions as part of SQL query. But it seems like AsyncTableFunction cannot
>> be used as UDTF currently.
>> 2. LookupableTableSource to access data in Temporal table but its
>> only supported in blink.
>>
>> Can you please suggest any other options I can try out?
>>
>> Thanks,
>> -spurthi
>>
>
--
-spurthi
Re: Async IO with SQL API
Posted by Till Rohrmann <tr...@apache.org>.
Hi Spurthi,
thanks for reaching out to the Flink community. Have you tried using the
Blink planner where these features are available?
Pulling in Jark and Timo who worked on this feature and who might be able
to explain to you the rationale behind making the LookupableTableSource and
AsyncTableFunction a Blink planner feature only.
Cheers,
Till
On Tue, Aug 18, 2020 at 8:11 PM Spurthi Chaganti <sp...@gmail.com>
wrote:
> Hello folks,
>
> We are using flink 1.9 SQL API and we are NOT using blink planner. Our
> platform users express their flink jobs as SQL queries. We currently have a
> use case of asynchronously lookup data from third parties for every event
> we read from kafka stream and populate additional fields which we use later
> in our group bys.
>
> We were trying to prototype the best way we can asynchronously enrich our
> events and two options we could think of:
> 1. UDTF extending AsyncTableFunction and users can use these functions
> as part of SQL query. But it seems like AsyncTableFunction cannot be used
> as UDTF currently.
> 2. LookupableTableSource to access data in Temporal table but its only
> supported in blink.
>
> Can you please suggest any other options I can try out?
>
> Thanks,
> -spurthi
>