Best way to perform look up with historical data in Flink

[Surendra Lalwani <su...@swiggy.in>]

Hi Team,

Since Flink does not support Stream-Batch Join, we need to perform a lookup
with some historical data, what could be the best way in Flink to do that.
I am aware of one such approach using Broadcasted stream, not sure if there
are any other proper solutions. Any help in this will be helpful.

Regards ,
Surendra Lalwani

-- 

IMPORTANT NOTICE: This e-mail, including any attachments, may contain 
confidential information and is intended only for the addressee(s) named 
above. If you are not the intended recipient(s), you should not 
disseminate, distribute, or copy this e-mail. Please notify the sender by 
reply e-mail immediately if you have received this e-mail in error and 
permanently delete all copies of the original message from your system. 
E-mail transmission cannot be guaranteed to be secure as it could be 
intercepted, corrupted, lost, destroyed, arrive late or incomplete, or 
contain viruses. Company accepts no liability for any damage or loss of 
confidential information caused by this email or due to any virus 
transmitted by this email or otherwise.

Re: Best way to perform look up with historical data in Flink

[Konstantin Knauf <kn...@apache.org>]

Hi Surendra,

where does your historical data reside? In a database? Files? Behind a REST
API?

Depending on the answer and which API you use, the AsyncIO [1] operator
(DataStream API) or a Lookup Table Join [2]  might be good options. There
are other ways, too. A while back I did a webinar [3] on this. It is a bit
outdated, but still covers a few concepts and ideas around this.

Cheers,

Konstantin

[1]
https://nightlies.apache.org/flink/flink-docs-release-1.15/docs/dev/datastream/operators/asyncio/
[2]
https://nightlies.apache.org/flink/flink-docs-release-1.15/docs/dev/table/sql/queries/joins/#lookup-join
[3] https://www.youtube.com/watch?v=cJS18iKLUIY&t=2s

Am Mo., 6. Juni 2022 um 07:36 Uhr schrieb Surendra Lalwani <
surendra.lalwani@swiggy.in>:

> Hi Team,
>
> Since Flink does not support Stream-Batch Join, we need to perform a
> lookup with some historical data, what could be the best way in Flink to do
> that. I am aware of one such approach using Broadcasted stream, not sure if
> there are any other proper solutions. Any help in this will be helpful.
>
> Regards ,
> Surendra Lalwani
>
>
> ------------------------------
> IMPORTANT NOTICE: This e-mail, including any attachments, may contain
> confidential information and is intended only for the addressee(s) named
> above. If you are not the intended recipient(s), you should not
> disseminate, distribute, or copy this e-mail. Please notify the sender by
> reply e-mail immediately if you have received this e-mail in error and
> permanently delete all copies of the original message from your system.
> E-mail transmission cannot be guaranteed to be secure as it could be
> intercepted, corrupted, lost, destroyed, arrive late or incomplete, or
> contain viruses. Company accepts no liability for any damage or loss of
> confidential information caused by this email or due to any virus
> transmitted by this email or otherwise.



-- 
https://twitter.com/snntrable
https://github.com/knaufk