Enrich streaming data with small lookup data that slowly changes over time

[Mu Kong <ko...@gmail.com>]

Hi community,

I have a stream of traffic data with a service_id in it.
I'm enriching this data with a map of (service_id, service_name), which
only has 10 ~ 20 pairs and is read from config file.

The problem I'm facing now is, this map changes from time to time, and I
don't want to redeploy the application to just change the map in the config
file.

Is there an existing solution for solving this problem?

Thanks in advance!

Best regards,
Mu

Re: Enrich streaming data with small lookup data that slowly changes over time

[Mu Kong <ko...@gmail.com>]

Hi Jark Wu,

Thanks for your help!
I gave the document a quick glimpse, it seems method [1] fits my purpose
better.
Let me give it a deeper look.

Thank you very much!!

Best,
Mu


On Tue, Apr 21, 2020 at 12:06 PM Jark Wu <im...@gmail.com> wrote:

> Hi Mu,
>
> Flink SQL does support dimension table join. There are two ways to join
> the dimension table.
> If the data is in your database (e.g. MySQL, HBase), you can use this way
> [1] to join the data in your database in realtime and enrich fresh data.
> If the data is in a log stream (change stream), you can use this way [2]
> to join the data.
>
> Best,
> Jark
>
> [1]:
> https://ci.apache.org/projects/flink/flink-docs-master/dev/table/streaming/joins.html#join-with-a-temporal-table
> [2]:
> https://ci.apache.org/projects/flink/flink-docs-master/dev/table/streaming/joins.html#join-with-a-temporal-table-function
>
> On Tue, 21 Apr 2020 at 10:51, Mu Kong <ko...@gmail.com> wrote:
>
>> Hi community,
>>
>> I have a stream of traffic data with a service_id in it.
>> I'm enriching this data with a map of (service_id, service_name), which
>> only has 10 ~ 20 pairs and is read from config file.
>>
>> The problem I'm facing now is, this map changes from time to time, and I
>> don't want to redeploy the application to just change the map in the config
>> file.
>>
>> Is there an existing solution for solving this problem?
>>
>> Thanks in advance!
>>
>> Best regards,
>> Mu
>>
>

Re: Enrich streaming data with small lookup data that slowly changes over time

[Jark Wu <im...@gmail.com>]

Hi Mu,

Flink SQL does support dimension table join. There are two ways to join the
dimension table.
If the data is in your database (e.g. MySQL, HBase), you can use this way
[1] to join the data in your database in realtime and enrich fresh data.
If the data is in a log stream (change stream), you can use this way [2] to
join the data.

Best,
Jark

[1]:
https://ci.apache.org/projects/flink/flink-docs-master/dev/table/streaming/joins.html#join-with-a-temporal-table
[2]:
https://ci.apache.org/projects/flink/flink-docs-master/dev/table/streaming/joins.html#join-with-a-temporal-table-function

On Tue, 21 Apr 2020 at 10:51, Mu Kong <ko...@gmail.com> wrote:

> Hi community,
>
> I have a stream of traffic data with a service_id in it.
> I'm enriching this data with a map of (service_id, service_name), which
> only has 10 ~ 20 pairs and is read from config file.
>
> The problem I'm facing now is, this map changes from time to time, and I
> don't want to redeploy the application to just change the map in the config
> file.
>
> Is there an existing solution for solving this problem?
>
> Thanks in advance!
>
> Best regards,
> Mu
>