You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Mohit Anchlia <mo...@gmail.com> on 2017/07/13 00:16:59 UTC
Reading static data
What is the best way to read a map of lookup data? This lookup data is like
a small short lived data that is available in transformation to do things
like filtering, additional augmentation of data etc.
Re: Reading static data
Posted by Fabian Hueske <fh...@gmail.com>.
You could either use a broadcast variable [1] or the distributed cache [2].
Best,
Fabian
[1]
https://ci.apache.org/projects/flink/flink-docs-release-1.3/dev/batch/index.html#broadcast-variables
[2]
https://ci.apache.org/projects/flink/flink-docs-release-1.3/dev/batch/index.html#distributed-cache
2017-07-14 20:18 GMT+02:00 Mohit Anchlia <mo...@gmail.com>:
> Is there a way to accomplish this for the batch operations?
>
> On Thu, Jul 13, 2017 at 4:59 AM, Timo Walther <tw...@apache.org> wrote:
>
>> Hi Mohit,
>>
>> do you plan to implement a batch or streaming job? If it is a streaming
>> job: You can use a connected stream (see [1], Slide 34). The static data is
>> one side of the stream that could be updated from time to time and will
>> always propagated (using a broadcast()) to all workers that do filtering,
>> augmentation etc.
>>
>> [1] http://training.data-artisans.com/dataStream/1-intro.html
>>
>> I hope this helps.
>>
>> Timo
>>
>>
>> Am 13.07.17 um 02:16 schrieb Mohit Anchlia:
>>
>> What is the best way to read a map of lookup data? This lookup data is
>>> like a small short lived data that is available in transformation to do
>>> things like filtering, additional augmentation of data etc.
>>>
>>
>>
>>
>
Re: Reading static data
Posted by Mohit Anchlia <mo...@gmail.com>.
Is there a way to accomplish this for the batch operations?
On Thu, Jul 13, 2017 at 4:59 AM, Timo Walther <tw...@apache.org> wrote:
> Hi Mohit,
>
> do you plan to implement a batch or streaming job? If it is a streaming
> job: You can use a connected stream (see [1], Slide 34). The static data is
> one side of the stream that could be updated from time to time and will
> always propagated (using a broadcast()) to all workers that do filtering,
> augmentation etc.
>
> [1] http://training.data-artisans.com/dataStream/1-intro.html
>
> I hope this helps.
>
> Timo
>
>
> Am 13.07.17 um 02:16 schrieb Mohit Anchlia:
>
> What is the best way to read a map of lookup data? This lookup data is
>> like a small short lived data that is available in transformation to do
>> things like filtering, additional augmentation of data etc.
>>
>
>
>
Re: Reading static data
Posted by Timo Walther <tw...@apache.org>.
Hi Mohit,
do you plan to implement a batch or streaming job? If it is a streaming
job: You can use a connected stream (see [1], Slide 34). The static data
is one side of the stream that could be updated from time to time and
will always propagated (using a broadcast()) to all workers that do
filtering, augmentation etc.
[1] http://training.data-artisans.com/dataStream/1-intro.html
I hope this helps.
Timo
Am 13.07.17 um 02:16 schrieb Mohit Anchlia:
> What is the best way to read a map of lookup data? This lookup data is
> like a small short lived data that is available in transformation to
> do things like filtering, additional augmentation of data etc.