You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by William Saar <wi...@saar.se> on 2019/02/04 16:59:42 UTC
How to add caching to async function?
Hi,
I am trying to implement an async function that looks up a value in a
cache or, if the value doesn't exist in the cache, queries a web
service, but I'm having trouble creating the cache. I've tried to
create a RichAsyncFunction and add a map state as cache, but I'm
getting: State is not supported in rich async functions.
What is the best practice for doing this? I guess I could have a
previous step with state and send the responses from the rich function
back as an iteration, but I would guess that's the wrong approach...
Thanks,
William
Re: How to add caching to async function?
Posted by William Saar <wi...@saar.se>.
Ah, thanks, missed it when I only looked at the slides. Yes, have
heard deadlocks are a problem with iterations but hoped it had been
fixed. Pity, had been hoping to replace an external service with the
Flink job, but will keep the service around for the caching,
----- Original Message -----
From: "Lasse Nedergaard" <la...@gmail.com>
To:"William Saar" <wi...@saar.se>
Cc:"Fabian Hueske" <fh...@gmail.com>, "user" <us...@flink.apache.org>
Sent:Tue, 5 Feb 2019 10:41:41 +0100
Subject:Re: How to add caching to async function?
Hi William
No iterations isn’t the solution as you can (will) end up in a
deadlock. We concluded that storing the results from external lookup
in Kafka and use these data as input to the cache was the only way
Med venlig hilsen / Best regards
Lasse Nedergaard
Den 5. feb. 2019 kl. 10.22 skrev William Saar <william@saar.se [1]>:
Thanks! Looks like iterations is indeed the way to go for now then...
----- Original Message -----
From: "Lasse Nedergaard" <lassenedergaard@gmail.com [2]>
To:"Fabian Hueske" <fhueske@gmail.com [3]>
Cc:"William Saar" <william@saar.se [4]>, "user" <user@flink.apache.org
[5]>
Sent:Mon, 4 Feb 2019 20:20:30 +0100
Subject:Re: How to add caching to async function?
Hi William
We have created a solution that do it. Please take a look at my
presentation from Flink forward.
https://www.slideshare.net/mobile/FlinkForward/flink-forward-berlin-2018-lasse-nedergaard-our-successful-journey-with-flink
[6]
Hopefully you can get inspired.
Med venlig hilsen / Best regards
Lasse Nedergaard
Den 4. feb. 2019 kl. 19.04 skrev Fabian Hueske <fhueske@gmail.com
[7]>:
Hi William,
Does the cache need to be fault tolerant?
If not you could use a regular in-memory map as cache (+some LRU
cleaning).
Or do you expect the cache to group too large for the memory?
Best, Fabian
Am Mo., 4. Feb. 2019 um 18:00 Uhr schrieb William Saar
<william@saar.se [8]>:
Hi,
I am trying to implement an async function that looks up a value in a
cache or, if the value doesn't exist in the cache, queries a web
service, but I'm having trouble creating the cache. I've tried to
create a RichAsyncFunction and add a map state as cache, but I'm
getting: State is not supported in rich async functions.
What is the best practice for doing this? I guess I could have a
previous step with state and send the responses from the rich function
back as an iteration, but I would guess that's the wrong approach...
Thanks,
William
Links:
------
[1] mailto:william@saar.se
[2] mailto:lassenedergaard@gmail.com
[3] mailto:fhueske@gmail.com
[4] mailto:william@saar.se
[5] mailto:user@flink.apache.org
[6]
https://www.slideshare.net/mobile/FlinkForward/flink-forward-berlin-2018-lasse-nedergaard-our-successful-journey-with-flink
[7] mailto:fhueske@gmail.com
[8] mailto:william@saar.se
Re: How to add caching to async function?
Posted by Lasse Nedergaard <la...@gmail.com>.
Hi William
No iterations isn’t the solution as you can (will) end up in a deadlock. We concluded that storing the results from external lookup in Kafka and use these data as input to the cache was the only way
Med venlig hilsen / Best regards
Lasse Nedergaard
> Den 5. feb. 2019 kl. 10.22 skrev William Saar <wi...@saar.se>:
>
> Thanks! Looks like iterations is indeed the way to go for now then...
>
>
>
> ----- Original Message -----
> From: "Lasse Nedergaard" <la...@gmail.com>
> To:"Fabian Hueske" <fh...@gmail.com>
> Cc:"William Saar" <wi...@saar.se>, "user" <us...@flink.apache.org>
> Sent:Mon, 4 Feb 2019 20:20:30 +0100
> Subject:Re: How to add caching to async function?
>
>
> Hi William
>
> We have created a solution that do it. Please take a look at my presentation from Flink forward.
> https://www.slideshare.net/mobile/FlinkForward/flink-forward-berlin-2018-lasse-nedergaard-our-successful-journey-with-flink
> Hopefully you can get inspired.
>
> Med venlig hilsen / Best regards
> Lasse Nedergaard
>
>
> Den 4. feb. 2019 kl. 19.04 skrev Fabian Hueske <fh...@gmail.com>:
>
> Hi William,
>
> Does the cache need to be fault tolerant?
> If not you could use a regular in-memory map as cache (+some LRU cleaning).
> Or do you expect the cache to group too large for the memory?
>
> Best, Fabian
>
>
>> Am Mo., 4. Feb. 2019 um 18:00 Uhr schrieb William Saar <wi...@saar.se>:
>> Hi,
>> I am trying to implement an async function that looks up a value in a cache or, if the value doesn't exist in the cache, queries a web service, but I'm having trouble creating the cache. I've tried to create a RichAsyncFunction and add a map state as cache, but I'm getting: State is not supported in rich async functions.
>>
>> What is the best practice for doing this? I guess I could have a previous step with state and send the responses from the rich function back as an iteration, but I would guess that's the wrong approach...
>>
>> Thanks,
>> William
>>
>>
Re: How to add caching to async function?
Posted by William Saar <wi...@saar.se>.
Thanks! Looks like iterations is indeed the way to go for now then...
----- Original Message -----
From: "Lasse Nedergaard" <la...@gmail.com>
To:"Fabian Hueske" <fh...@gmail.com>
Cc:"William Saar" <wi...@saar.se>, "user" <us...@flink.apache.org>
Sent:Mon, 4 Feb 2019 20:20:30 +0100
Subject:Re: How to add caching to async function?
Hi William
We have created a solution that do it. Please take a look at my
presentation from Flink forward.
https://www.slideshare.net/mobile/FlinkForward/flink-forward-berlin-2018-lasse-nedergaard-our-successful-journey-with-flink
[1]
Hopefully you can get inspired.
Med venlig hilsen / Best regards
Lasse Nedergaard
Den 4. feb. 2019 kl. 19.04 skrev Fabian Hueske <fhueske@gmail.com
[2]>:
Hi William,
Does the cache need to be fault tolerant?
If not you could use a regular in-memory map as cache (+some LRU
cleaning).
Or do you expect the cache to group too large for the memory?
Best, Fabian
Am Mo., 4. Feb. 2019 um 18:00 Uhr schrieb William Saar
<william@saar.se [3]>:
Hi,
I am trying to implement an async function that looks up a value in a
cache or, if the value doesn't exist in the cache, queries a web
service, but I'm having trouble creating the cache. I've tried to
create a RichAsyncFunction and add a map state as cache, but I'm
getting: State is not supported in rich async functions.
What is the best practice for doing this? I guess I could have a
previous step with state and send the responses from the rich function
back as an iteration, but I would guess that's the wrong approach...
Thanks,
William
Links:
------
[1]
https://www.slideshare.net/mobile/FlinkForward/flink-forward-berlin-2018-lasse-nedergaard-our-successful-journey-with-flink
[2] mailto:fhueske@gmail.com
[3] mailto:william@saar.se
Re: How to add caching to async function?
Posted by Lasse Nedergaard <la...@gmail.com>.
Hi William
We have created a solution that do it. Please take a look at my presentation from Flink forward.
https://www.slideshare.net/mobile/FlinkForward/flink-forward-berlin-2018-lasse-nedergaard-our-successful-journey-with-flink
Hopefully you can get inspired.
Med venlig hilsen / Best regards
Lasse Nedergaard
> Den 4. feb. 2019 kl. 19.04 skrev Fabian Hueske <fh...@gmail.com>:
>
> Hi William,
>
> Does the cache need to be fault tolerant?
> If not you could use a regular in-memory map as cache (+some LRU cleaning).
> Or do you expect the cache to group too large for the memory?
>
> Best, Fabian
>
>
>> Am Mo., 4. Feb. 2019 um 18:00 Uhr schrieb William Saar <wi...@saar.se>:
>> Hi,
>> I am trying to implement an async function that looks up a value in a cache or, if the value doesn't exist in the cache, queries a web service, but I'm having trouble creating the cache. I've tried to create a RichAsyncFunction and add a map state as cache, but I'm getting: State is not supported in rich async functions.
>>
>> What is the best practice for doing this? I guess I could have a previous step with state and send the responses from the rich function back as an iteration, but I would guess that's the wrong approach...
>>
>> Thanks,
>> William
>>
>>
Re: How to add caching to async function?
Posted by Fabian Hueske <fh...@gmail.com>.
Hi William,
Does the cache need to be fault tolerant?
If not you could use a regular in-memory map as cache (+some LRU cleaning).
Or do you expect the cache to group too large for the memory?
Best, Fabian
Am Mo., 4. Feb. 2019 um 18:00 Uhr schrieb William Saar <wi...@saar.se>:
> Hi,
> I am trying to implement an async function that looks up a value in a
> cache or, if the value doesn't exist in the cache, queries a web service,
> but I'm having trouble creating the cache. I've tried to create a
> RichAsyncFunction and add a map state as cache, but I'm getting: State is
> not supported in rich async functions.
>
> What is the best practice for doing this? I guess I could have a previous
> step with state and send the responses from the rich function back as an
> iteration, but I would guess that's the wrong approach...
>
> Thanks,
> William
>
>
>