RichAsyncFunction + Cache or Map State?

[Clayton Wohl <cl...@gmail.com>]

I have a RichAsyncFunction that does async queries to an external database.
I'm using a Guava cache within the Flink app. I'd like this Guava cache to
be serialized with the rest of Flink state in checkpoint/savepoints.
However, RichAsyncFunction doesn't support the state functionality at all.

There is one Guava cache for the entire Flink app which might make this
scenario simpler.

Is there a recommended way to handle this situation?

Also, the Flink MapState interface doesn't support check-and-set type
functionality and doesn't support lock-free concurrent use like
java.util.concurrent.ConcurrentMap and Guava's cache do. I need both of
these features for proper concurrent operation. So even if I could use
Flink MapState, that doesn't seem like a practical solution.

Re: RichAsyncFunction + Cache or Map State?

[Dan Hill <qu...@gmail.com>]

Hi.  Any advice on this?  I just hit this too.

Some ideas:
1. Manage our own separate cache (disk, Redis, etc).
2. Use two operators (first one a cache one and the second is the
RichAsyncFunction).  Have a feedback loop by using another Kafka topic or
S3 File source/sink.


On Wed, Feb 9, 2022 at 7:11 AM Clayton Wohl <cl...@gmail.com> wrote:

> I have a RichAsyncFunction that does async queries to an external
> database. I'm using a Guava cache within the Flink app. I'd like this Guava
> cache to be serialized with the rest of Flink state in
> checkpoint/savepoints. However, RichAsyncFunction doesn't support the state
> functionality at all.
>
> There is one Guava cache for the entire Flink app which might make this
> scenario simpler.
>
> Is there a recommended way to handle this situation?
>
> Also, the Flink MapState interface doesn't support check-and-set type
> functionality and doesn't support lock-free concurrent use like
> java.util.concurrent.ConcurrentMap and Guava's cache do. I need both of
> these features for proper concurrent operation. So even if I could use
> Flink MapState, that doesn't seem like a practical solution.
>