Re: Calling a stateful fuction from Flink Job - DataStream Integration

[Igal Shilman <ig...@ververica.com>]

Hi Deniz,
Sorry for the late reply!

Let me first say that if your use case is a simple enrichment/side-lookup
then perhaps it is simpler to use Flink's Async operator that
calls into your HTTP service. StateFun comes with a programming model, an
SDK, and few limitations (for example only supports processing time)

If you think that StateFun is a fit for your use case then, let me try to
answer your questions:

1) No, HTTP does not have to be an ingress for StateFun, in fact, the only
ingress currently supported out of the box are Kafka/Kinesis.
2) Yes, you are right, you can use the Data stream integration API to call
into remote functions. In this case the ingress is the events coming off
the DataStream.
Please note however that currently the DataStream to remote functions
integration has some rough edges.
3) I'm not sure exactly what are the common patterns of working with
AsyncIO in Flink, perhaps other members of the community might chime in :-)

All the best,
Igal.

On Wed, Jul 28, 2021 at 12:05 PM Deniz Koçak <le...@gmail.com> wrote:

> Hi,
>
> We would like to host a separate service (logically and possibly
> physically) from the Flink Job which also we would like to call from
> our Flink Job. We have been considering remote functions via HTTP
> which seems to be good option on cloud. We have been considering Async
> I/o and statefun to use the remote HTTP based endpoint. However, we
> have few questions not clear for us.
>
> 1) In statefun documentation says an ingress is mandatory, so we have
> to define one I think? In here -
>
> https://ci.apache.org/projects/flink/flink-statefun-docs-release-2.2/concepts/application-building-blocks.html
> - it says `This can be anything from a Kafka topic, to a messsage
> queue, to an http request - anything that can get data into the system
> and trigger the intitial functions to begin computation.` and mentions
> HTTP as an ingress. However, I could not see an example on that? Can
> we use the statefun without an ingress or an HTTP should be the
> ingress to our statefun?
>
> 2) I believe
> https://ci.apache.org/projects/flink/flink-statefun-docs-release-2.2/sdk/flink-datastream.html
> is the API we have to use for calling statefun from Flink Job.
> However, related to question 1, how it communicates with the statefun
> just via HTTP? Even in that integration do we need an ingress or what
> will it be?
>
> 3) We are intrested in statefun for tault tolerancy, retries and etc.
> Also considering Async I/O for Flink I wonder if there is any pattern
> that we can benefit from in terms of fault tolerancy for Async I/O? If
> we can handle all these via Async I/O, statefun would be a heavy
> lifting for us.
>
> Thanks,
>