REST API call in stream transformation

["G.S.Vijay Raajaa" <gs...@gmail.com>]

HI,

I have just started to explore Flink and have couple of questions. I  am
wondering if its possible to call a rest endpoint asynchronously and pipe
the response to the next state of my transformation on the stream. The idea
is such that after charging my data in a predefined time window, I would
like to apply some algorithm/transformation on the window of data external
to Flink. The algos have been exposed as REST endpoints.

My seconds question is an extension to the previous one, if i need to
transform my window data by applying three different algorithms in parallel
exposed via subsequent rest endpoints, how do wait unless the individual
rest endpoints respond back . Potentially I need to join the results of the
three algorithms before I call the sink.

Regards,
Vijay Raajaa

Re: REST API call in stream transformation

["Tzu-Li (Gordon) Tai" <tz...@apache.org>]

Hi Vijay,

Generally, for asynchronous operations to enrich (or in your case, fetching the algorithm for the actual transformation of the data), you’ll want to look at Flink’s Async I/O [1].

For your second question, I can see it as a stateful `FlatMapFunction` that keeps the seen results as managed state. Once all results are seen (i.e. the last result arrives at the operator), you join them and emit your final result further downstream.

Does this help with the use case you have in mind?

Cheers,
Gordon

[1] https://ci.apache.org/projects/flink/flink-docs-release-1.2/dev/stream/asyncio.html


On 27 April 2017 at 5:29:05 PM, G.S.Vijay Raajaa (gsvijayraajaa@gmail.com) wrote:

HI,

I have just started to explore Flink and have couple of questions. I  am wondering if its possible to call a rest endpoint asynchronously and pipe the response to the next state of my transformation on the stream. The idea is such that after charging my data in a predefined time window, I would like to apply some algorithm/transformation on the window of data external to Flink. The algos have been exposed as REST endpoints.

My seconds question is an extension to the previous one, if i need to transform my window data by applying three different algorithms in parallel exposed via subsequent rest endpoints, how do wait unless the individual rest endpoints respond back . Potentially I need to join the results of the three algorithms before I call the sink.

Regards,
Vijay Raajaa