You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Ejaskhan S <ia...@gmail.com> on 2021/02/03 05:59:10 UTC
Question on Flink and Rest API
Team,
It's just a random thought.
Can I make the Flink application exposing a rest endpoint for the data
source? So a client could send data to this endpoint. Subsequently, Flink
processed this data and responded to the client application through the
endpoint, like a client-server model.
Thanks
*EK*
Re: Question on Flink and Rest API
Posted by Ejaskhan S <ia...@gmail.com>.
Hi Raghavendar,
Yes , you are right. Your approach is correct ,and it is the most
straightforward one.but I was just thinking about the possibilities of my
question mentioned.
Thanks
EK
On Wed, Feb 3, 2021, 12:02 PM Raghavendar T S <ra...@gmail.com>
wrote:
> Hi Ejaskhan
>
> As per my understanding, this approach will require your data source to
> run a HTTP server within itself (embedded web server) and I am not sure If
> it is a good design. It looks like you are trying to build a
> synchronous(client-server model) processing model in Flink. But Flink is
> meant for processing data asynchronously (streaming/batch). Can you
> elaborate on the use case which you are trying to address? What kind of
> processing are you planning to do in Flink?
>
> Check If the following approach makes sense for your use case.
> Accept the events from HTTP client and persist the events to a data store
> (SQL/NoSQL) and publish the same to Kafka(topic)/RabbitMQ(queue). Let Flink
> data source listen to these topics/queues and update the status
> (SUCCESS/FAILURE) in the data store. If your clients are ok to see some lag
> in events, you can directly publish the events to
> Kafka(topic)/RabbitMQ(queue) without persisting it in the data store. Let
> Flink do all the processing and finally write to the data store.
>
> Thank you
> Raghavendar T S
> https://www.linkedin.com/in/raghavendar-ts
>
> On Wed, Feb 3, 2021 at 11:29 AM Ejaskhan S <ia...@gmail.com> wrote:
>
>> Team,
>>
>> It's just a random thought.
>>
>> Can I make the Flink application exposing a rest endpoint for the data
>> source? So a client could send data to this endpoint. Subsequently, Flink
>> processed this data and responded to the client application through the
>> endpoint, like a client-server model.
>>
>> Thanks
>> *EK*
>>
>>
>>
>
> --
> Raghavendar T S
> www.teknosrc.com
>
Re: Question on Flink and Rest API
Posted by Raghavendar T S <ra...@gmail.com>.
Hi Ejaskhan
As per my understanding, this approach will require your data source to run
a HTTP server within itself (embedded web server) and I am not sure If it
is a good design. It looks like you are trying to build a
synchronous(client-server model) processing model in Flink. But Flink is
meant for processing data asynchronously (streaming/batch). Can you
elaborate on the use case which you are trying to address? What kind of
processing are you planning to do in Flink?
Check If the following approach makes sense for your use case.
Accept the events from HTTP client and persist the events to a data store
(SQL/NoSQL) and publish the same to Kafka(topic)/RabbitMQ(queue). Let Flink
data source listen to these topics/queues and update the status
(SUCCESS/FAILURE) in the data store. If your clients are ok to see some lag
in events, you can directly publish the events to
Kafka(topic)/RabbitMQ(queue) without persisting it in the data store. Let
Flink do all the processing and finally write to the data store.
Thank you
Raghavendar T S
https://www.linkedin.com/in/raghavendar-ts
On Wed, Feb 3, 2021 at 11:29 AM Ejaskhan S <ia...@gmail.com> wrote:
> Team,
>
> It's just a random thought.
>
> Can I make the Flink application exposing a rest endpoint for the data
> source? So a client could send data to this endpoint. Subsequently, Flink
> processed this data and responded to the client application through the
> endpoint, like a client-server model.
>
> Thanks
> *EK*
>
>
>
--
Raghavendar T S
www.teknosrc.com
Re: Question on Flink and Rest API
Posted by Ejaskhan S <ia...@gmail.com>.
Yes Gordon, it's obviously gave me a starting point to think about.
On Wed, Feb 3, 2021, 12:02 PM Tzu-Li (Gordon) Tai <tz...@apache.org>
wrote:
> Hi,
>
> There is no out-of-box Flink source/sink connector for this, but it isn't
> unheard of that users have implemented something to support what you
> outlined.
>
> One way to possibly achieve this is: in terms of a Flink streaming job
> graph, what you would need to do is co-locate the source (which exposes the
> endpoint and maintains a pool of open client connections mapped by request
> ID), and the sink operators (which receives processed results with the
> original request IDs attached, and is in charge for replying to the
> original
> requests). The open client connections need to be process-wide accessible
> (e.g. via a static reference), so that when a co-located sink operator
> receives a result, it can directly fetch the corresponding client
> connection
> and return a response.
>
> The specifics are of course a bit more evolved; probably need some digging
> around previous Flink Forward conference talks to get a better picture.
> Hopefully this gives you a starting point to think about.
>
> Cheers,
> Gordon
>
>
>
> --
> Sent from:
> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/
>
Re: Question on Flink and Rest API
Posted by "Tzu-Li (Gordon) Tai" <tz...@apache.org>.
Hi,
There is no out-of-box Flink source/sink connector for this, but it isn't
unheard of that users have implemented something to support what you
outlined.
One way to possibly achieve this is: in terms of a Flink streaming job
graph, what you would need to do is co-locate the source (which exposes the
endpoint and maintains a pool of open client connections mapped by request
ID), and the sink operators (which receives processed results with the
original request IDs attached, and is in charge for replying to the original
requests). The open client connections need to be process-wide accessible
(e.g. via a static reference), so that when a co-located sink operator
receives a result, it can directly fetch the corresponding client connection
and return a response.
The specifics are of course a bit more evolved; probably need some digging
around previous Flink Forward conference talks to get a better picture.
Hopefully this gives you a starting point to think about.
Cheers,
Gordon
--
Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/