You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by "igor.berman" <ig...@gmail.com> on 2016/04/07 09:26:38 UTC
Flink event processing immediate feedback
Hi,
Suppose I have web facing frontend that gets stream of events(http calls). I
need to process event stream and do some aggregations over those events and
write aggregated statistics to Hbase - so far Flink seems as perfect match.
However in some cases event should trigger some alert and frontend needs to
get this alert in synchronous way - here I'm a bit lost. I thought about
some kind of following flow:
frontend -> queue -> flink -> redis(pub/sub)<- frontend
I.e. I have two major use cases - async aggregated analytics/stats computing
and "synchronous" response to frontend. Frontend might be node/play or any
other technology that won't have a problem of "waiting" for the response, so
the only question - how to implement this feedback ?
Might be some kind of Sink?
Any ideas would be appreciated,
Igor
--
View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Flink-event-processing-immediate-feedback-tp5978.html
Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.
Re: Flink event processing immediate feedback
Posted by Igor Berman <ig...@gmail.com>.
Yes, indeed this is direction we are trying currently
thanks
On 14 April 2016 at 18:31, Aljoscha Krettek <al...@apache.org> wrote:
> Could still be, as I described it by using a message queue to do the
> communication between Flink and the front end.
>
> On Thu, 14 Apr 2016 at 17:30 Igor Berman <ig...@gmail.com> wrote:
>
>> Hi Aljoscha,
>> thanks for the response
>>
>> Synchronous - in our case means that request by end-client to
>> frontend(say some REST call) needs to wait until processing in
>> backend(Flink) is done and should return response(e.g. alert) back to
>> end-client(i.e. end-client -> frontend -> kafka-> flink)
>> those request are minority and will be defined by different type of
>> request to frontend or different parameters
>>
>> Might be Flink is not best option for this use case?
>>
>> Igor
>>
>>
>>
>>
>>
>> On 14 April 2016 at 16:37, Aljoscha Krettek <al...@apache.org> wrote:
>>
>>> Hi,
>>> what do you mean by "synchronous". If I understood it correctly then
>>> some events entering the Flink pipeline would trigger an alert while some
>>> others would not trigger an alert. How would the component that receives
>>> such alerts know when to wait and when to don't wait.
>>>
>>> As I see it you can push these alerts into some sort of message queue
>>> and have a request in the front-end that returns as soon as something is
>>> available in that queue. Then you display it and start a new query that
>>> returns as soon as more data is available in the alert queue.
>>>
>>> Cheers,
>>> Aljoscha
>>>
>>> On Thu, 7 Apr 2016 at 09:57 igor.berman <ig...@gmail.com> wrote:
>>>
>>>> Hi,
>>>> Suppose I have web facing frontend that gets stream of events(http
>>>> calls). I
>>>> need to process event stream and do some aggregations over those events
>>>> and
>>>> write aggregated statistics to Hbase - so far Flink seems as perfect
>>>> match.
>>>> However in some cases event should trigger some alert and frontend
>>>> needs to
>>>> get this alert in synchronous way - here I'm a bit lost. I thought about
>>>> some kind of following flow:
>>>> frontend -> queue -> flink -> redis(pub/sub)<- frontend
>>>>
>>>> I.e. I have two major use cases - async aggregated analytics/stats
>>>> computing
>>>> and "synchronous" response to frontend. Frontend might be node/play or
>>>> any
>>>> other technology that won't have a problem of "waiting" for the
>>>> response, so
>>>> the only question - how to implement this feedback ?
>>>> Might be some kind of Sink?
>>>>
>>>> Any ideas would be appreciated,
>>>> Igor
>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> View this message in context:
>>>> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Flink-event-processing-immediate-feedback-tp5978.html
>>>> Sent from the Apache Flink User Mailing List archive. mailing list
>>>> archive at Nabble.com.
>>>>
>>>
>>
Re: Flink event processing immediate feedback
Posted by Aljoscha Krettek <al...@apache.org>.
Could still be, as I described it by using a message queue to do the
communication between Flink and the front end.
On Thu, 14 Apr 2016 at 17:30 Igor Berman <ig...@gmail.com> wrote:
> Hi Aljoscha,
> thanks for the response
>
> Synchronous - in our case means that request by end-client to frontend(say
> some REST call) needs to wait until processing in backend(Flink) is done
> and should return response(e.g. alert) back to end-client(i.e. end-client
> -> frontend -> kafka-> flink)
> those request are minority and will be defined by different type of
> request to frontend or different parameters
>
> Might be Flink is not best option for this use case?
>
> Igor
>
>
>
>
>
> On 14 April 2016 at 16:37, Aljoscha Krettek <al...@apache.org> wrote:
>
>> Hi,
>> what do you mean by "synchronous". If I understood it correctly then some
>> events entering the Flink pipeline would trigger an alert while some others
>> would not trigger an alert. How would the component that receives such
>> alerts know when to wait and when to don't wait.
>>
>> As I see it you can push these alerts into some sort of message queue
>> and have a request in the front-end that returns as soon as something is
>> available in that queue. Then you display it and start a new query that
>> returns as soon as more data is available in the alert queue.
>>
>> Cheers,
>> Aljoscha
>>
>> On Thu, 7 Apr 2016 at 09:57 igor.berman <ig...@gmail.com> wrote:
>>
>>> Hi,
>>> Suppose I have web facing frontend that gets stream of events(http
>>> calls). I
>>> need to process event stream and do some aggregations over those events
>>> and
>>> write aggregated statistics to Hbase - so far Flink seems as perfect
>>> match.
>>> However in some cases event should trigger some alert and frontend needs
>>> to
>>> get this alert in synchronous way - here I'm a bit lost. I thought about
>>> some kind of following flow:
>>> frontend -> queue -> flink -> redis(pub/sub)<- frontend
>>>
>>> I.e. I have two major use cases - async aggregated analytics/stats
>>> computing
>>> and "synchronous" response to frontend. Frontend might be node/play or
>>> any
>>> other technology that won't have a problem of "waiting" for the
>>> response, so
>>> the only question - how to implement this feedback ?
>>> Might be some kind of Sink?
>>>
>>> Any ideas would be appreciated,
>>> Igor
>>>
>>>
>>>
>>>
>>> --
>>> View this message in context:
>>> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Flink-event-processing-immediate-feedback-tp5978.html
>>> Sent from the Apache Flink User Mailing List archive. mailing list
>>> archive at Nabble.com.
>>>
>>
>
Re: Flink event processing immediate feedback
Posted by Igor Berman <ig...@gmail.com>.
Hi Aljoscha,
thanks for the response
Synchronous - in our case means that request by end-client to frontend(say
some REST call) needs to wait until processing in backend(Flink) is done
and should return response(e.g. alert) back to end-client(i.e. end-client
-> frontend -> kafka-> flink)
those request are minority and will be defined by different type of request
to frontend or different parameters
Might be Flink is not best option for this use case?
Igor
On 14 April 2016 at 16:37, Aljoscha Krettek <al...@apache.org> wrote:
> Hi,
> what do you mean by "synchronous". If I understood it correctly then some
> events entering the Flink pipeline would trigger an alert while some others
> would not trigger an alert. How would the component that receives such
> alerts know when to wait and when to don't wait.
>
> As I see it you can push these alerts into some sort of message queue and
> have a request in the front-end that returns as soon as something is
> available in that queue. Then you display it and start a new query that
> returns as soon as more data is available in the alert queue.
>
> Cheers,
> Aljoscha
>
> On Thu, 7 Apr 2016 at 09:57 igor.berman <ig...@gmail.com> wrote:
>
>> Hi,
>> Suppose I have web facing frontend that gets stream of events(http
>> calls). I
>> need to process event stream and do some aggregations over those events
>> and
>> write aggregated statistics to Hbase - so far Flink seems as perfect
>> match.
>> However in some cases event should trigger some alert and frontend needs
>> to
>> get this alert in synchronous way - here I'm a bit lost. I thought about
>> some kind of following flow:
>> frontend -> queue -> flink -> redis(pub/sub)<- frontend
>>
>> I.e. I have two major use cases - async aggregated analytics/stats
>> computing
>> and "synchronous" response to frontend. Frontend might be node/play or any
>> other technology that won't have a problem of "waiting" for the response,
>> so
>> the only question - how to implement this feedback ?
>> Might be some kind of Sink?
>>
>> Any ideas would be appreciated,
>> Igor
>>
>>
>>
>>
>> --
>> View this message in context:
>> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Flink-event-processing-immediate-feedback-tp5978.html
>> Sent from the Apache Flink User Mailing List archive. mailing list
>> archive at Nabble.com.
>>
>
Re: Flink event processing immediate feedback
Posted by Aljoscha Krettek <al...@apache.org>.
Hi,
what do you mean by "synchronous". If I understood it correctly then some
events entering the Flink pipeline would trigger an alert while some others
would not trigger an alert. How would the component that receives such
alerts know when to wait and when to don't wait.
As I see it you can push these alerts into some sort of message queue and
have a request in the front-end that returns as soon as something is
available in that queue. Then you display it and start a new query that
returns as soon as more data is available in the alert queue.
Cheers,
Aljoscha
On Thu, 7 Apr 2016 at 09:57 igor.berman <ig...@gmail.com> wrote:
> Hi,
> Suppose I have web facing frontend that gets stream of events(http calls).
> I
> need to process event stream and do some aggregations over those events and
> write aggregated statistics to Hbase - so far Flink seems as perfect match.
> However in some cases event should trigger some alert and frontend needs to
> get this alert in synchronous way - here I'm a bit lost. I thought about
> some kind of following flow:
> frontend -> queue -> flink -> redis(pub/sub)<- frontend
>
> I.e. I have two major use cases - async aggregated analytics/stats
> computing
> and "synchronous" response to frontend. Frontend might be node/play or any
> other technology that won't have a problem of "waiting" for the response,
> so
> the only question - how to implement this feedback ?
> Might be some kind of Sink?
>
> Any ideas would be appreciated,
> Igor
>
>
>
>
> --
> View this message in context:
> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Flink-event-processing-immediate-feedback-tp5978.html
> Sent from the Apache Flink User Mailing List archive. mailing list archive
> at Nabble.com.
>