You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by zanqing zhang <za...@gmail.com> on 2017/11/28 20:53:46 UTC
user driven stream processing
Hi All,
Has anyone done any stream processing driven by a user request? What's the
recommended way of doing this? Or is this completely wrong direction to go
for applications running on top of Flink?
Basically we need to tweak the stream processing based on parameters
provided by a user, e.g. show me the total # of application failures due to
"ABC", which is provided by the user. We are thinking of starting a flink
job with "ABC" as a parameter but this would result in a huge number of
flink jobs, is there a better way for this? Can we trigger the calculation
on a running job?
Thanks in advance.
KZ
Re: user driven stream processing
Posted by zanqing zhang <za...@gmail.com>.
Thanks Fabian and Tony for the info. It's very helpful.
Looks like the general approach is to implement a job topology containing
parameterized (CoXXXMapFunction) operators. The user defined parameters
will be ingested using the extra input the CoXXXMapFunction take.
Ken
On Wed, Nov 29, 2017 at 5:27 AM, Fabian Hueske <fh...@gmail.com> wrote:
> Another example is King's RBEA platform [1] which was built on Flink.
> In a nutshell, RBEA runs a single large Flink job, to which users can add
> queries that should be computed.
> Of course, the query language is restricted because they queries must
> match on the structure of the running job.
>
> Hope this helps,
> Fabian
>
> [1] http://2016.flink-forward.org/kb_sessions/rbea-scalable-
> real-time-analytics-at-king/
>
> 2017-11-29 3:32 GMT+01:00 Tony Wei <to...@gmail.com>:
>
>> Hi KZ,
>>
>> https://data-artisans.com/blog/real-time-fraud-detection-
>> ing-bank-apache-flink
>> This article seems to be a good example to trigger a new calculation on a
>> running job. Maybe you can get some help from it.
>>
>> Best Regards,
>> Tony Wei
>>
>> 2017-11-29 4:53 GMT+08:00 zanqing zhang <za...@gmail.com>:
>>
>>> Hi All,
>>>
>>> Has anyone done any stream processing driven by a user request? What's
>>> the recommended way of doing this? Or is this completely wrong direction to
>>> go for applications running on top of Flink?
>>>
>>> Basically we need to tweak the stream processing based on parameters
>>> provided by a user, e.g. show me the total # of application failures due to
>>> "ABC", which is provided by the user. We are thinking of starting a flink
>>> job with "ABC" as a parameter but this would result in a huge number of
>>> flink jobs, is there a better way for this? Can we trigger the calculation
>>> on a running job?
>>>
>>> Thanks in advance.
>>>
>>> KZ
>>>
>>>
>>
>
--
Ken Zhang
-----------------------------------------------------------
*Smart && Creative && Open == Innovative*
Re: user driven stream processing
Posted by Fabian Hueske <fh...@gmail.com>.
Another example is King's RBEA platform [1] which was built on Flink.
In a nutshell, RBEA runs a single large Flink job, to which users can add
queries that should be computed.
Of course, the query language is restricted because they queries must match
on the structure of the running job.
Hope this helps,
Fabian
[1]
http://2016.flink-forward.org/kb_sessions/rbea-scalable-real-time-analytics-at-king/
2017-11-29 3:32 GMT+01:00 Tony Wei <to...@gmail.com>:
> Hi KZ,
>
> https://data-artisans.com/blog/real-time-fraud-detection-ing-bank-apache-
> flink
> This article seems to be a good example to trigger a new calculation on a
> running job. Maybe you can get some help from it.
>
> Best Regards,
> Tony Wei
>
> 2017-11-29 4:53 GMT+08:00 zanqing zhang <za...@gmail.com>:
>
>> Hi All,
>>
>> Has anyone done any stream processing driven by a user request? What's
>> the recommended way of doing this? Or is this completely wrong direction to
>> go for applications running on top of Flink?
>>
>> Basically we need to tweak the stream processing based on parameters
>> provided by a user, e.g. show me the total # of application failures due to
>> "ABC", which is provided by the user. We are thinking of starting a flink
>> job with "ABC" as a parameter but this would result in a huge number of
>> flink jobs, is there a better way for this? Can we trigger the calculation
>> on a running job?
>>
>> Thanks in advance.
>>
>> KZ
>>
>>
>
Re: user driven stream processing
Posted by Tony Wei <to...@gmail.com>.
Hi KZ,
https://data-artisans.com/blog/real-time-fraud-detection-ing-bank-apache-flink
This article seems to be a good example to trigger a new calculation on a
running job. Maybe you can get some help from it.
Best Regards,
Tony Wei
2017-11-29 4:53 GMT+08:00 zanqing zhang <za...@gmail.com>:
> Hi All,
>
> Has anyone done any stream processing driven by a user request? What's the
> recommended way of doing this? Or is this completely wrong direction to go
> for applications running on top of Flink?
>
> Basically we need to tweak the stream processing based on parameters
> provided by a user, e.g. show me the total # of application failures due to
> "ABC", which is provided by the user. We are thinking of starting a flink
> job with "ABC" as a parameter but this would result in a huge number of
> flink jobs, is there a better way for this? Can we trigger the calculation
> on a running job?
>
> Thanks in advance.
>
> KZ
>
>