Re: Integrating Flink CEP with a Rules Engine

[Anton <ku...@gmail.com>]

We also have a requirement of using Drools in Flink. Drools brings a very
mature and usable business rules editor. And to be able to integrate Drools
into Flink would be very useful.

On 23 June 2017 at 22:09, Suneel Marthi <sm...@apache.org> wrote:

> Sorry I didn't read the whole thread.
>
> We have a similar rqmt wherein the users would like to add/update/delete
> CEP patterns via UX or REST api and we started discussing building a REST
> api for that, glad to see that this is a common ask and if there's already
> a community effort around this - that's great to know.
>
> On Fri, Jun 23, 2017 at 9:54 AM, Sridhar Chellappa <fl...@gmail.com>
> wrote:
>
>> Folks,
>>
>> Plenty of very good points but I see this discussion digressing from what
>> I originally asked for. We need a dashboard to let the Business Analysts to
>> define rules and the CEP to run them.
>>
>> My original question was how to solve this with Flink CEP?
>>
>> From what I see, this is not a solved problem. Correct me if I am wrong.
>>
>> On Fri, Jun 23, 2017 at 6:52 PM, Kostas Kloudas <
>> k.kloudas@data-artisans.com> wrote:
>>
>>> Hi all,
>>>
>>> Currently there is an ongoing effort to integrate FlinkCEP with Flink's
>>> SQL API.
>>> There is already an open FLIP for this:
>>>
>>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-20%3A
>>> +Integration+of+SQL+and+CEP
>>> <https://cwiki.apache.org/confluence/display/FLINK/FLIP-20:+Integration+of+SQL+and+CEP>
>>>
>>> So, if there was an effort for integration of different
>>> libraries/tools/functionality as well, it
>>> would be nice to go a bit more into details on i) what is already there,
>>> ii) what is planned to be
>>> integrated for the SQL effort, and iii) what else is required, and
>>> consolidate the resources
>>> available.
>>>
>>> This will allow the community to move faster and with a clear roadmap.
>>>
>>> Kostas
>>>
>>> On Jun 23, 2017, at 2:51 PM, Suneel Marthi <sm...@apache.org> wrote:
>>>
>>> FWIW, here's an old Cloudera blog about using Drools with Spark.
>>>
>>> https://blog.cloudera.com/blog/2015/11/how-to-build-a-comple
>>> x-event-processing-app-on-apache-spark-and-drools/
>>>
>>> It should be possible to invoke Drools from Flink in a similar way (I
>>> have not tried it).
>>>
>>> It all depends on what the use case and how much of present Flink CEP
>>> satisfies the use case before considering integration with more complex
>>> rule engines.
>>>
>>>
>>> Disclaimer: I work for Red Hat
>>>
>>> On Fri, Jun 23, 2017 at 8:43 AM, Ismaël Mejía <ie...@gmail.com> wrote:
>>>
>>>> Hello,
>>>>
>>>> It is really interesting to see this discussion because that was one
>>>> of the questions on the presentation on CEP at Berlin Buzzwords, and
>>>> this is one line of work that may eventually make sense to explore.
>>>>
>>>> Rule engines like drools implement the Rete algorithm that if I
>>>> understood correctly optimizes the analysis of a relatively big set of
>>>> facts (conditions) into a simpler evaluation graph. For more details
>>>> this is a really nice explanation.
>>>> https://www.sparklinglogic.com/rete-algorithm-demystified-part-2/
>>>>
>>>> On flink's CEP I have the impression that you define this graph by
>>>> hand. Using a rule engine you could infer an optimal graph from the
>>>> set of rules, and then this graph could be translated into CEP
>>>> patterns.
>>>>
>>>> Of course take all of this with a grain of salt because I am not an
>>>> expert on both CEP or the Rete algorithm, but I start to see the
>>>> connection of both worlds more clearly now. So if anyone else has
>>>> ideas of the feasibility of this or can see some other
>>>> issues/consequences please comment. I also have the impression that
>>>> distribution is less of an issue because the rete network is
>>>> calculated only once and updates are not 'dynamic' (but I might be
>>>> wrong).
>>>>
>>>> Ismaël
>>>>
>>>> ps. I add Thomas in copy who was who made the question in the
>>>> conference in case he has some comments/ideas.
>>>>
>>>>
>>>> On Fri, Jun 23, 2017 at 1:48 PM, Kostas Kloudas
>>>> <k....@data-artisans.com> wrote:
>>>> > Hi Jorn and Sridhar,
>>>> >
>>>> > It would be worth describing a bit more what these tools are and what
>>>> are
>>>> > your needs.
>>>> > In addition, and to see what the CEP library already offers here you
>>>> can
>>>> > find the documentation:
>>>> >
>>>> > https://ci.apache.org/projects/flink/flink-docs-release-1.3/
>>>> dev/libs/cep.html
>>>> >
>>>> >
>>>> > Thanks,
>>>> > Kostas
>>>> >
>>>> > On Jun 23, 2017, at 1:41 PM, Jörn Franke <jo...@gmail.com>
>>>> wrote:
>>>> >
>>>> > Hallo,
>>>> >
>>>> > It si possible, but some caveat : flink is a distributed system, but
>>>> in
>>>> > drools the fact are only locally available. This may lead to strange
>>>> effects
>>>> > when rules update the fact base.
>>>> >
>>>> > Best regards
>>>> >
>>>> > On 23. Jun 2017, at 12:49, Sridhar Chellappa <fl...@gmail.com>
>>>> wrote:
>>>> >
>>>> > Folks,
>>>> >
>>>> > I am new to Flink.
>>>> >
>>>> > One of the reasons why I am interested in Flink is because of its CEP
>>>> > library. Our CEP logic comprises of a set of complex business rules
>>>> which
>>>> > will have to be managed (Create, Update, Delete) by a bunch of
>>>> business
>>>> > analysts.
>>>> >
>>>> > Is there a way I can integrate other third party tools (Drools,
>>>> OpenRules)
>>>> > to let Business Analysts define rules and  execute them using Flink's
>>>> CEP
>>>> > library?
>>>> >
>>>> >
>>>>
>>>
>>>
>>>
>>
>