Business Rules / Decision Tree on Kafka Messages

[Greenhorn Techie <gr...@gmail.com>]

Hi,

Our Kafka message has 10-15 fields. On the consumer side, we need to take
some action based on a combination of fields. For example, if Field A = X
and Field B = Y, then push the message to Topic T1. However, if Field C = P
or Field D = Q, then push the message into Topic T2. Likewise, we have many
such rules depending on which certain action has to be taken.

Wondering whether there is standard pattern to implement such business
rules logic in any of the Kafka ecosystem components? For Kafka Streams?
Even otherwise, would be glad to understand how companies implement such
decision trees in practice on streaming data.

Thanks

Re: Business Rules / Decision Tree on Kafka Messages

[Russell Jurney <ru...@gmail.com>]

I just had to drop in and say that this may be a good fit for basic machine
learning in the form of simple decision trees
<https://en.wikipedia.org/wiki/Decision_tree>?

You could train simple decision tree classifier using examples of your
messages and their associated output message types. Then you would create
and deploy a worker that predicts the message type to send based on the
fields in the messages it reads. It would then send that message type.
Something like scikit-learn's DecisionTreeClassifier
<https://scikit-learn.org/stable/modules/generated/sklearn.tree.DecisionTreeClassifier.html#sklearn.tree.DecisionTreeClassifier>
may
work, or there is sure to be a decision tree library in the JVM world that
would work with Streams.

Just a thought,
Russell Jurney @rjurney <http://twitter.com/rjurney>
russell.jurney@gmail.com LI <http://linkedin.com/in/russelljurney> FB
<http://facebook.com/jurney> datasyndrome.com


On Mon, Mar 11, 2019 at 11:41 AM Daniel Hinojosa <
dhinojosa@evolutionnext.com> wrote:

> I would go with Kafka Streams. Kafka Streams is just a library. So you can
> make a simple or complex application. You can include a rules engine or
> machine learning model as a dependency as well as a kafka dependency and
> based on your message send to being different topics.  Run a number of your
> application equal to your partitions on different machines and let the
> magic happen.
>
> On Mon, Mar 11, 2019, 10:28 AM Greenhorn Techie <greenhorntechie@gmail.com
> >
> wrote:
>
> > Hi,
> >
> > Our Kafka message has 10-15 fields. On the consumer side, we need to take
> > some action based on a combination of fields. For example, if Field A = X
> > and Field B = Y, then push the message to Topic T1. However, if Field C
> = P
> > or Field D = Q, then push the message into Topic T2. Likewise, we have
> many
> > such rules depending on which certain action has to be taken.
> >
> > Wondering whether there is standard pattern to implement such business
> > rules logic in any of the Kafka ecosystem components? For Kafka Streams?
> > Even otherwise, would be glad to understand how companies implement such
> > decision trees in practice on streaming data.
> >
> > Thanks
> >
>

Re: Business Rules / Decision Tree on Kafka Messages

[Daniel Hinojosa <dh...@evolutionnext.com>]

I would go with Kafka Streams. Kafka Streams is just a library. So you can
make a simple or complex application. You can include a rules engine or
machine learning model as a dependency as well as a kafka dependency and
based on your message send to being different topics.  Run a number of your
application equal to your partitions on different machines and let the
magic happen.

On Mon, Mar 11, 2019, 10:28 AM Greenhorn Techie <gr...@gmail.com>
wrote:

> Hi,
>
> Our Kafka message has 10-15 fields. On the consumer side, we need to take
> some action based on a combination of fields. For example, if Field A = X
> and Field B = Y, then push the message to Topic T1. However, if Field C = P
> or Field D = Q, then push the message into Topic T2. Likewise, we have many
> such rules depending on which certain action has to be taken.
>
> Wondering whether there is standard pattern to implement such business
> rules logic in any of the Kafka ecosystem components? For Kafka Streams?
> Even otherwise, would be glad to understand how companies implement such
> decision trees in practice on streaming data.
>
> Thanks
>