Flush profiler based on condition and not based time

[Michel Sumbul <mi...@gmail.com>]

Hello Metron Guru,

I would like to know if there's way to have a profiler that will flush the
result to hbase not based on a time period but on some conditions.

For example, if we are tracking a specific sequence over the time for an
ip/user like (event A, event b and event C). If this entire sequence
happens inside the profil period duration its fine, but if this sequence
happens over 2 or more profiler periods then it will not be detected.
Moreover if the sequence occured in 1 sec but the profiler period is 15
minutes, then it will wait a long time before being flush to hbase.

Another use case will be, if you are looking at the average of something
and a threshold is reach to directly flush it and then generate an alert
asap. Like number of ssh connection over the last x minutes bigger than N,
then flush it. otherwise continue to profile the user.

My first question is, is it currently possible to do that, because I have
not found how.
Secondly if it not feasible for the moment, do you think that might be an
useful feature?

I was also thinking that the result might be flush to a specific kafka
topic and not to hbase. For example, if the profiler detect an anomaly on
the behavior like number of ssh connection, or sequence of event, it flush
the result to a kafka topic with all the "real-time" alert.

Maybe this already exist and profiler is not the good place to do that, to
be honest I'm not sure. But I suppose that Im not the first that imagine
that, so any comment on how to realize this is welcome :)

Best regard,
Michel