You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@metron.apache.org by Dima Kovalyov <Di...@sstech.us> on 2017/08/09 19:03:52 UTC
Profiler statistics NaN
Hello Metron Team,
I have created following profiler:
> {
> "profile": "host-talks-to",
> "onlyif": "exists(source_ip)",
> "foreach": "source_ip",
> "init": {
> "outcoming": "HLLP_INIT(5, 6)"
> },
> "update": { "outcoming": "HLLP_ADD(outcoming, destination_ip)" },
> "result": "HLLP_CARDINALITY(outcoming)"
> }
I have also created enrichment rule:
> {
> "enrichment" : {
> "fieldMap": {
> "stellar" : {
> "config" : {
> "numeric" : {
> "value_red_level_out": "STATS_PERCENTILE( STATS_MERGE(
> PROFILE_GET('host-being-talked-to', ip_src_addr, 1, 'HOURS')), 95)",
> "value_red_level_in": "STATS_PERCENTILE( STATS_MERGE(
> PROFILE_GET('host-talks-to', ip_src_addr, 1, 'HOURS')), 95)"
> },
> "text" : {
> "is_alert": "true"
> }
> }
> }
> }
> } }
However when I stream data to it I receive: "value_red_level_out": null,
I have checked in profiler client and here is what I got:
> [Stellar]>>> PROFILE_GET( "host-talks-to" , "99.191.183.156",
> PROFILE_FIXED(300, "MINUTES"))
> [1, 6, 6, 6, 6, 6, 3, 4, 5, 6, 4, 6, 6, 6, 1, 1, 6, 6, 1, 4, 1, 1, 4,
> 6, 6, 1, 6, 6, 1, 2, 6, 1, 1, 1, 6, 4, 6, 6, 3, 1, 6, 2, 1, 6, 1, 6]
> [Stellar]>>> STATS_PERCENTILE(STATS_MERGE(
> PROFILE_GET('host-talks-to', '99.191.183.156', PROFILE_FIXED(10,
> 'HOURS'))), 90)
> NaN
> [Stellar]>>> STATS_MERGE( PROFILE_GET('host-talks-to',
> '99.191.183.156', PROFILE_FIXED(10, 'HOURS')))
So the STATS_MERGE produces no results. Is this something expected or I
made a mistake somewhere? Please advise.
p.s. I am following this use cases:
https://github.com/hortonworks-gallery/metron-rules/tree/master/use-cases/DegreeOfHost
There were number of errors in the configs originally, which I have
corrected, maybe I missed something else.
- Dima
Re: Profiler statistics NaN
Posted by Otto Fowler <ot...@gmail.com>.
A couple of things come to mind, in no order
* Higher level compositing functions that bring all these things together…
maybe packaged as a snazzy extension ;)
* A more ‘structured’ version of the stellar shell text file input, where
you could configure variables in the file and just run it, so when the sell
reads it, it can prompt you for the inputs, then print out the plan and
execute…
* I think having attributes on stellar functions that give type info on
functions would help too, we could do an analysis tool that can
verify the types.
On August 9, 2017 at 16:44:10, Casey Stella (cestella@gmail.com) wrote:
Well, we need that too :) What're you thinking, procedures for stellar?
On Wed, Aug 9, 2017 at 4:42 PM, Otto Fowler <ot...@gmail.com> wrote:
> What we need, is a way to package up some ‘recipes’ for stellar.
> If many people are going to do this operation, then a more friendly set of
> facade functions,
> or some thing would work.
>
>
>
> On August 9, 2017 at 16:38:48, Casey Stella (cestella@gmail.com) wrote:
>
> Yeah, I'm leaning toward STATS_ADD or STATS_INIT taking a list of numbers.
> STATS_MERGE seems confusing.
>
> On Wed, Aug 9, 2017 at 4:37 PM, Nick Allen <ni...@nickallen.org> wrote:
>
> > Or even change the behavior of STATS_MERGE, too? If STATS_MERGE gets raw
> > numbers, it wraps those in a Stats object, then returns it. Then Dima's
> > example would just work as-is.
> >
> > I'm not sure I like that though. Maybe so flexible as to be confusing?
> > Thought I would throw it out as an alternative to consider.
> >
> >
> >
> >
> > On Wed, Aug 9, 2017 at 4:31 PM Nick Allen <ni...@nickallen.org> wrote:
> >
> > > Oh yeah, duh. Now I'm with you. That would be a good quick hit.
> > >
> > > The current behavior is a little nutty. If there is a list, it only
> > > consumes the first element in the list. I'd expect that it should
> either
> > > do what you describe or complain that it doesn't know how to handle a
> > > list. Easy fix though.
> > >
> > > [Stellar]>>> STATS_MEAN(STATS_ADD(null, 1, 2, 3))
> > > 2.0
> > >
> > > [Stellar]>>> STATS_MEAN(STATS_ADD(null, [1,2,3]))
> > > 1.0
> > >
> > > [Stellar]>>> STATS_COUNT(STATS_ADD(null, [1,2,3]))
> > > 1.0
> > >
> > > On Wed, Aug 9, 2017 at 4:17 PM Casey Stella <ce...@gmail.com>
> wrote:
> > >
> > >> outcoming is still a HLLP object, not a statistics object, so doing a
> > >> STATS_MERGE on a bunch of them wouldn't work either.
> > >>
> > >> On Wed, Aug 9, 2017 at 4:15 PM, Nick Allen <ni...@nickallen.org>
> wrote:
> > >>
> > >> > That is another problem. Isn't the simplest answer, to just change
> > >> this...
> > >> >
> > >> > "result": "HLLP_CARDINALITY(outcoming)"
> > >> >
> > >> > to this...
> > >> >
> > >> > "result": "outcoming"
> > >> >
> > >> > ?
> > >> >
> > >> > On Wed, Aug 9, 2017 at 3:48 PM Casey Stella <ce...@gmail.com>
> > wrote:
> > >> >
> > >> > > Ok, so the problem here is that your profile is returning integers
> > >> > > (specifically HLLP cardinalities) rather than stats objects. When
> > >> you're
> > >> > > doing:
> > >> > > STATS_PERCENTILE(STATS_MERGE( PROFILE_GET('host-talks-to',
> > >> > > '99.191.183.156', PROFILE_FIXED(10, 'HOURS')), 90)
> > >> > > You are calling STATS_MERGE on a list of integers and it takes a
> > list
> > >> of
> > >> > > statistics objects.
> > >> > >
> > >> > > What you can do instead is:
> > >> > > STATS_PERCENTILE( REDUCE( PROFILE_GET('host-talks-to',
> > >> > > '99.191.183.156', PROFILE_FIXED(10, 'HOURS'), (s, x) ->
> STATS_ADD(s,
> > >> x),
> > >> > > STATS_INIT()), 90)
> > >> > >
> > >> > > Ok, that looks horrible, doesn't it? Well, thankfully we added
> > >> temporary
> > >> > > variables for stellar enrichments in 0.4.1. Let's take that
> > "numeric"
> > >> > > stellar enrichment group and reimagine it. With temporary
> > variables,
> > >> you
> > >> > > would turn:
> > >> > >
> > >> > > "numeric" : {
> > >> > > "value_red_level_out": "STATS_PERCENTILE( REDUCE(
> > >> > > PROFILE_GET('host-being-talked-to', ip_src_addr, PROFILE_FIXED(1,
> > >> > > 'HOURS')), (s, x) -> STATS_ADD(s, x), STATS_INIT()), 95)",
> > >> > > "value_red_level_in": "STATS_PERCENTILE( REDUCE(
> > >> > > PROFILE_GET('host-talks-to',
> > >> > > ip_src_addr, PROFILE_FIXED(1, 'HOURS')), (s, x) -> STATS_ADD(s,
> x),
> > >> > > STATS_INIT()), 95)"
> > >> > > },
> > >> > >
> > >> > > into:
> > >> > > "numeric" : [
> > >> > > "profile_duration := PROFILE_FIXED(1, 'HOURS')",
> > >> > > "host_being_talked_to := PROFILE_GET('host-being-
> > >> > talked-to',
> > >> > > ip_src_addr, profile_duration)",
> > >> > > "host_talks_to := PROFILE_GET('host-talks-to',
> > >> ip_src_addr,
> > >> > > profile_duration)",
> > >> > > "host_being_talked_to_stats := REDUCE(
> > >> host_being_talked_to,
> > >> > > (s, x) -> STATS_ADD(s, x), STATS_INIT())",
> > >> > > "host_talks_to_stats := REDUCE(host_talks_to, (s, x) ->
> > >> > > STATS_ADD(s, x), STATS_INIT())",
> > >> > > "value_red_level_out": "STATS_PERCENTILE(
> > >> > > host_being_talked_to_stats, 95)",
> > >> > > "value_red_level_in": "STATS_PERCENTILE(
> > >> > host_talks_to_stats,
> > >> > > 95)",
> > >> > > "profile_duration := null",
> > >> > > "host_being_talked_to := null",
> > >> > > "host_talks_to := null",
> > >> > > "host_being_talked_to_stats := null",
> > >> > > host_talks_to_stats := null"
> > >> > > ],
> > >> > >
> > >> > > That's a lot more to type, but it allows you to reuse and take the
> > >> pieces
> > >> > > in chunks.
> > >> > >
> > >> > > Ok, so now I find myself thinking "a pox on both your houses"
> since
> > >> both
> > >> > > examples now kinda look long and convoluted. So, why are they?
> > Well,
> > >> > that
> > >> > > REDUCE is likely the culprit. It's supposed to get us out of bad
> > >> > > situations not show up in what could be argued is the 80% case.
> How
> > >> > about,
> > >> > > instead, we allow STATS_ADD or STATS_INIT to take a list of
> > >> numbers? If
> > >> > > so, we could pretty easily make that nicer:
> > >> > > STATS_PERCENTILE( STATS_ADD( PROFILE_GET('host-being-
> > talked-to',
> > >> > > ip_src_addr, PROFILE_FIXED(1, 'HOURS'))), 95)
> > >> > >
> > >> > > or
> > >> > > STATS_PERCENTILE( STATS_INIT(
> > >> PROFILE_GET('host-being-talked-to',
> > >> > > ip_src_addr, PROFILE_FIXED(1, 'HOURS'))), 95)
> > >> > >
> > >> > >
> > >> > > We should make some sort of candy like that so we can avoid some
> of
> > >> the
> > >> > > complexity in the normal case.
> > >> > >
> > >> > > On Wed, Aug 9, 2017 at 3:03 PM, Dima Kovalyov <
> > >> Dima.Kovalyov@sstech.us>
> > >> > > wrote:
> > >> > >
> > >> > > > Hello Metron Team,
> > >> > > >
> > >> > > > I have created following profiler:
> > >> > > > > {
> > >> > > > > "profile": "host-talks-to",
> > >> > > > > "onlyif": "exists(source_ip)",
> > >> > > > > "foreach": "source_ip",
> > >> > > > > "init": {
> > >> > > > > "outcoming": "HLLP_INIT(5, 6)"
> > >> > > > > },
> > >> > > > > "update": { "outcoming": "HLLP_ADD(outcoming, destination_ip)"
> > >> },
> > >> > > > > "result": "HLLP_CARDINALITY(outcoming)"
> > >> > > > > }
> > >> > > > I have also created enrichment rule:
> > >> > > > > {
> > >> > > > > "enrichment" : {
> > >> > > > > "fieldMap": {
> > >> > > > > "stellar" : {
> > >> > > > > "config" : {
> > >> > > > > "numeric" : {
> > >> > > > > "value_red_level_out": "STATS_PERCENTILE(
> > STATS_MERGE(
> > >> > > > > PROFILE_GET('host-being-talked-to', ip_src_addr, 1,
> 'HOURS')),
> > >> 95)",
> > >> > > > > "value_red_level_in": "STATS_PERCENTILE(
> > STATS_MERGE(
> > >> > > > > PROFILE_GET('host-talks-to', ip_src_addr, 1, 'HOURS')), 95)"
> > >> > > > > },
> > >> > > > > "text" : {
> > >> > > > > "is_alert": "true"
> > >> > > > > }
> > >> > > > > }
> > >> > > > > }
> > >> > > > > }
> > >> > > > > } }
> > >> > > > However when I stream data to it I receive:
> "value_red_level_out":
> > >> > null,
> > >> > > >
> > >> > > > I have checked in profiler client and here is what I got:
> > >> > > > > [Stellar]>>> PROFILE_GET( "host-talks-to" , "99.191.183.156",
> > >> > > > > PROFILE_FIXED(300, "MINUTES"))
> > >> > > > > [1, 6, 6, 6, 6, 6, 3, 4, 5, 6, 4, 6, 6, 6, 1, 1, 6, 6, 1, 4,
> 1,
> > >> 1, 4,
> > >> > > > > 6, 6, 1, 6, 6, 1, 2, 6, 1, 1, 1, 6, 4, 6, 6, 3, 1, 6, 2, 1, 6,
> > 1,
> > >> 6]
> > >> > > > > [Stellar]>>> STATS_PERCENTILE(STATS_MERGE(
> > >> > > > > PROFILE_GET('host-talks-to', '99.191.183.156',
> PROFILE_FIXED(10,
> > >> > > > > 'HOURS'))), 90)
> > >> > > > > NaN
> > >> > > > > [Stellar]>>> STATS_MERGE( PROFILE_GET('host-talks-to',
> > >> > > > > '99.191.183.156', PROFILE_FIXED(10, 'HOURS')))
> > >> > > > So the STATS_MERGE produces no results. Is this something
> expected
> > >> or I
> > >> > > > made a mistake somewhere? Please advise.
> > >> > > >
> > >> > > >
> > >> > > > p.s. I am following this use cases:
> > >> > > >
> > >> > > https://github.com/hortonworks-gallery/metron-
> > >> > rules/tree/master/use-cases/
> > >> > > > DegreeOfHost
> > >> > > > There were number of errors in the configs originally, which I
> > have
> > >> > > > corrected, maybe I missed something else.
> > >> > > >
> > >> > > > - Dima
> > >> > > >
> > >> > > >
> > >> > > >
> > >> > > >
> > >> > > >
> > >> > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> > >
> >
>
>
Re: Profiler statistics NaN
Posted by Nick Allen <ni...@nickallen.org>.
I like it, Otto.
I see the recipe idea implemented as a collection of GUI wizards. The user
can login to a web interface and choose from a collection of recipes. The
user interacts with a recipe via a GUI wizard-like mechanism. The wizard
gathers the input needed from the user to implement a recipe. The recipe
itself then takes that input and creates all the ugly JSON and Stellar
configuration required to actually implement the recipe in Metron.
These recipes would be great way for new users to hit the ground running
with Metron. Advanced users could still get under-the-hood and create
their own Profiles, Triage rules, etc using JSON and Stellar without using
these recipes.
We'd also want to make the recipe infrastructure pluggable. We want the
community to be able to quickly and easily create new recipes without
having to build a custom HTML/JS wizard page for each recipe.
On Wed, Aug 9, 2017 at 4:44 PM Casey Stella <ce...@gmail.com> wrote:
> Well, we need that too :) What're you thinking, procedures for stellar?
>
> On Wed, Aug 9, 2017 at 4:42 PM, Otto Fowler <ot...@gmail.com>
> wrote:
>
> > What we need, is a way to package up some ‘recipes’ for stellar.
> > If many people are going to do this operation, then a more friendly set
> of
> > facade functions,
> > or some thing would work.
> >
> >
> >
> > On August 9, 2017 at 16:38:48, Casey Stella (cestella@gmail.com) wrote:
> >
> > Yeah, I'm leaning toward STATS_ADD or STATS_INIT taking a list of
> numbers.
> > STATS_MERGE seems confusing.
> >
> > On Wed, Aug 9, 2017 at 4:37 PM, Nick Allen <ni...@nickallen.org> wrote:
> >
> > > Or even change the behavior of STATS_MERGE, too? If STATS_MERGE gets
> raw
> > > numbers, it wraps those in a Stats object, then returns it. Then Dima's
> > > example would just work as-is.
> > >
> > > I'm not sure I like that though. Maybe so flexible as to be confusing?
> > > Thought I would throw it out as an alternative to consider.
> > >
> > >
> > >
> > >
> > > On Wed, Aug 9, 2017 at 4:31 PM Nick Allen <ni...@nickallen.org> wrote:
> > >
> > > > Oh yeah, duh. Now I'm with you. That would be a good quick hit.
> > > >
> > > > The current behavior is a little nutty. If there is a list, it only
> > > > consumes the first element in the list. I'd expect that it should
> > either
> > > > do what you describe or complain that it doesn't know how to handle a
> > > > list. Easy fix though.
> > > >
> > > > [Stellar]>>> STATS_MEAN(STATS_ADD(null, 1, 2, 3))
> > > > 2.0
> > > >
> > > > [Stellar]>>> STATS_MEAN(STATS_ADD(null, [1,2,3]))
> > > > 1.0
> > > >
> > > > [Stellar]>>> STATS_COUNT(STATS_ADD(null, [1,2,3]))
> > > > 1.0
> > > >
> > > > On Wed, Aug 9, 2017 at 4:17 PM Casey Stella <ce...@gmail.com>
> > wrote:
> > > >
> > > >> outcoming is still a HLLP object, not a statistics object, so doing
> a
> > > >> STATS_MERGE on a bunch of them wouldn't work either.
> > > >>
> > > >> On Wed, Aug 9, 2017 at 4:15 PM, Nick Allen <ni...@nickallen.org>
> > wrote:
> > > >>
> > > >> > That is another problem. Isn't the simplest answer, to just change
> > > >> this...
> > > >> >
> > > >> > "result": "HLLP_CARDINALITY(outcoming)"
> > > >> >
> > > >> > to this...
> > > >> >
> > > >> > "result": "outcoming"
> > > >> >
> > > >> > ?
> > > >> >
> > > >> > On Wed, Aug 9, 2017 at 3:48 PM Casey Stella <ce...@gmail.com>
> > > wrote:
> > > >> >
> > > >> > > Ok, so the problem here is that your profile is returning
> > integers
> > > >> > > (specifically HLLP cardinalities) rather than stats objects.
> When
> > > >> you're
> > > >> > > doing:
> > > >> > > STATS_PERCENTILE(STATS_MERGE( PROFILE_GET('host-talks-to',
> > > >> > > '99.191.183.156', PROFILE_FIXED(10, 'HOURS')), 90)
> > > >> > > You are calling STATS_MERGE on a list of integers and it takes a
> > > list
> > > >> of
> > > >> > > statistics objects.
> > > >> > >
> > > >> > > What you can do instead is:
> > > >> > > STATS_PERCENTILE( REDUCE( PROFILE_GET('host-talks-to',
> > > >> > > '99.191.183.156', PROFILE_FIXED(10, 'HOURS'), (s, x) ->
> > STATS_ADD(s,
> > > >> x),
> > > >> > > STATS_INIT()), 90)
> > > >> > >
> > > >> > > Ok, that looks horrible, doesn't it? Well, thankfully we added
> > > >> temporary
> > > >> > > variables for stellar enrichments in 0.4.1. Let's take that
> > > "numeric"
> > > >> > > stellar enrichment group and reimagine it. With temporary
> > > variables,
> > > >> you
> > > >> > > would turn:
> > > >> > >
> > > >> > > "numeric" : {
> > > >> > > "value_red_level_out": "STATS_PERCENTILE( REDUCE(
> > > >> > > PROFILE_GET('host-being-talked-to', ip_src_addr,
> > PROFILE_FIXED(1,
> > > >> > > 'HOURS')), (s, x) -> STATS_ADD(s, x), STATS_INIT()), 95)",
> > > >> > > "value_red_level_in": "STATS_PERCENTILE( REDUCE(
> > > >> > > PROFILE_GET('host-talks-to',
> > > >> > > ip_src_addr, PROFILE_FIXED(1, 'HOURS')), (s, x) -> STATS_ADD(s,
> > x),
> > > >> > > STATS_INIT()), 95)"
> > > >> > > },
> > > >> > >
> > > >> > > into:
> > > >> > > "numeric" : [
> > > >> > > "profile_duration := PROFILE_FIXED(1, 'HOURS')",
> > > >> > > "host_being_talked_to := PROFILE_GET('host-being-
> > > >> > talked-to',
> > > >> > > ip_src_addr, profile_duration)",
> > > >> > > "host_talks_to := PROFILE_GET('host-talks-to',
> > > >> ip_src_addr,
> > > >> > > profile_duration)",
> > > >> > > "host_being_talked_to_stats := REDUCE(
> > > >> host_being_talked_to,
> > > >> > > (s, x) -> STATS_ADD(s, x), STATS_INIT())",
> > > >> > > "host_talks_to_stats := REDUCE(host_talks_to, (s, x) ->
> > > >> > > STATS_ADD(s, x), STATS_INIT())",
> > > >> > > "value_red_level_out": "STATS_PERCENTILE(
> > > >> > > host_being_talked_to_stats, 95)",
> > > >> > > "value_red_level_in": "STATS_PERCENTILE(
> > > >> > host_talks_to_stats,
> > > >> > > 95)",
> > > >> > > "profile_duration := null",
> > > >> > > "host_being_talked_to := null",
> > > >> > > "host_talks_to := null",
> > > >> > > "host_being_talked_to_stats := null",
> > > >> > > host_talks_to_stats := null"
> > > >> > > ],
> > > >> > >
> > > >> > > That's a lot more to type, but it allows you to reuse and take
> > the
> > > >> pieces
> > > >> > > in chunks.
> > > >> > >
> > > >> > > Ok, so now I find myself thinking "a pox on both your houses"
> > since
> > > >> both
> > > >> > > examples now kinda look long and convoluted. So, why are they?
> > > Well,
> > > >> > that
> > > >> > > REDUCE is likely the culprit. It's supposed to get us out of bad
> > > >> > > situations not show up in what could be argued is the 80% case.
> > How
> > > >> > about,
> > > >> > > instead, we allow STATS_ADD or STATS_INIT to take a list of
> > > >> numbers? If
> > > >> > > so, we could pretty easily make that nicer:
> > > >> > > STATS_PERCENTILE( STATS_ADD( PROFILE_GET('host-being-
> > > talked-to',
> > > >> > > ip_src_addr, PROFILE_FIXED(1, 'HOURS'))), 95)
> > > >> > >
> > > >> > > or
> > > >> > > STATS_PERCENTILE( STATS_INIT(
> > > >> PROFILE_GET('host-being-talked-to',
> > > >> > > ip_src_addr, PROFILE_FIXED(1, 'HOURS'))), 95)
> > > >> > >
> > > >> > >
> > > >> > > We should make some sort of candy like that so we can avoid some
> > of
> > > >> the
> > > >> > > complexity in the normal case.
> > > >> > >
> > > >> > > On Wed, Aug 9, 2017 at 3:03 PM, Dima Kovalyov <
> > > >> Dima.Kovalyov@sstech.us>
> > > >> > > wrote:
> > > >> > >
> > > >> > > > Hello Metron Team,
> > > >> > > >
> > > >> > > > I have created following profiler:
> > > >> > > > > {
> > > >> > > > > "profile": "host-talks-to",
> > > >> > > > > "onlyif": "exists(source_ip)",
> > > >> > > > > "foreach": "source_ip",
> > > >> > > > > "init": {
> > > >> > > > > "outcoming": "HLLP_INIT(5, 6)"
> > > >> > > > > },
> > > >> > > > > "update": { "outcoming": "HLLP_ADD(outcoming,
> > destination_ip)"
> > > >> },
> > > >> > > > > "result": "HLLP_CARDINALITY(outcoming)"
> > > >> > > > > }
> > > >> > > > I have also created enrichment rule:
> > > >> > > > > {
> > > >> > > > > "enrichment" : {
> > > >> > > > > "fieldMap": {
> > > >> > > > > "stellar" : {
> > > >> > > > > "config" : {
> > > >> > > > > "numeric" : {
> > > >> > > > > "value_red_level_out": "STATS_PERCENTILE(
> > > STATS_MERGE(
> > > >> > > > > PROFILE_GET('host-being-talked-to', ip_src_addr, 1,
> > 'HOURS')),
> > > >> 95)",
> > > >> > > > > "value_red_level_in": "STATS_PERCENTILE(
> > > STATS_MERGE(
> > > >> > > > > PROFILE_GET('host-talks-to', ip_src_addr, 1, 'HOURS')), 95)"
> > > >> > > > > },
> > > >> > > > > "text" : {
> > > >> > > > > "is_alert": "true"
> > > >> > > > > }
> > > >> > > > > }
> > > >> > > > > }
> > > >> > > > > }
> > > >> > > > > } }
> > > >> > > > However when I stream data to it I receive:
> > "value_red_level_out":
> > > >> > null,
> > > >> > > >
> > > >> > > > I have checked in profiler client and here is what I got:
> > > >> > > > > [Stellar]>>> PROFILE_GET( "host-talks-to" ,
> "99.191.183.156",
> > > >> > > > > PROFILE_FIXED(300, "MINUTES"))
> > > >> > > > > [1, 6, 6, 6, 6, 6, 3, 4, 5, 6, 4, 6, 6, 6, 1, 1, 6, 6, 1, 4,
> > 1,
> > > >> 1, 4,
> > > >> > > > > 6, 6, 1, 6, 6, 1, 2, 6, 1, 1, 1, 6, 4, 6, 6, 3, 1, 6, 2, 1,
> > 6,
> > > 1,
> > > >> 6]
> > > >> > > > > [Stellar]>>> STATS_PERCENTILE(STATS_MERGE(
> > > >> > > > > PROFILE_GET('host-talks-to', '99.191.183.156',
> > PROFILE_FIXED(10,
> > > >> > > > > 'HOURS'))), 90)
> > > >> > > > > NaN
> > > >> > > > > [Stellar]>>> STATS_MERGE( PROFILE_GET('host-talks-to',
> > > >> > > > > '99.191.183.156', PROFILE_FIXED(10, 'HOURS')))
> > > >> > > > So the STATS_MERGE produces no results. Is this something
> > expected
> > > >> or I
> > > >> > > > made a mistake somewhere? Please advise.
> > > >> > > >
> > > >> > > >
> > > >> > > > p.s. I am following this use cases:
> > > >> > > >
> > > >> > > https://github.com/hortonworks-gallery/metron-
> > > >> > rules/tree/master/use-cases/
> > > >> > > > DegreeOfHost
> > > >> > > > There were number of errors in the configs originally, which I
> > > have
> > > >> > > > corrected, maybe I missed something else.
> > > >> > > >
> > > >> > > > - Dima
> > > >> > > >
> > > >> > > >
> > > >> > > >
> > > >> > > >
> > > >> > > >
> > > >> > > >
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > > >
> > >
> >
> >
>
Re: Profiler statistics NaN
Posted by Casey Stella <ce...@gmail.com>.
Well, we need that too :) What're you thinking, procedures for stellar?
On Wed, Aug 9, 2017 at 4:42 PM, Otto Fowler <ot...@gmail.com> wrote:
> What we need, is a way to package up some ‘recipes’ for stellar.
> If many people are going to do this operation, then a more friendly set of
> facade functions,
> or some thing would work.
>
>
>
> On August 9, 2017 at 16:38:48, Casey Stella (cestella@gmail.com) wrote:
>
> Yeah, I'm leaning toward STATS_ADD or STATS_INIT taking a list of numbers.
> STATS_MERGE seems confusing.
>
> On Wed, Aug 9, 2017 at 4:37 PM, Nick Allen <ni...@nickallen.org> wrote:
>
> > Or even change the behavior of STATS_MERGE, too? If STATS_MERGE gets raw
> > numbers, it wraps those in a Stats object, then returns it. Then Dima's
> > example would just work as-is.
> >
> > I'm not sure I like that though. Maybe so flexible as to be confusing?
> > Thought I would throw it out as an alternative to consider.
> >
> >
> >
> >
> > On Wed, Aug 9, 2017 at 4:31 PM Nick Allen <ni...@nickallen.org> wrote:
> >
> > > Oh yeah, duh. Now I'm with you. That would be a good quick hit.
> > >
> > > The current behavior is a little nutty. If there is a list, it only
> > > consumes the first element in the list. I'd expect that it should
> either
> > > do what you describe or complain that it doesn't know how to handle a
> > > list. Easy fix though.
> > >
> > > [Stellar]>>> STATS_MEAN(STATS_ADD(null, 1, 2, 3))
> > > 2.0
> > >
> > > [Stellar]>>> STATS_MEAN(STATS_ADD(null, [1,2,3]))
> > > 1.0
> > >
> > > [Stellar]>>> STATS_COUNT(STATS_ADD(null, [1,2,3]))
> > > 1.0
> > >
> > > On Wed, Aug 9, 2017 at 4:17 PM Casey Stella <ce...@gmail.com>
> wrote:
> > >
> > >> outcoming is still a HLLP object, not a statistics object, so doing a
> > >> STATS_MERGE on a bunch of them wouldn't work either.
> > >>
> > >> On Wed, Aug 9, 2017 at 4:15 PM, Nick Allen <ni...@nickallen.org>
> wrote:
> > >>
> > >> > That is another problem. Isn't the simplest answer, to just change
> > >> this...
> > >> >
> > >> > "result": "HLLP_CARDINALITY(outcoming)"
> > >> >
> > >> > to this...
> > >> >
> > >> > "result": "outcoming"
> > >> >
> > >> > ?
> > >> >
> > >> > On Wed, Aug 9, 2017 at 3:48 PM Casey Stella <ce...@gmail.com>
> > wrote:
> > >> >
> > >> > > Ok, so the problem here is that your profile is returning
> integers
> > >> > > (specifically HLLP cardinalities) rather than stats objects. When
> > >> you're
> > >> > > doing:
> > >> > > STATS_PERCENTILE(STATS_MERGE( PROFILE_GET('host-talks-to',
> > >> > > '99.191.183.156', PROFILE_FIXED(10, 'HOURS')), 90)
> > >> > > You are calling STATS_MERGE on a list of integers and it takes a
> > list
> > >> of
> > >> > > statistics objects.
> > >> > >
> > >> > > What you can do instead is:
> > >> > > STATS_PERCENTILE( REDUCE( PROFILE_GET('host-talks-to',
> > >> > > '99.191.183.156', PROFILE_FIXED(10, 'HOURS'), (s, x) ->
> STATS_ADD(s,
> > >> x),
> > >> > > STATS_INIT()), 90)
> > >> > >
> > >> > > Ok, that looks horrible, doesn't it? Well, thankfully we added
> > >> temporary
> > >> > > variables for stellar enrichments in 0.4.1. Let's take that
> > "numeric"
> > >> > > stellar enrichment group and reimagine it. With temporary
> > variables,
> > >> you
> > >> > > would turn:
> > >> > >
> > >> > > "numeric" : {
> > >> > > "value_red_level_out": "STATS_PERCENTILE( REDUCE(
> > >> > > PROFILE_GET('host-being-talked-to', ip_src_addr,
> PROFILE_FIXED(1,
> > >> > > 'HOURS')), (s, x) -> STATS_ADD(s, x), STATS_INIT()), 95)",
> > >> > > "value_red_level_in": "STATS_PERCENTILE( REDUCE(
> > >> > > PROFILE_GET('host-talks-to',
> > >> > > ip_src_addr, PROFILE_FIXED(1, 'HOURS')), (s, x) -> STATS_ADD(s,
> x),
> > >> > > STATS_INIT()), 95)"
> > >> > > },
> > >> > >
> > >> > > into:
> > >> > > "numeric" : [
> > >> > > "profile_duration := PROFILE_FIXED(1, 'HOURS')",
> > >> > > "host_being_talked_to := PROFILE_GET('host-being-
> > >> > talked-to',
> > >> > > ip_src_addr, profile_duration)",
> > >> > > "host_talks_to := PROFILE_GET('host-talks-to',
> > >> ip_src_addr,
> > >> > > profile_duration)",
> > >> > > "host_being_talked_to_stats := REDUCE(
> > >> host_being_talked_to,
> > >> > > (s, x) -> STATS_ADD(s, x), STATS_INIT())",
> > >> > > "host_talks_to_stats := REDUCE(host_talks_to, (s, x) ->
> > >> > > STATS_ADD(s, x), STATS_INIT())",
> > >> > > "value_red_level_out": "STATS_PERCENTILE(
> > >> > > host_being_talked_to_stats, 95)",
> > >> > > "value_red_level_in": "STATS_PERCENTILE(
> > >> > host_talks_to_stats,
> > >> > > 95)",
> > >> > > "profile_duration := null",
> > >> > > "host_being_talked_to := null",
> > >> > > "host_talks_to := null",
> > >> > > "host_being_talked_to_stats := null",
> > >> > > host_talks_to_stats := null"
> > >> > > ],
> > >> > >
> > >> > > That's a lot more to type, but it allows you to reuse and take
> the
> > >> pieces
> > >> > > in chunks.
> > >> > >
> > >> > > Ok, so now I find myself thinking "a pox on both your houses"
> since
> > >> both
> > >> > > examples now kinda look long and convoluted. So, why are they?
> > Well,
> > >> > that
> > >> > > REDUCE is likely the culprit. It's supposed to get us out of bad
> > >> > > situations not show up in what could be argued is the 80% case.
> How
> > >> > about,
> > >> > > instead, we allow STATS_ADD or STATS_INIT to take a list of
> > >> numbers? If
> > >> > > so, we could pretty easily make that nicer:
> > >> > > STATS_PERCENTILE( STATS_ADD( PROFILE_GET('host-being-
> > talked-to',
> > >> > > ip_src_addr, PROFILE_FIXED(1, 'HOURS'))), 95)
> > >> > >
> > >> > > or
> > >> > > STATS_PERCENTILE( STATS_INIT(
> > >> PROFILE_GET('host-being-talked-to',
> > >> > > ip_src_addr, PROFILE_FIXED(1, 'HOURS'))), 95)
> > >> > >
> > >> > >
> > >> > > We should make some sort of candy like that so we can avoid some
> of
> > >> the
> > >> > > complexity in the normal case.
> > >> > >
> > >> > > On Wed, Aug 9, 2017 at 3:03 PM, Dima Kovalyov <
> > >> Dima.Kovalyov@sstech.us>
> > >> > > wrote:
> > >> > >
> > >> > > > Hello Metron Team,
> > >> > > >
> > >> > > > I have created following profiler:
> > >> > > > > {
> > >> > > > > "profile": "host-talks-to",
> > >> > > > > "onlyif": "exists(source_ip)",
> > >> > > > > "foreach": "source_ip",
> > >> > > > > "init": {
> > >> > > > > "outcoming": "HLLP_INIT(5, 6)"
> > >> > > > > },
> > >> > > > > "update": { "outcoming": "HLLP_ADD(outcoming,
> destination_ip)"
> > >> },
> > >> > > > > "result": "HLLP_CARDINALITY(outcoming)"
> > >> > > > > }
> > >> > > > I have also created enrichment rule:
> > >> > > > > {
> > >> > > > > "enrichment" : {
> > >> > > > > "fieldMap": {
> > >> > > > > "stellar" : {
> > >> > > > > "config" : {
> > >> > > > > "numeric" : {
> > >> > > > > "value_red_level_out": "STATS_PERCENTILE(
> > STATS_MERGE(
> > >> > > > > PROFILE_GET('host-being-talked-to', ip_src_addr, 1,
> 'HOURS')),
> > >> 95)",
> > >> > > > > "value_red_level_in": "STATS_PERCENTILE(
> > STATS_MERGE(
> > >> > > > > PROFILE_GET('host-talks-to', ip_src_addr, 1, 'HOURS')), 95)"
> > >> > > > > },
> > >> > > > > "text" : {
> > >> > > > > "is_alert": "true"
> > >> > > > > }
> > >> > > > > }
> > >> > > > > }
> > >> > > > > }
> > >> > > > > } }
> > >> > > > However when I stream data to it I receive:
> "value_red_level_out":
> > >> > null,
> > >> > > >
> > >> > > > I have checked in profiler client and here is what I got:
> > >> > > > > [Stellar]>>> PROFILE_GET( "host-talks-to" , "99.191.183.156",
> > >> > > > > PROFILE_FIXED(300, "MINUTES"))
> > >> > > > > [1, 6, 6, 6, 6, 6, 3, 4, 5, 6, 4, 6, 6, 6, 1, 1, 6, 6, 1, 4,
> 1,
> > >> 1, 4,
> > >> > > > > 6, 6, 1, 6, 6, 1, 2, 6, 1, 1, 1, 6, 4, 6, 6, 3, 1, 6, 2, 1,
> 6,
> > 1,
> > >> 6]
> > >> > > > > [Stellar]>>> STATS_PERCENTILE(STATS_MERGE(
> > >> > > > > PROFILE_GET('host-talks-to', '99.191.183.156',
> PROFILE_FIXED(10,
> > >> > > > > 'HOURS'))), 90)
> > >> > > > > NaN
> > >> > > > > [Stellar]>>> STATS_MERGE( PROFILE_GET('host-talks-to',
> > >> > > > > '99.191.183.156', PROFILE_FIXED(10, 'HOURS')))
> > >> > > > So the STATS_MERGE produces no results. Is this something
> expected
> > >> or I
> > >> > > > made a mistake somewhere? Please advise.
> > >> > > >
> > >> > > >
> > >> > > > p.s. I am following this use cases:
> > >> > > >
> > >> > > https://github.com/hortonworks-gallery/metron-
> > >> > rules/tree/master/use-cases/
> > >> > > > DegreeOfHost
> > >> > > > There were number of errors in the configs originally, which I
> > have
> > >> > > > corrected, maybe I missed something else.
> > >> > > >
> > >> > > > - Dima
> > >> > > >
> > >> > > >
> > >> > > >
> > >> > > >
> > >> > > >
> > >> > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> > >
> >
>
>
Re: Profiler statistics NaN
Posted by Otto Fowler <ot...@gmail.com>.
What we need, is a way to package up some ‘recipes’ for stellar.
If many people are going to do this operation, then a more friendly set of
facade functions,
or some thing would work.
On August 9, 2017 at 16:38:48, Casey Stella (cestella@gmail.com) wrote:
Yeah, I'm leaning toward STATS_ADD or STATS_INIT taking a list of numbers.
STATS_MERGE seems confusing.
On Wed, Aug 9, 2017 at 4:37 PM, Nick Allen <ni...@nickallen.org> wrote:
> Or even change the behavior of STATS_MERGE, too? If STATS_MERGE gets raw
> numbers, it wraps those in a Stats object, then returns it. Then Dima's
> example would just work as-is.
>
> I'm not sure I like that though. Maybe so flexible as to be confusing?
> Thought I would throw it out as an alternative to consider.
>
>
>
>
> On Wed, Aug 9, 2017 at 4:31 PM Nick Allen <ni...@nickallen.org> wrote:
>
> > Oh yeah, duh. Now I'm with you. That would be a good quick hit.
> >
> > The current behavior is a little nutty. If there is a list, it only
> > consumes the first element in the list. I'd expect that it should
either
> > do what you describe or complain that it doesn't know how to handle a
> > list. Easy fix though.
> >
> > [Stellar]>>> STATS_MEAN(STATS_ADD(null, 1, 2, 3))
> > 2.0
> >
> > [Stellar]>>> STATS_MEAN(STATS_ADD(null, [1,2,3]))
> > 1.0
> >
> > [Stellar]>>> STATS_COUNT(STATS_ADD(null, [1,2,3]))
> > 1.0
> >
> > On Wed, Aug 9, 2017 at 4:17 PM Casey Stella <ce...@gmail.com> wrote:
> >
> >> outcoming is still a HLLP object, not a statistics object, so doing a
> >> STATS_MERGE on a bunch of them wouldn't work either.
> >>
> >> On Wed, Aug 9, 2017 at 4:15 PM, Nick Allen <ni...@nickallen.org> wrote:
> >>
> >> > That is another problem. Isn't the simplest answer, to just change
> >> this...
> >> >
> >> > "result": "HLLP_CARDINALITY(outcoming)"
> >> >
> >> > to this...
> >> >
> >> > "result": "outcoming"
> >> >
> >> > ?
> >> >
> >> > On Wed, Aug 9, 2017 at 3:48 PM Casey Stella <ce...@gmail.com>
> wrote:
> >> >
> >> > > Ok, so the problem here is that your profile is returning integers
> >> > > (specifically HLLP cardinalities) rather than stats objects. When
> >> you're
> >> > > doing:
> >> > > STATS_PERCENTILE(STATS_MERGE( PROFILE_GET('host-talks-to',
> >> > > '99.191.183.156', PROFILE_FIXED(10, 'HOURS')), 90)
> >> > > You are calling STATS_MERGE on a list of integers and it takes a
> list
> >> of
> >> > > statistics objects.
> >> > >
> >> > > What you can do instead is:
> >> > > STATS_PERCENTILE( REDUCE( PROFILE_GET('host-talks-to',
> >> > > '99.191.183.156', PROFILE_FIXED(10, 'HOURS'), (s, x) ->
STATS_ADD(s,
> >> x),
> >> > > STATS_INIT()), 90)
> >> > >
> >> > > Ok, that looks horrible, doesn't it? Well, thankfully we added
> >> temporary
> >> > > variables for stellar enrichments in 0.4.1. Let's take that
> "numeric"
> >> > > stellar enrichment group and reimagine it. With temporary
> variables,
> >> you
> >> > > would turn:
> >> > >
> >> > > "numeric" : {
> >> > > "value_red_level_out": "STATS_PERCENTILE( REDUCE(
> >> > > PROFILE_GET('host-being-talked-to', ip_src_addr, PROFILE_FIXED(1,
> >> > > 'HOURS')), (s, x) -> STATS_ADD(s, x), STATS_INIT()), 95)",
> >> > > "value_red_level_in": "STATS_PERCENTILE( REDUCE(
> >> > > PROFILE_GET('host-talks-to',
> >> > > ip_src_addr, PROFILE_FIXED(1, 'HOURS')), (s, x) -> STATS_ADD(s,
x),
> >> > > STATS_INIT()), 95)"
> >> > > },
> >> > >
> >> > > into:
> >> > > "numeric" : [
> >> > > "profile_duration := PROFILE_FIXED(1, 'HOURS')",
> >> > > "host_being_talked_to := PROFILE_GET('host-being-
> >> > talked-to',
> >> > > ip_src_addr, profile_duration)",
> >> > > "host_talks_to := PROFILE_GET('host-talks-to',
> >> ip_src_addr,
> >> > > profile_duration)",
> >> > > "host_being_talked_to_stats := REDUCE(
> >> host_being_talked_to,
> >> > > (s, x) -> STATS_ADD(s, x), STATS_INIT())",
> >> > > "host_talks_to_stats := REDUCE(host_talks_to, (s, x) ->
> >> > > STATS_ADD(s, x), STATS_INIT())",
> >> > > "value_red_level_out": "STATS_PERCENTILE(
> >> > > host_being_talked_to_stats, 95)",
> >> > > "value_red_level_in": "STATS_PERCENTILE(
> >> > host_talks_to_stats,
> >> > > 95)",
> >> > > "profile_duration := null",
> >> > > "host_being_talked_to := null",
> >> > > "host_talks_to := null",
> >> > > "host_being_talked_to_stats := null",
> >> > > host_talks_to_stats := null"
> >> > > ],
> >> > >
> >> > > That's a lot more to type, but it allows you to reuse and take the
> >> pieces
> >> > > in chunks.
> >> > >
> >> > > Ok, so now I find myself thinking "a pox on both your houses"
since
> >> both
> >> > > examples now kinda look long and convoluted. So, why are they?
> Well,
> >> > that
> >> > > REDUCE is likely the culprit. It's supposed to get us out of bad
> >> > > situations not show up in what could be argued is the 80% case.
How
> >> > about,
> >> > > instead, we allow STATS_ADD or STATS_INIT to take a list of
> >> numbers? If
> >> > > so, we could pretty easily make that nicer:
> >> > > STATS_PERCENTILE( STATS_ADD( PROFILE_GET('host-being-
> talked-to',
> >> > > ip_src_addr, PROFILE_FIXED(1, 'HOURS'))), 95)
> >> > >
> >> > > or
> >> > > STATS_PERCENTILE( STATS_INIT(
> >> PROFILE_GET('host-being-talked-to',
> >> > > ip_src_addr, PROFILE_FIXED(1, 'HOURS'))), 95)
> >> > >
> >> > >
> >> > > We should make some sort of candy like that so we can avoid some
of
> >> the
> >> > > complexity in the normal case.
> >> > >
> >> > > On Wed, Aug 9, 2017 at 3:03 PM, Dima Kovalyov <
> >> Dima.Kovalyov@sstech.us>
> >> > > wrote:
> >> > >
> >> > > > Hello Metron Team,
> >> > > >
> >> > > > I have created following profiler:
> >> > > > > {
> >> > > > > "profile": "host-talks-to",
> >> > > > > "onlyif": "exists(source_ip)",
> >> > > > > "foreach": "source_ip",
> >> > > > > "init": {
> >> > > > > "outcoming": "HLLP_INIT(5, 6)"
> >> > > > > },
> >> > > > > "update": { "outcoming": "HLLP_ADD(outcoming, destination_ip)"
> >> },
> >> > > > > "result": "HLLP_CARDINALITY(outcoming)"
> >> > > > > }
> >> > > > I have also created enrichment rule:
> >> > > > > {
> >> > > > > "enrichment" : {
> >> > > > > "fieldMap": {
> >> > > > > "stellar" : {
> >> > > > > "config" : {
> >> > > > > "numeric" : {
> >> > > > > "value_red_level_out": "STATS_PERCENTILE(
> STATS_MERGE(
> >> > > > > PROFILE_GET('host-being-talked-to', ip_src_addr, 1, 'HOURS')),
> >> 95)",
> >> > > > > "value_red_level_in": "STATS_PERCENTILE(
> STATS_MERGE(
> >> > > > > PROFILE_GET('host-talks-to', ip_src_addr, 1, 'HOURS')), 95)"
> >> > > > > },
> >> > > > > "text" : {
> >> > > > > "is_alert": "true"
> >> > > > > }
> >> > > > > }
> >> > > > > }
> >> > > > > }
> >> > > > > } }
> >> > > > However when I stream data to it I receive:
"value_red_level_out":
> >> > null,
> >> > > >
> >> > > > I have checked in profiler client and here is what I got:
> >> > > > > [Stellar]>>> PROFILE_GET( "host-talks-to" , "99.191.183.156",
> >> > > > > PROFILE_FIXED(300, "MINUTES"))
> >> > > > > [1, 6, 6, 6, 6, 6, 3, 4, 5, 6, 4, 6, 6, 6, 1, 1, 6, 6, 1, 4,
1,
> >> 1, 4,
> >> > > > > 6, 6, 1, 6, 6, 1, 2, 6, 1, 1, 1, 6, 4, 6, 6, 3, 1, 6, 2, 1, 6,
> 1,
> >> 6]
> >> > > > > [Stellar]>>> STATS_PERCENTILE(STATS_MERGE(
> >> > > > > PROFILE_GET('host-talks-to', '99.191.183.156',
PROFILE_FIXED(10,
> >> > > > > 'HOURS'))), 90)
> >> > > > > NaN
> >> > > > > [Stellar]>>> STATS_MERGE( PROFILE_GET('host-talks-to',
> >> > > > > '99.191.183.156', PROFILE_FIXED(10, 'HOURS')))
> >> > > > So the STATS_MERGE produces no results. Is this something
expected
> >> or I
> >> > > > made a mistake somewhere? Please advise.
> >> > > >
> >> > > >
> >> > > > p.s. I am following this use cases:
> >> > > >
> >> > > https://github.com/hortonworks-gallery/metron-
> >> > rules/tree/master/use-cases/
> >> > > > DegreeOfHost
> >> > > > There were number of errors in the configs originally, which I
> have
> >> > > > corrected, maybe I missed something else.
> >> > > >
> >> > > > - Dima
> >> > > >
> >> > > >
> >> > > >
> >> > > >
> >> > > >
> >> > > >
> >> > > >
> >> > >
> >> >
> >>
> >
>
Re: Profiler statistics NaN
Posted by Casey Stella <ce...@gmail.com>.
Yeah, I'm leaning toward STATS_ADD or STATS_INIT taking a list of numbers.
STATS_MERGE seems confusing.
On Wed, Aug 9, 2017 at 4:37 PM, Nick Allen <ni...@nickallen.org> wrote:
> Or even change the behavior of STATS_MERGE, too? If STATS_MERGE gets raw
> numbers, it wraps those in a Stats object, then returns it. Then Dima's
> example would just work as-is.
>
> I'm not sure I like that though. Maybe so flexible as to be confusing?
> Thought I would throw it out as an alternative to consider.
>
>
>
>
> On Wed, Aug 9, 2017 at 4:31 PM Nick Allen <ni...@nickallen.org> wrote:
>
> > Oh yeah, duh. Now I'm with you. That would be a good quick hit.
> >
> > The current behavior is a little nutty. If there is a list, it only
> > consumes the first element in the list. I'd expect that it should either
> > do what you describe or complain that it doesn't know how to handle a
> > list. Easy fix though.
> >
> > [Stellar]>>> STATS_MEAN(STATS_ADD(null, 1, 2, 3))
> > 2.0
> >
> > [Stellar]>>> STATS_MEAN(STATS_ADD(null, [1,2,3]))
> > 1.0
> >
> > [Stellar]>>> STATS_COUNT(STATS_ADD(null, [1,2,3]))
> > 1.0
> >
> > On Wed, Aug 9, 2017 at 4:17 PM Casey Stella <ce...@gmail.com> wrote:
> >
> >> outcoming is still a HLLP object, not a statistics object, so doing a
> >> STATS_MERGE on a bunch of them wouldn't work either.
> >>
> >> On Wed, Aug 9, 2017 at 4:15 PM, Nick Allen <ni...@nickallen.org> wrote:
> >>
> >> > That is another problem. Isn't the simplest answer, to just change
> >> this...
> >> >
> >> > "result": "HLLP_CARDINALITY(outcoming)"
> >> >
> >> > to this...
> >> >
> >> > "result": "outcoming"
> >> >
> >> > ?
> >> >
> >> > On Wed, Aug 9, 2017 at 3:48 PM Casey Stella <ce...@gmail.com>
> wrote:
> >> >
> >> > > Ok, so the problem here is that your profile is returning integers
> >> > > (specifically HLLP cardinalities) rather than stats objects. When
> >> you're
> >> > > doing:
> >> > > STATS_PERCENTILE(STATS_MERGE( PROFILE_GET('host-talks-to',
> >> > > '99.191.183.156', PROFILE_FIXED(10, 'HOURS')), 90)
> >> > > You are calling STATS_MERGE on a list of integers and it takes a
> list
> >> of
> >> > > statistics objects.
> >> > >
> >> > > What you can do instead is:
> >> > > STATS_PERCENTILE( REDUCE( PROFILE_GET('host-talks-to',
> >> > > '99.191.183.156', PROFILE_FIXED(10, 'HOURS'), (s, x) -> STATS_ADD(s,
> >> x),
> >> > > STATS_INIT()), 90)
> >> > >
> >> > > Ok, that looks horrible, doesn't it? Well, thankfully we added
> >> temporary
> >> > > variables for stellar enrichments in 0.4.1. Let's take that
> "numeric"
> >> > > stellar enrichment group and reimagine it. With temporary
> variables,
> >> you
> >> > > would turn:
> >> > >
> >> > > "numeric" : {
> >> > > "value_red_level_out": "STATS_PERCENTILE( REDUCE(
> >> > > PROFILE_GET('host-being-talked-to', ip_src_addr, PROFILE_FIXED(1,
> >> > > 'HOURS')), (s, x) -> STATS_ADD(s, x), STATS_INIT()), 95)",
> >> > > "value_red_level_in": "STATS_PERCENTILE( REDUCE(
> >> > > PROFILE_GET('host-talks-to',
> >> > > ip_src_addr, PROFILE_FIXED(1, 'HOURS')), (s, x) -> STATS_ADD(s, x),
> >> > > STATS_INIT()), 95)"
> >> > > },
> >> > >
> >> > > into:
> >> > > "numeric" : [
> >> > > "profile_duration := PROFILE_FIXED(1, 'HOURS')",
> >> > > "host_being_talked_to := PROFILE_GET('host-being-
> >> > talked-to',
> >> > > ip_src_addr, profile_duration)",
> >> > > "host_talks_to := PROFILE_GET('host-talks-to',
> >> ip_src_addr,
> >> > > profile_duration)",
> >> > > "host_being_talked_to_stats := REDUCE(
> >> host_being_talked_to,
> >> > > (s, x) -> STATS_ADD(s, x), STATS_INIT())",
> >> > > "host_talks_to_stats := REDUCE(host_talks_to, (s, x) ->
> >> > > STATS_ADD(s, x), STATS_INIT())",
> >> > > "value_red_level_out": "STATS_PERCENTILE(
> >> > > host_being_talked_to_stats, 95)",
> >> > > "value_red_level_in": "STATS_PERCENTILE(
> >> > host_talks_to_stats,
> >> > > 95)",
> >> > > "profile_duration := null",
> >> > > "host_being_talked_to := null",
> >> > > "host_talks_to := null",
> >> > > "host_being_talked_to_stats := null",
> >> > > host_talks_to_stats := null"
> >> > > ],
> >> > >
> >> > > That's a lot more to type, but it allows you to reuse and take the
> >> pieces
> >> > > in chunks.
> >> > >
> >> > > Ok, so now I find myself thinking "a pox on both your houses" since
> >> both
> >> > > examples now kinda look long and convoluted. So, why are they?
> Well,
> >> > that
> >> > > REDUCE is likely the culprit. It's supposed to get us out of bad
> >> > > situations not show up in what could be argued is the 80% case. How
> >> > about,
> >> > > instead, we allow STATS_ADD or STATS_INIT to take a list of
> >> numbers? If
> >> > > so, we could pretty easily make that nicer:
> >> > > STATS_PERCENTILE( STATS_ADD( PROFILE_GET('host-being-
> talked-to',
> >> > > ip_src_addr, PROFILE_FIXED(1, 'HOURS'))), 95)
> >> > >
> >> > > or
> >> > > STATS_PERCENTILE( STATS_INIT(
> >> PROFILE_GET('host-being-talked-to',
> >> > > ip_src_addr, PROFILE_FIXED(1, 'HOURS'))), 95)
> >> > >
> >> > >
> >> > > We should make some sort of candy like that so we can avoid some of
> >> the
> >> > > complexity in the normal case.
> >> > >
> >> > > On Wed, Aug 9, 2017 at 3:03 PM, Dima Kovalyov <
> >> Dima.Kovalyov@sstech.us>
> >> > > wrote:
> >> > >
> >> > > > Hello Metron Team,
> >> > > >
> >> > > > I have created following profiler:
> >> > > > > {
> >> > > > > "profile": "host-talks-to",
> >> > > > > "onlyif": "exists(source_ip)",
> >> > > > > "foreach": "source_ip",
> >> > > > > "init": {
> >> > > > > "outcoming": "HLLP_INIT(5, 6)"
> >> > > > > },
> >> > > > > "update": { "outcoming": "HLLP_ADD(outcoming, destination_ip)"
> >> },
> >> > > > > "result": "HLLP_CARDINALITY(outcoming)"
> >> > > > > }
> >> > > > I have also created enrichment rule:
> >> > > > > {
> >> > > > > "enrichment" : {
> >> > > > > "fieldMap": {
> >> > > > > "stellar" : {
> >> > > > > "config" : {
> >> > > > > "numeric" : {
> >> > > > > "value_red_level_out": "STATS_PERCENTILE(
> STATS_MERGE(
> >> > > > > PROFILE_GET('host-being-talked-to', ip_src_addr, 1, 'HOURS')),
> >> 95)",
> >> > > > > "value_red_level_in": "STATS_PERCENTILE(
> STATS_MERGE(
> >> > > > > PROFILE_GET('host-talks-to', ip_src_addr, 1, 'HOURS')), 95)"
> >> > > > > },
> >> > > > > "text" : {
> >> > > > > "is_alert": "true"
> >> > > > > }
> >> > > > > }
> >> > > > > }
> >> > > > > }
> >> > > > > } }
> >> > > > However when I stream data to it I receive: "value_red_level_out":
> >> > null,
> >> > > >
> >> > > > I have checked in profiler client and here is what I got:
> >> > > > > [Stellar]>>> PROFILE_GET( "host-talks-to" , "99.191.183.156",
> >> > > > > PROFILE_FIXED(300, "MINUTES"))
> >> > > > > [1, 6, 6, 6, 6, 6, 3, 4, 5, 6, 4, 6, 6, 6, 1, 1, 6, 6, 1, 4, 1,
> >> 1, 4,
> >> > > > > 6, 6, 1, 6, 6, 1, 2, 6, 1, 1, 1, 6, 4, 6, 6, 3, 1, 6, 2, 1, 6,
> 1,
> >> 6]
> >> > > > > [Stellar]>>> STATS_PERCENTILE(STATS_MERGE(
> >> > > > > PROFILE_GET('host-talks-to', '99.191.183.156', PROFILE_FIXED(10,
> >> > > > > 'HOURS'))), 90)
> >> > > > > NaN
> >> > > > > [Stellar]>>> STATS_MERGE( PROFILE_GET('host-talks-to',
> >> > > > > '99.191.183.156', PROFILE_FIXED(10, 'HOURS')))
> >> > > > So the STATS_MERGE produces no results. Is this something expected
> >> or I
> >> > > > made a mistake somewhere? Please advise.
> >> > > >
> >> > > >
> >> > > > p.s. I am following this use cases:
> >> > > >
> >> > > https://github.com/hortonworks-gallery/metron-
> >> > rules/tree/master/use-cases/
> >> > > > DegreeOfHost
> >> > > > There were number of errors in the configs originally, which I
> have
> >> > > > corrected, maybe I missed something else.
> >> > > >
> >> > > > - Dima
> >> > > >
> >> > > >
> >> > > >
> >> > > >
> >> > > >
> >> > > >
> >> > > >
> >> > >
> >> >
> >>
> >
>
Re: Profiler statistics NaN
Posted by Nick Allen <ni...@nickallen.org>.
Or even change the behavior of STATS_MERGE, too? If STATS_MERGE gets raw
numbers, it wraps those in a Stats object, then returns it. Then Dima's
example would just work as-is.
I'm not sure I like that though. Maybe so flexible as to be confusing?
Thought I would throw it out as an alternative to consider.
On Wed, Aug 9, 2017 at 4:31 PM Nick Allen <ni...@nickallen.org> wrote:
> Oh yeah, duh. Now I'm with you. That would be a good quick hit.
>
> The current behavior is a little nutty. If there is a list, it only
> consumes the first element in the list. I'd expect that it should either
> do what you describe or complain that it doesn't know how to handle a
> list. Easy fix though.
>
> [Stellar]>>> STATS_MEAN(STATS_ADD(null, 1, 2, 3))
> 2.0
>
> [Stellar]>>> STATS_MEAN(STATS_ADD(null, [1,2,3]))
> 1.0
>
> [Stellar]>>> STATS_COUNT(STATS_ADD(null, [1,2,3]))
> 1.0
>
> On Wed, Aug 9, 2017 at 4:17 PM Casey Stella <ce...@gmail.com> wrote:
>
>> outcoming is still a HLLP object, not a statistics object, so doing a
>> STATS_MERGE on a bunch of them wouldn't work either.
>>
>> On Wed, Aug 9, 2017 at 4:15 PM, Nick Allen <ni...@nickallen.org> wrote:
>>
>> > That is another problem. Isn't the simplest answer, to just change
>> this...
>> >
>> > "result": "HLLP_CARDINALITY(outcoming)"
>> >
>> > to this...
>> >
>> > "result": "outcoming"
>> >
>> > ?
>> >
>> > On Wed, Aug 9, 2017 at 3:48 PM Casey Stella <ce...@gmail.com> wrote:
>> >
>> > > Ok, so the problem here is that your profile is returning integers
>> > > (specifically HLLP cardinalities) rather than stats objects. When
>> you're
>> > > doing:
>> > > STATS_PERCENTILE(STATS_MERGE( PROFILE_GET('host-talks-to',
>> > > '99.191.183.156', PROFILE_FIXED(10, 'HOURS')), 90)
>> > > You are calling STATS_MERGE on a list of integers and it takes a list
>> of
>> > > statistics objects.
>> > >
>> > > What you can do instead is:
>> > > STATS_PERCENTILE( REDUCE( PROFILE_GET('host-talks-to',
>> > > '99.191.183.156', PROFILE_FIXED(10, 'HOURS'), (s, x) -> STATS_ADD(s,
>> x),
>> > > STATS_INIT()), 90)
>> > >
>> > > Ok, that looks horrible, doesn't it? Well, thankfully we added
>> temporary
>> > > variables for stellar enrichments in 0.4.1. Let's take that "numeric"
>> > > stellar enrichment group and reimagine it. With temporary variables,
>> you
>> > > would turn:
>> > >
>> > > "numeric" : {
>> > > "value_red_level_out": "STATS_PERCENTILE( REDUCE(
>> > > PROFILE_GET('host-being-talked-to', ip_src_addr, PROFILE_FIXED(1,
>> > > 'HOURS')), (s, x) -> STATS_ADD(s, x), STATS_INIT()), 95)",
>> > > "value_red_level_in": "STATS_PERCENTILE( REDUCE(
>> > > PROFILE_GET('host-talks-to',
>> > > ip_src_addr, PROFILE_FIXED(1, 'HOURS')), (s, x) -> STATS_ADD(s, x),
>> > > STATS_INIT()), 95)"
>> > > },
>> > >
>> > > into:
>> > > "numeric" : [
>> > > "profile_duration := PROFILE_FIXED(1, 'HOURS')",
>> > > "host_being_talked_to := PROFILE_GET('host-being-
>> > talked-to',
>> > > ip_src_addr, profile_duration)",
>> > > "host_talks_to := PROFILE_GET('host-talks-to',
>> ip_src_addr,
>> > > profile_duration)",
>> > > "host_being_talked_to_stats := REDUCE(
>> host_being_talked_to,
>> > > (s, x) -> STATS_ADD(s, x), STATS_INIT())",
>> > > "host_talks_to_stats := REDUCE(host_talks_to, (s, x) ->
>> > > STATS_ADD(s, x), STATS_INIT())",
>> > > "value_red_level_out": "STATS_PERCENTILE(
>> > > host_being_talked_to_stats, 95)",
>> > > "value_red_level_in": "STATS_PERCENTILE(
>> > host_talks_to_stats,
>> > > 95)",
>> > > "profile_duration := null",
>> > > "host_being_talked_to := null",
>> > > "host_talks_to := null",
>> > > "host_being_talked_to_stats := null",
>> > > host_talks_to_stats := null"
>> > > ],
>> > >
>> > > That's a lot more to type, but it allows you to reuse and take the
>> pieces
>> > > in chunks.
>> > >
>> > > Ok, so now I find myself thinking "a pox on both your houses" since
>> both
>> > > examples now kinda look long and convoluted. So, why are they? Well,
>> > that
>> > > REDUCE is likely the culprit. It's supposed to get us out of bad
>> > > situations not show up in what could be argued is the 80% case. How
>> > about,
>> > > instead, we allow STATS_ADD or STATS_INIT to take a list of
>> numbers? If
>> > > so, we could pretty easily make that nicer:
>> > > STATS_PERCENTILE( STATS_ADD( PROFILE_GET('host-being-talked-to',
>> > > ip_src_addr, PROFILE_FIXED(1, 'HOURS'))), 95)
>> > >
>> > > or
>> > > STATS_PERCENTILE( STATS_INIT(
>> PROFILE_GET('host-being-talked-to',
>> > > ip_src_addr, PROFILE_FIXED(1, 'HOURS'))), 95)
>> > >
>> > >
>> > > We should make some sort of candy like that so we can avoid some of
>> the
>> > > complexity in the normal case.
>> > >
>> > > On Wed, Aug 9, 2017 at 3:03 PM, Dima Kovalyov <
>> Dima.Kovalyov@sstech.us>
>> > > wrote:
>> > >
>> > > > Hello Metron Team,
>> > > >
>> > > > I have created following profiler:
>> > > > > {
>> > > > > "profile": "host-talks-to",
>> > > > > "onlyif": "exists(source_ip)",
>> > > > > "foreach": "source_ip",
>> > > > > "init": {
>> > > > > "outcoming": "HLLP_INIT(5, 6)"
>> > > > > },
>> > > > > "update": { "outcoming": "HLLP_ADD(outcoming, destination_ip)"
>> },
>> > > > > "result": "HLLP_CARDINALITY(outcoming)"
>> > > > > }
>> > > > I have also created enrichment rule:
>> > > > > {
>> > > > > "enrichment" : {
>> > > > > "fieldMap": {
>> > > > > "stellar" : {
>> > > > > "config" : {
>> > > > > "numeric" : {
>> > > > > "value_red_level_out": "STATS_PERCENTILE( STATS_MERGE(
>> > > > > PROFILE_GET('host-being-talked-to', ip_src_addr, 1, 'HOURS')),
>> 95)",
>> > > > > "value_red_level_in": "STATS_PERCENTILE( STATS_MERGE(
>> > > > > PROFILE_GET('host-talks-to', ip_src_addr, 1, 'HOURS')), 95)"
>> > > > > },
>> > > > > "text" : {
>> > > > > "is_alert": "true"
>> > > > > }
>> > > > > }
>> > > > > }
>> > > > > }
>> > > > > } }
>> > > > However when I stream data to it I receive: "value_red_level_out":
>> > null,
>> > > >
>> > > > I have checked in profiler client and here is what I got:
>> > > > > [Stellar]>>> PROFILE_GET( "host-talks-to" , "99.191.183.156",
>> > > > > PROFILE_FIXED(300, "MINUTES"))
>> > > > > [1, 6, 6, 6, 6, 6, 3, 4, 5, 6, 4, 6, 6, 6, 1, 1, 6, 6, 1, 4, 1,
>> 1, 4,
>> > > > > 6, 6, 1, 6, 6, 1, 2, 6, 1, 1, 1, 6, 4, 6, 6, 3, 1, 6, 2, 1, 6, 1,
>> 6]
>> > > > > [Stellar]>>> STATS_PERCENTILE(STATS_MERGE(
>> > > > > PROFILE_GET('host-talks-to', '99.191.183.156', PROFILE_FIXED(10,
>> > > > > 'HOURS'))), 90)
>> > > > > NaN
>> > > > > [Stellar]>>> STATS_MERGE( PROFILE_GET('host-talks-to',
>> > > > > '99.191.183.156', PROFILE_FIXED(10, 'HOURS')))
>> > > > So the STATS_MERGE produces no results. Is this something expected
>> or I
>> > > > made a mistake somewhere? Please advise.
>> > > >
>> > > >
>> > > > p.s. I am following this use cases:
>> > > >
>> > > https://github.com/hortonworks-gallery/metron-
>> > rules/tree/master/use-cases/
>> > > > DegreeOfHost
>> > > > There were number of errors in the configs originally, which I have
>> > > > corrected, maybe I missed something else.
>> > > >
>> > > > - Dima
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > >
>> >
>>
>
Re: Profiler statistics NaN
Posted by Nick Allen <ni...@nickallen.org>.
Oh yeah, duh. Now I'm with you. That would be a good quick hit.
The current behavior is a little nutty. If there is a list, it only
consumes the first element in the list. I'd expect that it should either
do what you describe or complain that it doesn't know how to handle a
list. Easy fix though.
[Stellar]>>> STATS_MEAN(STATS_ADD(null, 1, 2, 3))
2.0
[Stellar]>>> STATS_MEAN(STATS_ADD(null, [1,2,3]))
1.0
[Stellar]>>> STATS_COUNT(STATS_ADD(null, [1,2,3]))
1.0
On Wed, Aug 9, 2017 at 4:17 PM Casey Stella <ce...@gmail.com> wrote:
> outcoming is still a HLLP object, not a statistics object, so doing a
> STATS_MERGE on a bunch of them wouldn't work either.
>
> On Wed, Aug 9, 2017 at 4:15 PM, Nick Allen <ni...@nickallen.org> wrote:
>
> > That is another problem. Isn't the simplest answer, to just change
> this...
> >
> > "result": "HLLP_CARDINALITY(outcoming)"
> >
> > to this...
> >
> > "result": "outcoming"
> >
> > ?
> >
> > On Wed, Aug 9, 2017 at 3:48 PM Casey Stella <ce...@gmail.com> wrote:
> >
> > > Ok, so the problem here is that your profile is returning integers
> > > (specifically HLLP cardinalities) rather than stats objects. When
> you're
> > > doing:
> > > STATS_PERCENTILE(STATS_MERGE( PROFILE_GET('host-talks-to',
> > > '99.191.183.156', PROFILE_FIXED(10, 'HOURS')), 90)
> > > You are calling STATS_MERGE on a list of integers and it takes a list
> of
> > > statistics objects.
> > >
> > > What you can do instead is:
> > > STATS_PERCENTILE( REDUCE( PROFILE_GET('host-talks-to',
> > > '99.191.183.156', PROFILE_FIXED(10, 'HOURS'), (s, x) -> STATS_ADD(s,
> x),
> > > STATS_INIT()), 90)
> > >
> > > Ok, that looks horrible, doesn't it? Well, thankfully we added
> temporary
> > > variables for stellar enrichments in 0.4.1. Let's take that "numeric"
> > > stellar enrichment group and reimagine it. With temporary variables,
> you
> > > would turn:
> > >
> > > "numeric" : {
> > > "value_red_level_out": "STATS_PERCENTILE( REDUCE(
> > > PROFILE_GET('host-being-talked-to', ip_src_addr, PROFILE_FIXED(1,
> > > 'HOURS')), (s, x) -> STATS_ADD(s, x), STATS_INIT()), 95)",
> > > "value_red_level_in": "STATS_PERCENTILE( REDUCE(
> > > PROFILE_GET('host-talks-to',
> > > ip_src_addr, PROFILE_FIXED(1, 'HOURS')), (s, x) -> STATS_ADD(s, x),
> > > STATS_INIT()), 95)"
> > > },
> > >
> > > into:
> > > "numeric" : [
> > > "profile_duration := PROFILE_FIXED(1, 'HOURS')",
> > > "host_being_talked_to := PROFILE_GET('host-being-
> > talked-to',
> > > ip_src_addr, profile_duration)",
> > > "host_talks_to := PROFILE_GET('host-talks-to',
> ip_src_addr,
> > > profile_duration)",
> > > "host_being_talked_to_stats := REDUCE(
> host_being_talked_to,
> > > (s, x) -> STATS_ADD(s, x), STATS_INIT())",
> > > "host_talks_to_stats := REDUCE(host_talks_to, (s, x) ->
> > > STATS_ADD(s, x), STATS_INIT())",
> > > "value_red_level_out": "STATS_PERCENTILE(
> > > host_being_talked_to_stats, 95)",
> > > "value_red_level_in": "STATS_PERCENTILE(
> > host_talks_to_stats,
> > > 95)",
> > > "profile_duration := null",
> > > "host_being_talked_to := null",
> > > "host_talks_to := null",
> > > "host_being_talked_to_stats := null",
> > > host_talks_to_stats := null"
> > > ],
> > >
> > > That's a lot more to type, but it allows you to reuse and take the
> pieces
> > > in chunks.
> > >
> > > Ok, so now I find myself thinking "a pox on both your houses" since
> both
> > > examples now kinda look long and convoluted. So, why are they? Well,
> > that
> > > REDUCE is likely the culprit. It's supposed to get us out of bad
> > > situations not show up in what could be argued is the 80% case. How
> > about,
> > > instead, we allow STATS_ADD or STATS_INIT to take a list of numbers?
> If
> > > so, we could pretty easily make that nicer:
> > > STATS_PERCENTILE( STATS_ADD( PROFILE_GET('host-being-talked-to',
> > > ip_src_addr, PROFILE_FIXED(1, 'HOURS'))), 95)
> > >
> > > or
> > > STATS_PERCENTILE( STATS_INIT( PROFILE_GET('host-being-talked-to',
> > > ip_src_addr, PROFILE_FIXED(1, 'HOURS'))), 95)
> > >
> > >
> > > We should make some sort of candy like that so we can avoid some of the
> > > complexity in the normal case.
> > >
> > > On Wed, Aug 9, 2017 at 3:03 PM, Dima Kovalyov <Dima.Kovalyov@sstech.us
> >
> > > wrote:
> > >
> > > > Hello Metron Team,
> > > >
> > > > I have created following profiler:
> > > > > {
> > > > > "profile": "host-talks-to",
> > > > > "onlyif": "exists(source_ip)",
> > > > > "foreach": "source_ip",
> > > > > "init": {
> > > > > "outcoming": "HLLP_INIT(5, 6)"
> > > > > },
> > > > > "update": { "outcoming": "HLLP_ADD(outcoming, destination_ip)" },
> > > > > "result": "HLLP_CARDINALITY(outcoming)"
> > > > > }
> > > > I have also created enrichment rule:
> > > > > {
> > > > > "enrichment" : {
> > > > > "fieldMap": {
> > > > > "stellar" : {
> > > > > "config" : {
> > > > > "numeric" : {
> > > > > "value_red_level_out": "STATS_PERCENTILE( STATS_MERGE(
> > > > > PROFILE_GET('host-being-talked-to', ip_src_addr, 1, 'HOURS')),
> 95)",
> > > > > "value_red_level_in": "STATS_PERCENTILE( STATS_MERGE(
> > > > > PROFILE_GET('host-talks-to', ip_src_addr, 1, 'HOURS')), 95)"
> > > > > },
> > > > > "text" : {
> > > > > "is_alert": "true"
> > > > > }
> > > > > }
> > > > > }
> > > > > }
> > > > > } }
> > > > However when I stream data to it I receive: "value_red_level_out":
> > null,
> > > >
> > > > I have checked in profiler client and here is what I got:
> > > > > [Stellar]>>> PROFILE_GET( "host-talks-to" , "99.191.183.156",
> > > > > PROFILE_FIXED(300, "MINUTES"))
> > > > > [1, 6, 6, 6, 6, 6, 3, 4, 5, 6, 4, 6, 6, 6, 1, 1, 6, 6, 1, 4, 1, 1,
> 4,
> > > > > 6, 6, 1, 6, 6, 1, 2, 6, 1, 1, 1, 6, 4, 6, 6, 3, 1, 6, 2, 1, 6, 1,
> 6]
> > > > > [Stellar]>>> STATS_PERCENTILE(STATS_MERGE(
> > > > > PROFILE_GET('host-talks-to', '99.191.183.156', PROFILE_FIXED(10,
> > > > > 'HOURS'))), 90)
> > > > > NaN
> > > > > [Stellar]>>> STATS_MERGE( PROFILE_GET('host-talks-to',
> > > > > '99.191.183.156', PROFILE_FIXED(10, 'HOURS')))
> > > > So the STATS_MERGE produces no results. Is this something expected
> or I
> > > > made a mistake somewhere? Please advise.
> > > >
> > > >
> > > > p.s. I am following this use cases:
> > > >
> > > https://github.com/hortonworks-gallery/metron-
> > rules/tree/master/use-cases/
> > > > DegreeOfHost
> > > > There were number of errors in the configs originally, which I have
> > > > corrected, maybe I missed something else.
> > > >
> > > > - Dima
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > >
> >
>
Re: Profiler statistics NaN
Posted by Casey Stella <ce...@gmail.com>.
outcoming is still a HLLP object, not a statistics object, so doing a
STATS_MERGE on a bunch of them wouldn't work either.
On Wed, Aug 9, 2017 at 4:15 PM, Nick Allen <ni...@nickallen.org> wrote:
> That is another problem. Isn't the simplest answer, to just change this...
>
> "result": "HLLP_CARDINALITY(outcoming)"
>
> to this...
>
> "result": "outcoming"
>
> ?
>
> On Wed, Aug 9, 2017 at 3:48 PM Casey Stella <ce...@gmail.com> wrote:
>
> > Ok, so the problem here is that your profile is returning integers
> > (specifically HLLP cardinalities) rather than stats objects. When you're
> > doing:
> > STATS_PERCENTILE(STATS_MERGE( PROFILE_GET('host-talks-to',
> > '99.191.183.156', PROFILE_FIXED(10, 'HOURS')), 90)
> > You are calling STATS_MERGE on a list of integers and it takes a list of
> > statistics objects.
> >
> > What you can do instead is:
> > STATS_PERCENTILE( REDUCE( PROFILE_GET('host-talks-to',
> > '99.191.183.156', PROFILE_FIXED(10, 'HOURS'), (s, x) -> STATS_ADD(s, x),
> > STATS_INIT()), 90)
> >
> > Ok, that looks horrible, doesn't it? Well, thankfully we added temporary
> > variables for stellar enrichments in 0.4.1. Let's take that "numeric"
> > stellar enrichment group and reimagine it. With temporary variables, you
> > would turn:
> >
> > "numeric" : {
> > "value_red_level_out": "STATS_PERCENTILE( REDUCE(
> > PROFILE_GET('host-being-talked-to', ip_src_addr, PROFILE_FIXED(1,
> > 'HOURS')), (s, x) -> STATS_ADD(s, x), STATS_INIT()), 95)",
> > "value_red_level_in": "STATS_PERCENTILE( REDUCE(
> > PROFILE_GET('host-talks-to',
> > ip_src_addr, PROFILE_FIXED(1, 'HOURS')), (s, x) -> STATS_ADD(s, x),
> > STATS_INIT()), 95)"
> > },
> >
> > into:
> > "numeric" : [
> > "profile_duration := PROFILE_FIXED(1, 'HOURS')",
> > "host_being_talked_to := PROFILE_GET('host-being-
> talked-to',
> > ip_src_addr, profile_duration)",
> > "host_talks_to := PROFILE_GET('host-talks-to', ip_src_addr,
> > profile_duration)",
> > "host_being_talked_to_stats := REDUCE( host_being_talked_to,
> > (s, x) -> STATS_ADD(s, x), STATS_INIT())",
> > "host_talks_to_stats := REDUCE(host_talks_to, (s, x) ->
> > STATS_ADD(s, x), STATS_INIT())",
> > "value_red_level_out": "STATS_PERCENTILE(
> > host_being_talked_to_stats, 95)",
> > "value_red_level_in": "STATS_PERCENTILE(
> host_talks_to_stats,
> > 95)",
> > "profile_duration := null",
> > "host_being_talked_to := null",
> > "host_talks_to := null",
> > "host_being_talked_to_stats := null",
> > host_talks_to_stats := null"
> > ],
> >
> > That's a lot more to type, but it allows you to reuse and take the pieces
> > in chunks.
> >
> > Ok, so now I find myself thinking "a pox on both your houses" since both
> > examples now kinda look long and convoluted. So, why are they? Well,
> that
> > REDUCE is likely the culprit. It's supposed to get us out of bad
> > situations not show up in what could be argued is the 80% case. How
> about,
> > instead, we allow STATS_ADD or STATS_INIT to take a list of numbers? If
> > so, we could pretty easily make that nicer:
> > STATS_PERCENTILE( STATS_ADD( PROFILE_GET('host-being-talked-to',
> > ip_src_addr, PROFILE_FIXED(1, 'HOURS'))), 95)
> >
> > or
> > STATS_PERCENTILE( STATS_INIT( PROFILE_GET('host-being-talked-to',
> > ip_src_addr, PROFILE_FIXED(1, 'HOURS'))), 95)
> >
> >
> > We should make some sort of candy like that so we can avoid some of the
> > complexity in the normal case.
> >
> > On Wed, Aug 9, 2017 at 3:03 PM, Dima Kovalyov <Di...@sstech.us>
> > wrote:
> >
> > > Hello Metron Team,
> > >
> > > I have created following profiler:
> > > > {
> > > > "profile": "host-talks-to",
> > > > "onlyif": "exists(source_ip)",
> > > > "foreach": "source_ip",
> > > > "init": {
> > > > "outcoming": "HLLP_INIT(5, 6)"
> > > > },
> > > > "update": { "outcoming": "HLLP_ADD(outcoming, destination_ip)" },
> > > > "result": "HLLP_CARDINALITY(outcoming)"
> > > > }
> > > I have also created enrichment rule:
> > > > {
> > > > "enrichment" : {
> > > > "fieldMap": {
> > > > "stellar" : {
> > > > "config" : {
> > > > "numeric" : {
> > > > "value_red_level_out": "STATS_PERCENTILE( STATS_MERGE(
> > > > PROFILE_GET('host-being-talked-to', ip_src_addr, 1, 'HOURS')), 95)",
> > > > "value_red_level_in": "STATS_PERCENTILE( STATS_MERGE(
> > > > PROFILE_GET('host-talks-to', ip_src_addr, 1, 'HOURS')), 95)"
> > > > },
> > > > "text" : {
> > > > "is_alert": "true"
> > > > }
> > > > }
> > > > }
> > > > }
> > > > } }
> > > However when I stream data to it I receive: "value_red_level_out":
> null,
> > >
> > > I have checked in profiler client and here is what I got:
> > > > [Stellar]>>> PROFILE_GET( "host-talks-to" , "99.191.183.156",
> > > > PROFILE_FIXED(300, "MINUTES"))
> > > > [1, 6, 6, 6, 6, 6, 3, 4, 5, 6, 4, 6, 6, 6, 1, 1, 6, 6, 1, 4, 1, 1, 4,
> > > > 6, 6, 1, 6, 6, 1, 2, 6, 1, 1, 1, 6, 4, 6, 6, 3, 1, 6, 2, 1, 6, 1, 6]
> > > > [Stellar]>>> STATS_PERCENTILE(STATS_MERGE(
> > > > PROFILE_GET('host-talks-to', '99.191.183.156', PROFILE_FIXED(10,
> > > > 'HOURS'))), 90)
> > > > NaN
> > > > [Stellar]>>> STATS_MERGE( PROFILE_GET('host-talks-to',
> > > > '99.191.183.156', PROFILE_FIXED(10, 'HOURS')))
> > > So the STATS_MERGE produces no results. Is this something expected or I
> > > made a mistake somewhere? Please advise.
> > >
> > >
> > > p.s. I am following this use cases:
> > >
> > https://github.com/hortonworks-gallery/metron-
> rules/tree/master/use-cases/
> > > DegreeOfHost
> > > There were number of errors in the configs originally, which I have
> > > corrected, maybe I missed something else.
> > >
> > > - Dima
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> >
>
Re: Profiler statistics NaN
Posted by Nick Allen <ni...@nickallen.org>.
That is another problem. Isn't the simplest answer, to just change this...
"result": "HLLP_CARDINALITY(outcoming)"
to this...
"result": "outcoming"
?
On Wed, Aug 9, 2017 at 3:48 PM Casey Stella <ce...@gmail.com> wrote:
> Ok, so the problem here is that your profile is returning integers
> (specifically HLLP cardinalities) rather than stats objects. When you're
> doing:
> STATS_PERCENTILE(STATS_MERGE( PROFILE_GET('host-talks-to',
> '99.191.183.156', PROFILE_FIXED(10, 'HOURS')), 90)
> You are calling STATS_MERGE on a list of integers and it takes a list of
> statistics objects.
>
> What you can do instead is:
> STATS_PERCENTILE( REDUCE( PROFILE_GET('host-talks-to',
> '99.191.183.156', PROFILE_FIXED(10, 'HOURS'), (s, x) -> STATS_ADD(s, x),
> STATS_INIT()), 90)
>
> Ok, that looks horrible, doesn't it? Well, thankfully we added temporary
> variables for stellar enrichments in 0.4.1. Let's take that "numeric"
> stellar enrichment group and reimagine it. With temporary variables, you
> would turn:
>
> "numeric" : {
> "value_red_level_out": "STATS_PERCENTILE( REDUCE(
> PROFILE_GET('host-being-talked-to', ip_src_addr, PROFILE_FIXED(1,
> 'HOURS')), (s, x) -> STATS_ADD(s, x), STATS_INIT()), 95)",
> "value_red_level_in": "STATS_PERCENTILE( REDUCE(
> PROFILE_GET('host-talks-to',
> ip_src_addr, PROFILE_FIXED(1, 'HOURS')), (s, x) -> STATS_ADD(s, x),
> STATS_INIT()), 95)"
> },
>
> into:
> "numeric" : [
> "profile_duration := PROFILE_FIXED(1, 'HOURS')",
> "host_being_talked_to := PROFILE_GET('host-being-talked-to',
> ip_src_addr, profile_duration)",
> "host_talks_to := PROFILE_GET('host-talks-to', ip_src_addr,
> profile_duration)",
> "host_being_talked_to_stats := REDUCE( host_being_talked_to,
> (s, x) -> STATS_ADD(s, x), STATS_INIT())",
> "host_talks_to_stats := REDUCE(host_talks_to, (s, x) ->
> STATS_ADD(s, x), STATS_INIT())",
> "value_red_level_out": "STATS_PERCENTILE(
> host_being_talked_to_stats, 95)",
> "value_red_level_in": "STATS_PERCENTILE( host_talks_to_stats,
> 95)",
> "profile_duration := null",
> "host_being_talked_to := null",
> "host_talks_to := null",
> "host_being_talked_to_stats := null",
> host_talks_to_stats := null"
> ],
>
> That's a lot more to type, but it allows you to reuse and take the pieces
> in chunks.
>
> Ok, so now I find myself thinking "a pox on both your houses" since both
> examples now kinda look long and convoluted. So, why are they? Well, that
> REDUCE is likely the culprit. It's supposed to get us out of bad
> situations not show up in what could be argued is the 80% case. How about,
> instead, we allow STATS_ADD or STATS_INIT to take a list of numbers? If
> so, we could pretty easily make that nicer:
> STATS_PERCENTILE( STATS_ADD( PROFILE_GET('host-being-talked-to',
> ip_src_addr, PROFILE_FIXED(1, 'HOURS'))), 95)
>
> or
> STATS_PERCENTILE( STATS_INIT( PROFILE_GET('host-being-talked-to',
> ip_src_addr, PROFILE_FIXED(1, 'HOURS'))), 95)
>
>
> We should make some sort of candy like that so we can avoid some of the
> complexity in the normal case.
>
> On Wed, Aug 9, 2017 at 3:03 PM, Dima Kovalyov <Di...@sstech.us>
> wrote:
>
> > Hello Metron Team,
> >
> > I have created following profiler:
> > > {
> > > "profile": "host-talks-to",
> > > "onlyif": "exists(source_ip)",
> > > "foreach": "source_ip",
> > > "init": {
> > > "outcoming": "HLLP_INIT(5, 6)"
> > > },
> > > "update": { "outcoming": "HLLP_ADD(outcoming, destination_ip)" },
> > > "result": "HLLP_CARDINALITY(outcoming)"
> > > }
> > I have also created enrichment rule:
> > > {
> > > "enrichment" : {
> > > "fieldMap": {
> > > "stellar" : {
> > > "config" : {
> > > "numeric" : {
> > > "value_red_level_out": "STATS_PERCENTILE( STATS_MERGE(
> > > PROFILE_GET('host-being-talked-to', ip_src_addr, 1, 'HOURS')), 95)",
> > > "value_red_level_in": "STATS_PERCENTILE( STATS_MERGE(
> > > PROFILE_GET('host-talks-to', ip_src_addr, 1, 'HOURS')), 95)"
> > > },
> > > "text" : {
> > > "is_alert": "true"
> > > }
> > > }
> > > }
> > > }
> > > } }
> > However when I stream data to it I receive: "value_red_level_out": null,
> >
> > I have checked in profiler client and here is what I got:
> > > [Stellar]>>> PROFILE_GET( "host-talks-to" , "99.191.183.156",
> > > PROFILE_FIXED(300, "MINUTES"))
> > > [1, 6, 6, 6, 6, 6, 3, 4, 5, 6, 4, 6, 6, 6, 1, 1, 6, 6, 1, 4, 1, 1, 4,
> > > 6, 6, 1, 6, 6, 1, 2, 6, 1, 1, 1, 6, 4, 6, 6, 3, 1, 6, 2, 1, 6, 1, 6]
> > > [Stellar]>>> STATS_PERCENTILE(STATS_MERGE(
> > > PROFILE_GET('host-talks-to', '99.191.183.156', PROFILE_FIXED(10,
> > > 'HOURS'))), 90)
> > > NaN
> > > [Stellar]>>> STATS_MERGE( PROFILE_GET('host-talks-to',
> > > '99.191.183.156', PROFILE_FIXED(10, 'HOURS')))
> > So the STATS_MERGE produces no results. Is this something expected or I
> > made a mistake somewhere? Please advise.
> >
> >
> > p.s. I am following this use cases:
> >
> https://github.com/hortonworks-gallery/metron-rules/tree/master/use-cases/
> > DegreeOfHost
> > There were number of errors in the configs originally, which I have
> > corrected, maybe I missed something else.
> >
> > - Dima
> >
> >
> >
> >
> >
> >
> >
>
Re: Profiler statistics NaN
Posted by Casey Stella <ce...@gmail.com>.
Ok, so the problem here is that your profile is returning integers
(specifically HLLP cardinalities) rather than stats objects. When you're
doing:
STATS_PERCENTILE(STATS_MERGE( PROFILE_GET('host-talks-to',
'99.191.183.156', PROFILE_FIXED(10, 'HOURS')), 90)
You are calling STATS_MERGE on a list of integers and it takes a list of
statistics objects.
What you can do instead is:
STATS_PERCENTILE( REDUCE( PROFILE_GET('host-talks-to',
'99.191.183.156', PROFILE_FIXED(10, 'HOURS'), (s, x) -> STATS_ADD(s, x),
STATS_INIT()), 90)
Ok, that looks horrible, doesn't it? Well, thankfully we added temporary
variables for stellar enrichments in 0.4.1. Let's take that "numeric"
stellar enrichment group and reimagine it. With temporary variables, you
would turn:
"numeric" : {
"value_red_level_out": "STATS_PERCENTILE( REDUCE(
PROFILE_GET('host-being-talked-to', ip_src_addr, PROFILE_FIXED(1,
'HOURS')), (s, x) -> STATS_ADD(s, x), STATS_INIT()), 95)",
"value_red_level_in": "STATS_PERCENTILE( REDUCE(
PROFILE_GET('host-talks-to',
ip_src_addr, PROFILE_FIXED(1, 'HOURS')), (s, x) -> STATS_ADD(s, x),
STATS_INIT()), 95)"
},
into:
"numeric" : [
"profile_duration := PROFILE_FIXED(1, 'HOURS')",
"host_being_talked_to := PROFILE_GET('host-being-talked-to',
ip_src_addr, profile_duration)",
"host_talks_to := PROFILE_GET('host-talks-to', ip_src_addr,
profile_duration)",
"host_being_talked_to_stats := REDUCE( host_being_talked_to,
(s, x) -> STATS_ADD(s, x), STATS_INIT())",
"host_talks_to_stats := REDUCE(host_talks_to, (s, x) ->
STATS_ADD(s, x), STATS_INIT())",
"value_red_level_out": "STATS_PERCENTILE(
host_being_talked_to_stats, 95)",
"value_red_level_in": "STATS_PERCENTILE( host_talks_to_stats,
95)",
"profile_duration := null",
"host_being_talked_to := null",
"host_talks_to := null",
"host_being_talked_to_stats := null",
host_talks_to_stats := null"
],
That's a lot more to type, but it allows you to reuse and take the pieces
in chunks.
Ok, so now I find myself thinking "a pox on both your houses" since both
examples now kinda look long and convoluted. So, why are they? Well, that
REDUCE is likely the culprit. It's supposed to get us out of bad
situations not show up in what could be argued is the 80% case. How about,
instead, we allow STATS_ADD or STATS_INIT to take a list of numbers? If
so, we could pretty easily make that nicer:
STATS_PERCENTILE( STATS_ADD( PROFILE_GET('host-being-talked-to',
ip_src_addr, PROFILE_FIXED(1, 'HOURS'))), 95)
or
STATS_PERCENTILE( STATS_INIT( PROFILE_GET('host-being-talked-to',
ip_src_addr, PROFILE_FIXED(1, 'HOURS'))), 95)
We should make some sort of candy like that so we can avoid some of the
complexity in the normal case.
On Wed, Aug 9, 2017 at 3:03 PM, Dima Kovalyov <Di...@sstech.us>
wrote:
> Hello Metron Team,
>
> I have created following profiler:
> > {
> > "profile": "host-talks-to",
> > "onlyif": "exists(source_ip)",
> > "foreach": "source_ip",
> > "init": {
> > "outcoming": "HLLP_INIT(5, 6)"
> > },
> > "update": { "outcoming": "HLLP_ADD(outcoming, destination_ip)" },
> > "result": "HLLP_CARDINALITY(outcoming)"
> > }
> I have also created enrichment rule:
> > {
> > "enrichment" : {
> > "fieldMap": {
> > "stellar" : {
> > "config" : {
> > "numeric" : {
> > "value_red_level_out": "STATS_PERCENTILE( STATS_MERGE(
> > PROFILE_GET('host-being-talked-to', ip_src_addr, 1, 'HOURS')), 95)",
> > "value_red_level_in": "STATS_PERCENTILE( STATS_MERGE(
> > PROFILE_GET('host-talks-to', ip_src_addr, 1, 'HOURS')), 95)"
> > },
> > "text" : {
> > "is_alert": "true"
> > }
> > }
> > }
> > }
> > } }
> However when I stream data to it I receive: "value_red_level_out": null,
>
> I have checked in profiler client and here is what I got:
> > [Stellar]>>> PROFILE_GET( "host-talks-to" , "99.191.183.156",
> > PROFILE_FIXED(300, "MINUTES"))
> > [1, 6, 6, 6, 6, 6, 3, 4, 5, 6, 4, 6, 6, 6, 1, 1, 6, 6, 1, 4, 1, 1, 4,
> > 6, 6, 1, 6, 6, 1, 2, 6, 1, 1, 1, 6, 4, 6, 6, 3, 1, 6, 2, 1, 6, 1, 6]
> > [Stellar]>>> STATS_PERCENTILE(STATS_MERGE(
> > PROFILE_GET('host-talks-to', '99.191.183.156', PROFILE_FIXED(10,
> > 'HOURS'))), 90)
> > NaN
> > [Stellar]>>> STATS_MERGE( PROFILE_GET('host-talks-to',
> > '99.191.183.156', PROFILE_FIXED(10, 'HOURS')))
> So the STATS_MERGE produces no results. Is this something expected or I
> made a mistake somewhere? Please advise.
>
>
> p.s. I am following this use cases:
> https://github.com/hortonworks-gallery/metron-rules/tree/master/use-cases/
> DegreeOfHost
> There were number of errors in the configs originally, which I have
> corrected, maybe I missed something else.
>
> - Dima
>
>
>
>
>
>
>
Re: Profiler statistics NaN
Posted by Nick Allen <ni...@nickallen.org>.
It seems that you are using the Profiler Client API correctly from the
REPL, but you are using it incorrectly in your triage rules. Change your
triage rules to match what you ran in the REPL.
Correct:
PROFILE_GET( "host-talks-to" , "99.191.183.156", PROFILE_FIXED(300,
"MINUTES"))
Incorrect:
PROFILE_GET('host-being-talked-to', ip_src_addr, 1, 'HOURS')
On Wed, Aug 9, 2017 at 3:04 PM Dima Kovalyov <Di...@sstech.us>
wrote:
> Hello Metron Team,
>
> I have created following profiler:
> > {
> > "profile": "host-talks-to",
> > "onlyif": "exists(source_ip)",
> > "foreach": "source_ip",
> > "init": {
> > "outcoming": "HLLP_INIT(5, 6)"
> > },
> > "update": { "outcoming": "HLLP_ADD(outcoming, destination_ip)" },
> > "result": "HLLP_CARDINALITY(outcoming)"
> > }
> I have also created enrichment rule:
> > {
> > "enrichment" : {
> > "fieldMap": {
> > "stellar" : {
> > "config" : {
> > "numeric" : {
> > "value_red_level_out": "STATS_PERCENTILE( STATS_MERGE(
> > PROFILE_GET('host-being-talked-to', ip_src_addr, 1, 'HOURS')), 95)",
> > "value_red_level_in": "STATS_PERCENTILE( STATS_MERGE(
> > PROFILE_GET('host-talks-to', ip_src_addr, 1, 'HOURS')), 95)"
> > },
> > "text" : {
> > "is_alert": "true"
> > }
> > }
> > }
> > }
> > } }
> However when I stream data to it I receive: "value_red_level_out": null,
>
> I have checked in profiler client and here is what I got:
> > [Stellar]>>> PROFILE_GET( "host-talks-to" , "99.191.183.156",
> > PROFILE_FIXED(300, "MINUTES"))
> > [1, 6, 6, 6, 6, 6, 3, 4, 5, 6, 4, 6, 6, 6, 1, 1, 6, 6, 1, 4, 1, 1, 4,
> > 6, 6, 1, 6, 6, 1, 2, 6, 1, 1, 1, 6, 4, 6, 6, 3, 1, 6, 2, 1, 6, 1, 6]
> > [Stellar]>>> STATS_PERCENTILE(STATS_MERGE(
> > PROFILE_GET('host-talks-to', '99.191.183.156', PROFILE_FIXED(10,
> > 'HOURS'))), 90)
> > NaN
> > [Stellar]>>> STATS_MERGE( PROFILE_GET('host-talks-to',
> > '99.191.183.156', PROFILE_FIXED(10, 'HOURS')))
> So the STATS_MERGE produces no results. Is this something expected or I
> made a mistake somewhere? Please advise.
>
>
> p.s. I am following this use cases:
>
> https://github.com/hortonworks-gallery/metron-rules/tree/master/use-cases/DegreeOfHost
> There were number of errors in the configs originally, which I have
> corrected, maybe I missed something else.
>
> - Dima
>
>
>
>
>
>
>