You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@metron.apache.org by Anil Donthireddy <An...@sstech.us> on 2019/03/12 00:15:48 UTC
Issues with Metron profiler
Hi,
I am seeing below issues with Metron profiler.
1) Profiler is writing the duplicate data to Hbase. For example I have inserted few records couple of days back to my test sensor topic and written a profile that counts the number of records. I see that profiler is still storing the same count continuously in to Hbase. As I observe when I see the WARN message below in profiler log, I see the profiler inserts data to Hbase that number of times (I am not sure if there is any relation)
a. 2019-03-07 15:56:28.419 o.a.m.p.DefaultMessageDistributor pool-17-thread-1 [WARN] Profile expired from active cache; profile=demo_user_activity_count_ar, entity=Test1User5
2) I am storing the STAT objects to Hbase and trying to calculate the bin in another profiler by pulling Stats from Hbase. I see below exception after a while
a. 2019-03-11 11:24:51.039 o.a.m.p.DefaultProfileBuilder pool-18-thread-1 [ERROR] Bad 'update' expression: error='Unable to parse: if EXISTS(s) then STATS_BIN(s, activity_count, 'DECILE') else -1 due to: Your bins must be non-decreasing with relevant variables s=org.apache.metron.statistics.OnlineStatisticsProvider@808c9b26,activity_count=24', expr='if EXISTS(s) then STATS_BIN(s, activity_count, 'DECILE') else -1', profile='demo_user_activity_bin_ar', entity='Test2User4', variables-available='[period.start, period, enrichmentsplitterbolt.splitter.end.ts, bin, profile, enrichmentsplitterbolt.splitter.begin.ts, is_alert, source.type, threatintelsplitterbolt.splitter.end.ts, s, activity_count, threatinteljoinbolt.joiner.ts, enrichmentjoinbolt.joiner.ts, period.end, threatintelsplitterbolt.splitter.begin.ts, entity, timestamp]'
b. org.apache.metron.stellar.dsl.ParseException: Unable to parse: if EXISTS(s) then STATS_BIN(s, activity_count, 'DECILE') else -1 due to: Your bins must be non-decreasing with relevant variables s=org.apache.metron.statistics.OnlineStatisticsProvider@808c9b26,activity_count=24<mailto:s=org.apache.metron.statistics.OnlineStatisticsProvider@808c9b26,activity_count=24>
I would like to understand what does it meant by "[WARN] Profile expired from active cache; profile=demo_user_activity_count_ar, entity=Test1User5", when can this happen and how to resolve this?
I would also like to understand how to resolve the issue "[ERROR] Bad 'update' expression: error='Unable to parse: if EXISTS(s) then STATS_BIN(s, activity_count, 'DECILE') else -1 due to: Your bins must be non-decreasing with relevant variables s=org.apache.metron.statistics.OnlineStatisticsProvider@808c9b26,activity_count=24', expr='if EXISTS(s) then STATS_BIN(s, activity_count, 'DECILE') else -1', profile='demo_user_activity_bin_ar', entity='Test2User4', variables-available='[period.start, period, enrichmentsplitterbolt.splitter.end.ts, bin, profile, enrichmentsplitterbolt.splitter.begin.ts, is_alert, source.type, threatintelsplitterbolt.splitter.end.ts, s, activity_count, threatinteljoinbolt.joiner.ts, enrichmentjoinbolt.joiner.ts, period.end, threatintelsplitterbolt.splitter.begin.ts, entity, timestamp]'"
Looking forward for some help from Metron team or community to understand and resolve the above issues.
Thanking you.
Thanks,
Anil.
Re: Issues with Metron profiler
Posted by Michael Miklavcic <mi...@gmail.com>.
Hi Anil,
Can you share your profile definition? For the STATS_BIN issue you're
seeing, it seems like your bins are wrong, however you're using the
pre-defined DECILE bin list, which is unit tested (and the tests are
passing). It may be that the stats object isn't quite right, but I can't
tell from this snippet alone. Your activity_count at least appears
populated.
On Mon, Mar 11, 2019 at 6:16 PM Anil Donthireddy <An...@sstech.us>
wrote:
> Hi,
>
>
>
> I am seeing below issues with Metron profiler.
>
>
>
> 1) Profiler is writing the duplicate data to Hbase. For example I
> have inserted few records couple of days back to my test sensor topic and
> written a profile that counts the number of records. I see that profiler is
> still storing the same count continuously in to Hbase. As I observe when I
> see the WARN message below in profiler log, I see the profiler inserts data
> to Hbase that number of times (I am not sure if there is any relation)
>
> a. 2019-03-07 15:56:28.419 o.a.m.p.DefaultMessageDistributor
> pool-17-thread-1 [WARN] Profile expired from active cache;
> profile=demo_user_activity_count_ar, entity=Test1User5
>
> 2) I am storing the STAT objects to Hbase and trying to calculate
> the bin in another profiler by pulling Stats from Hbase. I see below
> exception after a while
>
> a. 2019-03-11 11:24:51.039 o.a.m.p.DefaultProfileBuilder
> pool-18-thread-1 [ERROR] Bad 'update' expression: error='Unable to parse:
> if EXISTS(s) then STATS_BIN(s, activity_count, 'DECILE') else -1 due to:
> Your bins must be non-decreasing with relevant variables
> s=org.apache.metron.statistics.OnlineStatisticsProvider@808c9b26,activity_count=24',
> expr='if EXISTS(s) then STATS_BIN(s, activity_count, 'DECILE') else -1',
> profile='demo_user_activity_bin_ar', entity='Test2User4',
> variables-available='[period.start, period,
> enrichmentsplitterbolt.splitter.end.ts, bin, profile,
> enrichmentsplitterbolt.splitter.begin.ts, is_alert, source.type,
> threatintelsplitterbolt.splitter.end.ts, s, activity_count,
> threatinteljoinbolt.joiner.ts, enrichmentjoinbolt.joiner.ts, period.end,
> threatintelsplitterbolt.splitter.begin.ts, entity, timestamp]'
>
> b. org.apache.metron.stellar.dsl.ParseException: Unable to parse: if
> EXISTS(s) then STATS_BIN(s, activity_count, 'DECILE') else -1 due to: Your
> bins must be non-decreasing with relevant variables
> s=org.apache.metron.statistics.OnlineStatisticsProvider@808c9b26,activity_count=24
>
>
>
> I would like to understand what does it meant by “[WARN] Profile expired
> from active cache; profile=demo_user_activity_count_ar, entity=Test1User5”,
> when can this happen and how to resolve this?
>
>
>
> I would also like to understand how to resolve the issue “[ERROR] Bad
> 'update' expression: error='Unable to parse: if EXISTS(s) then STATS_BIN(s,
> activity_count, 'DECILE') else -1 due to: Your bins must be non-decreasing
> with relevant variables
> s=org.apache.metron.statistics.OnlineStatisticsProvider@808c9b26,activity_count=24',
> expr='if EXISTS(s) then STATS_BIN(s, activity_count, 'DECILE') else -1',
> profile='demo_user_activity_bin_ar', entity='Test2User4',
> variables-available='[period.start, period,
> enrichmentsplitterbolt.splitter.end.ts, bin, profile,
> enrichmentsplitterbolt.splitter.begin.ts, is_alert, source.type,
> threatintelsplitterbolt.splitter.end.ts, s, activity_count,
> threatinteljoinbolt.joiner.ts, enrichmentjoinbolt.joiner.ts, period.end,
> threatintelsplitterbolt.splitter.begin.ts, entity, timestamp]'”
>
>
>
> Looking forward for some help from Metron team or community to understand
> and resolve the above issues.
>
>
>
> Thanking you.
>
>
>
> Thanks,
>
> Anil.
>
Re: Issues with Metron profiler
Posted by Michael Miklavcic <mi...@gmail.com>.
Hi Anil,
Can you share your profile definition? For the STATS_BIN issue you're
seeing, it seems like your bins are wrong, however you're using the
pre-defined DECILE bin list, which is unit tested (and the tests are
passing). It may be that the stats object isn't quite right, but I can't
tell from this snippet alone. Your activity_count at least appears
populated.
On Mon, Mar 11, 2019 at 6:16 PM Anil Donthireddy <An...@sstech.us>
wrote:
> Hi,
>
>
>
> I am seeing below issues with Metron profiler.
>
>
>
> 1) Profiler is writing the duplicate data to Hbase. For example I
> have inserted few records couple of days back to my test sensor topic and
> written a profile that counts the number of records. I see that profiler is
> still storing the same count continuously in to Hbase. As I observe when I
> see the WARN message below in profiler log, I see the profiler inserts data
> to Hbase that number of times (I am not sure if there is any relation)
>
> a. 2019-03-07 15:56:28.419 o.a.m.p.DefaultMessageDistributor
> pool-17-thread-1 [WARN] Profile expired from active cache;
> profile=demo_user_activity_count_ar, entity=Test1User5
>
> 2) I am storing the STAT objects to Hbase and trying to calculate
> the bin in another profiler by pulling Stats from Hbase. I see below
> exception after a while
>
> a. 2019-03-11 11:24:51.039 o.a.m.p.DefaultProfileBuilder
> pool-18-thread-1 [ERROR] Bad 'update' expression: error='Unable to parse:
> if EXISTS(s) then STATS_BIN(s, activity_count, 'DECILE') else -1 due to:
> Your bins must be non-decreasing with relevant variables
> s=org.apache.metron.statistics.OnlineStatisticsProvider@808c9b26,activity_count=24',
> expr='if EXISTS(s) then STATS_BIN(s, activity_count, 'DECILE') else -1',
> profile='demo_user_activity_bin_ar', entity='Test2User4',
> variables-available='[period.start, period,
> enrichmentsplitterbolt.splitter.end.ts, bin, profile,
> enrichmentsplitterbolt.splitter.begin.ts, is_alert, source.type,
> threatintelsplitterbolt.splitter.end.ts, s, activity_count,
> threatinteljoinbolt.joiner.ts, enrichmentjoinbolt.joiner.ts, period.end,
> threatintelsplitterbolt.splitter.begin.ts, entity, timestamp]'
>
> b. org.apache.metron.stellar.dsl.ParseException: Unable to parse: if
> EXISTS(s) then STATS_BIN(s, activity_count, 'DECILE') else -1 due to: Your
> bins must be non-decreasing with relevant variables
> s=org.apache.metron.statistics.OnlineStatisticsProvider@808c9b26,activity_count=24
>
>
>
> I would like to understand what does it meant by “[WARN] Profile expired
> from active cache; profile=demo_user_activity_count_ar, entity=Test1User5”,
> when can this happen and how to resolve this?
>
>
>
> I would also like to understand how to resolve the issue “[ERROR] Bad
> 'update' expression: error='Unable to parse: if EXISTS(s) then STATS_BIN(s,
> activity_count, 'DECILE') else -1 due to: Your bins must be non-decreasing
> with relevant variables
> s=org.apache.metron.statistics.OnlineStatisticsProvider@808c9b26,activity_count=24',
> expr='if EXISTS(s) then STATS_BIN(s, activity_count, 'DECILE') else -1',
> profile='demo_user_activity_bin_ar', entity='Test2User4',
> variables-available='[period.start, period,
> enrichmentsplitterbolt.splitter.end.ts, bin, profile,
> enrichmentsplitterbolt.splitter.begin.ts, is_alert, source.type,
> threatintelsplitterbolt.splitter.end.ts, s, activity_count,
> threatinteljoinbolt.joiner.ts, enrichmentjoinbolt.joiner.ts, period.end,
> threatintelsplitterbolt.splitter.begin.ts, entity, timestamp]'”
>
>
>
> Looking forward for some help from Metron team or community to understand
> and resolve the above issues.
>
>
>
> Thanking you.
>
>
>
> Thanks,
>
> Anil.
>