You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@metron.apache.org by shoggi <sh...@gmail.com> on 2017/02/27 22:29:13 UTC
Median Absolute Deviation - Profiler Error
Hi all
Has anyone come across following profiler error, when trying to work with
the MAD functions? The configuration loads fine and I only get the error
when the profiler is pushing its data. I don't think its a typo but maybe
the logic is not sound.
Error from the profiler worker:
2017-02-27 23:20:43.368 o.a.m.p.b.ProfileBuilderBolt [INFO] Flushing
profile: profile=sketchy_mad, entity=krxd.net
2017-02-27 23:20:43.368 o.a.m.p.b.ProfileBuilderBolt [ERROR] Unexpected
failure: message='null', tuple='source: __system:-1, stream: __tick, id:
{}, [900]'
java.lang.NullPointerException
2017-02-27 23:20:43.368 o.a.s.d.executor [ERROR]
java.lang.NullPointerException
My Profiler config:
{
"profile": "sketchy_mad",
"foreach": "if exists(domain_without_subdomains) then
domain_without_subdomains else 'n/a'",
"onlyif": "exists(domain_without_subdomains) && source.type ==
'squid'",
"init" : {
"s": "OUTLIER_MAD_STATE_MERGE(PROFILE_GET('sketchy_mad',
domain_without_subdomains, 5, 'MINUTES'))"
},
"tickUpdate" : {
"s": "OUTLIER_MAD_STATE_MERGE(PROFILE_GET('sketchy_mad',
domain_without_subdomains, 5, 'MINUTES'), s)"
},
"update": {
"s": "OUTLIER_MAD_ADD(s, bytes)"
},
"result": "s"
}
Regards
shoggi
Re: Median Absolute Deviation - Profiler Error
Posted by shoggi <sh...@gmail.com>.
Hi Casey
I have attached the full workers log of the profiler after I started the
profiler topology again.
Thank you
shoggi
On Tue, Feb 28, 2017 at 3:23 PM, Casey Stella <ce...@gmail.com> wrote:
> So, I do recall in 0.3.0 a bug where if you modify a profile, it still
> continues to run the old copy of the profile. I was going to have you
> bounce the profiler topology. Did you do that? If not, then please do
> it. If so, then are the NPE's the same? I've seen 2 distinct exceptions
> in the stacktraces, one around casting a MAD$State objct to a List and
> another as a real NPE.
>
> Let's level-set, can you bounce the profiler topology and then send the
> NPE exception and stack trace?
>
> Casey
>
> On Tue, Feb 28, 2017 at 3:56 AM, shoggi <sh...@gmail.com> wrote:
>
>> I restarted Storm and ingest some data over the last few hours. I still
>> get the casual NPE but some data gets profiled. It definitely skips data as
>> I fed 160 unique tld's but only get 9 mad profile entries. The list seems
>> random as I would least expect my top talkers to show up.
>>
>>
>> [Stellar]>>> OUTLIER_MAD_STATE_MERGE(PROFILE_GET('sketchy_mad', 'krxd.net',
>> 5, 'HOURS'))
>> org.apache.metron.statistics.outlier.MedianAbsoluteDeviation
>> Functions$State@5b307115
>> [Stellar]>>> PROFILE_GET('sketchy_mad', 'krxd.net', 5, 'HOURS')
>> []
>> [Stellar]>>> OUTLIER_MAD_STATE_MERGE(PROFILE_GET('sketchy_mad', 'krxd.net',
>> 5, 'MINUTES'))
>> org.apache.metron.statistics.outlier.MedianAbsoluteDeviation
>> Functions$State@714c575b
>> [Stellar]>>> PROFILE_GET('sketchy_mad', 'krxd.net', 5, 'MINUTES')
>> []
>>
>> [Stellar]>>> bytes := 700
>> [Stellar]>>> OUTLIER_MAD_SCORE( OUTLIER_MAD_STATE_MERGE( PROFILE_GET(
>> 'sketchy_mad', 'krxd.net', 5, 'MINUTES')), bytes)
>> NaN
>> [Stellar]>>> OUTLIER_MAD_SCORE( OUTLIER_MAD_STATE_MERGE( PROFILE_GET(
>> 'sketchy_mad', 'krxd.net', 5, 'HOURS')), bytes)
>> NaN
>> [Stellar]>>> OUTLIER_MAD_SCORE( OUTLIER_MAD_STATE_MERGE( PROFILE_GET(
>> 'sketchy_mad', 'krxd.net', 5, 'DAYS')), bytes)
>> 3.8971111111111107
>>
>>
>> # echo "scan 'profiler'" | hbase shell > /tmp/test.txt && egrep
>> 'sketchy_mad' /tmp/test.txt | awk '{print $1}'
>> \x00\x00\x00xsketchy_madoptimicdn.com\x00\x00\x00\x00\x00\x19;s
>> \x00\x00\x01dsketchy_madajax.googleapis.com\x00\x00\x00\x00\x00\x19;g
>> \x00\x00\x02\x86sketchy_madrlcdn.com\x00\x00\x00\x00\x00\x19;c
>> \x00\x00\x02\x86sketchy_madshareaholic.com\x00\x00\x00\x00\x00\x19;c
>> \x00\x00\x02\xDBsketchy_madnewsnetz.ch\x00\x00\x00\x00\x00\x19;k
>> \xFF\xFF\xFC\x8Esketchy_madajax.googleapis.com\x00\x00\x00\x00\x00\x19;r
>> \xFF\xFF\xFC\xEBsketchy_madkrxd.net\x00\x00\x00\x00\x00\x19;[
>> \xFF\xFF\xFE\x87sketchy_madkrxd.net\x00\x00\x00\x00\x00\x19;^
>> \xFF\xFF\xFE\xF2sketchy_madoptimicdn.com\x00\x00\x00\x00\x00\x19;\x5C
>>
>> On Tue, Feb 28, 2017 at 8:10 AM, Casey Stella <ce...@gmail.com> wrote:
>>
>>> Ok, from the Stellar REPL (started with -z and the zookeeper quorum),
>>> what does the following yield:
>>>
>>> - OUTLIER_MAD_STATE_MERGE(PROFILE_GET('sketchy_mad', 'krxd.net', 5,
>>> 'HOURS'))
>>> - PROFILE_GET('sketchy_mad', 'krxd.net', 5, 'HOURS')
>>> - OUTLIER_MAD_STATE_MERGE(PROFILE_GET('sketchy_mad', 'krxd.net', 5,
>>> 'MINUTES'))
>>> - PROFILE_GET('sketchy_mad', 'krxd.net', 5, 'MINUTES')
>>>
>>>
>>> On Tue, Feb 28, 2017 at 1:36 AM, shoggi <sh...@gmail.com> wrote:
>>>
>>>> This is what I've got..
>>>>
>>>> {
>>>> "profile": "sketchy_mad",
>>>> "foreach": "if exists(domain_without_subdomains) then
>>>> domain_without_subdomains else 'n/a'",
>>>> "onlyif": "exists(domain_without_subdomains) && source.type ==
>>>> 'squid'",
>>>> "init" : {
>>>> "s": "OUTLIER_MAD_STATE_MERGE(PROFILE_GET('sketchy_mad',
>>>> domain_without_subdomains, 5, 'MINUTES'))"
>>>> },
>>>> "update": {
>>>> "s": "OUTLIER_MAD_ADD(s, bytes)"
>>>> },
>>>> "result": "s"
>>>> }
>>>>
>>>> On Tue, Feb 28, 2017 at 3:34 AM, Casey Stella <ce...@gmail.com>
>>>> wrote:
>>>>
>>>>> Sorry, waht does the current profile config look like for that very
>>>>> last error?
>>>>>
>>>>> On Mon, Feb 27, 2017 at 6:39 PM, shoggi <sh...@gmail.com> wrote:
>>>>>
>>>>>> Seems not to be the culprit..
>>>>>>
>>>>>>
>>>>>> 2017-02-28 00:35:43.367 o.a.m.p.b.ProfileBuilderBolt [INFO] Flushing
>>>>>> profile: profile=sketchy_mad, entity=optimicdn.com
>>>>>> 2017-02-28 00:35:43.368 o.a.m.p.b.ProfileBuilderBolt [ERROR]
>>>>>> Unexpected failure: message='Unable to execute:
>>>>>> org.apache.metron.statistics.outlier.MedianAbsoluteDeviationFunctions$State
>>>>>> cannot be cast to java.util.List', tuple='source: __system:-1, stream:
>>>>>> __tick, id: {}, [900]'
>>>>>> org.apache.metron.common.dsl.ParseException: Unable to execute:
>>>>>> org.apache.metron.statistics.outlier.MedianAbsoluteDeviationFunctions$State
>>>>>> cannot be cast to java.util.List
>>>>>> at org.apache.metron.common.stellar.StellarCompiler.getResult(StellarCompiler.java:428)
>>>>>> ~[stormjar.jar:?]
>>>>>> at org.apache.metron.common.stellar.BaseStellarProcessor.parse(BaseStellarProcessor.java:90)
>>>>>> ~[stormjar.jar:?]
>>>>>> at org.apache.metron.profiler.stellar.DefaultStellarExecutor.ex
>>>>>> ecute(DefaultStellarExecutor.java:148) ~[stormjar.jar:?]
>>>>>> at org.apache.metron.profiler.stellar.DefaultStellarExecutor.as
>>>>>> sign(DefaultStellarExecutor.java:94) ~[stormjar.jar:?]
>>>>>> at org.apache.metron.profiler.bolt.ProfileBuilderBolt.lambda$nu
>>>>>> ll$2(ProfileBuilderBolt.java:258) ~[stormjar.jar:?]
>>>>>> at java.util.LinkedHashMap.forEach(LinkedHashMap.java:676)
>>>>>> ~[?:1.8.0_77]
>>>>>> at org.apache.metron.profiler.bolt.ProfileBuilderBolt.lambda$fl
>>>>>> ush$4(ProfileBuilderBolt.java:258) ~[stormjar.jar:?]
>>>>>> at java.util.concurrent.ConcurrentMap.forEach(ConcurrentMap.java:114)
>>>>>> ~[?:1.8.0_77]
>>>>>> at org.apache.metron.profiler.bolt.ProfileBuilderBolt.flush(ProfileBuilderBolt.java:237)
>>>>>> ~[stormjar.jar:?]
>>>>>> at org.apache.metron.profiler.bolt.ProfileBuilderBolt.doExecute(ProfileBuilderBolt.java:164)
>>>>>> ~[stormjar.jar:?]
>>>>>> at org.apache.metron.profiler.bolt.ProfileBuilderBolt.execute(ProfileBuilderBolt.java:144)
>>>>>> [stormjar.jar:?]
>>>>>> at org.apache.storm.daemon.executor$fn__6571$tuple_action_fn__6573.invoke(executor.clj:734)
>>>>>> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>>>>>> at org.apache.storm.daemon.executor$mk_task_receiver$fn__6492.invoke(executor.clj:469)
>>>>>> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>>>>>> at org.apache.storm.disruptor$clojure_handler$reify__6005.onEvent(disruptor.clj:40)
>>>>>> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>>>>>> at org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:451)
>>>>>> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>>>>>> at org.apache.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:430)
>>>>>> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>>>>>> at org.apache.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:73)
>>>>>> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>>>>>> at org.apache.storm.daemon.executor$fn__6571$fn__6584$fn__6637.invoke(executor.clj:853)
>>>>>> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>>>>>> at org.apache.storm.util$async_loop$fn__554.invoke(util.clj:484)
>>>>>> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>>>>>> at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
>>>>>> at java.lang.Thread.run(Thread.java:745) [?:1.8.0_77]
>>>>>> Caused by: java.lang.ClassCastException:
>>>>>> org.apache.metron.statistics.outlier.MedianAbsoluteDeviationFunctions$State
>>>>>> cannot be cast to java.util.List
>>>>>> at org.apache.metron.statistics.outlier.MedianAbsoluteDeviation
>>>>>> Functions$StateUpdate.apply(MedianAbsoluteDeviationFunctions.java:85)
>>>>>> ~[stormjar.jar:?]
>>>>>> at org.apache.metron.common.stellar.StellarCompiler.exitTransfo
>>>>>> rmationFunc(StellarCompiler.java:246) ~[stormjar.jar:?]
>>>>>> at org.apache.metron.common.stellar.generated.StellarParser$Tra
>>>>>> nsformationFuncContext.exitRule(StellarParser.java:1617)
>>>>>> ~[stormjar.jar:?]
>>>>>> at org.antlr.v4.runtime.Parser.triggerExitRuleEvent(Parser.java:422)
>>>>>> ~[stormjar.jar:?]
>>>>>> at org.antlr.v4.runtime.Parser.exitRule(Parser.java:632)
>>>>>> ~[stormjar.jar:?]
>>>>>> at org.apache.metron.common.stellar.generated.StellarParser.fun
>>>>>> ctions(StellarParser.java:1640) ~[stormjar.jar:?]
>>>>>> at org.apache.metron.common.stellar.generated.StellarParser.ari
>>>>>> thmetic_operands(StellarParser.java:1750) ~[stormjar.jar:?]
>>>>>> at org.apache.metron.common.stellar.generated.StellarParser.ari
>>>>>> thmetic_expr_mul(StellarParser.java:1537) ~[stormjar.jar:?]
>>>>>> at org.apache.metron.common.stellar.generated.StellarParser.ari
>>>>>> thmetic_expr(StellarParser.java:1397) ~[stormjar.jar:?]
>>>>>> at org.apache.metron.common.stellar.generated.StellarParser.tra
>>>>>> nsformation_expr(StellarParser.java:278) ~[stormjar.jar:?]
>>>>>> at org.apache.metron.common.stellar.generated.StellarParser.tra
>>>>>> nsformation(StellarParser.java:147) ~[stormjar.jar:?]
>>>>>> at org.apache.metron.common.stellar.BaseStellarProcessor.parse(BaseStellarProcessor.java:89)
>>>>>> ~[stormjar.jar:?]
>>>>>> ... 19 more
>>>>>>
>>>>>> On Tue, Feb 28, 2017 at 12:18 AM, Casey Stella <ce...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Ok, try something for me and tell me if it fixes it: adjust the
>>>>>>> config to remove the tickUpdate (it didn't work anyway and we have since
>>>>>>> removed it in 0.3.1).
>>>>>>>
>>>>>>>
>>>>>>> On Mon, Feb 27, 2017 at 6:07 PM, shoggi <sh...@gmail.com> wrote:
>>>>>>>
>>>>>>>> Yes, this is 0.3.0.
>>>>>>>>
>>>>>>>> Here is the full trace.
>>>>>>>>
>>>>>>>> 2017-02-28 00:05:43.367 o.a.m.p.b.ProfileBuilderBolt [INFO]
>>>>>>>> Flushing profile: profile=sketchy_mad, entity=krxd.net
>>>>>>>> 2017-02-28 00:05:43.374 o.a.m.p.b.ProfileBuilderBolt [ERROR]
>>>>>>>> Unexpected failure: message='Unable to execute:
>>>>>>>> org.apache.metron.statistics.outlier.MedianAbsoluteDeviationFunctions$State
>>>>>>>> cannot be cast to java.util.List', tuple='source: __system:-1, stream:
>>>>>>>> __tick, id: {}, [900]'
>>>>>>>> org.apache.metron.common.dsl.ParseException: Unable to execute:
>>>>>>>> org.apache.metron.statistics.outlier.MedianAbsoluteDeviationFunctions$State
>>>>>>>> cannot be cast to java.util.List
>>>>>>>> at org.apache.metron.common.stellar.StellarCompiler.getResult(StellarCompiler.java:428)
>>>>>>>> ~[stormjar.jar:?]
>>>>>>>> at org.apache.metron.common.stellar.BaseStellarProcessor.parse(BaseStellarProcessor.java:90)
>>>>>>>> ~[stormjar.jar:?]
>>>>>>>> at org.apache.metron.profiler.stellar.DefaultStellarExecutor.ex
>>>>>>>> ecute(DefaultStellarExecutor.java:148) ~[stormjar.jar:?]
>>>>>>>> at org.apache.metron.profiler.stellar.DefaultStellarExecutor.as
>>>>>>>> sign(DefaultStellarExecutor.java:94) ~[stormjar.jar:?]
>>>>>>>> at org.apache.metron.profiler.bolt.ProfileBuilderBolt.lambda$nu
>>>>>>>> ll$2(ProfileBuilderBolt.java:258) ~[stormjar.jar:?]
>>>>>>>> at java.util.LinkedHashMap.forEach(LinkedHashMap.java:676)
>>>>>>>> ~[?:1.8.0_77]
>>>>>>>> at org.apache.metron.profiler.bolt.ProfileBuilderBolt.lambda$fl
>>>>>>>> ush$4(ProfileBuilderBolt.java:258) ~[stormjar.jar:?]
>>>>>>>> at java.util.concurrent.ConcurrentMap.forEach(ConcurrentMap.java:114)
>>>>>>>> ~[?:1.8.0_77]
>>>>>>>> at org.apache.metron.profiler.bolt.ProfileBuilderBolt.flush(ProfileBuilderBolt.java:237)
>>>>>>>> ~[stormjar.jar:?]
>>>>>>>> at org.apache.metron.profiler.bolt.ProfileBuilderBolt.doExecute(ProfileBuilderBolt.java:164)
>>>>>>>> ~[stormjar.jar:?]
>>>>>>>> at org.apache.metron.profiler.bolt.ProfileBuilderBolt.execute(ProfileBuilderBolt.java:144)
>>>>>>>> [stormjar.jar:?]
>>>>>>>> at org.apache.storm.daemon.executor$fn__6571$tuple_action_fn__6573.invoke(executor.clj:734)
>>>>>>>> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>>>>>>>> at org.apache.storm.daemon.executor$mk_task_receiver$fn__6492.invoke(executor.clj:469)
>>>>>>>> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>>>>>>>> at org.apache.storm.disruptor$clojure_handler$reify__6005.onEvent(disruptor.clj:40)
>>>>>>>> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>>>>>>>> at org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:451)
>>>>>>>> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>>>>>>>> at org.apache.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:430)
>>>>>>>> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>>>>>>>> at org.apache.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:73)
>>>>>>>> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>>>>>>>> at org.apache.storm.daemon.executor$fn__6571$fn__6584$fn__6637.invoke(executor.clj:853)
>>>>>>>> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>>>>>>>> at org.apache.storm.util$async_loop$fn__554.invoke(util.clj:484)
>>>>>>>> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>>>>>>>> at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
>>>>>>>> at java.lang.Thread.run(Thread.java:745) [?:1.8.0_77]
>>>>>>>> Caused by: java.lang.ClassCastException:
>>>>>>>> org.apache.metron.statistics.outlier.MedianAbsoluteDeviationFunctions$State
>>>>>>>> cannot be cast to java.util.List
>>>>>>>> at org.apache.metron.statistics.outlier.MedianAbsoluteDeviation
>>>>>>>> Functions$StateUpdate.apply(MedianAbsoluteDeviationFunctions.java:85)
>>>>>>>> ~[stormjar.jar:?]
>>>>>>>> at org.apache.metron.common.stellar.StellarCompiler.exitTransfo
>>>>>>>> rmationFunc(StellarCompiler.java:246) ~[stormjar.jar:?]
>>>>>>>> at org.apache.metron.common.stellar.generated.StellarParser$Tra
>>>>>>>> nsformationFuncContext.exitRule(StellarParser.java:1617)
>>>>>>>> ~[stormjar.jar:?]
>>>>>>>> at org.antlr.v4.runtime.Parser.triggerExitRuleEvent(Parser.java:422)
>>>>>>>> ~[stormjar.jar:?]
>>>>>>>> at org.antlr.v4.runtime.Parser.exitRule(Parser.java:632)
>>>>>>>> ~[stormjar.jar:?]
>>>>>>>> at org.apache.metron.common.stellar.generated.StellarParser.fun
>>>>>>>> ctions(StellarParser.java:1640) ~[stormjar.jar:?]
>>>>>>>> at org.apache.metron.common.stellar.generated.StellarParser.ari
>>>>>>>> thmetic_operands(StellarParser.java:1750) ~[stormjar.jar:?]
>>>>>>>> at org.apache.metron.common.stellar.generated.StellarParser.ari
>>>>>>>> thmetic_expr_mul(StellarParser.java:1537) ~[stormjar.jar:?]
>>>>>>>> at org.apache.metron.common.stellar.generated.StellarParser.ari
>>>>>>>> thmetic_expr(StellarParser.java:1397) ~[stormjar.jar:?]
>>>>>>>> at org.apache.metron.common.stellar.generated.StellarParser.tra
>>>>>>>> nsformation_expr(StellarParser.java:278) ~[stormjar.jar:?]
>>>>>>>> at org.apache.metron.common.stellar.generated.StellarParser.tra
>>>>>>>> nsformation(StellarParser.java:147) ~[stormjar.jar:?]
>>>>>>>> at org.apache.metron.common.stellar.BaseStellarProcessor.parse(BaseStellarProcessor.java:89)
>>>>>>>> ~[stormjar.jar:?]
>>>>>>>> ... 19 more
>>>>>>>> 2017-02-28 00:05:43.377 o.a.s.d.executor [ERROR]
>>>>>>>> org.apache.metron.common.dsl.ParseException: Unable to execute:
>>>>>>>> org.apache.metron.statistics.outlier.MedianAbsoluteDeviationFunctions$State
>>>>>>>> cannot be cast to java.util.List
>>>>>>>> at org.apache.metron.common.stellar.StellarCompiler.getResult(StellarCompiler.java:428)
>>>>>>>> ~[stormjar.jar:?]
>>>>>>>> at org.apache.metron.common.stellar.BaseStellarProcessor.parse(BaseStellarProcessor.java:90)
>>>>>>>> ~[stormjar.jar:?]
>>>>>>>> at org.apache.metron.profiler.stellar.DefaultStellarExecutor.ex
>>>>>>>> ecute(DefaultStellarExecutor.java:148) ~[stormjar.jar:?]
>>>>>>>> at org.apache.metron.profiler.stellar.DefaultStellarExecutor.as
>>>>>>>> sign(DefaultStellarExecutor.java:94) ~[stormjar.jar:?]
>>>>>>>> at org.apache.metron.profiler.bolt.ProfileBuilderBolt.lambda$nu
>>>>>>>> ll$2(ProfileBuilderBolt.java:258) ~[stormjar.jar:?]
>>>>>>>> at java.util.LinkedHashMap.forEach(LinkedHashMap.java:676)
>>>>>>>> ~[?:1.8.0_77]
>>>>>>>> at org.apache.metron.profiler.bolt.ProfileBuilderBolt.lambda$fl
>>>>>>>> ush$4(ProfileBuilderBolt.java:258) ~[stormjar.jar:?]
>>>>>>>> at java.util.concurrent.ConcurrentMap.forEach(ConcurrentMap.java:114)
>>>>>>>> ~[?:1.8.0_77]
>>>>>>>> at org.apache.metron.profiler.bolt.ProfileBuilderBolt.flush(ProfileBuilderBolt.java:237)
>>>>>>>> ~[stormjar.jar:?]
>>>>>>>> at org.apache.metron.profiler.bolt.ProfileBuilderBolt.doExecute(ProfileBuilderBolt.java:164)
>>>>>>>> ~[stormjar.jar:?]
>>>>>>>> at org.apache.metron.profiler.bolt.ProfileBuilderBolt.execute(ProfileBuilderBolt.java:144)
>>>>>>>> [stormjar.jar:?]
>>>>>>>> at org.apache.storm.daemon.executor$fn__6571$tuple_action_fn__6573.invoke(executor.clj:734)
>>>>>>>> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>>>>>>>> at org.apache.storm.daemon.executor$mk_task_receiver$fn__6492.invoke(executor.clj:469)
>>>>>>>> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>>>>>>>> at org.apache.storm.disruptor$clojure_handler$reify__6005.onEvent(disruptor.clj:40)
>>>>>>>> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>>>>>>>> at org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:451)
>>>>>>>> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>>>>>>>> at org.apache.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:430)
>>>>>>>> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>>>>>>>> at org.apache.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:73)
>>>>>>>> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>>>>>>>> at org.apache.storm.daemon.executor$fn__6571$fn__6584$fn__6637.invoke(executor.clj:853)
>>>>>>>> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>>>>>>>> at org.apache.storm.util$async_loop$fn__554.invoke(util.clj:484)
>>>>>>>> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>>>>>>>> at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
>>>>>>>> at java.lang.Thread.run(Thread.java:745) [?:1.8.0_77]
>>>>>>>> Caused by: java.lang.ClassCastException:
>>>>>>>> org.apache.metron.statistics.outlier.MedianAbsoluteDeviationFunctions$State
>>>>>>>> cannot be cast to java.util.List
>>>>>>>> at org.apache.metron.statistics.outlier.MedianAbsoluteDeviation
>>>>>>>> Functions$StateUpdate.apply(MedianAbsoluteDeviationFunctions.java:85)
>>>>>>>> ~[stormjar.jar:?]
>>>>>>>> at org.apache.metron.common.stellar.StellarCompiler.exitTransfo
>>>>>>>> rmationFunc(StellarCompiler.java:246) ~[stormjar.jar:?]
>>>>>>>> at org.apache.metron.common.stellar.generated.StellarParser$Tra
>>>>>>>> nsformationFuncContext.exitRule(StellarParser.java:1617)
>>>>>>>> ~[stormjar.jar:?]
>>>>>>>> at org.antlr.v4.runtime.Parser.triggerExitRuleEvent(Parser.java:422)
>>>>>>>> ~[stormjar.jar:?]
>>>>>>>> at org.antlr.v4.runtime.Parser.exitRule(Parser.java:632)
>>>>>>>> ~[stormjar.jar:?]
>>>>>>>> at org.apache.metron.common.stellar.generated.StellarParser.fun
>>>>>>>> ctions(StellarParser.java:1640) ~[stormjar.jar:?]
>>>>>>>> at org.apache.metron.common.stellar.generated.StellarParser.ari
>>>>>>>> thmetic_operands(StellarParser.java:1750) ~[stormjar.jar:?]
>>>>>>>> at org.apache.metron.common.stellar.generated.StellarParser.ari
>>>>>>>> thmetic_expr_mul(StellarParser.java:1537) ~[stormjar.jar:?]
>>>>>>>> at org.apache.metron.common.stellar.generated.StellarParser.ari
>>>>>>>> thmetic_expr(StellarParser.java:1397) ~[stormjar.jar:?]
>>>>>>>> at org.apache.metron.common.stellar.generated.StellarParser.tra
>>>>>>>> nsformation_expr(StellarParser.java:278) ~[stormjar.jar:?]
>>>>>>>> at org.apache.metron.common.stellar.generated.StellarParser.tra
>>>>>>>> nsformation(StellarParser.java:147) ~[stormjar.jar:?]
>>>>>>>> at org.apache.metron.common.stellar.BaseStellarProcessor.parse(BaseStellarProcessor.java:89)
>>>>>>>> ~[stormjar.jar:?]
>>>>>>>> ... 19 more
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Mon, Feb 27, 2017 at 11:33 PM, Casey Stella <ce...@gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Any chance there's a full stacktrace associated with that NPE
>>>>>>>>> anywhere? Also, I assume this is 0.3.0, right?
>>>>>>>>>
>>>>>>>>> On Mon, Feb 27, 2017 at 5:29 PM, shoggi <sh...@gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>> Hi all
>>>>>>>>>>
>>>>>>>>>> Has anyone come across following profiler error, when trying to
>>>>>>>>>> work with the MAD functions? The configuration loads fine and I only get
>>>>>>>>>> the error when the profiler is pushing its data. I don't think its a typo
>>>>>>>>>> but maybe the logic is not sound.
>>>>>>>>>>
>>>>>>>>>> Error from the profiler worker:
>>>>>>>>>> 2017-02-27 23:20:43.368 o.a.m.p.b.ProfileBuilderBolt [INFO]
>>>>>>>>>> Flushing profile: profile=sketchy_mad, entity=krxd.net
>>>>>>>>>> 2017-02-27 23:20:43.368 o.a.m.p.b.ProfileBuilderBolt [ERROR]
>>>>>>>>>> Unexpected failure: message='null', tuple='source: __system:-1, stream:
>>>>>>>>>> __tick, id: {}, [900]'
>>>>>>>>>> java.lang.NullPointerException
>>>>>>>>>> 2017-02-27 23:20:43.368 o.a.s.d.executor [ERROR]
>>>>>>>>>> java.lang.NullPointerException
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> My Profiler config:
>>>>>>>>>>
>>>>>>>>>> {
>>>>>>>>>> "profile": "sketchy_mad",
>>>>>>>>>> "foreach": "if exists(domain_without_subdomains) then
>>>>>>>>>> domain_without_subdomains else 'n/a'",
>>>>>>>>>> "onlyif": "exists(domain_without_subdomains) &&
>>>>>>>>>> source.type == 'squid'",
>>>>>>>>>> "init" : {
>>>>>>>>>> "s": "OUTLIER_MAD_STATE_MERGE(PROFILE_GET('sketchy_mad',
>>>>>>>>>> domain_without_subdomains, 5, 'MINUTES'))"
>>>>>>>>>> },
>>>>>>>>>> "tickUpdate" : {
>>>>>>>>>> "s": "OUTLIER_MAD_STATE_MERGE(PROFILE_GET('sketchy_mad',
>>>>>>>>>> domain_without_subdomains, 5, 'MINUTES'), s)"
>>>>>>>>>> },
>>>>>>>>>> "update": {
>>>>>>>>>> "s": "OUTLIER_MAD_ADD(s, bytes)"
>>>>>>>>>> },
>>>>>>>>>> "result": "s"
>>>>>>>>>> }
>>>>>>>>>>
>>>>>>>>>> Regards
>>>>>>>>>> shoggi
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
Re: Median Absolute Deviation - Profiler Error
Posted by Casey Stella <ce...@gmail.com>.
So, I do recall in 0.3.0 a bug where if you modify a profile, it still
continues to run the old copy of the profile. I was going to have you
bounce the profiler topology. Did you do that? If not, then please do
it. If so, then are the NPE's the same? I've seen 2 distinct exceptions
in the stacktraces, one around casting a MAD$State objct to a List and
another as a real NPE.
Let's level-set, can you bounce the profiler topology and then send the NPE
exception and stack trace?
Casey
On Tue, Feb 28, 2017 at 3:56 AM, shoggi <sh...@gmail.com> wrote:
> I restarted Storm and ingest some data over the last few hours. I still
> get the casual NPE but some data gets profiled. It definitely skips data as
> I fed 160 unique tld's but only get 9 mad profile entries. The list seems
> random as I would least expect my top talkers to show up.
>
>
> [Stellar]>>> OUTLIER_MAD_STATE_MERGE(PROFILE_GET('sketchy_mad', 'krxd.net',
> 5, 'HOURS'))
> org.apache.metron.statistics.outlier.MedianAbsoluteDeviationFunctio
> ns$State@5b307115
> [Stellar]>>> PROFILE_GET('sketchy_mad', 'krxd.net', 5, 'HOURS')
> []
> [Stellar]>>> OUTLIER_MAD_STATE_MERGE(PROFILE_GET('sketchy_mad', 'krxd.net',
> 5, 'MINUTES'))
> org.apache.metron.statistics.outlier.MedianAbsoluteDeviationFunctio
> ns$State@714c575b
> [Stellar]>>> PROFILE_GET('sketchy_mad', 'krxd.net', 5, 'MINUTES')
> []
>
> [Stellar]>>> bytes := 700
> [Stellar]>>> OUTLIER_MAD_SCORE( OUTLIER_MAD_STATE_MERGE( PROFILE_GET(
> 'sketchy_mad', 'krxd.net', 5, 'MINUTES')), bytes)
> NaN
> [Stellar]>>> OUTLIER_MAD_SCORE( OUTLIER_MAD_STATE_MERGE( PROFILE_GET(
> 'sketchy_mad', 'krxd.net', 5, 'HOURS')), bytes)
> NaN
> [Stellar]>>> OUTLIER_MAD_SCORE( OUTLIER_MAD_STATE_MERGE( PROFILE_GET(
> 'sketchy_mad', 'krxd.net', 5, 'DAYS')), bytes)
> 3.8971111111111107
>
>
> # echo "scan 'profiler'" | hbase shell > /tmp/test.txt && egrep
> 'sketchy_mad' /tmp/test.txt | awk '{print $1}'
> \x00\x00\x00xsketchy_madoptimicdn.com\x00\x00\x00\x00\x00\x19;s
> \x00\x00\x01dsketchy_madajax.googleapis.com\x00\x00\x00\x00\x00\x19;g
> \x00\x00\x02\x86sketchy_madrlcdn.com\x00\x00\x00\x00\x00\x19;c
> \x00\x00\x02\x86sketchy_madshareaholic.com\x00\x00\x00\x00\x00\x19;c
> \x00\x00\x02\xDBsketchy_madnewsnetz.ch\x00\x00\x00\x00\x00\x19;k
> \xFF\xFF\xFC\x8Esketchy_madajax.googleapis.com\x00\x00\x00\x00\x00\x19;r
> \xFF\xFF\xFC\xEBsketchy_madkrxd.net\x00\x00\x00\x00\x00\x19;[
> \xFF\xFF\xFE\x87sketchy_madkrxd.net\x00\x00\x00\x00\x00\x19;^
> \xFF\xFF\xFE\xF2sketchy_madoptimicdn.com\x00\x00\x00\x00\x00\x19;\x5C
>
> On Tue, Feb 28, 2017 at 8:10 AM, Casey Stella <ce...@gmail.com> wrote:
>
>> Ok, from the Stellar REPL (started with -z and the zookeeper quorum),
>> what does the following yield:
>>
>> - OUTLIER_MAD_STATE_MERGE(PROFILE_GET('sketchy_mad', 'krxd.net', 5,
>> 'HOURS'))
>> - PROFILE_GET('sketchy_mad', 'krxd.net', 5, 'HOURS')
>> - OUTLIER_MAD_STATE_MERGE(PROFILE_GET('sketchy_mad', 'krxd.net', 5,
>> 'MINUTES'))
>> - PROFILE_GET('sketchy_mad', 'krxd.net', 5, 'MINUTES')
>>
>>
>> On Tue, Feb 28, 2017 at 1:36 AM, shoggi <sh...@gmail.com> wrote:
>>
>>> This is what I've got..
>>>
>>> {
>>> "profile": "sketchy_mad",
>>> "foreach": "if exists(domain_without_subdomains) then
>>> domain_without_subdomains else 'n/a'",
>>> "onlyif": "exists(domain_without_subdomains) && source.type ==
>>> 'squid'",
>>> "init" : {
>>> "s": "OUTLIER_MAD_STATE_MERGE(PROFILE_GET('sketchy_mad',
>>> domain_without_subdomains, 5, 'MINUTES'))"
>>> },
>>> "update": {
>>> "s": "OUTLIER_MAD_ADD(s, bytes)"
>>> },
>>> "result": "s"
>>> }
>>>
>>> On Tue, Feb 28, 2017 at 3:34 AM, Casey Stella <ce...@gmail.com>
>>> wrote:
>>>
>>>> Sorry, waht does the current profile config look like for that very
>>>> last error?
>>>>
>>>> On Mon, Feb 27, 2017 at 6:39 PM, shoggi <sh...@gmail.com> wrote:
>>>>
>>>>> Seems not to be the culprit..
>>>>>
>>>>>
>>>>> 2017-02-28 00:35:43.367 o.a.m.p.b.ProfileBuilderBolt [INFO] Flushing
>>>>> profile: profile=sketchy_mad, entity=optimicdn.com
>>>>> 2017-02-28 00:35:43.368 o.a.m.p.b.ProfileBuilderBolt [ERROR]
>>>>> Unexpected failure: message='Unable to execute:
>>>>> org.apache.metron.statistics.outlier.MedianAbsoluteDeviationFunctions$State
>>>>> cannot be cast to java.util.List', tuple='source: __system:-1, stream:
>>>>> __tick, id: {}, [900]'
>>>>> org.apache.metron.common.dsl.ParseException: Unable to execute:
>>>>> org.apache.metron.statistics.outlier.MedianAbsoluteDeviationFunctions$State
>>>>> cannot be cast to java.util.List
>>>>> at org.apache.metron.common.stellar.StellarCompiler.getResult(StellarCompiler.java:428)
>>>>> ~[stormjar.jar:?]
>>>>> at org.apache.metron.common.stellar.BaseStellarProcessor.parse(BaseStellarProcessor.java:90)
>>>>> ~[stormjar.jar:?]
>>>>> at org.apache.metron.profiler.stellar.DefaultStellarExecutor.ex
>>>>> ecute(DefaultStellarExecutor.java:148) ~[stormjar.jar:?]
>>>>> at org.apache.metron.profiler.stellar.DefaultStellarExecutor.as
>>>>> sign(DefaultStellarExecutor.java:94) ~[stormjar.jar:?]
>>>>> at org.apache.metron.profiler.bolt.ProfileBuilderBolt.lambda$nu
>>>>> ll$2(ProfileBuilderBolt.java:258) ~[stormjar.jar:?]
>>>>> at java.util.LinkedHashMap.forEach(LinkedHashMap.java:676)
>>>>> ~[?:1.8.0_77]
>>>>> at org.apache.metron.profiler.bolt.ProfileBuilderBolt.lambda$fl
>>>>> ush$4(ProfileBuilderBolt.java:258) ~[stormjar.jar:?]
>>>>> at java.util.concurrent.ConcurrentMap.forEach(ConcurrentMap.java:114)
>>>>> ~[?:1.8.0_77]
>>>>> at org.apache.metron.profiler.bolt.ProfileBuilderBolt.flush(ProfileBuilderBolt.java:237)
>>>>> ~[stormjar.jar:?]
>>>>> at org.apache.metron.profiler.bolt.ProfileBuilderBolt.doExecute(ProfileBuilderBolt.java:164)
>>>>> ~[stormjar.jar:?]
>>>>> at org.apache.metron.profiler.bolt.ProfileBuilderBolt.execute(ProfileBuilderBolt.java:144)
>>>>> [stormjar.jar:?]
>>>>> at org.apache.storm.daemon.executor$fn__6571$tuple_action_fn__6573.invoke(executor.clj:734)
>>>>> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>>>>> at org.apache.storm.daemon.executor$mk_task_receiver$fn__6492.invoke(executor.clj:469)
>>>>> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>>>>> at org.apache.storm.disruptor$clojure_handler$reify__6005.onEvent(disruptor.clj:40)
>>>>> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>>>>> at org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:451)
>>>>> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>>>>> at org.apache.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:430)
>>>>> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>>>>> at org.apache.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:73)
>>>>> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>>>>> at org.apache.storm.daemon.executor$fn__6571$fn__6584$fn__6637.invoke(executor.clj:853)
>>>>> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>>>>> at org.apache.storm.util$async_loop$fn__554.invoke(util.clj:484)
>>>>> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>>>>> at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
>>>>> at java.lang.Thread.run(Thread.java:745) [?:1.8.0_77]
>>>>> Caused by: java.lang.ClassCastException: org.apache.metron.statistics.o
>>>>> utlier.MedianAbsoluteDeviationFunctions$State cannot be cast to
>>>>> java.util.List
>>>>> at org.apache.metron.statistics.outlier.MedianAbsoluteDeviation
>>>>> Functions$StateUpdate.apply(MedianAbsoluteDeviationFunctions.java:85)
>>>>> ~[stormjar.jar:?]
>>>>> at org.apache.metron.common.stellar.StellarCompiler.exitTransfo
>>>>> rmationFunc(StellarCompiler.java:246) ~[stormjar.jar:?]
>>>>> at org.apache.metron.common.stellar.generated.StellarParser$Tra
>>>>> nsformationFuncContext.exitRule(StellarParser.java:1617)
>>>>> ~[stormjar.jar:?]
>>>>> at org.antlr.v4.runtime.Parser.triggerExitRuleEvent(Parser.java:422)
>>>>> ~[stormjar.jar:?]
>>>>> at org.antlr.v4.runtime.Parser.exitRule(Parser.java:632)
>>>>> ~[stormjar.jar:?]
>>>>> at org.apache.metron.common.stellar.generated.StellarParser.fun
>>>>> ctions(StellarParser.java:1640) ~[stormjar.jar:?]
>>>>> at org.apache.metron.common.stellar.generated.StellarParser.ari
>>>>> thmetic_operands(StellarParser.java:1750) ~[stormjar.jar:?]
>>>>> at org.apache.metron.common.stellar.generated.StellarParser.ari
>>>>> thmetic_expr_mul(StellarParser.java:1537) ~[stormjar.jar:?]
>>>>> at org.apache.metron.common.stellar.generated.StellarParser.ari
>>>>> thmetic_expr(StellarParser.java:1397) ~[stormjar.jar:?]
>>>>> at org.apache.metron.common.stellar.generated.StellarParser.tra
>>>>> nsformation_expr(StellarParser.java:278) ~[stormjar.jar:?]
>>>>> at org.apache.metron.common.stellar.generated.StellarParser.tra
>>>>> nsformation(StellarParser.java:147) ~[stormjar.jar:?]
>>>>> at org.apache.metron.common.stellar.BaseStellarProcessor.parse(BaseStellarProcessor.java:89)
>>>>> ~[stormjar.jar:?]
>>>>> ... 19 more
>>>>>
>>>>> On Tue, Feb 28, 2017 at 12:18 AM, Casey Stella <ce...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Ok, try something for me and tell me if it fixes it: adjust the
>>>>>> config to remove the tickUpdate (it didn't work anyway and we have since
>>>>>> removed it in 0.3.1).
>>>>>>
>>>>>>
>>>>>> On Mon, Feb 27, 2017 at 6:07 PM, shoggi <sh...@gmail.com> wrote:
>>>>>>
>>>>>>> Yes, this is 0.3.0.
>>>>>>>
>>>>>>> Here is the full trace.
>>>>>>>
>>>>>>> 2017-02-28 00:05:43.367 o.a.m.p.b.ProfileBuilderBolt [INFO] Flushing
>>>>>>> profile: profile=sketchy_mad, entity=krxd.net
>>>>>>> 2017-02-28 00:05:43.374 o.a.m.p.b.ProfileBuilderBolt [ERROR]
>>>>>>> Unexpected failure: message='Unable to execute:
>>>>>>> org.apache.metron.statistics.outlier.MedianAbsoluteDeviationFunctions$State
>>>>>>> cannot be cast to java.util.List', tuple='source: __system:-1, stream:
>>>>>>> __tick, id: {}, [900]'
>>>>>>> org.apache.metron.common.dsl.ParseException: Unable to execute:
>>>>>>> org.apache.metron.statistics.outlier.MedianAbsoluteDeviationFunctions$State
>>>>>>> cannot be cast to java.util.List
>>>>>>> at org.apache.metron.common.stellar.StellarCompiler.getResult(StellarCompiler.java:428)
>>>>>>> ~[stormjar.jar:?]
>>>>>>> at org.apache.metron.common.stellar.BaseStellarProcessor.parse(BaseStellarProcessor.java:90)
>>>>>>> ~[stormjar.jar:?]
>>>>>>> at org.apache.metron.profiler.stellar.DefaultStellarExecutor.ex
>>>>>>> ecute(DefaultStellarExecutor.java:148) ~[stormjar.jar:?]
>>>>>>> at org.apache.metron.profiler.stellar.DefaultStellarExecutor.as
>>>>>>> sign(DefaultStellarExecutor.java:94) ~[stormjar.jar:?]
>>>>>>> at org.apache.metron.profiler.bolt.ProfileBuilderBolt.lambda$nu
>>>>>>> ll$2(ProfileBuilderBolt.java:258) ~[stormjar.jar:?]
>>>>>>> at java.util.LinkedHashMap.forEach(LinkedHashMap.java:676)
>>>>>>> ~[?:1.8.0_77]
>>>>>>> at org.apache.metron.profiler.bolt.ProfileBuilderBolt.lambda$fl
>>>>>>> ush$4(ProfileBuilderBolt.java:258) ~[stormjar.jar:?]
>>>>>>> at java.util.concurrent.ConcurrentMap.forEach(ConcurrentMap.java:114)
>>>>>>> ~[?:1.8.0_77]
>>>>>>> at org.apache.metron.profiler.bolt.ProfileBuilderBolt.flush(ProfileBuilderBolt.java:237)
>>>>>>> ~[stormjar.jar:?]
>>>>>>> at org.apache.metron.profiler.bolt.ProfileBuilderBolt.doExecute(ProfileBuilderBolt.java:164)
>>>>>>> ~[stormjar.jar:?]
>>>>>>> at org.apache.metron.profiler.bolt.ProfileBuilderBolt.execute(ProfileBuilderBolt.java:144)
>>>>>>> [stormjar.jar:?]
>>>>>>> at org.apache.storm.daemon.executor$fn__6571$tuple_action_fn__6573.invoke(executor.clj:734)
>>>>>>> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>>>>>>> at org.apache.storm.daemon.executor$mk_task_receiver$fn__6492.invoke(executor.clj:469)
>>>>>>> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>>>>>>> at org.apache.storm.disruptor$clojure_handler$reify__6005.onEvent(disruptor.clj:40)
>>>>>>> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>>>>>>> at org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:451)
>>>>>>> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>>>>>>> at org.apache.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:430)
>>>>>>> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>>>>>>> at org.apache.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:73)
>>>>>>> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>>>>>>> at org.apache.storm.daemon.executor$fn__6571$fn__6584$fn__6637.invoke(executor.clj:853)
>>>>>>> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>>>>>>> at org.apache.storm.util$async_loop$fn__554.invoke(util.clj:484)
>>>>>>> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>>>>>>> at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
>>>>>>> at java.lang.Thread.run(Thread.java:745) [?:1.8.0_77]
>>>>>>> Caused by: java.lang.ClassCastException:
>>>>>>> org.apache.metron.statistics.outlier.MedianAbsoluteDeviationFunctions$State
>>>>>>> cannot be cast to java.util.List
>>>>>>> at org.apache.metron.statistics.outlier.MedianAbsoluteDeviation
>>>>>>> Functions$StateUpdate.apply(MedianAbsoluteDeviationFunctions.java:85)
>>>>>>> ~[stormjar.jar:?]
>>>>>>> at org.apache.metron.common.stellar.StellarCompiler.exitTransfo
>>>>>>> rmationFunc(StellarCompiler.java:246) ~[stormjar.jar:?]
>>>>>>> at org.apache.metron.common.stellar.generated.StellarParser$Tra
>>>>>>> nsformationFuncContext.exitRule(StellarParser.java:1617)
>>>>>>> ~[stormjar.jar:?]
>>>>>>> at org.antlr.v4.runtime.Parser.triggerExitRuleEvent(Parser.java:422)
>>>>>>> ~[stormjar.jar:?]
>>>>>>> at org.antlr.v4.runtime.Parser.exitRule(Parser.java:632)
>>>>>>> ~[stormjar.jar:?]
>>>>>>> at org.apache.metron.common.stellar.generated.StellarParser.fun
>>>>>>> ctions(StellarParser.java:1640) ~[stormjar.jar:?]
>>>>>>> at org.apache.metron.common.stellar.generated.StellarParser.ari
>>>>>>> thmetic_operands(StellarParser.java:1750) ~[stormjar.jar:?]
>>>>>>> at org.apache.metron.common.stellar.generated.StellarParser.ari
>>>>>>> thmetic_expr_mul(StellarParser.java:1537) ~[stormjar.jar:?]
>>>>>>> at org.apache.metron.common.stellar.generated.StellarParser.ari
>>>>>>> thmetic_expr(StellarParser.java:1397) ~[stormjar.jar:?]
>>>>>>> at org.apache.metron.common.stellar.generated.StellarParser.tra
>>>>>>> nsformation_expr(StellarParser.java:278) ~[stormjar.jar:?]
>>>>>>> at org.apache.metron.common.stellar.generated.StellarParser.tra
>>>>>>> nsformation(StellarParser.java:147) ~[stormjar.jar:?]
>>>>>>> at org.apache.metron.common.stellar.BaseStellarProcessor.parse(BaseStellarProcessor.java:89)
>>>>>>> ~[stormjar.jar:?]
>>>>>>> ... 19 more
>>>>>>> 2017-02-28 00:05:43.377 o.a.s.d.executor [ERROR]
>>>>>>> org.apache.metron.common.dsl.ParseException: Unable to execute:
>>>>>>> org.apache.metron.statistics.outlier.MedianAbsoluteDeviationFunctions$State
>>>>>>> cannot be cast to java.util.List
>>>>>>> at org.apache.metron.common.stellar.StellarCompiler.getResult(StellarCompiler.java:428)
>>>>>>> ~[stormjar.jar:?]
>>>>>>> at org.apache.metron.common.stellar.BaseStellarProcessor.parse(BaseStellarProcessor.java:90)
>>>>>>> ~[stormjar.jar:?]
>>>>>>> at org.apache.metron.profiler.stellar.DefaultStellarExecutor.ex
>>>>>>> ecute(DefaultStellarExecutor.java:148) ~[stormjar.jar:?]
>>>>>>> at org.apache.metron.profiler.stellar.DefaultStellarExecutor.as
>>>>>>> sign(DefaultStellarExecutor.java:94) ~[stormjar.jar:?]
>>>>>>> at org.apache.metron.profiler.bolt.ProfileBuilderBolt.lambda$nu
>>>>>>> ll$2(ProfileBuilderBolt.java:258) ~[stormjar.jar:?]
>>>>>>> at java.util.LinkedHashMap.forEach(LinkedHashMap.java:676)
>>>>>>> ~[?:1.8.0_77]
>>>>>>> at org.apache.metron.profiler.bolt.ProfileBuilderBolt.lambda$fl
>>>>>>> ush$4(ProfileBuilderBolt.java:258) ~[stormjar.jar:?]
>>>>>>> at java.util.concurrent.ConcurrentMap.forEach(ConcurrentMap.java:114)
>>>>>>> ~[?:1.8.0_77]
>>>>>>> at org.apache.metron.profiler.bolt.ProfileBuilderBolt.flush(ProfileBuilderBolt.java:237)
>>>>>>> ~[stormjar.jar:?]
>>>>>>> at org.apache.metron.profiler.bolt.ProfileBuilderBolt.doExecute(ProfileBuilderBolt.java:164)
>>>>>>> ~[stormjar.jar:?]
>>>>>>> at org.apache.metron.profiler.bolt.ProfileBuilderBolt.execute(ProfileBuilderBolt.java:144)
>>>>>>> [stormjar.jar:?]
>>>>>>> at org.apache.storm.daemon.executor$fn__6571$tuple_action_fn__6573.invoke(executor.clj:734)
>>>>>>> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>>>>>>> at org.apache.storm.daemon.executor$mk_task_receiver$fn__6492.invoke(executor.clj:469)
>>>>>>> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>>>>>>> at org.apache.storm.disruptor$clojure_handler$reify__6005.onEvent(disruptor.clj:40)
>>>>>>> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>>>>>>> at org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:451)
>>>>>>> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>>>>>>> at org.apache.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:430)
>>>>>>> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>>>>>>> at org.apache.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:73)
>>>>>>> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>>>>>>> at org.apache.storm.daemon.executor$fn__6571$fn__6584$fn__6637.invoke(executor.clj:853)
>>>>>>> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>>>>>>> at org.apache.storm.util$async_loop$fn__554.invoke(util.clj:484)
>>>>>>> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>>>>>>> at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
>>>>>>> at java.lang.Thread.run(Thread.java:745) [?:1.8.0_77]
>>>>>>> Caused by: java.lang.ClassCastException:
>>>>>>> org.apache.metron.statistics.outlier.MedianAbsoluteDeviationFunctions$State
>>>>>>> cannot be cast to java.util.List
>>>>>>> at org.apache.metron.statistics.outlier.MedianAbsoluteDeviation
>>>>>>> Functions$StateUpdate.apply(MedianAbsoluteDeviationFunctions.java:85)
>>>>>>> ~[stormjar.jar:?]
>>>>>>> at org.apache.metron.common.stellar.StellarCompiler.exitTransfo
>>>>>>> rmationFunc(StellarCompiler.java:246) ~[stormjar.jar:?]
>>>>>>> at org.apache.metron.common.stellar.generated.StellarParser$Tra
>>>>>>> nsformationFuncContext.exitRule(StellarParser.java:1617)
>>>>>>> ~[stormjar.jar:?]
>>>>>>> at org.antlr.v4.runtime.Parser.triggerExitRuleEvent(Parser.java:422)
>>>>>>> ~[stormjar.jar:?]
>>>>>>> at org.antlr.v4.runtime.Parser.exitRule(Parser.java:632)
>>>>>>> ~[stormjar.jar:?]
>>>>>>> at org.apache.metron.common.stellar.generated.StellarParser.fun
>>>>>>> ctions(StellarParser.java:1640) ~[stormjar.jar:?]
>>>>>>> at org.apache.metron.common.stellar.generated.StellarParser.ari
>>>>>>> thmetic_operands(StellarParser.java:1750) ~[stormjar.jar:?]
>>>>>>> at org.apache.metron.common.stellar.generated.StellarParser.ari
>>>>>>> thmetic_expr_mul(StellarParser.java:1537) ~[stormjar.jar:?]
>>>>>>> at org.apache.metron.common.stellar.generated.StellarParser.ari
>>>>>>> thmetic_expr(StellarParser.java:1397) ~[stormjar.jar:?]
>>>>>>> at org.apache.metron.common.stellar.generated.StellarParser.tra
>>>>>>> nsformation_expr(StellarParser.java:278) ~[stormjar.jar:?]
>>>>>>> at org.apache.metron.common.stellar.generated.StellarParser.tra
>>>>>>> nsformation(StellarParser.java:147) ~[stormjar.jar:?]
>>>>>>> at org.apache.metron.common.stellar.BaseStellarProcessor.parse(BaseStellarProcessor.java:89)
>>>>>>> ~[stormjar.jar:?]
>>>>>>> ... 19 more
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Mon, Feb 27, 2017 at 11:33 PM, Casey Stella <ce...@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Any chance there's a full stacktrace associated with that NPE
>>>>>>>> anywhere? Also, I assume this is 0.3.0, right?
>>>>>>>>
>>>>>>>> On Mon, Feb 27, 2017 at 5:29 PM, shoggi <sh...@gmail.com> wrote:
>>>>>>>>
>>>>>>>>> Hi all
>>>>>>>>>
>>>>>>>>> Has anyone come across following profiler error, when trying to
>>>>>>>>> work with the MAD functions? The configuration loads fine and I only get
>>>>>>>>> the error when the profiler is pushing its data. I don't think its a typo
>>>>>>>>> but maybe the logic is not sound.
>>>>>>>>>
>>>>>>>>> Error from the profiler worker:
>>>>>>>>> 2017-02-27 23:20:43.368 o.a.m.p.b.ProfileBuilderBolt [INFO]
>>>>>>>>> Flushing profile: profile=sketchy_mad, entity=krxd.net
>>>>>>>>> 2017-02-27 23:20:43.368 o.a.m.p.b.ProfileBuilderBolt [ERROR]
>>>>>>>>> Unexpected failure: message='null', tuple='source: __system:-1, stream:
>>>>>>>>> __tick, id: {}, [900]'
>>>>>>>>> java.lang.NullPointerException
>>>>>>>>> 2017-02-27 23:20:43.368 o.a.s.d.executor [ERROR]
>>>>>>>>> java.lang.NullPointerException
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> My Profiler config:
>>>>>>>>>
>>>>>>>>> {
>>>>>>>>> "profile": "sketchy_mad",
>>>>>>>>> "foreach": "if exists(domain_without_subdomains) then
>>>>>>>>> domain_without_subdomains else 'n/a'",
>>>>>>>>> "onlyif": "exists(domain_without_subdomains) && source.type
>>>>>>>>> == 'squid'",
>>>>>>>>> "init" : {
>>>>>>>>> "s": "OUTLIER_MAD_STATE_MERGE(PROFILE_GET('sketchy_mad',
>>>>>>>>> domain_without_subdomains, 5, 'MINUTES'))"
>>>>>>>>> },
>>>>>>>>> "tickUpdate" : {
>>>>>>>>> "s": "OUTLIER_MAD_STATE_MERGE(PROFILE_GET('sketchy_mad',
>>>>>>>>> domain_without_subdomains, 5, 'MINUTES'), s)"
>>>>>>>>> },
>>>>>>>>> "update": {
>>>>>>>>> "s": "OUTLIER_MAD_ADD(s, bytes)"
>>>>>>>>> },
>>>>>>>>> "result": "s"
>>>>>>>>> }
>>>>>>>>>
>>>>>>>>> Regards
>>>>>>>>> shoggi
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
Re: Median Absolute Deviation - Profiler Error
Posted by shoggi <sh...@gmail.com>.
I restarted Storm and ingest some data over the last few hours. I still get
the casual NPE but some data gets profiled. It definitely skips data as I
fed 160 unique tld's but only get 9 mad profile entries. The list seems
random as I would least expect my top talkers to show up.
[Stellar]>>> OUTLIER_MAD_STATE_MERGE(PROFILE_GET('sketchy_mad', 'krxd.net',
5, 'HOURS'))
org.apache.metron.statistics.outlier.MedianAbsoluteDeviationFunctions$State@5b307115
[Stellar]>>> PROFILE_GET('sketchy_mad', 'krxd.net', 5, 'HOURS')
[]
[Stellar]>>> OUTLIER_MAD_STATE_MERGE(PROFILE_GET('sketchy_mad', 'krxd.net',
5, 'MINUTES'))
org.apache.metron.statistics.outlier.MedianAbsoluteDeviationFunctions$State@714c575b
[Stellar]>>> PROFILE_GET('sketchy_mad', 'krxd.net', 5, 'MINUTES')
[]
[Stellar]>>> bytes := 700
[Stellar]>>> OUTLIER_MAD_SCORE( OUTLIER_MAD_STATE_MERGE( PROFILE_GET(
'sketchy_mad', 'krxd.net', 5, 'MINUTES')), bytes)
NaN
[Stellar]>>> OUTLIER_MAD_SCORE( OUTLIER_MAD_STATE_MERGE( PROFILE_GET(
'sketchy_mad', 'krxd.net', 5, 'HOURS')), bytes)
NaN
[Stellar]>>> OUTLIER_MAD_SCORE( OUTLIER_MAD_STATE_MERGE( PROFILE_GET(
'sketchy_mad', 'krxd.net', 5, 'DAYS')), bytes)
3.8971111111111107
# echo "scan 'profiler'" | hbase shell > /tmp/test.txt && egrep
'sketchy_mad' /tmp/test.txt | awk '{print $1}'
\x00\x00\x00xsketchy_madoptimicdn.com\x00\x00\x00\x00\x00\x19;s
\x00\x00\x01dsketchy_madajax.googleapis.com\x00\x00\x00\x00\x00\x19;g
\x00\x00\x02\x86sketchy_madrlcdn.com\x00\x00\x00\x00\x00\x19;c
\x00\x00\x02\x86sketchy_madshareaholic.com\x00\x00\x00\x00\x00\x19;c
\x00\x00\x02\xDBsketchy_madnewsnetz.ch\x00\x00\x00\x00\x00\x19;k
\xFF\xFF\xFC\x8Esketchy_madajax.googleapis.com\x00\x00\x00\x00\x00\x19;r
\xFF\xFF\xFC\xEBsketchy_madkrxd.net\x00\x00\x00\x00\x00\x19;[
\xFF\xFF\xFE\x87sketchy_madkrxd.net\x00\x00\x00\x00\x00\x19;^
\xFF\xFF\xFE\xF2sketchy_madoptimicdn.com\x00\x00\x00\x00\x00\x19;\x5C
On Tue, Feb 28, 2017 at 8:10 AM, Casey Stella <ce...@gmail.com> wrote:
> Ok, from the Stellar REPL (started with -z and the zookeeper quorum), what
> does the following yield:
>
> - OUTLIER_MAD_STATE_MERGE(PROFILE_GET('sketchy_mad', 'krxd.net', 5,
> 'HOURS'))
> - PROFILE_GET('sketchy_mad', 'krxd.net', 5, 'HOURS')
> - OUTLIER_MAD_STATE_MERGE(PROFILE_GET('sketchy_mad', 'krxd.net', 5,
> 'MINUTES'))
> - PROFILE_GET('sketchy_mad', 'krxd.net', 5, 'MINUTES')
>
>
> On Tue, Feb 28, 2017 at 1:36 AM, shoggi <sh...@gmail.com> wrote:
>
>> This is what I've got..
>>
>> {
>> "profile": "sketchy_mad",
>> "foreach": "if exists(domain_without_subdomains) then
>> domain_without_subdomains else 'n/a'",
>> "onlyif": "exists(domain_without_subdomains) && source.type ==
>> 'squid'",
>> "init" : {
>> "s": "OUTLIER_MAD_STATE_MERGE(PROFILE_GET('sketchy_mad',
>> domain_without_subdomains, 5, 'MINUTES'))"
>> },
>> "update": {
>> "s": "OUTLIER_MAD_ADD(s, bytes)"
>> },
>> "result": "s"
>> }
>>
>> On Tue, Feb 28, 2017 at 3:34 AM, Casey Stella <ce...@gmail.com> wrote:
>>
>>> Sorry, waht does the current profile config look like for that very last
>>> error?
>>>
>>> On Mon, Feb 27, 2017 at 6:39 PM, shoggi <sh...@gmail.com> wrote:
>>>
>>>> Seems not to be the culprit..
>>>>
>>>>
>>>> 2017-02-28 00:35:43.367 o.a.m.p.b.ProfileBuilderBolt [INFO] Flushing
>>>> profile: profile=sketchy_mad, entity=optimicdn.com
>>>> 2017-02-28 00:35:43.368 o.a.m.p.b.ProfileBuilderBolt [ERROR] Unexpected
>>>> failure: message='Unable to execute: org.apache.metron.statistics.o
>>>> utlier.MedianAbsoluteDeviationFunctions$State cannot be cast to
>>>> java.util.List', tuple='source: __system:-1, stream: __tick, id: {}, [900]'
>>>> org.apache.metron.common.dsl.ParseException: Unable to execute:
>>>> org.apache.metron.statistics.outlier.MedianAbsoluteDeviationFunctions$State
>>>> cannot be cast to java.util.List
>>>> at org.apache.metron.common.stellar.StellarCompiler.getResult(StellarCompiler.java:428)
>>>> ~[stormjar.jar:?]
>>>> at org.apache.metron.common.stellar.BaseStellarProcessor.parse(BaseStellarProcessor.java:90)
>>>> ~[stormjar.jar:?]
>>>> at org.apache.metron.profiler.stellar.DefaultStellarExecutor.ex
>>>> ecute(DefaultStellarExecutor.java:148) ~[stormjar.jar:?]
>>>> at org.apache.metron.profiler.stellar.DefaultStellarExecutor.as
>>>> sign(DefaultStellarExecutor.java:94) ~[stormjar.jar:?]
>>>> at org.apache.metron.profiler.bolt.ProfileBuilderBolt.lambda$nu
>>>> ll$2(ProfileBuilderBolt.java:258) ~[stormjar.jar:?]
>>>> at java.util.LinkedHashMap.forEach(LinkedHashMap.java:676)
>>>> ~[?:1.8.0_77]
>>>> at org.apache.metron.profiler.bolt.ProfileBuilderBolt.lambda$fl
>>>> ush$4(ProfileBuilderBolt.java:258) ~[stormjar.jar:?]
>>>> at java.util.concurrent.ConcurrentMap.forEach(ConcurrentMap.java:114)
>>>> ~[?:1.8.0_77]
>>>> at org.apache.metron.profiler.bolt.ProfileBuilderBolt.flush(ProfileBuilderBolt.java:237)
>>>> ~[stormjar.jar:?]
>>>> at org.apache.metron.profiler.bolt.ProfileBuilderBolt.doExecute(ProfileBuilderBolt.java:164)
>>>> ~[stormjar.jar:?]
>>>> at org.apache.metron.profiler.bolt.ProfileBuilderBolt.execute(ProfileBuilderBolt.java:144)
>>>> [stormjar.jar:?]
>>>> at org.apache.storm.daemon.executor$fn__6571$tuple_action_fn__6573.invoke(executor.clj:734)
>>>> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>>>> at org.apache.storm.daemon.executor$mk_task_receiver$fn__6492.invoke(executor.clj:469)
>>>> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>>>> at org.apache.storm.disruptor$clojure_handler$reify__6005.onEvent(disruptor.clj:40)
>>>> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>>>> at org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:451)
>>>> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>>>> at org.apache.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:430)
>>>> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>>>> at org.apache.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:73)
>>>> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>>>> at org.apache.storm.daemon.executor$fn__6571$fn__6584$fn__6637.invoke(executor.clj:853)
>>>> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>>>> at org.apache.storm.util$async_loop$fn__554.invoke(util.clj:484)
>>>> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>>>> at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
>>>> at java.lang.Thread.run(Thread.java:745) [?:1.8.0_77]
>>>> Caused by: java.lang.ClassCastException: org.apache.metron.statistics.o
>>>> utlier.MedianAbsoluteDeviationFunctions$State cannot be cast to
>>>> java.util.List
>>>> at org.apache.metron.statistics.outlier.MedianAbsoluteDeviation
>>>> Functions$StateUpdate.apply(MedianAbsoluteDeviationFunctions.java:85)
>>>> ~[stormjar.jar:?]
>>>> at org.apache.metron.common.stellar.StellarCompiler.exitTransfo
>>>> rmationFunc(StellarCompiler.java:246) ~[stormjar.jar:?]
>>>> at org.apache.metron.common.stellar.generated.StellarParser$Tra
>>>> nsformationFuncContext.exitRule(StellarParser.java:1617)
>>>> ~[stormjar.jar:?]
>>>> at org.antlr.v4.runtime.Parser.triggerExitRuleEvent(Parser.java:422)
>>>> ~[stormjar.jar:?]
>>>> at org.antlr.v4.runtime.Parser.exitRule(Parser.java:632)
>>>> ~[stormjar.jar:?]
>>>> at org.apache.metron.common.stellar.generated.StellarParser.fun
>>>> ctions(StellarParser.java:1640) ~[stormjar.jar:?]
>>>> at org.apache.metron.common.stellar.generated.StellarParser.ari
>>>> thmetic_operands(StellarParser.java:1750) ~[stormjar.jar:?]
>>>> at org.apache.metron.common.stellar.generated.StellarParser.ari
>>>> thmetic_expr_mul(StellarParser.java:1537) ~[stormjar.jar:?]
>>>> at org.apache.metron.common.stellar.generated.StellarParser.ari
>>>> thmetic_expr(StellarParser.java:1397) ~[stormjar.jar:?]
>>>> at org.apache.metron.common.stellar.generated.StellarParser.tra
>>>> nsformation_expr(StellarParser.java:278) ~[stormjar.jar:?]
>>>> at org.apache.metron.common.stellar.generated.StellarParser.tra
>>>> nsformation(StellarParser.java:147) ~[stormjar.jar:?]
>>>> at org.apache.metron.common.stellar.BaseStellarProcessor.parse(BaseStellarProcessor.java:89)
>>>> ~[stormjar.jar:?]
>>>> ... 19 more
>>>>
>>>> On Tue, Feb 28, 2017 at 12:18 AM, Casey Stella <ce...@gmail.com>
>>>> wrote:
>>>>
>>>>> Ok, try something for me and tell me if it fixes it: adjust the config
>>>>> to remove the tickUpdate (it didn't work anyway and we have since removed
>>>>> it in 0.3.1).
>>>>>
>>>>>
>>>>> On Mon, Feb 27, 2017 at 6:07 PM, shoggi <sh...@gmail.com> wrote:
>>>>>
>>>>>> Yes, this is 0.3.0.
>>>>>>
>>>>>> Here is the full trace.
>>>>>>
>>>>>> 2017-02-28 00:05:43.367 o.a.m.p.b.ProfileBuilderBolt [INFO] Flushing
>>>>>> profile: profile=sketchy_mad, entity=krxd.net
>>>>>> 2017-02-28 00:05:43.374 o.a.m.p.b.ProfileBuilderBolt [ERROR]
>>>>>> Unexpected failure: message='Unable to execute:
>>>>>> org.apache.metron.statistics.outlier.MedianAbsoluteDeviationFunctions$State
>>>>>> cannot be cast to java.util.List', tuple='source: __system:-1, stream:
>>>>>> __tick, id: {}, [900]'
>>>>>> org.apache.metron.common.dsl.ParseException: Unable to execute:
>>>>>> org.apache.metron.statistics.outlier.MedianAbsoluteDeviationFunctions$State
>>>>>> cannot be cast to java.util.List
>>>>>> at org.apache.metron.common.stellar.StellarCompiler.getResult(StellarCompiler.java:428)
>>>>>> ~[stormjar.jar:?]
>>>>>> at org.apache.metron.common.stellar.BaseStellarProcessor.parse(BaseStellarProcessor.java:90)
>>>>>> ~[stormjar.jar:?]
>>>>>> at org.apache.metron.profiler.stellar.DefaultStellarExecutor.ex
>>>>>> ecute(DefaultStellarExecutor.java:148) ~[stormjar.jar:?]
>>>>>> at org.apache.metron.profiler.stellar.DefaultStellarExecutor.as
>>>>>> sign(DefaultStellarExecutor.java:94) ~[stormjar.jar:?]
>>>>>> at org.apache.metron.profiler.bolt.ProfileBuilderBolt.lambda$nu
>>>>>> ll$2(ProfileBuilderBolt.java:258) ~[stormjar.jar:?]
>>>>>> at java.util.LinkedHashMap.forEach(LinkedHashMap.java:676)
>>>>>> ~[?:1.8.0_77]
>>>>>> at org.apache.metron.profiler.bolt.ProfileBuilderBolt.lambda$fl
>>>>>> ush$4(ProfileBuilderBolt.java:258) ~[stormjar.jar:?]
>>>>>> at java.util.concurrent.ConcurrentMap.forEach(ConcurrentMap.java:114)
>>>>>> ~[?:1.8.0_77]
>>>>>> at org.apache.metron.profiler.bolt.ProfileBuilderBolt.flush(ProfileBuilderBolt.java:237)
>>>>>> ~[stormjar.jar:?]
>>>>>> at org.apache.metron.profiler.bolt.ProfileBuilderBolt.doExecute(ProfileBuilderBolt.java:164)
>>>>>> ~[stormjar.jar:?]
>>>>>> at org.apache.metron.profiler.bolt.ProfileBuilderBolt.execute(ProfileBuilderBolt.java:144)
>>>>>> [stormjar.jar:?]
>>>>>> at org.apache.storm.daemon.executor$fn__6571$tuple_action_fn__6573.invoke(executor.clj:734)
>>>>>> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>>>>>> at org.apache.storm.daemon.executor$mk_task_receiver$fn__6492.invoke(executor.clj:469)
>>>>>> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>>>>>> at org.apache.storm.disruptor$clojure_handler$reify__6005.onEvent(disruptor.clj:40)
>>>>>> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>>>>>> at org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:451)
>>>>>> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>>>>>> at org.apache.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:430)
>>>>>> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>>>>>> at org.apache.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:73)
>>>>>> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>>>>>> at org.apache.storm.daemon.executor$fn__6571$fn__6584$fn__6637.invoke(executor.clj:853)
>>>>>> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>>>>>> at org.apache.storm.util$async_loop$fn__554.invoke(util.clj:484)
>>>>>> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>>>>>> at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
>>>>>> at java.lang.Thread.run(Thread.java:745) [?:1.8.0_77]
>>>>>> Caused by: java.lang.ClassCastException:
>>>>>> org.apache.metron.statistics.outlier.MedianAbsoluteDeviationFunctions$State
>>>>>> cannot be cast to java.util.List
>>>>>> at org.apache.metron.statistics.outlier.MedianAbsoluteDeviation
>>>>>> Functions$StateUpdate.apply(MedianAbsoluteDeviationFunctions.java:85)
>>>>>> ~[stormjar.jar:?]
>>>>>> at org.apache.metron.common.stellar.StellarCompiler.exitTransfo
>>>>>> rmationFunc(StellarCompiler.java:246) ~[stormjar.jar:?]
>>>>>> at org.apache.metron.common.stellar.generated.StellarParser$Tra
>>>>>> nsformationFuncContext.exitRule(StellarParser.java:1617)
>>>>>> ~[stormjar.jar:?]
>>>>>> at org.antlr.v4.runtime.Parser.triggerExitRuleEvent(Parser.java:422)
>>>>>> ~[stormjar.jar:?]
>>>>>> at org.antlr.v4.runtime.Parser.exitRule(Parser.java:632)
>>>>>> ~[stormjar.jar:?]
>>>>>> at org.apache.metron.common.stellar.generated.StellarParser.fun
>>>>>> ctions(StellarParser.java:1640) ~[stormjar.jar:?]
>>>>>> at org.apache.metron.common.stellar.generated.StellarParser.ari
>>>>>> thmetic_operands(StellarParser.java:1750) ~[stormjar.jar:?]
>>>>>> at org.apache.metron.common.stellar.generated.StellarParser.ari
>>>>>> thmetic_expr_mul(StellarParser.java:1537) ~[stormjar.jar:?]
>>>>>> at org.apache.metron.common.stellar.generated.StellarParser.ari
>>>>>> thmetic_expr(StellarParser.java:1397) ~[stormjar.jar:?]
>>>>>> at org.apache.metron.common.stellar.generated.StellarParser.tra
>>>>>> nsformation_expr(StellarParser.java:278) ~[stormjar.jar:?]
>>>>>> at org.apache.metron.common.stellar.generated.StellarParser.tra
>>>>>> nsformation(StellarParser.java:147) ~[stormjar.jar:?]
>>>>>> at org.apache.metron.common.stellar.BaseStellarProcessor.parse(BaseStellarProcessor.java:89)
>>>>>> ~[stormjar.jar:?]
>>>>>> ... 19 more
>>>>>> 2017-02-28 00:05:43.377 o.a.s.d.executor [ERROR]
>>>>>> org.apache.metron.common.dsl.ParseException: Unable to execute:
>>>>>> org.apache.metron.statistics.outlier.MedianAbsoluteDeviationFunctions$State
>>>>>> cannot be cast to java.util.List
>>>>>> at org.apache.metron.common.stellar.StellarCompiler.getResult(StellarCompiler.java:428)
>>>>>> ~[stormjar.jar:?]
>>>>>> at org.apache.metron.common.stellar.BaseStellarProcessor.parse(BaseStellarProcessor.java:90)
>>>>>> ~[stormjar.jar:?]
>>>>>> at org.apache.metron.profiler.stellar.DefaultStellarExecutor.ex
>>>>>> ecute(DefaultStellarExecutor.java:148) ~[stormjar.jar:?]
>>>>>> at org.apache.metron.profiler.stellar.DefaultStellarExecutor.as
>>>>>> sign(DefaultStellarExecutor.java:94) ~[stormjar.jar:?]
>>>>>> at org.apache.metron.profiler.bolt.ProfileBuilderBolt.lambda$nu
>>>>>> ll$2(ProfileBuilderBolt.java:258) ~[stormjar.jar:?]
>>>>>> at java.util.LinkedHashMap.forEach(LinkedHashMap.java:676)
>>>>>> ~[?:1.8.0_77]
>>>>>> at org.apache.metron.profiler.bolt.ProfileBuilderBolt.lambda$fl
>>>>>> ush$4(ProfileBuilderBolt.java:258) ~[stormjar.jar:?]
>>>>>> at java.util.concurrent.ConcurrentMap.forEach(ConcurrentMap.java:114)
>>>>>> ~[?:1.8.0_77]
>>>>>> at org.apache.metron.profiler.bolt.ProfileBuilderBolt.flush(ProfileBuilderBolt.java:237)
>>>>>> ~[stormjar.jar:?]
>>>>>> at org.apache.metron.profiler.bolt.ProfileBuilderBolt.doExecute(ProfileBuilderBolt.java:164)
>>>>>> ~[stormjar.jar:?]
>>>>>> at org.apache.metron.profiler.bolt.ProfileBuilderBolt.execute(ProfileBuilderBolt.java:144)
>>>>>> [stormjar.jar:?]
>>>>>> at org.apache.storm.daemon.executor$fn__6571$tuple_action_fn__6573.invoke(executor.clj:734)
>>>>>> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>>>>>> at org.apache.storm.daemon.executor$mk_task_receiver$fn__6492.invoke(executor.clj:469)
>>>>>> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>>>>>> at org.apache.storm.disruptor$clojure_handler$reify__6005.onEvent(disruptor.clj:40)
>>>>>> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>>>>>> at org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:451)
>>>>>> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>>>>>> at org.apache.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:430)
>>>>>> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>>>>>> at org.apache.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:73)
>>>>>> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>>>>>> at org.apache.storm.daemon.executor$fn__6571$fn__6584$fn__6637.invoke(executor.clj:853)
>>>>>> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>>>>>> at org.apache.storm.util$async_loop$fn__554.invoke(util.clj:484)
>>>>>> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>>>>>> at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
>>>>>> at java.lang.Thread.run(Thread.java:745) [?:1.8.0_77]
>>>>>> Caused by: java.lang.ClassCastException:
>>>>>> org.apache.metron.statistics.outlier.MedianAbsoluteDeviationFunctions$State
>>>>>> cannot be cast to java.util.List
>>>>>> at org.apache.metron.statistics.outlier.MedianAbsoluteDeviation
>>>>>> Functions$StateUpdate.apply(MedianAbsoluteDeviationFunctions.java:85)
>>>>>> ~[stormjar.jar:?]
>>>>>> at org.apache.metron.common.stellar.StellarCompiler.exitTransfo
>>>>>> rmationFunc(StellarCompiler.java:246) ~[stormjar.jar:?]
>>>>>> at org.apache.metron.common.stellar.generated.StellarParser$Tra
>>>>>> nsformationFuncContext.exitRule(StellarParser.java:1617)
>>>>>> ~[stormjar.jar:?]
>>>>>> at org.antlr.v4.runtime.Parser.triggerExitRuleEvent(Parser.java:422)
>>>>>> ~[stormjar.jar:?]
>>>>>> at org.antlr.v4.runtime.Parser.exitRule(Parser.java:632)
>>>>>> ~[stormjar.jar:?]
>>>>>> at org.apache.metron.common.stellar.generated.StellarParser.fun
>>>>>> ctions(StellarParser.java:1640) ~[stormjar.jar:?]
>>>>>> at org.apache.metron.common.stellar.generated.StellarParser.ari
>>>>>> thmetic_operands(StellarParser.java:1750) ~[stormjar.jar:?]
>>>>>> at org.apache.metron.common.stellar.generated.StellarParser.ari
>>>>>> thmetic_expr_mul(StellarParser.java:1537) ~[stormjar.jar:?]
>>>>>> at org.apache.metron.common.stellar.generated.StellarParser.ari
>>>>>> thmetic_expr(StellarParser.java:1397) ~[stormjar.jar:?]
>>>>>> at org.apache.metron.common.stellar.generated.StellarParser.tra
>>>>>> nsformation_expr(StellarParser.java:278) ~[stormjar.jar:?]
>>>>>> at org.apache.metron.common.stellar.generated.StellarParser.tra
>>>>>> nsformation(StellarParser.java:147) ~[stormjar.jar:?]
>>>>>> at org.apache.metron.common.stellar.BaseStellarProcessor.parse(BaseStellarProcessor.java:89)
>>>>>> ~[stormjar.jar:?]
>>>>>> ... 19 more
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Mon, Feb 27, 2017 at 11:33 PM, Casey Stella <ce...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Any chance there's a full stacktrace associated with that NPE
>>>>>>> anywhere? Also, I assume this is 0.3.0, right?
>>>>>>>
>>>>>>> On Mon, Feb 27, 2017 at 5:29 PM, shoggi <sh...@gmail.com> wrote:
>>>>>>>
>>>>>>>> Hi all
>>>>>>>>
>>>>>>>> Has anyone come across following profiler error, when trying to
>>>>>>>> work with the MAD functions? The configuration loads fine and I only get
>>>>>>>> the error when the profiler is pushing its data. I don't think its a typo
>>>>>>>> but maybe the logic is not sound.
>>>>>>>>
>>>>>>>> Error from the profiler worker:
>>>>>>>> 2017-02-27 23:20:43.368 o.a.m.p.b.ProfileBuilderBolt [INFO]
>>>>>>>> Flushing profile: profile=sketchy_mad, entity=krxd.net
>>>>>>>> 2017-02-27 23:20:43.368 o.a.m.p.b.ProfileBuilderBolt [ERROR]
>>>>>>>> Unexpected failure: message='null', tuple='source: __system:-1, stream:
>>>>>>>> __tick, id: {}, [900]'
>>>>>>>> java.lang.NullPointerException
>>>>>>>> 2017-02-27 23:20:43.368 o.a.s.d.executor [ERROR]
>>>>>>>> java.lang.NullPointerException
>>>>>>>>
>>>>>>>>
>>>>>>>> My Profiler config:
>>>>>>>>
>>>>>>>> {
>>>>>>>> "profile": "sketchy_mad",
>>>>>>>> "foreach": "if exists(domain_without_subdomains) then
>>>>>>>> domain_without_subdomains else 'n/a'",
>>>>>>>> "onlyif": "exists(domain_without_subdomains) && source.type
>>>>>>>> == 'squid'",
>>>>>>>> "init" : {
>>>>>>>> "s": "OUTLIER_MAD_STATE_MERGE(PROFILE_GET('sketchy_mad',
>>>>>>>> domain_without_subdomains, 5, 'MINUTES'))"
>>>>>>>> },
>>>>>>>> "tickUpdate" : {
>>>>>>>> "s": "OUTLIER_MAD_STATE_MERGE(PROFILE_GET('sketchy_mad',
>>>>>>>> domain_without_subdomains, 5, 'MINUTES'), s)"
>>>>>>>> },
>>>>>>>> "update": {
>>>>>>>> "s": "OUTLIER_MAD_ADD(s, bytes)"
>>>>>>>> },
>>>>>>>> "result": "s"
>>>>>>>> }
>>>>>>>>
>>>>>>>> Regards
>>>>>>>> shoggi
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
Re: Median Absolute Deviation - Profiler Error
Posted by Casey Stella <ce...@gmail.com>.
Ok, from the Stellar REPL (started with -z and the zookeeper quorum), what
does the following yield:
- OUTLIER_MAD_STATE_MERGE(PROFILE_GET('sketchy_mad', 'krxd.net', 5,
'HOURS'))
- PROFILE_GET('sketchy_mad', 'krxd.net', 5, 'HOURS')
- OUTLIER_MAD_STATE_MERGE(PROFILE_GET('sketchy_mad', 'krxd.net', 5,
'MINUTES'))
- PROFILE_GET('sketchy_mad', 'krxd.net', 5, 'MINUTES')
On Tue, Feb 28, 2017 at 1:36 AM, shoggi <sh...@gmail.com> wrote:
> This is what I've got..
>
> {
> "profile": "sketchy_mad",
> "foreach": "if exists(domain_without_subdomains) then
> domain_without_subdomains else 'n/a'",
> "onlyif": "exists(domain_without_subdomains) && source.type ==
> 'squid'",
> "init" : {
> "s": "OUTLIER_MAD_STATE_MERGE(PROFILE_GET('sketchy_mad',
> domain_without_subdomains, 5, 'MINUTES'))"
> },
> "update": {
> "s": "OUTLIER_MAD_ADD(s, bytes)"
> },
> "result": "s"
> }
>
> On Tue, Feb 28, 2017 at 3:34 AM, Casey Stella <ce...@gmail.com> wrote:
>
>> Sorry, waht does the current profile config look like for that very last
>> error?
>>
>> On Mon, Feb 27, 2017 at 6:39 PM, shoggi <sh...@gmail.com> wrote:
>>
>>> Seems not to be the culprit..
>>>
>>>
>>> 2017-02-28 00:35:43.367 o.a.m.p.b.ProfileBuilderBolt [INFO] Flushing
>>> profile: profile=sketchy_mad, entity=optimicdn.com
>>> 2017-02-28 00:35:43.368 o.a.m.p.b.ProfileBuilderBolt [ERROR] Unexpected
>>> failure: message='Unable to execute: org.apache.metron.statistics.o
>>> utlier.MedianAbsoluteDeviationFunctions$State cannot be cast to
>>> java.util.List', tuple='source: __system:-1, stream: __tick, id: {}, [900]'
>>> org.apache.metron.common.dsl.ParseException: Unable to execute:
>>> org.apache.metron.statistics.outlier.MedianAbsoluteDeviationFunctions$State
>>> cannot be cast to java.util.List
>>> at org.apache.metron.common.stellar.StellarCompiler.getResult(StellarCompiler.java:428)
>>> ~[stormjar.jar:?]
>>> at org.apache.metron.common.stellar.BaseStellarProcessor.parse(BaseStellarProcessor.java:90)
>>> ~[stormjar.jar:?]
>>> at org.apache.metron.profiler.stellar.DefaultStellarExecutor.ex
>>> ecute(DefaultStellarExecutor.java:148) ~[stormjar.jar:?]
>>> at org.apache.metron.profiler.stellar.DefaultStellarExecutor.as
>>> sign(DefaultStellarExecutor.java:94) ~[stormjar.jar:?]
>>> at org.apache.metron.profiler.bolt.ProfileBuilderBolt.lambda$nu
>>> ll$2(ProfileBuilderBolt.java:258) ~[stormjar.jar:?]
>>> at java.util.LinkedHashMap.forEach(LinkedHashMap.java:676) ~[?:1.8.0_77]
>>> at org.apache.metron.profiler.bolt.ProfileBuilderBolt.lambda$fl
>>> ush$4(ProfileBuilderBolt.java:258) ~[stormjar.jar:?]
>>> at java.util.concurrent.ConcurrentMap.forEach(ConcurrentMap.java:114)
>>> ~[?:1.8.0_77]
>>> at org.apache.metron.profiler.bolt.ProfileBuilderBolt.flush(ProfileBuilderBolt.java:237)
>>> ~[stormjar.jar:?]
>>> at org.apache.metron.profiler.bolt.ProfileBuilderBolt.doExecute(ProfileBuilderBolt.java:164)
>>> ~[stormjar.jar:?]
>>> at org.apache.metron.profiler.bolt.ProfileBuilderBolt.execute(ProfileBuilderBolt.java:144)
>>> [stormjar.jar:?]
>>> at org.apache.storm.daemon.executor$fn__6571$tuple_action_fn__6573.invoke(executor.clj:734)
>>> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>>> at org.apache.storm.daemon.executor$mk_task_receiver$fn__6492.invoke(executor.clj:469)
>>> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>>> at org.apache.storm.disruptor$clojure_handler$reify__6005.onEvent(disruptor.clj:40)
>>> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>>> at org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:451)
>>> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>>> at org.apache.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:430)
>>> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>>> at org.apache.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:73)
>>> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>>> at org.apache.storm.daemon.executor$fn__6571$fn__6584$fn__6637.invoke(executor.clj:853)
>>> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>>> at org.apache.storm.util$async_loop$fn__554.invoke(util.clj:484)
>>> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>>> at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
>>> at java.lang.Thread.run(Thread.java:745) [?:1.8.0_77]
>>> Caused by: java.lang.ClassCastException: org.apache.metron.statistics.o
>>> utlier.MedianAbsoluteDeviationFunctions$State cannot be cast to
>>> java.util.List
>>> at org.apache.metron.statistics.outlier.MedianAbsoluteDeviation
>>> Functions$StateUpdate.apply(MedianAbsoluteDeviationFunctions.java:85)
>>> ~[stormjar.jar:?]
>>> at org.apache.metron.common.stellar.StellarCompiler.exitTransfo
>>> rmationFunc(StellarCompiler.java:246) ~[stormjar.jar:?]
>>> at org.apache.metron.common.stellar.generated.StellarParser$Tra
>>> nsformationFuncContext.exitRule(StellarParser.java:1617)
>>> ~[stormjar.jar:?]
>>> at org.antlr.v4.runtime.Parser.triggerExitRuleEvent(Parser.java:422)
>>> ~[stormjar.jar:?]
>>> at org.antlr.v4.runtime.Parser.exitRule(Parser.java:632)
>>> ~[stormjar.jar:?]
>>> at org.apache.metron.common.stellar.generated.StellarParser.fun
>>> ctions(StellarParser.java:1640) ~[stormjar.jar:?]
>>> at org.apache.metron.common.stellar.generated.StellarParser.ari
>>> thmetic_operands(StellarParser.java:1750) ~[stormjar.jar:?]
>>> at org.apache.metron.common.stellar.generated.StellarParser.ari
>>> thmetic_expr_mul(StellarParser.java:1537) ~[stormjar.jar:?]
>>> at org.apache.metron.common.stellar.generated.StellarParser.ari
>>> thmetic_expr(StellarParser.java:1397) ~[stormjar.jar:?]
>>> at org.apache.metron.common.stellar.generated.StellarParser.tra
>>> nsformation_expr(StellarParser.java:278) ~[stormjar.jar:?]
>>> at org.apache.metron.common.stellar.generated.StellarParser.tra
>>> nsformation(StellarParser.java:147) ~[stormjar.jar:?]
>>> at org.apache.metron.common.stellar.BaseStellarProcessor.parse(BaseStellarProcessor.java:89)
>>> ~[stormjar.jar:?]
>>> ... 19 more
>>>
>>> On Tue, Feb 28, 2017 at 12:18 AM, Casey Stella <ce...@gmail.com>
>>> wrote:
>>>
>>>> Ok, try something for me and tell me if it fixes it: adjust the config
>>>> to remove the tickUpdate (it didn't work anyway and we have since removed
>>>> it in 0.3.1).
>>>>
>>>>
>>>> On Mon, Feb 27, 2017 at 6:07 PM, shoggi <sh...@gmail.com> wrote:
>>>>
>>>>> Yes, this is 0.3.0.
>>>>>
>>>>> Here is the full trace.
>>>>>
>>>>> 2017-02-28 00:05:43.367 o.a.m.p.b.ProfileBuilderBolt [INFO] Flushing
>>>>> profile: profile=sketchy_mad, entity=krxd.net
>>>>> 2017-02-28 00:05:43.374 o.a.m.p.b.ProfileBuilderBolt [ERROR]
>>>>> Unexpected failure: message='Unable to execute:
>>>>> org.apache.metron.statistics.outlier.MedianAbsoluteDeviationFunctions$State
>>>>> cannot be cast to java.util.List', tuple='source: __system:-1, stream:
>>>>> __tick, id: {}, [900]'
>>>>> org.apache.metron.common.dsl.ParseException: Unable to execute:
>>>>> org.apache.metron.statistics.outlier.MedianAbsoluteDeviationFunctions$State
>>>>> cannot be cast to java.util.List
>>>>> at org.apache.metron.common.stellar.StellarCompiler.getResult(StellarCompiler.java:428)
>>>>> ~[stormjar.jar:?]
>>>>> at org.apache.metron.common.stellar.BaseStellarProcessor.parse(BaseStellarProcessor.java:90)
>>>>> ~[stormjar.jar:?]
>>>>> at org.apache.metron.profiler.stellar.DefaultStellarExecutor.ex
>>>>> ecute(DefaultStellarExecutor.java:148) ~[stormjar.jar:?]
>>>>> at org.apache.metron.profiler.stellar.DefaultStellarExecutor.as
>>>>> sign(DefaultStellarExecutor.java:94) ~[stormjar.jar:?]
>>>>> at org.apache.metron.profiler.bolt.ProfileBuilderBolt.lambda$nu
>>>>> ll$2(ProfileBuilderBolt.java:258) ~[stormjar.jar:?]
>>>>> at java.util.LinkedHashMap.forEach(LinkedHashMap.java:676)
>>>>> ~[?:1.8.0_77]
>>>>> at org.apache.metron.profiler.bolt.ProfileBuilderBolt.lambda$fl
>>>>> ush$4(ProfileBuilderBolt.java:258) ~[stormjar.jar:?]
>>>>> at java.util.concurrent.ConcurrentMap.forEach(ConcurrentMap.java:114)
>>>>> ~[?:1.8.0_77]
>>>>> at org.apache.metron.profiler.bolt.ProfileBuilderBolt.flush(ProfileBuilderBolt.java:237)
>>>>> ~[stormjar.jar:?]
>>>>> at org.apache.metron.profiler.bolt.ProfileBuilderBolt.doExecute(ProfileBuilderBolt.java:164)
>>>>> ~[stormjar.jar:?]
>>>>> at org.apache.metron.profiler.bolt.ProfileBuilderBolt.execute(ProfileBuilderBolt.java:144)
>>>>> [stormjar.jar:?]
>>>>> at org.apache.storm.daemon.executor$fn__6571$tuple_action_fn__6573.invoke(executor.clj:734)
>>>>> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>>>>> at org.apache.storm.daemon.executor$mk_task_receiver$fn__6492.invoke(executor.clj:469)
>>>>> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>>>>> at org.apache.storm.disruptor$clojure_handler$reify__6005.onEvent(disruptor.clj:40)
>>>>> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>>>>> at org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:451)
>>>>> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>>>>> at org.apache.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:430)
>>>>> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>>>>> at org.apache.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:73)
>>>>> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>>>>> at org.apache.storm.daemon.executor$fn__6571$fn__6584$fn__6637.invoke(executor.clj:853)
>>>>> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>>>>> at org.apache.storm.util$async_loop$fn__554.invoke(util.clj:484)
>>>>> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>>>>> at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
>>>>> at java.lang.Thread.run(Thread.java:745) [?:1.8.0_77]
>>>>> Caused by: java.lang.ClassCastException: org.apache.metron.statistics.o
>>>>> utlier.MedianAbsoluteDeviationFunctions$State cannot be cast to
>>>>> java.util.List
>>>>> at org.apache.metron.statistics.outlier.MedianAbsoluteDeviation
>>>>> Functions$StateUpdate.apply(MedianAbsoluteDeviationFunctions.java:85)
>>>>> ~[stormjar.jar:?]
>>>>> at org.apache.metron.common.stellar.StellarCompiler.exitTransfo
>>>>> rmationFunc(StellarCompiler.java:246) ~[stormjar.jar:?]
>>>>> at org.apache.metron.common.stellar.generated.StellarParser$Tra
>>>>> nsformationFuncContext.exitRule(StellarParser.java:1617)
>>>>> ~[stormjar.jar:?]
>>>>> at org.antlr.v4.runtime.Parser.triggerExitRuleEvent(Parser.java:422)
>>>>> ~[stormjar.jar:?]
>>>>> at org.antlr.v4.runtime.Parser.exitRule(Parser.java:632)
>>>>> ~[stormjar.jar:?]
>>>>> at org.apache.metron.common.stellar.generated.StellarParser.fun
>>>>> ctions(StellarParser.java:1640) ~[stormjar.jar:?]
>>>>> at org.apache.metron.common.stellar.generated.StellarParser.ari
>>>>> thmetic_operands(StellarParser.java:1750) ~[stormjar.jar:?]
>>>>> at org.apache.metron.common.stellar.generated.StellarParser.ari
>>>>> thmetic_expr_mul(StellarParser.java:1537) ~[stormjar.jar:?]
>>>>> at org.apache.metron.common.stellar.generated.StellarParser.ari
>>>>> thmetic_expr(StellarParser.java:1397) ~[stormjar.jar:?]
>>>>> at org.apache.metron.common.stellar.generated.StellarParser.tra
>>>>> nsformation_expr(StellarParser.java:278) ~[stormjar.jar:?]
>>>>> at org.apache.metron.common.stellar.generated.StellarParser.tra
>>>>> nsformation(StellarParser.java:147) ~[stormjar.jar:?]
>>>>> at org.apache.metron.common.stellar.BaseStellarProcessor.parse(BaseStellarProcessor.java:89)
>>>>> ~[stormjar.jar:?]
>>>>> ... 19 more
>>>>> 2017-02-28 00:05:43.377 o.a.s.d.executor [ERROR]
>>>>> org.apache.metron.common.dsl.ParseException: Unable to execute:
>>>>> org.apache.metron.statistics.outlier.MedianAbsoluteDeviationFunctions$State
>>>>> cannot be cast to java.util.List
>>>>> at org.apache.metron.common.stellar.StellarCompiler.getResult(StellarCompiler.java:428)
>>>>> ~[stormjar.jar:?]
>>>>> at org.apache.metron.common.stellar.BaseStellarProcessor.parse(BaseStellarProcessor.java:90)
>>>>> ~[stormjar.jar:?]
>>>>> at org.apache.metron.profiler.stellar.DefaultStellarExecutor.ex
>>>>> ecute(DefaultStellarExecutor.java:148) ~[stormjar.jar:?]
>>>>> at org.apache.metron.profiler.stellar.DefaultStellarExecutor.as
>>>>> sign(DefaultStellarExecutor.java:94) ~[stormjar.jar:?]
>>>>> at org.apache.metron.profiler.bolt.ProfileBuilderBolt.lambda$nu
>>>>> ll$2(ProfileBuilderBolt.java:258) ~[stormjar.jar:?]
>>>>> at java.util.LinkedHashMap.forEach(LinkedHashMap.java:676)
>>>>> ~[?:1.8.0_77]
>>>>> at org.apache.metron.profiler.bolt.ProfileBuilderBolt.lambda$fl
>>>>> ush$4(ProfileBuilderBolt.java:258) ~[stormjar.jar:?]
>>>>> at java.util.concurrent.ConcurrentMap.forEach(ConcurrentMap.java:114)
>>>>> ~[?:1.8.0_77]
>>>>> at org.apache.metron.profiler.bolt.ProfileBuilderBolt.flush(ProfileBuilderBolt.java:237)
>>>>> ~[stormjar.jar:?]
>>>>> at org.apache.metron.profiler.bolt.ProfileBuilderBolt.doExecute(ProfileBuilderBolt.java:164)
>>>>> ~[stormjar.jar:?]
>>>>> at org.apache.metron.profiler.bolt.ProfileBuilderBolt.execute(ProfileBuilderBolt.java:144)
>>>>> [stormjar.jar:?]
>>>>> at org.apache.storm.daemon.executor$fn__6571$tuple_action_fn__6573.invoke(executor.clj:734)
>>>>> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>>>>> at org.apache.storm.daemon.executor$mk_task_receiver$fn__6492.invoke(executor.clj:469)
>>>>> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>>>>> at org.apache.storm.disruptor$clojure_handler$reify__6005.onEvent(disruptor.clj:40)
>>>>> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>>>>> at org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:451)
>>>>> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>>>>> at org.apache.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:430)
>>>>> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>>>>> at org.apache.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:73)
>>>>> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>>>>> at org.apache.storm.daemon.executor$fn__6571$fn__6584$fn__6637.invoke(executor.clj:853)
>>>>> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>>>>> at org.apache.storm.util$async_loop$fn__554.invoke(util.clj:484)
>>>>> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>>>>> at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
>>>>> at java.lang.Thread.run(Thread.java:745) [?:1.8.0_77]
>>>>> Caused by: java.lang.ClassCastException: org.apache.metron.statistics.o
>>>>> utlier.MedianAbsoluteDeviationFunctions$State cannot be cast to
>>>>> java.util.List
>>>>> at org.apache.metron.statistics.outlier.MedianAbsoluteDeviation
>>>>> Functions$StateUpdate.apply(MedianAbsoluteDeviationFunctions.java:85)
>>>>> ~[stormjar.jar:?]
>>>>> at org.apache.metron.common.stellar.StellarCompiler.exitTransfo
>>>>> rmationFunc(StellarCompiler.java:246) ~[stormjar.jar:?]
>>>>> at org.apache.metron.common.stellar.generated.StellarParser$Tra
>>>>> nsformationFuncContext.exitRule(StellarParser.java:1617)
>>>>> ~[stormjar.jar:?]
>>>>> at org.antlr.v4.runtime.Parser.triggerExitRuleEvent(Parser.java:422)
>>>>> ~[stormjar.jar:?]
>>>>> at org.antlr.v4.runtime.Parser.exitRule(Parser.java:632)
>>>>> ~[stormjar.jar:?]
>>>>> at org.apache.metron.common.stellar.generated.StellarParser.fun
>>>>> ctions(StellarParser.java:1640) ~[stormjar.jar:?]
>>>>> at org.apache.metron.common.stellar.generated.StellarParser.ari
>>>>> thmetic_operands(StellarParser.java:1750) ~[stormjar.jar:?]
>>>>> at org.apache.metron.common.stellar.generated.StellarParser.ari
>>>>> thmetic_expr_mul(StellarParser.java:1537) ~[stormjar.jar:?]
>>>>> at org.apache.metron.common.stellar.generated.StellarParser.ari
>>>>> thmetic_expr(StellarParser.java:1397) ~[stormjar.jar:?]
>>>>> at org.apache.metron.common.stellar.generated.StellarParser.tra
>>>>> nsformation_expr(StellarParser.java:278) ~[stormjar.jar:?]
>>>>> at org.apache.metron.common.stellar.generated.StellarParser.tra
>>>>> nsformation(StellarParser.java:147) ~[stormjar.jar:?]
>>>>> at org.apache.metron.common.stellar.BaseStellarProcessor.parse(BaseStellarProcessor.java:89)
>>>>> ~[stormjar.jar:?]
>>>>> ... 19 more
>>>>>
>>>>>
>>>>>
>>>>> On Mon, Feb 27, 2017 at 11:33 PM, Casey Stella <ce...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Any chance there's a full stacktrace associated with that NPE
>>>>>> anywhere? Also, I assume this is 0.3.0, right?
>>>>>>
>>>>>> On Mon, Feb 27, 2017 at 5:29 PM, shoggi <sh...@gmail.com> wrote:
>>>>>>
>>>>>>> Hi all
>>>>>>>
>>>>>>> Has anyone come across following profiler error, when trying to work
>>>>>>> with the MAD functions? The configuration loads fine and I only get the
>>>>>>> error when the profiler is pushing its data. I don't think its a typo but
>>>>>>> maybe the logic is not sound.
>>>>>>>
>>>>>>> Error from the profiler worker:
>>>>>>> 2017-02-27 23:20:43.368 o.a.m.p.b.ProfileBuilderBolt [INFO] Flushing
>>>>>>> profile: profile=sketchy_mad, entity=krxd.net
>>>>>>> 2017-02-27 23:20:43.368 o.a.m.p.b.ProfileBuilderBolt [ERROR]
>>>>>>> Unexpected failure: message='null', tuple='source: __system:-1, stream:
>>>>>>> __tick, id: {}, [900]'
>>>>>>> java.lang.NullPointerException
>>>>>>> 2017-02-27 23:20:43.368 o.a.s.d.executor [ERROR]
>>>>>>> java.lang.NullPointerException
>>>>>>>
>>>>>>>
>>>>>>> My Profiler config:
>>>>>>>
>>>>>>> {
>>>>>>> "profile": "sketchy_mad",
>>>>>>> "foreach": "if exists(domain_without_subdomains) then
>>>>>>> domain_without_subdomains else 'n/a'",
>>>>>>> "onlyif": "exists(domain_without_subdomains) && source.type
>>>>>>> == 'squid'",
>>>>>>> "init" : {
>>>>>>> "s": "OUTLIER_MAD_STATE_MERGE(PROFILE_GET('sketchy_mad',
>>>>>>> domain_without_subdomains, 5, 'MINUTES'))"
>>>>>>> },
>>>>>>> "tickUpdate" : {
>>>>>>> "s": "OUTLIER_MAD_STATE_MERGE(PROFILE_GET('sketchy_mad',
>>>>>>> domain_without_subdomains, 5, 'MINUTES'), s)"
>>>>>>> },
>>>>>>> "update": {
>>>>>>> "s": "OUTLIER_MAD_ADD(s, bytes)"
>>>>>>> },
>>>>>>> "result": "s"
>>>>>>> }
>>>>>>>
>>>>>>> Regards
>>>>>>> shoggi
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
Re: Median Absolute Deviation - Profiler Error
Posted by shoggi <sh...@gmail.com>.
This is what I've got..
{
"profile": "sketchy_mad",
"foreach": "if exists(domain_without_subdomains) then
domain_without_subdomains else 'n/a'",
"onlyif": "exists(domain_without_subdomains) && source.type ==
'squid'",
"init" : {
"s": "OUTLIER_MAD_STATE_MERGE(PROFILE_GET('sketchy_mad',
domain_without_subdomains, 5, 'MINUTES'))"
},
"update": {
"s": "OUTLIER_MAD_ADD(s, bytes)"
},
"result": "s"
}
On Tue, Feb 28, 2017 at 3:34 AM, Casey Stella <ce...@gmail.com> wrote:
> Sorry, waht does the current profile config look like for that very last
> error?
>
> On Mon, Feb 27, 2017 at 6:39 PM, shoggi <sh...@gmail.com> wrote:
>
>> Seems not to be the culprit..
>>
>>
>> 2017-02-28 00:35:43.367 o.a.m.p.b.ProfileBuilderBolt [INFO] Flushing
>> profile: profile=sketchy_mad, entity=optimicdn.com
>> 2017-02-28 00:35:43.368 o.a.m.p.b.ProfileBuilderBolt [ERROR] Unexpected
>> failure: message='Unable to execute: org.apache.metron.statistics.o
>> utlier.MedianAbsoluteDeviationFunctions$State cannot be cast to
>> java.util.List', tuple='source: __system:-1, stream: __tick, id: {}, [900]'
>> org.apache.metron.common.dsl.ParseException: Unable to execute:
>> org.apache.metron.statistics.outlier.MedianAbsoluteDeviationFunctions$State
>> cannot be cast to java.util.List
>> at org.apache.metron.common.stellar.StellarCompiler.getResult(StellarCompiler.java:428)
>> ~[stormjar.jar:?]
>> at org.apache.metron.common.stellar.BaseStellarProcessor.parse(BaseStellarProcessor.java:90)
>> ~[stormjar.jar:?]
>> at org.apache.metron.profiler.stellar.DefaultStellarExecutor.ex
>> ecute(DefaultStellarExecutor.java:148) ~[stormjar.jar:?]
>> at org.apache.metron.profiler.stellar.DefaultStellarExecutor.
>> assign(DefaultStellarExecutor.java:94) ~[stormjar.jar:?]
>> at org.apache.metron.profiler.bolt.ProfileBuilderBolt.lambda$
>> null$2(ProfileBuilderBolt.java:258) ~[stormjar.jar:?]
>> at java.util.LinkedHashMap.forEach(LinkedHashMap.java:676) ~[?:1.8.0_77]
>> at org.apache.metron.profiler.bolt.ProfileBuilderBolt.lambda$
>> flush$4(ProfileBuilderBolt.java:258) ~[stormjar.jar:?]
>> at java.util.concurrent.ConcurrentMap.forEach(ConcurrentMap.java:114)
>> ~[?:1.8.0_77]
>> at org.apache.metron.profiler.bolt.ProfileBuilderBolt.flush(ProfileBuilderBolt.java:237)
>> ~[stormjar.jar:?]
>> at org.apache.metron.profiler.bolt.ProfileBuilderBolt.doExecute(ProfileBuilderBolt.java:164)
>> ~[stormjar.jar:?]
>> at org.apache.metron.profiler.bolt.ProfileBuilderBolt.execute(ProfileBuilderBolt.java:144)
>> [stormjar.jar:?]
>> at org.apache.storm.daemon.executor$fn__6571$tuple_action_fn__6573.invoke(executor.clj:734)
>> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>> at org.apache.storm.daemon.executor$mk_task_receiver$fn__6492.invoke(executor.clj:469)
>> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>> at org.apache.storm.disruptor$clojure_handler$reify__6005.onEvent(disruptor.clj:40)
>> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>> at org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:451)
>> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>> at org.apache.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:430)
>> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>> at org.apache.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:73)
>> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>> at org.apache.storm.daemon.executor$fn__6571$fn__6584$fn__6637.invoke(executor.clj:853)
>> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>> at org.apache.storm.util$async_loop$fn__554.invoke(util.clj:484)
>> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>> at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
>> at java.lang.Thread.run(Thread.java:745) [?:1.8.0_77]
>> Caused by: java.lang.ClassCastException: org.apache.metron.statistics.o
>> utlier.MedianAbsoluteDeviationFunctions$State cannot be cast to
>> java.util.List
>> at org.apache.metron.statistics.outlier.MedianAbsoluteDeviation
>> Functions$StateUpdate.apply(MedianAbsoluteDeviationFunctions.java:85)
>> ~[stormjar.jar:?]
>> at org.apache.metron.common.stellar.StellarCompiler.exitTransfo
>> rmationFunc(StellarCompiler.java:246) ~[stormjar.jar:?]
>> at org.apache.metron.common.stellar.generated.StellarParser$Tra
>> nsformationFuncContext.exitRule(StellarParser.java:1617)
>> ~[stormjar.jar:?]
>> at org.antlr.v4.runtime.Parser.triggerExitRuleEvent(Parser.java:422)
>> ~[stormjar.jar:?]
>> at org.antlr.v4.runtime.Parser.exitRule(Parser.java:632)
>> ~[stormjar.jar:?]
>> at org.apache.metron.common.stellar.generated.StellarParser.
>> functions(StellarParser.java:1640) ~[stormjar.jar:?]
>> at org.apache.metron.common.stellar.generated.StellarParser.
>> arithmetic_operands(StellarParser.java:1750) ~[stormjar.jar:?]
>> at org.apache.metron.common.stellar.generated.StellarParser.
>> arithmetic_expr_mul(StellarParser.java:1537) ~[stormjar.jar:?]
>> at org.apache.metron.common.stellar.generated.StellarParser.
>> arithmetic_expr(StellarParser.java:1397) ~[stormjar.jar:?]
>> at org.apache.metron.common.stellar.generated.StellarParser.
>> transformation_expr(StellarParser.java:278) ~[stormjar.jar:?]
>> at org.apache.metron.common.stellar.generated.StellarParser.
>> transformation(StellarParser.java:147) ~[stormjar.jar:?]
>> at org.apache.metron.common.stellar.BaseStellarProcessor.parse(BaseStellarProcessor.java:89)
>> ~[stormjar.jar:?]
>> ... 19 more
>>
>> On Tue, Feb 28, 2017 at 12:18 AM, Casey Stella <ce...@gmail.com>
>> wrote:
>>
>>> Ok, try something for me and tell me if it fixes it: adjust the config
>>> to remove the tickUpdate (it didn't work anyway and we have since removed
>>> it in 0.3.1).
>>>
>>>
>>> On Mon, Feb 27, 2017 at 6:07 PM, shoggi <sh...@gmail.com> wrote:
>>>
>>>> Yes, this is 0.3.0.
>>>>
>>>> Here is the full trace.
>>>>
>>>> 2017-02-28 00:05:43.367 o.a.m.p.b.ProfileBuilderBolt [INFO] Flushing
>>>> profile: profile=sketchy_mad, entity=krxd.net
>>>> 2017-02-28 00:05:43.374 o.a.m.p.b.ProfileBuilderBolt [ERROR] Unexpected
>>>> failure: message='Unable to execute: org.apache.metron.statistics.o
>>>> utlier.MedianAbsoluteDeviationFunctions$State cannot be cast to
>>>> java.util.List', tuple='source: __system:-1, stream: __tick, id: {}, [900]'
>>>> org.apache.metron.common.dsl.ParseException: Unable to execute:
>>>> org.apache.metron.statistics.outlier.MedianAbsoluteDeviationFunctions$State
>>>> cannot be cast to java.util.List
>>>> at org.apache.metron.common.stellar.StellarCompiler.getResult(StellarCompiler.java:428)
>>>> ~[stormjar.jar:?]
>>>> at org.apache.metron.common.stellar.BaseStellarProcessor.parse(BaseStellarProcessor.java:90)
>>>> ~[stormjar.jar:?]
>>>> at org.apache.metron.profiler.stellar.DefaultStellarExecutor.ex
>>>> ecute(DefaultStellarExecutor.java:148) ~[stormjar.jar:?]
>>>> at org.apache.metron.profiler.stellar.DefaultStellarExecutor.as
>>>> sign(DefaultStellarExecutor.java:94) ~[stormjar.jar:?]
>>>> at org.apache.metron.profiler.bolt.ProfileBuilderBolt.lambda$nu
>>>> ll$2(ProfileBuilderBolt.java:258) ~[stormjar.jar:?]
>>>> at java.util.LinkedHashMap.forEach(LinkedHashMap.java:676)
>>>> ~[?:1.8.0_77]
>>>> at org.apache.metron.profiler.bolt.ProfileBuilderBolt.lambda$fl
>>>> ush$4(ProfileBuilderBolt.java:258) ~[stormjar.jar:?]
>>>> at java.util.concurrent.ConcurrentMap.forEach(ConcurrentMap.java:114)
>>>> ~[?:1.8.0_77]
>>>> at org.apache.metron.profiler.bolt.ProfileBuilderBolt.flush(ProfileBuilderBolt.java:237)
>>>> ~[stormjar.jar:?]
>>>> at org.apache.metron.profiler.bolt.ProfileBuilderBolt.doExecute(ProfileBuilderBolt.java:164)
>>>> ~[stormjar.jar:?]
>>>> at org.apache.metron.profiler.bolt.ProfileBuilderBolt.execute(ProfileBuilderBolt.java:144)
>>>> [stormjar.jar:?]
>>>> at org.apache.storm.daemon.executor$fn__6571$tuple_action_fn__6573.invoke(executor.clj:734)
>>>> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>>>> at org.apache.storm.daemon.executor$mk_task_receiver$fn__6492.invoke(executor.clj:469)
>>>> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>>>> at org.apache.storm.disruptor$clojure_handler$reify__6005.onEvent(disruptor.clj:40)
>>>> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>>>> at org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:451)
>>>> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>>>> at org.apache.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:430)
>>>> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>>>> at org.apache.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:73)
>>>> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>>>> at org.apache.storm.daemon.executor$fn__6571$fn__6584$fn__6637.invoke(executor.clj:853)
>>>> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>>>> at org.apache.storm.util$async_loop$fn__554.invoke(util.clj:484)
>>>> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>>>> at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
>>>> at java.lang.Thread.run(Thread.java:745) [?:1.8.0_77]
>>>> Caused by: java.lang.ClassCastException: org.apache.metron.statistics.o
>>>> utlier.MedianAbsoluteDeviationFunctions$State cannot be cast to
>>>> java.util.List
>>>> at org.apache.metron.statistics.outlier.MedianAbsoluteDeviation
>>>> Functions$StateUpdate.apply(MedianAbsoluteDeviationFunctions.java:85)
>>>> ~[stormjar.jar:?]
>>>> at org.apache.metron.common.stellar.StellarCompiler.exitTransfo
>>>> rmationFunc(StellarCompiler.java:246) ~[stormjar.jar:?]
>>>> at org.apache.metron.common.stellar.generated.StellarParser$Tra
>>>> nsformationFuncContext.exitRule(StellarParser.java:1617)
>>>> ~[stormjar.jar:?]
>>>> at org.antlr.v4.runtime.Parser.triggerExitRuleEvent(Parser.java:422)
>>>> ~[stormjar.jar:?]
>>>> at org.antlr.v4.runtime.Parser.exitRule(Parser.java:632)
>>>> ~[stormjar.jar:?]
>>>> at org.apache.metron.common.stellar.generated.StellarParser.fun
>>>> ctions(StellarParser.java:1640) ~[stormjar.jar:?]
>>>> at org.apache.metron.common.stellar.generated.StellarParser.ari
>>>> thmetic_operands(StellarParser.java:1750) ~[stormjar.jar:?]
>>>> at org.apache.metron.common.stellar.generated.StellarParser.ari
>>>> thmetic_expr_mul(StellarParser.java:1537) ~[stormjar.jar:?]
>>>> at org.apache.metron.common.stellar.generated.StellarParser.ari
>>>> thmetic_expr(StellarParser.java:1397) ~[stormjar.jar:?]
>>>> at org.apache.metron.common.stellar.generated.StellarParser.tra
>>>> nsformation_expr(StellarParser.java:278) ~[stormjar.jar:?]
>>>> at org.apache.metron.common.stellar.generated.StellarParser.tra
>>>> nsformation(StellarParser.java:147) ~[stormjar.jar:?]
>>>> at org.apache.metron.common.stellar.BaseStellarProcessor.parse(BaseStellarProcessor.java:89)
>>>> ~[stormjar.jar:?]
>>>> ... 19 more
>>>> 2017-02-28 00:05:43.377 o.a.s.d.executor [ERROR]
>>>> org.apache.metron.common.dsl.ParseException: Unable to execute:
>>>> org.apache.metron.statistics.outlier.MedianAbsoluteDeviationFunctions$State
>>>> cannot be cast to java.util.List
>>>> at org.apache.metron.common.stellar.StellarCompiler.getResult(StellarCompiler.java:428)
>>>> ~[stormjar.jar:?]
>>>> at org.apache.metron.common.stellar.BaseStellarProcessor.parse(BaseStellarProcessor.java:90)
>>>> ~[stormjar.jar:?]
>>>> at org.apache.metron.profiler.stellar.DefaultStellarExecutor.ex
>>>> ecute(DefaultStellarExecutor.java:148) ~[stormjar.jar:?]
>>>> at org.apache.metron.profiler.stellar.DefaultStellarExecutor.as
>>>> sign(DefaultStellarExecutor.java:94) ~[stormjar.jar:?]
>>>> at org.apache.metron.profiler.bolt.ProfileBuilderBolt.lambda$nu
>>>> ll$2(ProfileBuilderBolt.java:258) ~[stormjar.jar:?]
>>>> at java.util.LinkedHashMap.forEach(LinkedHashMap.java:676)
>>>> ~[?:1.8.0_77]
>>>> at org.apache.metron.profiler.bolt.ProfileBuilderBolt.lambda$fl
>>>> ush$4(ProfileBuilderBolt.java:258) ~[stormjar.jar:?]
>>>> at java.util.concurrent.ConcurrentMap.forEach(ConcurrentMap.java:114)
>>>> ~[?:1.8.0_77]
>>>> at org.apache.metron.profiler.bolt.ProfileBuilderBolt.flush(ProfileBuilderBolt.java:237)
>>>> ~[stormjar.jar:?]
>>>> at org.apache.metron.profiler.bolt.ProfileBuilderBolt.doExecute(ProfileBuilderBolt.java:164)
>>>> ~[stormjar.jar:?]
>>>> at org.apache.metron.profiler.bolt.ProfileBuilderBolt.execute(ProfileBuilderBolt.java:144)
>>>> [stormjar.jar:?]
>>>> at org.apache.storm.daemon.executor$fn__6571$tuple_action_fn__6573.invoke(executor.clj:734)
>>>> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>>>> at org.apache.storm.daemon.executor$mk_task_receiver$fn__6492.invoke(executor.clj:469)
>>>> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>>>> at org.apache.storm.disruptor$clojure_handler$reify__6005.onEvent(disruptor.clj:40)
>>>> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>>>> at org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:451)
>>>> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>>>> at org.apache.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:430)
>>>> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>>>> at org.apache.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:73)
>>>> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>>>> at org.apache.storm.daemon.executor$fn__6571$fn__6584$fn__6637.invoke(executor.clj:853)
>>>> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>>>> at org.apache.storm.util$async_loop$fn__554.invoke(util.clj:484)
>>>> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>>>> at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
>>>> at java.lang.Thread.run(Thread.java:745) [?:1.8.0_77]
>>>> Caused by: java.lang.ClassCastException: org.apache.metron.statistics.o
>>>> utlier.MedianAbsoluteDeviationFunctions$State cannot be cast to
>>>> java.util.List
>>>> at org.apache.metron.statistics.outlier.MedianAbsoluteDeviation
>>>> Functions$StateUpdate.apply(MedianAbsoluteDeviationFunctions.java:85)
>>>> ~[stormjar.jar:?]
>>>> at org.apache.metron.common.stellar.StellarCompiler.exitTransfo
>>>> rmationFunc(StellarCompiler.java:246) ~[stormjar.jar:?]
>>>> at org.apache.metron.common.stellar.generated.StellarParser$Tra
>>>> nsformationFuncContext.exitRule(StellarParser.java:1617)
>>>> ~[stormjar.jar:?]
>>>> at org.antlr.v4.runtime.Parser.triggerExitRuleEvent(Parser.java:422)
>>>> ~[stormjar.jar:?]
>>>> at org.antlr.v4.runtime.Parser.exitRule(Parser.java:632)
>>>> ~[stormjar.jar:?]
>>>> at org.apache.metron.common.stellar.generated.StellarParser.fun
>>>> ctions(StellarParser.java:1640) ~[stormjar.jar:?]
>>>> at org.apache.metron.common.stellar.generated.StellarParser.ari
>>>> thmetic_operands(StellarParser.java:1750) ~[stormjar.jar:?]
>>>> at org.apache.metron.common.stellar.generated.StellarParser.ari
>>>> thmetic_expr_mul(StellarParser.java:1537) ~[stormjar.jar:?]
>>>> at org.apache.metron.common.stellar.generated.StellarParser.ari
>>>> thmetic_expr(StellarParser.java:1397) ~[stormjar.jar:?]
>>>> at org.apache.metron.common.stellar.generated.StellarParser.tra
>>>> nsformation_expr(StellarParser.java:278) ~[stormjar.jar:?]
>>>> at org.apache.metron.common.stellar.generated.StellarParser.tra
>>>> nsformation(StellarParser.java:147) ~[stormjar.jar:?]
>>>> at org.apache.metron.common.stellar.BaseStellarProcessor.parse(BaseStellarProcessor.java:89)
>>>> ~[stormjar.jar:?]
>>>> ... 19 more
>>>>
>>>>
>>>>
>>>> On Mon, Feb 27, 2017 at 11:33 PM, Casey Stella <ce...@gmail.com>
>>>> wrote:
>>>>
>>>>> Any chance there's a full stacktrace associated with that NPE
>>>>> anywhere? Also, I assume this is 0.3.0, right?
>>>>>
>>>>> On Mon, Feb 27, 2017 at 5:29 PM, shoggi <sh...@gmail.com> wrote:
>>>>>
>>>>>> Hi all
>>>>>>
>>>>>> Has anyone come across following profiler error, when trying to work
>>>>>> with the MAD functions? The configuration loads fine and I only get the
>>>>>> error when the profiler is pushing its data. I don't think its a typo but
>>>>>> maybe the logic is not sound.
>>>>>>
>>>>>> Error from the profiler worker:
>>>>>> 2017-02-27 23:20:43.368 o.a.m.p.b.ProfileBuilderBolt [INFO] Flushing
>>>>>> profile: profile=sketchy_mad, entity=krxd.net
>>>>>> 2017-02-27 23:20:43.368 o.a.m.p.b.ProfileBuilderBolt [ERROR]
>>>>>> Unexpected failure: message='null', tuple='source: __system:-1, stream:
>>>>>> __tick, id: {}, [900]'
>>>>>> java.lang.NullPointerException
>>>>>> 2017-02-27 23:20:43.368 o.a.s.d.executor [ERROR]
>>>>>> java.lang.NullPointerException
>>>>>>
>>>>>>
>>>>>> My Profiler config:
>>>>>>
>>>>>> {
>>>>>> "profile": "sketchy_mad",
>>>>>> "foreach": "if exists(domain_without_subdomains) then
>>>>>> domain_without_subdomains else 'n/a'",
>>>>>> "onlyif": "exists(domain_without_subdomains) && source.type ==
>>>>>> 'squid'",
>>>>>> "init" : {
>>>>>> "s": "OUTLIER_MAD_STATE_MERGE(PROFILE_GET('sketchy_mad',
>>>>>> domain_without_subdomains, 5, 'MINUTES'))"
>>>>>> },
>>>>>> "tickUpdate" : {
>>>>>> "s": "OUTLIER_MAD_STATE_MERGE(PROFILE_GET('sketchy_mad',
>>>>>> domain_without_subdomains, 5, 'MINUTES'), s)"
>>>>>> },
>>>>>> "update": {
>>>>>> "s": "OUTLIER_MAD_ADD(s, bytes)"
>>>>>> },
>>>>>> "result": "s"
>>>>>> }
>>>>>>
>>>>>> Regards
>>>>>> shoggi
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
>
Re: Median Absolute Deviation - Profiler Error
Posted by Casey Stella <ce...@gmail.com>.
Sorry, waht does the current profile config look like for that very last
error?
On Mon, Feb 27, 2017 at 6:39 PM, shoggi <sh...@gmail.com> wrote:
> Seems not to be the culprit..
>
>
> 2017-02-28 00:35:43.367 o.a.m.p.b.ProfileBuilderBolt [INFO] Flushing
> profile: profile=sketchy_mad, entity=optimicdn.com
> 2017-02-28 00:35:43.368 o.a.m.p.b.ProfileBuilderBolt [ERROR] Unexpected
> failure: message='Unable to execute: org.apache.metron.statistics.outlier.
> MedianAbsoluteDeviationFunctions$State cannot be cast to java.util.List',
> tuple='source: __system:-1, stream: __tick, id: {}, [900]'
> org.apache.metron.common.dsl.ParseException: Unable to execute:
> org.apache.metron.statistics.outlier.MedianAbsoluteDeviationFunctions$State
> cannot be cast to java.util.List
> at org.apache.metron.common.stellar.StellarCompiler.
> getResult(StellarCompiler.java:428) ~[stormjar.jar:?]
> at org.apache.metron.common.stellar.BaseStellarProcessor.
> parse(BaseStellarProcessor.java:90) ~[stormjar.jar:?]
> at org.apache.metron.profiler.stellar.DefaultStellarExecutor.execute(
> DefaultStellarExecutor.java:148) ~[stormjar.jar:?]
> at org.apache.metron.profiler.stellar.DefaultStellarExecutor.assign(
> DefaultStellarExecutor.java:94) ~[stormjar.jar:?]
> at org.apache.metron.profiler.bolt.ProfileBuilderBolt.lambda$null$2(ProfileBuilderBolt.java:258)
> ~[stormjar.jar:?]
> at java.util.LinkedHashMap.forEach(LinkedHashMap.java:676) ~[?:1.8.0_77]
> at org.apache.metron.profiler.bolt.ProfileBuilderBolt.lambda$flush$4(ProfileBuilderBolt.java:258)
> ~[stormjar.jar:?]
> at java.util.concurrent.ConcurrentMap.forEach(ConcurrentMap.java:114)
> ~[?:1.8.0_77]
> at org.apache.metron.profiler.bolt.ProfileBuilderBolt.flush(ProfileBuilderBolt.java:237)
> ~[stormjar.jar:?]
> at org.apache.metron.profiler.bolt.ProfileBuilderBolt.
> doExecute(ProfileBuilderBolt.java:164) ~[stormjar.jar:?]
> at org.apache.metron.profiler.bolt.ProfileBuilderBolt.
> execute(ProfileBuilderBolt.java:144) [stormjar.jar:?]
> at org.apache.storm.daemon.executor$fn__6571$tuple_action_fn__6573.invoke(executor.clj:734)
> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
> at org.apache.storm.daemon.executor$mk_task_receiver$fn__6492.invoke(executor.clj:469)
> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
> at org.apache.storm.disruptor$clojure_handler$reify__6005.onEvent(disruptor.clj:40)
> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
> at org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:451)
> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
> at org.apache.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:430)
> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
> at org.apache.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:73)
> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
> at org.apache.storm.daemon.executor$fn__6571$fn__6584$fn__6637.invoke(executor.clj:853)
> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
> at org.apache.storm.util$async_loop$fn__554.invoke(util.clj:484)
> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
> at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
> at java.lang.Thread.run(Thread.java:745) [?:1.8.0_77]
> Caused by: java.lang.ClassCastException: org.apache.metron.statistics.
> outlier.MedianAbsoluteDeviationFunctions$State cannot be cast to
> java.util.List
> at org.apache.metron.statistics.outlier.MedianAbsoluteDeviationFunctio
> ns$StateUpdate.apply(MedianAbsoluteDeviationFunctions.java:85)
> ~[stormjar.jar:?]
> at org.apache.metron.common.stellar.StellarCompiler.
> exitTransformationFunc(StellarCompiler.java:246) ~[stormjar.jar:?]
> at org.apache.metron.common.stellar.generated.StellarParser$
> TransformationFuncContext.exitRule(StellarParser.java:1617)
> ~[stormjar.jar:?]
> at org.antlr.v4.runtime.Parser.triggerExitRuleEvent(Parser.java:422)
> ~[stormjar.jar:?]
> at org.antlr.v4.runtime.Parser.exitRule(Parser.java:632) ~[stormjar.jar:?]
> at org.apache.metron.common.stellar.generated.StellarParser.functions(StellarParser.java:1640)
> ~[stormjar.jar:?]
> at org.apache.metron.common.stellar.generated.StellarParser.arithmetic_
> operands(StellarParser.java:1750) ~[stormjar.jar:?]
> at org.apache.metron.common.stellar.generated.
> StellarParser.arithmetic_expr_mul(StellarParser.java:1537)
> ~[stormjar.jar:?]
> at org.apache.metron.common.stellar.generated.
> StellarParser.arithmetic_expr(StellarParser.java:1397) ~[stormjar.jar:?]
> at org.apache.metron.common.stellar.generated.
> StellarParser.transformation_expr(StellarParser.java:278)
> ~[stormjar.jar:?]
> at org.apache.metron.common.stellar.generated.
> StellarParser.transformation(StellarParser.java:147) ~[stormjar.jar:?]
> at org.apache.metron.common.stellar.BaseStellarProcessor.
> parse(BaseStellarProcessor.java:89) ~[stormjar.jar:?]
> ... 19 more
>
> On Tue, Feb 28, 2017 at 12:18 AM, Casey Stella <ce...@gmail.com> wrote:
>
>> Ok, try something for me and tell me if it fixes it: adjust the config to
>> remove the tickUpdate (it didn't work anyway and we have since removed it
>> in 0.3.1).
>>
>>
>> On Mon, Feb 27, 2017 at 6:07 PM, shoggi <sh...@gmail.com> wrote:
>>
>>> Yes, this is 0.3.0.
>>>
>>> Here is the full trace.
>>>
>>> 2017-02-28 00:05:43.367 o.a.m.p.b.ProfileBuilderBolt [INFO] Flushing
>>> profile: profile=sketchy_mad, entity=krxd.net
>>> 2017-02-28 00:05:43.374 o.a.m.p.b.ProfileBuilderBolt [ERROR] Unexpected
>>> failure: message='Unable to execute: org.apache.metron.statistics.o
>>> utlier.MedianAbsoluteDeviationFunctions$State cannot be cast to
>>> java.util.List', tuple='source: __system:-1, stream: __tick, id: {}, [900]'
>>> org.apache.metron.common.dsl.ParseException: Unable to execute:
>>> org.apache.metron.statistics.outlier.MedianAbsoluteDeviationFunctions$State
>>> cannot be cast to java.util.List
>>> at org.apache.metron.common.stellar.StellarCompiler.getResult(StellarCompiler.java:428)
>>> ~[stormjar.jar:?]
>>> at org.apache.metron.common.stellar.BaseStellarProcessor.parse(BaseStellarProcessor.java:90)
>>> ~[stormjar.jar:?]
>>> at org.apache.metron.profiler.stellar.DefaultStellarExecutor.ex
>>> ecute(DefaultStellarExecutor.java:148) ~[stormjar.jar:?]
>>> at org.apache.metron.profiler.stellar.DefaultStellarExecutor.as
>>> sign(DefaultStellarExecutor.java:94) ~[stormjar.jar:?]
>>> at org.apache.metron.profiler.bolt.ProfileBuilderBolt.lambda$nu
>>> ll$2(ProfileBuilderBolt.java:258) ~[stormjar.jar:?]
>>> at java.util.LinkedHashMap.forEach(LinkedHashMap.java:676) ~[?:1.8.0_77]
>>> at org.apache.metron.profiler.bolt.ProfileBuilderBolt.lambda$fl
>>> ush$4(ProfileBuilderBolt.java:258) ~[stormjar.jar:?]
>>> at java.util.concurrent.ConcurrentMap.forEach(ConcurrentMap.java:114)
>>> ~[?:1.8.0_77]
>>> at org.apache.metron.profiler.bolt.ProfileBuilderBolt.flush(ProfileBuilderBolt.java:237)
>>> ~[stormjar.jar:?]
>>> at org.apache.metron.profiler.bolt.ProfileBuilderBolt.doExecute(ProfileBuilderBolt.java:164)
>>> ~[stormjar.jar:?]
>>> at org.apache.metron.profiler.bolt.ProfileBuilderBolt.execute(ProfileBuilderBolt.java:144)
>>> [stormjar.jar:?]
>>> at org.apache.storm.daemon.executor$fn__6571$tuple_action_fn__6573.invoke(executor.clj:734)
>>> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>>> at org.apache.storm.daemon.executor$mk_task_receiver$fn__6492.invoke(executor.clj:469)
>>> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>>> at org.apache.storm.disruptor$clojure_handler$reify__6005.onEvent(disruptor.clj:40)
>>> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>>> at org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:451)
>>> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>>> at org.apache.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:430)
>>> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>>> at org.apache.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:73)
>>> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>>> at org.apache.storm.daemon.executor$fn__6571$fn__6584$fn__6637.invoke(executor.clj:853)
>>> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>>> at org.apache.storm.util$async_loop$fn__554.invoke(util.clj:484)
>>> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>>> at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
>>> at java.lang.Thread.run(Thread.java:745) [?:1.8.0_77]
>>> Caused by: java.lang.ClassCastException: org.apache.metron.statistics.o
>>> utlier.MedianAbsoluteDeviationFunctions$State cannot be cast to
>>> java.util.List
>>> at org.apache.metron.statistics.outlier.MedianAbsoluteDeviation
>>> Functions$StateUpdate.apply(MedianAbsoluteDeviationFunctions.java:85)
>>> ~[stormjar.jar:?]
>>> at org.apache.metron.common.stellar.StellarCompiler.exitTransfo
>>> rmationFunc(StellarCompiler.java:246) ~[stormjar.jar:?]
>>> at org.apache.metron.common.stellar.generated.StellarParser$Tra
>>> nsformationFuncContext.exitRule(StellarParser.java:1617)
>>> ~[stormjar.jar:?]
>>> at org.antlr.v4.runtime.Parser.triggerExitRuleEvent(Parser.java:422)
>>> ~[stormjar.jar:?]
>>> at org.antlr.v4.runtime.Parser.exitRule(Parser.java:632)
>>> ~[stormjar.jar:?]
>>> at org.apache.metron.common.stellar.generated.StellarParser.fun
>>> ctions(StellarParser.java:1640) ~[stormjar.jar:?]
>>> at org.apache.metron.common.stellar.generated.StellarParser.ari
>>> thmetic_operands(StellarParser.java:1750) ~[stormjar.jar:?]
>>> at org.apache.metron.common.stellar.generated.StellarParser.ari
>>> thmetic_expr_mul(StellarParser.java:1537) ~[stormjar.jar:?]
>>> at org.apache.metron.common.stellar.generated.StellarParser.ari
>>> thmetic_expr(StellarParser.java:1397) ~[stormjar.jar:?]
>>> at org.apache.metron.common.stellar.generated.StellarParser.tra
>>> nsformation_expr(StellarParser.java:278) ~[stormjar.jar:?]
>>> at org.apache.metron.common.stellar.generated.StellarParser.tra
>>> nsformation(StellarParser.java:147) ~[stormjar.jar:?]
>>> at org.apache.metron.common.stellar.BaseStellarProcessor.parse(BaseStellarProcessor.java:89)
>>> ~[stormjar.jar:?]
>>> ... 19 more
>>> 2017-02-28 00:05:43.377 o.a.s.d.executor [ERROR]
>>> org.apache.metron.common.dsl.ParseException: Unable to execute:
>>> org.apache.metron.statistics.outlier.MedianAbsoluteDeviationFunctions$State
>>> cannot be cast to java.util.List
>>> at org.apache.metron.common.stellar.StellarCompiler.getResult(StellarCompiler.java:428)
>>> ~[stormjar.jar:?]
>>> at org.apache.metron.common.stellar.BaseStellarProcessor.parse(BaseStellarProcessor.java:90)
>>> ~[stormjar.jar:?]
>>> at org.apache.metron.profiler.stellar.DefaultStellarExecutor.ex
>>> ecute(DefaultStellarExecutor.java:148) ~[stormjar.jar:?]
>>> at org.apache.metron.profiler.stellar.DefaultStellarExecutor.as
>>> sign(DefaultStellarExecutor.java:94) ~[stormjar.jar:?]
>>> at org.apache.metron.profiler.bolt.ProfileBuilderBolt.lambda$nu
>>> ll$2(ProfileBuilderBolt.java:258) ~[stormjar.jar:?]
>>> at java.util.LinkedHashMap.forEach(LinkedHashMap.java:676) ~[?:1.8.0_77]
>>> at org.apache.metron.profiler.bolt.ProfileBuilderBolt.lambda$fl
>>> ush$4(ProfileBuilderBolt.java:258) ~[stormjar.jar:?]
>>> at java.util.concurrent.ConcurrentMap.forEach(ConcurrentMap.java:114)
>>> ~[?:1.8.0_77]
>>> at org.apache.metron.profiler.bolt.ProfileBuilderBolt.flush(ProfileBuilderBolt.java:237)
>>> ~[stormjar.jar:?]
>>> at org.apache.metron.profiler.bolt.ProfileBuilderBolt.doExecute(ProfileBuilderBolt.java:164)
>>> ~[stormjar.jar:?]
>>> at org.apache.metron.profiler.bolt.ProfileBuilderBolt.execute(ProfileBuilderBolt.java:144)
>>> [stormjar.jar:?]
>>> at org.apache.storm.daemon.executor$fn__6571$tuple_action_fn__6573.invoke(executor.clj:734)
>>> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>>> at org.apache.storm.daemon.executor$mk_task_receiver$fn__6492.invoke(executor.clj:469)
>>> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>>> at org.apache.storm.disruptor$clojure_handler$reify__6005.onEvent(disruptor.clj:40)
>>> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>>> at org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:451)
>>> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>>> at org.apache.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:430)
>>> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>>> at org.apache.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:73)
>>> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>>> at org.apache.storm.daemon.executor$fn__6571$fn__6584$fn__6637.invoke(executor.clj:853)
>>> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>>> at org.apache.storm.util$async_loop$fn__554.invoke(util.clj:484)
>>> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>>> at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
>>> at java.lang.Thread.run(Thread.java:745) [?:1.8.0_77]
>>> Caused by: java.lang.ClassCastException: org.apache.metron.statistics.o
>>> utlier.MedianAbsoluteDeviationFunctions$State cannot be cast to
>>> java.util.List
>>> at org.apache.metron.statistics.outlier.MedianAbsoluteDeviation
>>> Functions$StateUpdate.apply(MedianAbsoluteDeviationFunctions.java:85)
>>> ~[stormjar.jar:?]
>>> at org.apache.metron.common.stellar.StellarCompiler.exitTransfo
>>> rmationFunc(StellarCompiler.java:246) ~[stormjar.jar:?]
>>> at org.apache.metron.common.stellar.generated.StellarParser$Tra
>>> nsformationFuncContext.exitRule(StellarParser.java:1617)
>>> ~[stormjar.jar:?]
>>> at org.antlr.v4.runtime.Parser.triggerExitRuleEvent(Parser.java:422)
>>> ~[stormjar.jar:?]
>>> at org.antlr.v4.runtime.Parser.exitRule(Parser.java:632)
>>> ~[stormjar.jar:?]
>>> at org.apache.metron.common.stellar.generated.StellarParser.fun
>>> ctions(StellarParser.java:1640) ~[stormjar.jar:?]
>>> at org.apache.metron.common.stellar.generated.StellarParser.ari
>>> thmetic_operands(StellarParser.java:1750) ~[stormjar.jar:?]
>>> at org.apache.metron.common.stellar.generated.StellarParser.ari
>>> thmetic_expr_mul(StellarParser.java:1537) ~[stormjar.jar:?]
>>> at org.apache.metron.common.stellar.generated.StellarParser.ari
>>> thmetic_expr(StellarParser.java:1397) ~[stormjar.jar:?]
>>> at org.apache.metron.common.stellar.generated.StellarParser.tra
>>> nsformation_expr(StellarParser.java:278) ~[stormjar.jar:?]
>>> at org.apache.metron.common.stellar.generated.StellarParser.tra
>>> nsformation(StellarParser.java:147) ~[stormjar.jar:?]
>>> at org.apache.metron.common.stellar.BaseStellarProcessor.parse(BaseStellarProcessor.java:89)
>>> ~[stormjar.jar:?]
>>> ... 19 more
>>>
>>>
>>>
>>> On Mon, Feb 27, 2017 at 11:33 PM, Casey Stella <ce...@gmail.com>
>>> wrote:
>>>
>>>> Any chance there's a full stacktrace associated with that NPE
>>>> anywhere? Also, I assume this is 0.3.0, right?
>>>>
>>>> On Mon, Feb 27, 2017 at 5:29 PM, shoggi <sh...@gmail.com> wrote:
>>>>
>>>>> Hi all
>>>>>
>>>>> Has anyone come across following profiler error, when trying to work
>>>>> with the MAD functions? The configuration loads fine and I only get the
>>>>> error when the profiler is pushing its data. I don't think its a typo but
>>>>> maybe the logic is not sound.
>>>>>
>>>>> Error from the profiler worker:
>>>>> 2017-02-27 23:20:43.368 o.a.m.p.b.ProfileBuilderBolt [INFO] Flushing
>>>>> profile: profile=sketchy_mad, entity=krxd.net
>>>>> 2017-02-27 23:20:43.368 o.a.m.p.b.ProfileBuilderBolt [ERROR]
>>>>> Unexpected failure: message='null', tuple='source: __system:-1, stream:
>>>>> __tick, id: {}, [900]'
>>>>> java.lang.NullPointerException
>>>>> 2017-02-27 23:20:43.368 o.a.s.d.executor [ERROR]
>>>>> java.lang.NullPointerException
>>>>>
>>>>>
>>>>> My Profiler config:
>>>>>
>>>>> {
>>>>> "profile": "sketchy_mad",
>>>>> "foreach": "if exists(domain_without_subdomains) then
>>>>> domain_without_subdomains else 'n/a'",
>>>>> "onlyif": "exists(domain_without_subdomains) && source.type ==
>>>>> 'squid'",
>>>>> "init" : {
>>>>> "s": "OUTLIER_MAD_STATE_MERGE(PROFILE_GET('sketchy_mad',
>>>>> domain_without_subdomains, 5, 'MINUTES'))"
>>>>> },
>>>>> "tickUpdate" : {
>>>>> "s": "OUTLIER_MAD_STATE_MERGE(PROFILE_GET('sketchy_mad',
>>>>> domain_without_subdomains, 5, 'MINUTES'), s)"
>>>>> },
>>>>> "update": {
>>>>> "s": "OUTLIER_MAD_ADD(s, bytes)"
>>>>> },
>>>>> "result": "s"
>>>>> }
>>>>>
>>>>> Regards
>>>>> shoggi
>>>>>
>>>>
>>>>
>>>
>>
>
Re: Median Absolute Deviation - Profiler Error
Posted by shoggi <sh...@gmail.com>.
Seems not to be the culprit..
2017-02-28 00:35:43.367 o.a.m.p.b.ProfileBuilderBolt [INFO] Flushing
profile: profile=sketchy_mad, entity=optimicdn.com
2017-02-28 00:35:43.368 o.a.m.p.b.ProfileBuilderBolt [ERROR] Unexpected
failure: message='Unable to execute:
org.apache.metron.statistics.outlier.MedianAbsoluteDeviationFunctions$State
cannot be cast to java.util.List', tuple='source: __system:-1, stream:
__tick, id: {}, [900]'
org.apache.metron.common.dsl.ParseException: Unable to execute:
org.apache.metron.statistics.outlier.MedianAbsoluteDeviationFunctions$State
cannot be cast to java.util.List
at
org.apache.metron.common.stellar.StellarCompiler.getResult(StellarCompiler.java:428)
~[stormjar.jar:?]
at
org.apache.metron.common.stellar.BaseStellarProcessor.parse(BaseStellarProcessor.java:90)
~[stormjar.jar:?]
at
org.apache.metron.profiler.stellar.DefaultStellarExecutor.execute(DefaultStellarExecutor.java:148)
~[stormjar.jar:?]
at
org.apache.metron.profiler.stellar.DefaultStellarExecutor.assign(DefaultStellarExecutor.java:94)
~[stormjar.jar:?]
at
org.apache.metron.profiler.bolt.ProfileBuilderBolt.lambda$null$2(ProfileBuilderBolt.java:258)
~[stormjar.jar:?]
at java.util.LinkedHashMap.forEach(LinkedHashMap.java:676) ~[?:1.8.0_77]
at
org.apache.metron.profiler.bolt.ProfileBuilderBolt.lambda$flush$4(ProfileBuilderBolt.java:258)
~[stormjar.jar:?]
at java.util.concurrent.ConcurrentMap.forEach(ConcurrentMap.java:114)
~[?:1.8.0_77]
at
org.apache.metron.profiler.bolt.ProfileBuilderBolt.flush(ProfileBuilderBolt.java:237)
~[stormjar.jar:?]
at
org.apache.metron.profiler.bolt.ProfileBuilderBolt.doExecute(ProfileBuilderBolt.java:164)
~[stormjar.jar:?]
at
org.apache.metron.profiler.bolt.ProfileBuilderBolt.execute(ProfileBuilderBolt.java:144)
[stormjar.jar:?]
at
org.apache.storm.daemon.executor$fn__6571$tuple_action_fn__6573.invoke(executor.clj:734)
[storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
at
org.apache.storm.daemon.executor$mk_task_receiver$fn__6492.invoke(executor.clj:469)
[storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
at
org.apache.storm.disruptor$clojure_handler$reify__6005.onEvent(disruptor.clj:40)
[storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
at
org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:451)
[storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
at
org.apache.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:430)
[storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
at
org.apache.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:73)
[storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
at
org.apache.storm.daemon.executor$fn__6571$fn__6584$fn__6637.invoke(executor.clj:853)
[storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
at org.apache.storm.util$async_loop$fn__554.invoke(util.clj:484)
[storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_77]
Caused by: java.lang.ClassCastException:
org.apache.metron.statistics.outlier.MedianAbsoluteDeviationFunctions$State
cannot be cast to java.util.List
at
org.apache.metron.statistics.outlier.MedianAbsoluteDeviationFunctions$StateUpdate.apply(MedianAbsoluteDeviationFunctions.java:85)
~[stormjar.jar:?]
at
org.apache.metron.common.stellar.StellarCompiler.exitTransformationFunc(StellarCompiler.java:246)
~[stormjar.jar:?]
at
org.apache.metron.common.stellar.generated.StellarParser$TransformationFuncContext.exitRule(StellarParser.java:1617)
~[stormjar.jar:?]
at org.antlr.v4.runtime.Parser.triggerExitRuleEvent(Parser.java:422)
~[stormjar.jar:?]
at org.antlr.v4.runtime.Parser.exitRule(Parser.java:632) ~[stormjar.jar:?]
at
org.apache.metron.common.stellar.generated.StellarParser.functions(StellarParser.java:1640)
~[stormjar.jar:?]
at
org.apache.metron.common.stellar.generated.StellarParser.arithmetic_operands(StellarParser.java:1750)
~[stormjar.jar:?]
at
org.apache.metron.common.stellar.generated.StellarParser.arithmetic_expr_mul(StellarParser.java:1537)
~[stormjar.jar:?]
at
org.apache.metron.common.stellar.generated.StellarParser.arithmetic_expr(StellarParser.java:1397)
~[stormjar.jar:?]
at
org.apache.metron.common.stellar.generated.StellarParser.transformation_expr(StellarParser.java:278)
~[stormjar.jar:?]
at
org.apache.metron.common.stellar.generated.StellarParser.transformation(StellarParser.java:147)
~[stormjar.jar:?]
at
org.apache.metron.common.stellar.BaseStellarProcessor.parse(BaseStellarProcessor.java:89)
~[stormjar.jar:?]
... 19 more
On Tue, Feb 28, 2017 at 12:18 AM, Casey Stella <ce...@gmail.com> wrote:
> Ok, try something for me and tell me if it fixes it: adjust the config to
> remove the tickUpdate (it didn't work anyway and we have since removed it
> in 0.3.1).
>
>
> On Mon, Feb 27, 2017 at 6:07 PM, shoggi <sh...@gmail.com> wrote:
>
>> Yes, this is 0.3.0.
>>
>> Here is the full trace.
>>
>> 2017-02-28 00:05:43.367 o.a.m.p.b.ProfileBuilderBolt [INFO] Flushing
>> profile: profile=sketchy_mad, entity=krxd.net
>> 2017-02-28 00:05:43.374 o.a.m.p.b.ProfileBuilderBolt [ERROR] Unexpected
>> failure: message='Unable to execute: org.apache.metron.statistics.o
>> utlier.MedianAbsoluteDeviationFunctions$State cannot be cast to
>> java.util.List', tuple='source: __system:-1, stream: __tick, id: {}, [900]'
>> org.apache.metron.common.dsl.ParseException: Unable to execute:
>> org.apache.metron.statistics.outlier.MedianAbsoluteDeviationFunctions$State
>> cannot be cast to java.util.List
>> at org.apache.metron.common.stellar.StellarCompiler.getResult(StellarCompiler.java:428)
>> ~[stormjar.jar:?]
>> at org.apache.metron.common.stellar.BaseStellarProcessor.parse(BaseStellarProcessor.java:90)
>> ~[stormjar.jar:?]
>> at org.apache.metron.profiler.stellar.DefaultStellarExecutor.ex
>> ecute(DefaultStellarExecutor.java:148) ~[stormjar.jar:?]
>> at org.apache.metron.profiler.stellar.DefaultStellarExecutor.
>> assign(DefaultStellarExecutor.java:94) ~[stormjar.jar:?]
>> at org.apache.metron.profiler.bolt.ProfileBuilderBolt.lambda$
>> null$2(ProfileBuilderBolt.java:258) ~[stormjar.jar:?]
>> at java.util.LinkedHashMap.forEach(LinkedHashMap.java:676) ~[?:1.8.0_77]
>> at org.apache.metron.profiler.bolt.ProfileBuilderBolt.lambda$
>> flush$4(ProfileBuilderBolt.java:258) ~[stormjar.jar:?]
>> at java.util.concurrent.ConcurrentMap.forEach(ConcurrentMap.java:114)
>> ~[?:1.8.0_77]
>> at org.apache.metron.profiler.bolt.ProfileBuilderBolt.flush(ProfileBuilderBolt.java:237)
>> ~[stormjar.jar:?]
>> at org.apache.metron.profiler.bolt.ProfileBuilderBolt.doExecute(ProfileBuilderBolt.java:164)
>> ~[stormjar.jar:?]
>> at org.apache.metron.profiler.bolt.ProfileBuilderBolt.execute(ProfileBuilderBolt.java:144)
>> [stormjar.jar:?]
>> at org.apache.storm.daemon.executor$fn__6571$tuple_action_fn__6573.invoke(executor.clj:734)
>> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>> at org.apache.storm.daemon.executor$mk_task_receiver$fn__6492.invoke(executor.clj:469)
>> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>> at org.apache.storm.disruptor$clojure_handler$reify__6005.onEvent(disruptor.clj:40)
>> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>> at org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:451)
>> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>> at org.apache.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:430)
>> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>> at org.apache.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:73)
>> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>> at org.apache.storm.daemon.executor$fn__6571$fn__6584$fn__6637.invoke(executor.clj:853)
>> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>> at org.apache.storm.util$async_loop$fn__554.invoke(util.clj:484)
>> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>> at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
>> at java.lang.Thread.run(Thread.java:745) [?:1.8.0_77]
>> Caused by: java.lang.ClassCastException: org.apache.metron.statistics.o
>> utlier.MedianAbsoluteDeviationFunctions$State cannot be cast to
>> java.util.List
>> at org.apache.metron.statistics.outlier.MedianAbsoluteDeviation
>> Functions$StateUpdate.apply(MedianAbsoluteDeviationFunctions.java:85)
>> ~[stormjar.jar:?]
>> at org.apache.metron.common.stellar.StellarCompiler.exitTransfo
>> rmationFunc(StellarCompiler.java:246) ~[stormjar.jar:?]
>> at org.apache.metron.common.stellar.generated.StellarParser$Tra
>> nsformationFuncContext.exitRule(StellarParser.java:1617)
>> ~[stormjar.jar:?]
>> at org.antlr.v4.runtime.Parser.triggerExitRuleEvent(Parser.java:422)
>> ~[stormjar.jar:?]
>> at org.antlr.v4.runtime.Parser.exitRule(Parser.java:632)
>> ~[stormjar.jar:?]
>> at org.apache.metron.common.stellar.generated.StellarParser.
>> functions(StellarParser.java:1640) ~[stormjar.jar:?]
>> at org.apache.metron.common.stellar.generated.StellarParser.
>> arithmetic_operands(StellarParser.java:1750) ~[stormjar.jar:?]
>> at org.apache.metron.common.stellar.generated.StellarParser.
>> arithmetic_expr_mul(StellarParser.java:1537) ~[stormjar.jar:?]
>> at org.apache.metron.common.stellar.generated.StellarParser.
>> arithmetic_expr(StellarParser.java:1397) ~[stormjar.jar:?]
>> at org.apache.metron.common.stellar.generated.StellarParser.
>> transformation_expr(StellarParser.java:278) ~[stormjar.jar:?]
>> at org.apache.metron.common.stellar.generated.StellarParser.
>> transformation(StellarParser.java:147) ~[stormjar.jar:?]
>> at org.apache.metron.common.stellar.BaseStellarProcessor.parse(BaseStellarProcessor.java:89)
>> ~[stormjar.jar:?]
>> ... 19 more
>> 2017-02-28 00:05:43.377 o.a.s.d.executor [ERROR]
>> org.apache.metron.common.dsl.ParseException: Unable to execute:
>> org.apache.metron.statistics.outlier.MedianAbsoluteDeviationFunctions$State
>> cannot be cast to java.util.List
>> at org.apache.metron.common.stellar.StellarCompiler.getResult(StellarCompiler.java:428)
>> ~[stormjar.jar:?]
>> at org.apache.metron.common.stellar.BaseStellarProcessor.parse(BaseStellarProcessor.java:90)
>> ~[stormjar.jar:?]
>> at org.apache.metron.profiler.stellar.DefaultStellarExecutor.ex
>> ecute(DefaultStellarExecutor.java:148) ~[stormjar.jar:?]
>> at org.apache.metron.profiler.stellar.DefaultStellarExecutor.
>> assign(DefaultStellarExecutor.java:94) ~[stormjar.jar:?]
>> at org.apache.metron.profiler.bolt.ProfileBuilderBolt.lambda$
>> null$2(ProfileBuilderBolt.java:258) ~[stormjar.jar:?]
>> at java.util.LinkedHashMap.forEach(LinkedHashMap.java:676) ~[?:1.8.0_77]
>> at org.apache.metron.profiler.bolt.ProfileBuilderBolt.lambda$
>> flush$4(ProfileBuilderBolt.java:258) ~[stormjar.jar:?]
>> at java.util.concurrent.ConcurrentMap.forEach(ConcurrentMap.java:114)
>> ~[?:1.8.0_77]
>> at org.apache.metron.profiler.bolt.ProfileBuilderBolt.flush(ProfileBuilderBolt.java:237)
>> ~[stormjar.jar:?]
>> at org.apache.metron.profiler.bolt.ProfileBuilderBolt.doExecute(ProfileBuilderBolt.java:164)
>> ~[stormjar.jar:?]
>> at org.apache.metron.profiler.bolt.ProfileBuilderBolt.execute(ProfileBuilderBolt.java:144)
>> [stormjar.jar:?]
>> at org.apache.storm.daemon.executor$fn__6571$tuple_action_fn__6573.invoke(executor.clj:734)
>> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>> at org.apache.storm.daemon.executor$mk_task_receiver$fn__6492.invoke(executor.clj:469)
>> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>> at org.apache.storm.disruptor$clojure_handler$reify__6005.onEvent(disruptor.clj:40)
>> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>> at org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:451)
>> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>> at org.apache.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:430)
>> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>> at org.apache.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:73)
>> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>> at org.apache.storm.daemon.executor$fn__6571$fn__6584$fn__6637.invoke(executor.clj:853)
>> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>> at org.apache.storm.util$async_loop$fn__554.invoke(util.clj:484)
>> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>> at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
>> at java.lang.Thread.run(Thread.java:745) [?:1.8.0_77]
>> Caused by: java.lang.ClassCastException: org.apache.metron.statistics.o
>> utlier.MedianAbsoluteDeviationFunctions$State cannot be cast to
>> java.util.List
>> at org.apache.metron.statistics.outlier.MedianAbsoluteDeviation
>> Functions$StateUpdate.apply(MedianAbsoluteDeviationFunctions.java:85)
>> ~[stormjar.jar:?]
>> at org.apache.metron.common.stellar.StellarCompiler.exitTransfo
>> rmationFunc(StellarCompiler.java:246) ~[stormjar.jar:?]
>> at org.apache.metron.common.stellar.generated.StellarParser$Tra
>> nsformationFuncContext.exitRule(StellarParser.java:1617)
>> ~[stormjar.jar:?]
>> at org.antlr.v4.runtime.Parser.triggerExitRuleEvent(Parser.java:422)
>> ~[stormjar.jar:?]
>> at org.antlr.v4.runtime.Parser.exitRule(Parser.java:632)
>> ~[stormjar.jar:?]
>> at org.apache.metron.common.stellar.generated.StellarParser.
>> functions(StellarParser.java:1640) ~[stormjar.jar:?]
>> at org.apache.metron.common.stellar.generated.StellarParser.
>> arithmetic_operands(StellarParser.java:1750) ~[stormjar.jar:?]
>> at org.apache.metron.common.stellar.generated.StellarParser.
>> arithmetic_expr_mul(StellarParser.java:1537) ~[stormjar.jar:?]
>> at org.apache.metron.common.stellar.generated.StellarParser.
>> arithmetic_expr(StellarParser.java:1397) ~[stormjar.jar:?]
>> at org.apache.metron.common.stellar.generated.StellarParser.
>> transformation_expr(StellarParser.java:278) ~[stormjar.jar:?]
>> at org.apache.metron.common.stellar.generated.StellarParser.
>> transformation(StellarParser.java:147) ~[stormjar.jar:?]
>> at org.apache.metron.common.stellar.BaseStellarProcessor.parse(BaseStellarProcessor.java:89)
>> ~[stormjar.jar:?]
>> ... 19 more
>>
>>
>>
>> On Mon, Feb 27, 2017 at 11:33 PM, Casey Stella <ce...@gmail.com>
>> wrote:
>>
>>> Any chance there's a full stacktrace associated with that NPE anywhere?
>>> Also, I assume this is 0.3.0, right?
>>>
>>> On Mon, Feb 27, 2017 at 5:29 PM, shoggi <sh...@gmail.com> wrote:
>>>
>>>> Hi all
>>>>
>>>> Has anyone come across following profiler error, when trying to work
>>>> with the MAD functions? The configuration loads fine and I only get the
>>>> error when the profiler is pushing its data. I don't think its a typo but
>>>> maybe the logic is not sound.
>>>>
>>>> Error from the profiler worker:
>>>> 2017-02-27 23:20:43.368 o.a.m.p.b.ProfileBuilderBolt [INFO] Flushing
>>>> profile: profile=sketchy_mad, entity=krxd.net
>>>> 2017-02-27 23:20:43.368 o.a.m.p.b.ProfileBuilderBolt [ERROR] Unexpected
>>>> failure: message='null', tuple='source: __system:-1, stream: __tick, id:
>>>> {}, [900]'
>>>> java.lang.NullPointerException
>>>> 2017-02-27 23:20:43.368 o.a.s.d.executor [ERROR]
>>>> java.lang.NullPointerException
>>>>
>>>>
>>>> My Profiler config:
>>>>
>>>> {
>>>> "profile": "sketchy_mad",
>>>> "foreach": "if exists(domain_without_subdomains) then
>>>> domain_without_subdomains else 'n/a'",
>>>> "onlyif": "exists(domain_without_subdomains) && source.type ==
>>>> 'squid'",
>>>> "init" : {
>>>> "s": "OUTLIER_MAD_STATE_MERGE(PROFILE_GET('sketchy_mad',
>>>> domain_without_subdomains, 5, 'MINUTES'))"
>>>> },
>>>> "tickUpdate" : {
>>>> "s": "OUTLIER_MAD_STATE_MERGE(PROFILE_GET('sketchy_mad',
>>>> domain_without_subdomains, 5, 'MINUTES'), s)"
>>>> },
>>>> "update": {
>>>> "s": "OUTLIER_MAD_ADD(s, bytes)"
>>>> },
>>>> "result": "s"
>>>> }
>>>>
>>>> Regards
>>>> shoggi
>>>>
>>>
>>>
>>
>
Re: Median Absolute Deviation - Profiler Error
Posted by Casey Stella <ce...@gmail.com>.
Ok, try something for me and tell me if it fixes it: adjust the config to
remove the tickUpdate (it didn't work anyway and we have since removed it
in 0.3.1).
On Mon, Feb 27, 2017 at 6:07 PM, shoggi <sh...@gmail.com> wrote:
> Yes, this is 0.3.0.
>
> Here is the full trace.
>
> 2017-02-28 00:05:43.367 o.a.m.p.b.ProfileBuilderBolt [INFO] Flushing
> profile: profile=sketchy_mad, entity=krxd.net
> 2017-02-28 00:05:43.374 o.a.m.p.b.ProfileBuilderBolt [ERROR] Unexpected
> failure: message='Unable to execute: org.apache.metron.statistics.outlier.
> MedianAbsoluteDeviationFunctions$State cannot be cast to java.util.List',
> tuple='source: __system:-1, stream: __tick, id: {}, [900]'
> org.apache.metron.common.dsl.ParseException: Unable to execute:
> org.apache.metron.statistics.outlier.MedianAbsoluteDeviationFunctions$State
> cannot be cast to java.util.List
> at org.apache.metron.common.stellar.StellarCompiler.
> getResult(StellarCompiler.java:428) ~[stormjar.jar:?]
> at org.apache.metron.common.stellar.BaseStellarProcessor.
> parse(BaseStellarProcessor.java:90) ~[stormjar.jar:?]
> at org.apache.metron.profiler.stellar.DefaultStellarExecutor.execute(
> DefaultStellarExecutor.java:148) ~[stormjar.jar:?]
> at org.apache.metron.profiler.stellar.DefaultStellarExecutor.assign(
> DefaultStellarExecutor.java:94) ~[stormjar.jar:?]
> at org.apache.metron.profiler.bolt.ProfileBuilderBolt.lambda$null$2(ProfileBuilderBolt.java:258)
> ~[stormjar.jar:?]
> at java.util.LinkedHashMap.forEach(LinkedHashMap.java:676) ~[?:1.8.0_77]
> at org.apache.metron.profiler.bolt.ProfileBuilderBolt.lambda$flush$4(ProfileBuilderBolt.java:258)
> ~[stormjar.jar:?]
> at java.util.concurrent.ConcurrentMap.forEach(ConcurrentMap.java:114)
> ~[?:1.8.0_77]
> at org.apache.metron.profiler.bolt.ProfileBuilderBolt.flush(ProfileBuilderBolt.java:237)
> ~[stormjar.jar:?]
> at org.apache.metron.profiler.bolt.ProfileBuilderBolt.
> doExecute(ProfileBuilderBolt.java:164) ~[stormjar.jar:?]
> at org.apache.metron.profiler.bolt.ProfileBuilderBolt.
> execute(ProfileBuilderBolt.java:144) [stormjar.jar:?]
> at org.apache.storm.daemon.executor$fn__6571$tuple_action_fn__6573.invoke(executor.clj:734)
> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
> at org.apache.storm.daemon.executor$mk_task_receiver$fn__6492.invoke(executor.clj:469)
> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
> at org.apache.storm.disruptor$clojure_handler$reify__6005.onEvent(disruptor.clj:40)
> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
> at org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:451)
> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
> at org.apache.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:430)
> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
> at org.apache.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:73)
> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
> at org.apache.storm.daemon.executor$fn__6571$fn__6584$fn__6637.invoke(executor.clj:853)
> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
> at org.apache.storm.util$async_loop$fn__554.invoke(util.clj:484)
> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
> at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
> at java.lang.Thread.run(Thread.java:745) [?:1.8.0_77]
> Caused by: java.lang.ClassCastException: org.apache.metron.statistics.
> outlier.MedianAbsoluteDeviationFunctions$State cannot be cast to
> java.util.List
> at org.apache.metron.statistics.outlier.MedianAbsoluteDeviationFunctio
> ns$StateUpdate.apply(MedianAbsoluteDeviationFunctions.java:85)
> ~[stormjar.jar:?]
> at org.apache.metron.common.stellar.StellarCompiler.
> exitTransformationFunc(StellarCompiler.java:246) ~[stormjar.jar:?]
> at org.apache.metron.common.stellar.generated.StellarParser$
> TransformationFuncContext.exitRule(StellarParser.java:1617)
> ~[stormjar.jar:?]
> at org.antlr.v4.runtime.Parser.triggerExitRuleEvent(Parser.java:422)
> ~[stormjar.jar:?]
> at org.antlr.v4.runtime.Parser.exitRule(Parser.java:632) ~[stormjar.jar:?]
> at org.apache.metron.common.stellar.generated.StellarParser.functions(StellarParser.java:1640)
> ~[stormjar.jar:?]
> at org.apache.metron.common.stellar.generated.StellarParser.arithmetic_
> operands(StellarParser.java:1750) ~[stormjar.jar:?]
> at org.apache.metron.common.stellar.generated.
> StellarParser.arithmetic_expr_mul(StellarParser.java:1537)
> ~[stormjar.jar:?]
> at org.apache.metron.common.stellar.generated.
> StellarParser.arithmetic_expr(StellarParser.java:1397) ~[stormjar.jar:?]
> at org.apache.metron.common.stellar.generated.
> StellarParser.transformation_expr(StellarParser.java:278)
> ~[stormjar.jar:?]
> at org.apache.metron.common.stellar.generated.
> StellarParser.transformation(StellarParser.java:147) ~[stormjar.jar:?]
> at org.apache.metron.common.stellar.BaseStellarProcessor.
> parse(BaseStellarProcessor.java:89) ~[stormjar.jar:?]
> ... 19 more
> 2017-02-28 00:05:43.377 o.a.s.d.executor [ERROR]
> org.apache.metron.common.dsl.ParseException: Unable to execute:
> org.apache.metron.statistics.outlier.MedianAbsoluteDeviationFunctions$State
> cannot be cast to java.util.List
> at org.apache.metron.common.stellar.StellarCompiler.
> getResult(StellarCompiler.java:428) ~[stormjar.jar:?]
> at org.apache.metron.common.stellar.BaseStellarProcessor.
> parse(BaseStellarProcessor.java:90) ~[stormjar.jar:?]
> at org.apache.metron.profiler.stellar.DefaultStellarExecutor.execute(
> DefaultStellarExecutor.java:148) ~[stormjar.jar:?]
> at org.apache.metron.profiler.stellar.DefaultStellarExecutor.assign(
> DefaultStellarExecutor.java:94) ~[stormjar.jar:?]
> at org.apache.metron.profiler.bolt.ProfileBuilderBolt.lambda$null$2(ProfileBuilderBolt.java:258)
> ~[stormjar.jar:?]
> at java.util.LinkedHashMap.forEach(LinkedHashMap.java:676) ~[?:1.8.0_77]
> at org.apache.metron.profiler.bolt.ProfileBuilderBolt.lambda$flush$4(ProfileBuilderBolt.java:258)
> ~[stormjar.jar:?]
> at java.util.concurrent.ConcurrentMap.forEach(ConcurrentMap.java:114)
> ~[?:1.8.0_77]
> at org.apache.metron.profiler.bolt.ProfileBuilderBolt.flush(ProfileBuilderBolt.java:237)
> ~[stormjar.jar:?]
> at org.apache.metron.profiler.bolt.ProfileBuilderBolt.
> doExecute(ProfileBuilderBolt.java:164) ~[stormjar.jar:?]
> at org.apache.metron.profiler.bolt.ProfileBuilderBolt.
> execute(ProfileBuilderBolt.java:144) [stormjar.jar:?]
> at org.apache.storm.daemon.executor$fn__6571$tuple_action_fn__6573.invoke(executor.clj:734)
> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
> at org.apache.storm.daemon.executor$mk_task_receiver$fn__6492.invoke(executor.clj:469)
> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
> at org.apache.storm.disruptor$clojure_handler$reify__6005.onEvent(disruptor.clj:40)
> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
> at org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:451)
> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
> at org.apache.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:430)
> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
> at org.apache.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:73)
> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
> at org.apache.storm.daemon.executor$fn__6571$fn__6584$fn__6637.invoke(executor.clj:853)
> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
> at org.apache.storm.util$async_loop$fn__554.invoke(util.clj:484)
> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
> at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
> at java.lang.Thread.run(Thread.java:745) [?:1.8.0_77]
> Caused by: java.lang.ClassCastException: org.apache.metron.statistics.
> outlier.MedianAbsoluteDeviationFunctions$State cannot be cast to
> java.util.List
> at org.apache.metron.statistics.outlier.MedianAbsoluteDeviationFunctio
> ns$StateUpdate.apply(MedianAbsoluteDeviationFunctions.java:85)
> ~[stormjar.jar:?]
> at org.apache.metron.common.stellar.StellarCompiler.
> exitTransformationFunc(StellarCompiler.java:246) ~[stormjar.jar:?]
> at org.apache.metron.common.stellar.generated.StellarParser$
> TransformationFuncContext.exitRule(StellarParser.java:1617)
> ~[stormjar.jar:?]
> at org.antlr.v4.runtime.Parser.triggerExitRuleEvent(Parser.java:422)
> ~[stormjar.jar:?]
> at org.antlr.v4.runtime.Parser.exitRule(Parser.java:632) ~[stormjar.jar:?]
> at org.apache.metron.common.stellar.generated.StellarParser.functions(StellarParser.java:1640)
> ~[stormjar.jar:?]
> at org.apache.metron.common.stellar.generated.StellarParser.arithmetic_
> operands(StellarParser.java:1750) ~[stormjar.jar:?]
> at org.apache.metron.common.stellar.generated.
> StellarParser.arithmetic_expr_mul(StellarParser.java:1537)
> ~[stormjar.jar:?]
> at org.apache.metron.common.stellar.generated.
> StellarParser.arithmetic_expr(StellarParser.java:1397) ~[stormjar.jar:?]
> at org.apache.metron.common.stellar.generated.
> StellarParser.transformation_expr(StellarParser.java:278)
> ~[stormjar.jar:?]
> at org.apache.metron.common.stellar.generated.
> StellarParser.transformation(StellarParser.java:147) ~[stormjar.jar:?]
> at org.apache.metron.common.stellar.BaseStellarProcessor.
> parse(BaseStellarProcessor.java:89) ~[stormjar.jar:?]
> ... 19 more
>
>
>
> On Mon, Feb 27, 2017 at 11:33 PM, Casey Stella <ce...@gmail.com> wrote:
>
>> Any chance there's a full stacktrace associated with that NPE anywhere?
>> Also, I assume this is 0.3.0, right?
>>
>> On Mon, Feb 27, 2017 at 5:29 PM, shoggi <sh...@gmail.com> wrote:
>>
>>> Hi all
>>>
>>> Has anyone come across following profiler error, when trying to work
>>> with the MAD functions? The configuration loads fine and I only get the
>>> error when the profiler is pushing its data. I don't think its a typo but
>>> maybe the logic is not sound.
>>>
>>> Error from the profiler worker:
>>> 2017-02-27 23:20:43.368 o.a.m.p.b.ProfileBuilderBolt [INFO] Flushing
>>> profile: profile=sketchy_mad, entity=krxd.net
>>> 2017-02-27 23:20:43.368 o.a.m.p.b.ProfileBuilderBolt [ERROR] Unexpected
>>> failure: message='null', tuple='source: __system:-1, stream: __tick, id:
>>> {}, [900]'
>>> java.lang.NullPointerException
>>> 2017-02-27 23:20:43.368 o.a.s.d.executor [ERROR]
>>> java.lang.NullPointerException
>>>
>>>
>>> My Profiler config:
>>>
>>> {
>>> "profile": "sketchy_mad",
>>> "foreach": "if exists(domain_without_subdomains) then
>>> domain_without_subdomains else 'n/a'",
>>> "onlyif": "exists(domain_without_subdomains) && source.type ==
>>> 'squid'",
>>> "init" : {
>>> "s": "OUTLIER_MAD_STATE_MERGE(PROFILE_GET('sketchy_mad',
>>> domain_without_subdomains, 5, 'MINUTES'))"
>>> },
>>> "tickUpdate" : {
>>> "s": "OUTLIER_MAD_STATE_MERGE(PROFILE_GET('sketchy_mad',
>>> domain_without_subdomains, 5, 'MINUTES'), s)"
>>> },
>>> "update": {
>>> "s": "OUTLIER_MAD_ADD(s, bytes)"
>>> },
>>> "result": "s"
>>> }
>>>
>>> Regards
>>> shoggi
>>>
>>
>>
>
Re: Median Absolute Deviation - Profiler Error
Posted by shoggi <sh...@gmail.com>.
Yes, this is 0.3.0.
Here is the full trace.
2017-02-28 00:05:43.367 o.a.m.p.b.ProfileBuilderBolt [INFO] Flushing
profile: profile=sketchy_mad, entity=krxd.net
2017-02-28 00:05:43.374 o.a.m.p.b.ProfileBuilderBolt [ERROR] Unexpected
failure: message='Unable to execute:
org.apache.metron.statistics.outlier.MedianAbsoluteDeviationFunctions$State
cannot be cast to java.util.List', tuple='source: __system:-1, stream:
__tick, id: {}, [900]'
org.apache.metron.common.dsl.ParseException: Unable to execute:
org.apache.metron.statistics.outlier.MedianAbsoluteDeviationFunctions$State
cannot be cast to java.util.List
at
org.apache.metron.common.stellar.StellarCompiler.getResult(StellarCompiler.java:428)
~[stormjar.jar:?]
at
org.apache.metron.common.stellar.BaseStellarProcessor.parse(BaseStellarProcessor.java:90)
~[stormjar.jar:?]
at
org.apache.metron.profiler.stellar.DefaultStellarExecutor.execute(DefaultStellarExecutor.java:148)
~[stormjar.jar:?]
at
org.apache.metron.profiler.stellar.DefaultStellarExecutor.assign(DefaultStellarExecutor.java:94)
~[stormjar.jar:?]
at
org.apache.metron.profiler.bolt.ProfileBuilderBolt.lambda$null$2(ProfileBuilderBolt.java:258)
~[stormjar.jar:?]
at java.util.LinkedHashMap.forEach(LinkedHashMap.java:676) ~[?:1.8.0_77]
at
org.apache.metron.profiler.bolt.ProfileBuilderBolt.lambda$flush$4(ProfileBuilderBolt.java:258)
~[stormjar.jar:?]
at java.util.concurrent.ConcurrentMap.forEach(ConcurrentMap.java:114)
~[?:1.8.0_77]
at
org.apache.metron.profiler.bolt.ProfileBuilderBolt.flush(ProfileBuilderBolt.java:237)
~[stormjar.jar:?]
at
org.apache.metron.profiler.bolt.ProfileBuilderBolt.doExecute(ProfileBuilderBolt.java:164)
~[stormjar.jar:?]
at
org.apache.metron.profiler.bolt.ProfileBuilderBolt.execute(ProfileBuilderBolt.java:144)
[stormjar.jar:?]
at
org.apache.storm.daemon.executor$fn__6571$tuple_action_fn__6573.invoke(executor.clj:734)
[storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
at
org.apache.storm.daemon.executor$mk_task_receiver$fn__6492.invoke(executor.clj:469)
[storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
at
org.apache.storm.disruptor$clojure_handler$reify__6005.onEvent(disruptor.clj:40)
[storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
at
org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:451)
[storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
at
org.apache.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:430)
[storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
at
org.apache.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:73)
[storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
at
org.apache.storm.daemon.executor$fn__6571$fn__6584$fn__6637.invoke(executor.clj:853)
[storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
at org.apache.storm.util$async_loop$fn__554.invoke(util.clj:484)
[storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_77]
Caused by: java.lang.ClassCastException:
org.apache.metron.statistics.outlier.MedianAbsoluteDeviationFunctions$State
cannot be cast to java.util.List
at
org.apache.metron.statistics.outlier.MedianAbsoluteDeviationFunctions$StateUpdate.apply(MedianAbsoluteDeviationFunctions.java:85)
~[stormjar.jar:?]
at
org.apache.metron.common.stellar.StellarCompiler.exitTransformationFunc(StellarCompiler.java:246)
~[stormjar.jar:?]
at
org.apache.metron.common.stellar.generated.StellarParser$TransformationFuncContext.exitRule(StellarParser.java:1617)
~[stormjar.jar:?]
at org.antlr.v4.runtime.Parser.triggerExitRuleEvent(Parser.java:422)
~[stormjar.jar:?]
at org.antlr.v4.runtime.Parser.exitRule(Parser.java:632) ~[stormjar.jar:?]
at
org.apache.metron.common.stellar.generated.StellarParser.functions(StellarParser.java:1640)
~[stormjar.jar:?]
at
org.apache.metron.common.stellar.generated.StellarParser.arithmetic_operands(StellarParser.java:1750)
~[stormjar.jar:?]
at
org.apache.metron.common.stellar.generated.StellarParser.arithmetic_expr_mul(StellarParser.java:1537)
~[stormjar.jar:?]
at
org.apache.metron.common.stellar.generated.StellarParser.arithmetic_expr(StellarParser.java:1397)
~[stormjar.jar:?]
at
org.apache.metron.common.stellar.generated.StellarParser.transformation_expr(StellarParser.java:278)
~[stormjar.jar:?]
at
org.apache.metron.common.stellar.generated.StellarParser.transformation(StellarParser.java:147)
~[stormjar.jar:?]
at
org.apache.metron.common.stellar.BaseStellarProcessor.parse(BaseStellarProcessor.java:89)
~[stormjar.jar:?]
... 19 more
2017-02-28 00:05:43.377 o.a.s.d.executor [ERROR]
org.apache.metron.common.dsl.ParseException: Unable to execute:
org.apache.metron.statistics.outlier.MedianAbsoluteDeviationFunctions$State
cannot be cast to java.util.List
at
org.apache.metron.common.stellar.StellarCompiler.getResult(StellarCompiler.java:428)
~[stormjar.jar:?]
at
org.apache.metron.common.stellar.BaseStellarProcessor.parse(BaseStellarProcessor.java:90)
~[stormjar.jar:?]
at
org.apache.metron.profiler.stellar.DefaultStellarExecutor.execute(DefaultStellarExecutor.java:148)
~[stormjar.jar:?]
at
org.apache.metron.profiler.stellar.DefaultStellarExecutor.assign(DefaultStellarExecutor.java:94)
~[stormjar.jar:?]
at
org.apache.metron.profiler.bolt.ProfileBuilderBolt.lambda$null$2(ProfileBuilderBolt.java:258)
~[stormjar.jar:?]
at java.util.LinkedHashMap.forEach(LinkedHashMap.java:676) ~[?:1.8.0_77]
at
org.apache.metron.profiler.bolt.ProfileBuilderBolt.lambda$flush$4(ProfileBuilderBolt.java:258)
~[stormjar.jar:?]
at java.util.concurrent.ConcurrentMap.forEach(ConcurrentMap.java:114)
~[?:1.8.0_77]
at
org.apache.metron.profiler.bolt.ProfileBuilderBolt.flush(ProfileBuilderBolt.java:237)
~[stormjar.jar:?]
at
org.apache.metron.profiler.bolt.ProfileBuilderBolt.doExecute(ProfileBuilderBolt.java:164)
~[stormjar.jar:?]
at
org.apache.metron.profiler.bolt.ProfileBuilderBolt.execute(ProfileBuilderBolt.java:144)
[stormjar.jar:?]
at
org.apache.storm.daemon.executor$fn__6571$tuple_action_fn__6573.invoke(executor.clj:734)
[storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
at
org.apache.storm.daemon.executor$mk_task_receiver$fn__6492.invoke(executor.clj:469)
[storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
at
org.apache.storm.disruptor$clojure_handler$reify__6005.onEvent(disruptor.clj:40)
[storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
at
org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:451)
[storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
at
org.apache.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:430)
[storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
at
org.apache.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:73)
[storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
at
org.apache.storm.daemon.executor$fn__6571$fn__6584$fn__6637.invoke(executor.clj:853)
[storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
at org.apache.storm.util$async_loop$fn__554.invoke(util.clj:484)
[storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_77]
Caused by: java.lang.ClassCastException:
org.apache.metron.statistics.outlier.MedianAbsoluteDeviationFunctions$State
cannot be cast to java.util.List
at
org.apache.metron.statistics.outlier.MedianAbsoluteDeviationFunctions$StateUpdate.apply(MedianAbsoluteDeviationFunctions.java:85)
~[stormjar.jar:?]
at
org.apache.metron.common.stellar.StellarCompiler.exitTransformationFunc(StellarCompiler.java:246)
~[stormjar.jar:?]
at
org.apache.metron.common.stellar.generated.StellarParser$TransformationFuncContext.exitRule(StellarParser.java:1617)
~[stormjar.jar:?]
at org.antlr.v4.runtime.Parser.triggerExitRuleEvent(Parser.java:422)
~[stormjar.jar:?]
at org.antlr.v4.runtime.Parser.exitRule(Parser.java:632) ~[stormjar.jar:?]
at
org.apache.metron.common.stellar.generated.StellarParser.functions(StellarParser.java:1640)
~[stormjar.jar:?]
at
org.apache.metron.common.stellar.generated.StellarParser.arithmetic_operands(StellarParser.java:1750)
~[stormjar.jar:?]
at
org.apache.metron.common.stellar.generated.StellarParser.arithmetic_expr_mul(StellarParser.java:1537)
~[stormjar.jar:?]
at
org.apache.metron.common.stellar.generated.StellarParser.arithmetic_expr(StellarParser.java:1397)
~[stormjar.jar:?]
at
org.apache.metron.common.stellar.generated.StellarParser.transformation_expr(StellarParser.java:278)
~[stormjar.jar:?]
at
org.apache.metron.common.stellar.generated.StellarParser.transformation(StellarParser.java:147)
~[stormjar.jar:?]
at
org.apache.metron.common.stellar.BaseStellarProcessor.parse(BaseStellarProcessor.java:89)
~[stormjar.jar:?]
... 19 more
On Mon, Feb 27, 2017 at 11:33 PM, Casey Stella <ce...@gmail.com> wrote:
> Any chance there's a full stacktrace associated with that NPE anywhere?
> Also, I assume this is 0.3.0, right?
>
> On Mon, Feb 27, 2017 at 5:29 PM, shoggi <sh...@gmail.com> wrote:
>
>> Hi all
>>
>> Has anyone come across following profiler error, when trying to work with
>> the MAD functions? The configuration loads fine and I only get the error
>> when the profiler is pushing its data. I don't think its a typo but maybe
>> the logic is not sound.
>>
>> Error from the profiler worker:
>> 2017-02-27 23:20:43.368 o.a.m.p.b.ProfileBuilderBolt [INFO] Flushing
>> profile: profile=sketchy_mad, entity=krxd.net
>> 2017-02-27 23:20:43.368 o.a.m.p.b.ProfileBuilderBolt [ERROR] Unexpected
>> failure: message='null', tuple='source: __system:-1, stream: __tick, id:
>> {}, [900]'
>> java.lang.NullPointerException
>> 2017-02-27 23:20:43.368 o.a.s.d.executor [ERROR]
>> java.lang.NullPointerException
>>
>>
>> My Profiler config:
>>
>> {
>> "profile": "sketchy_mad",
>> "foreach": "if exists(domain_without_subdomains) then
>> domain_without_subdomains else 'n/a'",
>> "onlyif": "exists(domain_without_subdomains) && source.type ==
>> 'squid'",
>> "init" : {
>> "s": "OUTLIER_MAD_STATE_MERGE(PROFILE_GET('sketchy_mad',
>> domain_without_subdomains, 5, 'MINUTES'))"
>> },
>> "tickUpdate" : {
>> "s": "OUTLIER_MAD_STATE_MERGE(PROFILE_GET('sketchy_mad',
>> domain_without_subdomains, 5, 'MINUTES'), s)"
>> },
>> "update": {
>> "s": "OUTLIER_MAD_ADD(s, bytes)"
>> },
>> "result": "s"
>> }
>>
>> Regards
>> shoggi
>>
>
>
Re: Median Absolute Deviation - Profiler Error
Posted by Casey Stella <ce...@gmail.com>.
Any chance there's a full stacktrace associated with that NPE anywhere?
Also, I assume this is 0.3.0, right?
On Mon, Feb 27, 2017 at 5:29 PM, shoggi <sh...@gmail.com> wrote:
> Hi all
>
> Has anyone come across following profiler error, when trying to work with
> the MAD functions? The configuration loads fine and I only get the error
> when the profiler is pushing its data. I don't think its a typo but maybe
> the logic is not sound.
>
> Error from the profiler worker:
> 2017-02-27 23:20:43.368 o.a.m.p.b.ProfileBuilderBolt [INFO] Flushing
> profile: profile=sketchy_mad, entity=krxd.net
> 2017-02-27 23:20:43.368 o.a.m.p.b.ProfileBuilderBolt [ERROR] Unexpected
> failure: message='null', tuple='source: __system:-1, stream: __tick, id:
> {}, [900]'
> java.lang.NullPointerException
> 2017-02-27 23:20:43.368 o.a.s.d.executor [ERROR]
> java.lang.NullPointerException
>
>
> My Profiler config:
>
> {
> "profile": "sketchy_mad",
> "foreach": "if exists(domain_without_subdomains) then
> domain_without_subdomains else 'n/a'",
> "onlyif": "exists(domain_without_subdomains) && source.type ==
> 'squid'",
> "init" : {
> "s": "OUTLIER_MAD_STATE_MERGE(PROFILE_GET('sketchy_mad',
> domain_without_subdomains, 5, 'MINUTES'))"
> },
> "tickUpdate" : {
> "s": "OUTLIER_MAD_STATE_MERGE(PROFILE_GET('sketchy_mad',
> domain_without_subdomains, 5, 'MINUTES'), s)"
> },
> "update": {
> "s": "OUTLIER_MAD_ADD(s, bytes)"
> },
> "result": "s"
> }
>
> Regards
> shoggi
>