You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@metron.apache.org by "Anand Subramanian (JIRA)" <ji...@apache.org> on 2017/08/23 13:34:00 UTC
[jira] [Created] (METRON-1128) MAAS_GET_ENDPOINT - Unable to
resolve function error seen for squid example
Anand Subramanian created METRON-1128:
-----------------------------------------
Summary: MAAS_GET_ENDPOINT - Unable to resolve function error seen for squid example
Key: METRON-1128
URL: https://issues.apache.org/jira/browse/METRON-1128
Project: Metron
Issue Type: Bug
Reporter: Anand Subramanian
Assignee: Casey Stella
While attempting to run the the DGA MaaS [example|https://github.com/apache/metron/tree/master/metron-analytics/metron-maas-service#example] with squid, a ParseException is seen saying it could not resolve the 'MAAS_GET_ENDPOINT' function. Full exception is pasted at the bottom of the description.
I confirmed that the metron-management jar is present under $METRON_HOME/lib.
Also, the MAAS functions are listed when I checked on the stellar REPL.
[Stellar]>>> ?MAAS_
?MAAS_GET_ENDPOINT ?MAAS_MODEL_APPLY
I have tried to kill and start the squid topology, but it did not resolve the issue.
Full exception:
{code}
017-08-23 09:11:01.113 o.a.s.d.executor [ERROR]
java.lang.IllegalStateException: Unable to process transformation: MAP_GET('is_malicious', MAAS_MODEL_APPLY(MAAS_GET_ENDPOINT('dga'), {'host' : domain_without_subdomains})) for is_malicious because Unable to resolve function named 'MAAS_GET_ENDPOINT'. Valid functions are IS_DATE,ZIP,IS_DOMAIN,MONTH,FORMAT,TRIM,SQRT,WEEK_OF_MONTH,JOIN,MAP_GET,LOG10,FLOOR,TO_INTEGER,YEAR,OUTLIER_MAD_SCORE,WEEK_OF_YEAR,PROTOCOL_TO_NAME,GET_FIRST,FILL_LEFT,STATS_MAX,CHOP,DECODE,PROFILE_FIXED,DOMAIN_REMOVE_SUBDOMAINS,STATS_SUM_SQUARES,REGEXP_GROUP_VAL,STATS_INIT,ENRICHMENT_GET,STATS_SD,HLLP_INIT,GEO_GET,GET,STATS_COUNT,IS_INTEGER,DAY_OF_WEEK,MAP,ENCODE,IS_EMPTY,DAY_OF_YEAR,MAP_EXISTS,TO_LOWER,STATS_SKEWNESS,ENRICHMENT_EXISTS,DAY_OF_MONTH,LENGTH,IN_SUBNET,SPLIT,PROFILER_APPLY,ABS,STATS_VARIANCE,STATS_ADD,PROFILER_FLUSH,TO_UPPER,OUTLIER_MAD_STATE_MERGE,TO_EPOCH_TIMESTAMP,BLOOM_EXISTS,GET_SUPPORTED_ENCODINGS,COS,BLOOM_INIT,URL_TO_PATH,OUTLIER_MAD_ADD,URL_TO_PROTOCOL,TO_FLOAT,STATS_SUM_LOGS,IS_IP,TO_LONG,STATS_MEAN,PROFILE_GET,CHOMP,TAN,LN,LIST_ADD,PROFILER_INIT,ENDS_WITH,DOMAIN_TO_TLD,COUNT_MATCHES,BYTEARRAY_MATCHER,TO_STRING,HLLP_MERGE,HLLP_CARDINALITY,APPEND_IF_MISSING,BLOOM_MERGE,STARTS_WITH,STATS_KURTOSIS,DOMAIN_REMOVE_TLD,LOG2,SYSTEM_ENV_GET,CEILING,HLLP_ADD,SIN,STATS_BIN,REDUCE,IS_EMAIL,REGEXP_MATCH,PROFILE_WINDOW,URL_TO_PORT,ROUND,GET_LAST,FILL_RIGHT,STATS_POPULATION_VARIANCE,SYSTEM_PROPERTY_GET,IS_ENCODING,FILTER,BIN,BLOOM_ADD,STATS_QUADRATIC_MEAN,STATS_GEOMETRIC_MEAN,STATS_MIN,PREPEND_IF_MISSING,URL_TO_HOST,ZIP_LONGEST,IS_URL,STATS_SUM,STATS_MERGE,STATS_PERCENTILE,EXP,STRING_ENTROPY,TO_DOUBLE
at org.apache.metron.common.field.transformation.StellarTransformation.map(StellarTransformation.java:61) ~[stormjar.jar:?]
at org.apache.metron.common.configuration.FieldTransformer.transform(FieldTransformer.java:108) ~[stormjar.jar:?]
at org.apache.metron.common.configuration.FieldTransformer.transformAndUpdate(FieldTransformer.java:120) ~[stormjar.jar:?]
at org.apache.metron.parsers.bolt.ParserBolt.execute(ParserBolt.java:190) [stormjar.jar:?]
at org.apache.storm.daemon.executor$fn__6573$tuple_action_fn__6575.invoke(executor.clj:734) [storm-core-1.0.1.2.5.6.0-40.jar:1.0.1.2.5.6.0-40]
at org.apache.storm.daemon.executor$mk_task_receiver$fn__6494.invoke(executor.clj:466) [storm-core-1.0.1.2.5.6.0-40.jar:1.0.1.2.5.6.0-40]
at org.apache.storm.disruptor$clojure_handler$reify__6007.onEvent(disruptor.clj:40) [storm-core-1.0.1.2.5.6.0-40.jar:1.0.1.2.5.6.0-40]
at org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:451) [storm-core-1.0.1.2.5.6.0-40.jar:1.0.1.2.5.6.0-40]
at org.apache.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:430) [storm-core-1.0.1.2.5.6.0-40.jar:1.0.1.2.5.6.0-40]
at org.apache.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:73) [storm-core-1.0.1.2.5.6.0-40.jar:1.0.1.2.5.6.0-40]
at org.apache.storm.daemon.executor$fn__6573$fn__6586$fn__6639.invoke(executor.clj:853) [storm-core-1.0.1.2.5.6.0-40.jar:1.0.1.2.5.6.0-40]
at org.apache.storm.util$async_loop$fn__554.invoke(util.clj:484) [storm-core-1.0.1.2.5.6.0-40.jar:1.0.1.2.5.6.0-40]
at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_141]
Caused by: org.apache.metron.stellar.dsl.ParseException: Unable to resolve function named 'MAAS_GET_ENDPOINT'. Valid functions are IS_DATE,ZIP,IS_DOMAIN,MONTH,FORMAT,TRIM,SQRT,WEEK_OF_MONTH,JOIN,MAP_GET,LOG10,FLOOR,TO_INTEGER,YEAR,OUTLIER_MAD_SCORE,WEEK_OF_YEAR,PROTOCOL_TO_NAME,GET_FIRST,FILL_LEFT,STATS_MAX,CHOP,DECODE,PROFILE_FIXED,DOMAIN_REMOVE_SUBDOMAINS,STATS_SUM_SQUARES,REGEXP_GROUP_VAL,STATS_INIT,ENRICHMENT_GET,STATS_SD,HLLP_INIT,GEO_GET,GET,STATS_COUNT,IS_INTEGER,DAY_OF_WEEK,MAP,ENCODE,IS_EMPTY,DAY_OF_YEAR,MAP_EXISTS,TO_LOWER,STATS_SKEWNESS,ENRICHMENT_EXISTS,DAY_OF_MONTH,LENGTH,IN_SUBNET,SPLIT,PROFILER_APPLY,ABS,STATS_VARIANCE,STATS_ADD,PROFILER_FLUSH,TO_UPPER,OUTLIER_MAD_STATE_MERGE,TO_EPOCH_TIMESTAMP,BLOOM_EXISTS,GET_SUPPORTED_ENCODINGS,COS,BLOOM_INIT,URL_TO_PATH,OUTLIER_MAD_ADD,URL_TO_PROTOCOL,TO_FLOAT,STATS_SUM_LOGS,IS_IP,TO_LONG,STATS_MEAN,PROFILE_GET,CHOMP,TAN,LN,LIST_ADD,PROFILER_INIT,ENDS_WITH,DOMAIN_TO_TLD,COUNT_MATCHES,BYTEARRAY_MATCHER,TO_STRING,HLLP_MERGE,HLLP_CARDINALITY,APPEND_IF_MISSING,BLOOM_MERGE,STARTS_WITH,STATS_KURTOSIS,DOMAIN_REMOVE_TLD,LOG2,SYSTEM_ENV_GET,CEILING,HLLP_ADD,SIN,STATS_BIN,REDUCE,IS_EMAIL,REGEXP_MATCH,PROFILE_WINDOW,URL_TO_PORT,ROUND,GET_LAST,FILL_RIGHT,STATS_POPULATION_VARIANCE,SYSTEM_PROPERTY_GET,IS_ENCODING,FILTER,BIN,BLOOM_ADD,STATS_QUADRATIC_MEAN,STATS_GEOMETRIC_MEAN,STATS_MIN,PREPEND_IF_MISSING,URL_TO_HOST,ZIP_LONGEST,IS_URL,STATS_SUM,STATS_MERGE,STATS_PERCENTILE,EXP,STRING_ENTROPY,TO_DOUBLE
at org.apache.metron.stellar.common.StellarCompiler.resolveFunction(StellarCompiler.java:588) ~[stormjar.jar:?]
at org.apache.metron.stellar.common.StellarCompiler.lambda$exitTransformationFunc$13(StellarCompiler.java:551) ~[stormjar.jar:?]
at org.apache.metron.stellar.common.StellarCompiler$Expression.apply(StellarCompiler.java:160) ~[stormjar.jar:?]
at org.apache.metron.stellar.common.BaseStellarProcessor.parse(BaseStellarProcessor.java:152) ~[stormjar.jar:?]
at org.apache.metron.common.field.transformation.StellarTransformation.map(StellarTransformation.java:49) ~[stormjar.jar:?]
... 13 more
Caused by: java.lang.IllegalStateException: Unknown function: `MAAS_GET_ENDPOINT`
at org.apache.metron.stellar.dsl.functions.resolver.BaseFunctionResolver.apply(BaseFunctionResolver.java:106) ~[stormjar.jar:?]
at org.apache.metron.stellar.dsl.functions.resolver.BaseFunctionResolver.apply(BaseFunctionResolver.java:47) ~[stormjar.jar:?]
at org.apache.metron.stellar.common.StellarCompiler.resolveFunction(StellarCompiler.java:583) ~[stormjar.jar:?]
at org.apache.metron.stellar.common.StellarCompiler.lambda$exitTransformationFunc$13(StellarCompiler.java:551) ~[stormjar.jar:?]
at org.apache.metron.stellar.common.StellarCompiler$Expression.apply(StellarCompiler.java:160) ~[stormjar.jar:?]
at org.apache.metron.stellar.common.BaseStellarProcessor.parse(BaseStellarProcessor.java:152) ~[stormjar.jar:?]
at org.apache.metron.common.field.transformation.StellarTransformation.map(StellarTransformation.java:49) ~[stormjar.jar:?]
... 13 more
2017-08-23 09:11:01.186 o.a.s.d.executor [ERROR]
java.lang.IllegalStateException: Unable to process transformation: MAP_GET('is_malicious', MAAS_MODEL_APPLY(MAAS_GET_ENDPOINT('dga'), {'host' : domain_without_subdomains})) for is_malicious because Unable to resolve function named 'MAAS_GET_ENDPOINT'. Valid functions are IS_DATE,ZIP,IS_DOMAIN,MONTH,FORMAT,TRIM,SQRT,WEEK_OF_MONTH,JOIN,MAP_GET,LOG10,FLOOR,TO_INTEGER,YEAR,OUTLIER_MAD_SCORE,WEEK_OF_YEAR,PROTOCOL_TO_NAME,GET_FIRST,FILL_LEFT,STATS_MAX,CHOP,DECODE,PROFILE_FIXED,DOMAIN_REMOVE_SUBDOMAINS,STATS_SUM_SQUARES,REGEXP_GROUP_VAL,STATS_INIT,ENRICHMENT_GET,STATS_SD,HLLP_INIT,GEO_GET,GET,STATS_COUNT,IS_INTEGER,DAY_OF_WEEK,MAP,ENCODE,IS_EMPTY,DAY_OF_YEAR,MAP_EXISTS,TO_LOWER,STATS_SKEWNESS,ENRICHMENT_EXISTS,DAY_OF_MONTH,LENGTH,IN_SUBNET,SPLIT,PROFILER_APPLY,ABS,STATS_VARIANCE,STATS_ADD,PROFILER_FLUSH,TO_UPPER,OUTLIER_MAD_STATE_MERGE,TO_EPOCH_TIMESTAMP,BLOOM_EXISTS,GET_SUPPORTED_ENCODINGS,COS,BLOOM_INIT,URL_TO_PATH,OUTLIER_MAD_ADD,URL_TO_PROTOCOL,TO_FLOAT,STATS_SUM_LOGS,IS_IP,TO_LONG,STATS_MEAN,PROFILE_GET,CHOMP,TAN,LN,LIST_ADD,PROFILER_INIT,ENDS_WITH,DOMAIN_TO_TLD,COUNT_MATCHES,BYTEARRAY_MATCHER,TO_STRING,HLLP_MERGE,HLLP_CARDINALITY,APPEND_IF_MISSING,BLOOM_MERGE,STARTS_WITH,STATS_KURTOSIS,DOMAIN_REMOVE_TLD,LOG2,SYSTEM_ENV_GET,CEILING,HLLP_ADD,SIN,STATS_BIN,REDUCE,IS_EMAIL,REGEXP_MATCH,PROFILE_WINDOW,URL_TO_PORT,ROUND,GET_LAST,FILL_RIGHT,STATS_POPULATION_VARIANCE,SYSTEM_PROPERTY_GET,IS_ENCODING,FILTER,BIN,BLOOM_ADD,STATS_QUADRATIC_MEAN,STATS_GEOMETRIC_MEAN,STATS_MIN,PREPEND_IF_MISSING,URL_TO_HOST,ZIP_LONGEST,IS_URL,STATS_SUM,STATS_MERGE,STATS_PERCENTILE,EXP,STRING_ENTROPY,TO_DOUBLE
at org.apache.metron.common.field.transformation.StellarTransformation.map(StellarTransformation.java:61) ~[stormjar.jar:?]
at org.apache.metron.common.configuration.FieldTransformer.transform(FieldTransformer.java:108) ~[stormjar.jar:?]
at org.apache.metron.common.configuration.FieldTransformer.transformAndUpdate(FieldTransformer.java:120) ~[stormjar.jar:?]
at org.apache.metron.parsers.bolt.ParserBolt.execute(ParserBolt.java:190) [stormjar.jar:?]
at org.apache.storm.daemon.executor$fn__6573$tuple_action_fn__6575.invoke(executor.clj:734) [storm-core-1.0.1.2.5.6.0-40.jar:1.0.1.2.5.6.0-40]
at org.apache.storm.daemon.executor$mk_task_receiver$fn__6494.invoke(executor.clj:466) [storm-core-1.0.1.2.5.6.0-40.jar:1.0.1.2.5.6.0-40]
at org.apache.storm.disruptor$clojure_handler$reify__6007.onEvent(disruptor.clj:40) [storm-core-1.0.1.2.5.6.0-40.jar:1.0.1.2.5.6.0-40]
at org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:451) [storm-core-1.0.1.2.5.6.0-40.jar:1.0.1.2.5.6.0-40]
at org.apache.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:430) [storm-core-1.0.1.2.5.6.0-40.jar:1.0.1.2.5.6.0-40]
at org.apache.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:73) [storm-core-1.0.1.2.5.6.0-40.jar:1.0.1.2.5.6.0-40]
at org.apache.storm.daemon.executor$fn__6573$fn__6586$fn__6639.invoke(executor.clj:853) [storm-core-1.0.1.2.5.6.0-40.jar:1.0.1.2.5.6.0-40]
at org.apache.storm.util$async_loop$fn__554.invoke(util.clj:484) [storm-core-1.0.1.2.5.6.0-40.jar:1.0.1.2.5.6.0-40]
at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_141]
Caused by: org.apache.metron.stellar.dsl.ParseException: Unable to resolve function named 'MAAS_GET_ENDPOINT'. Valid functions are IS_DATE,ZIP,IS_DOMAIN,MONTH,FORMAT,TRIM,SQRT,WEEK_OF_MONTH,JOIN,MAP_GET,LOG10,FLOOR,TO_INTEGER,YEAR,OUTLIER_MAD_SCORE,WEEK_OF_YEAR,PROTOCOL_TO_NAME,GET_FIRST,FILL_LEFT,STATS_MAX,CHOP,DECODE,PROFILE_FIXED,DOMAIN_REMOVE_SUBDOMAINS,STATS_SUM_SQUARES,REGEXP_GROUP_VAL,STATS_INIT,ENRICHMENT_GET,STATS_SD,HLLP_INIT,GEO_GET,GET,STATS_COUNT,IS_INTEGER,DAY_OF_WEEK,MAP,ENCODE,IS_EMPTY,DAY_OF_YEAR,MAP_EXISTS,TO_LOWER,STATS_SKEWNESS,ENRICHMENT_EXISTS,DAY_OF_MONTH,LENGTH,IN_SUBNET,SPLIT,PROFILER_APPLY,ABS,STATS_VARIANCE,STATS_ADD,PROFILER_FLUSH,TO_UPPER,OUTLIER_MAD_STATE_MERGE,TO_EPOCH_TIMESTAMP,BLOOM_EXISTS,GET_SUPPORTED_ENCODINGS,COS,BLOOM_INIT,URL_TO_PATH,OUTLIER_MAD_ADD,URL_TO_PROTOCOL,TO_FLOAT,STATS_SUM_LOGS,IS_IP,TO_LONG,STATS_MEAN,PROFILE_GET,CHOMP,TAN,LN,LIST_ADD,PROFILER_INIT,ENDS_WITH,DOMAIN_TO_TLD,COUNT_MATCHES,BYTEARRAY_MATCHER,TO_STRING,HLLP_MERGE,HLLP_CARDINALITY,APPEND_IF_MISSING,BLOOM_MERGE,STARTS_WITH,STATS_KURTOSIS,DOMAIN_REMOVE_TLD,LOG2,SYSTEM_ENV_GET,CEILING,HLLP_ADD,SIN,STATS_BIN,REDUCE,IS_EMAIL,REGEXP_MATCH,PROFILE_WINDOW,URL_TO_PORT,ROUND,GET_LAST,FILL_RIGHT,STATS_POPULATION_VARIANCE,SYSTEM_PROPERTY_GET,IS_ENCODING,FILTER,BIN,BLOOM_ADD,STATS_QUADRATIC_MEAN,STATS_GEOMETRIC_MEAN,STATS_MIN,PREPEND_IF_MISSING,URL_TO_HOST,ZIP_LONGEST,IS_URL,STATS_SUM,STATS_MERGE,STATS_PERCENTILE,EXP,STRING_ENTROPY,TO_DOUBLE
at org.apache.metron.stellar.common.StellarCompiler.resolveFunction(StellarCompiler.java:588) ~[stormjar.jar:?]
at org.apache.metron.stellar.common.StellarCompiler.lambda$exitTransformationFunc$13(StellarCompiler.java:551) ~[stormjar.jar:?]
at org.apache.metron.stellar.common.StellarCompiler$Expression.apply(StellarCompiler.java:160) ~[stormjar.jar:?]
at org.apache.metron.stellar.common.BaseStellarProcessor.parse(BaseStellarProcessor.java:152) ~[stormjar.jar:?]
at org.apache.metron.common.field.transformation.StellarTransformation.map(StellarTransformation.java:49) ~[stormjar.jar:?]
... 13 more
Caused by: java.lang.IllegalStateException: Unknown function: `MAAS_GET_ENDPOINT`
at org.apache.metron.stellar.dsl.functions.resolver.BaseFunctionResolver.apply(BaseFunctionResolver.java:106) ~[stormjar.jar:?]
at org.apache.metron.stellar.dsl.functions.resolver.BaseFunctionResolver.apply(BaseFunctionResolver.java:47) ~[stormjar.jar:?]
at org.apache.metron.stellar.common.StellarCompiler.resolveFunction(StellarCompiler.java:583) ~[stormjar.jar:?]
at org.apache.metron.stellar.common.StellarCompiler.lambda$exitTransformationFunc$13(StellarCompiler.java:551) ~[stormjar.jar:?]
at org.apache.metron.stellar.common.StellarCompiler$Expression.apply(StellarCompiler.java:160) ~[stormjar.jar:?]
at org.apache.metron.stellar.common.BaseStellarProcessor.parse(BaseStellarProcessor.java:152) ~[stormjar.jar:?]
at org.apache.metron.common.field.transformation.StellarTransformation.map(StellarTransformation.java:49) ~[stormjar.jar:?]
... 13 more
{code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)