You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@metron.apache.org by "Nick Allen (JIRA)" <ji...@apache.org> on 2019/02/19 21:17:00 UTC
[jira] [Updated] (METRON-1986) Batch Profiler Fails to Resolve
Stats Stellar Functions
[ https://issues.apache.org/jira/browse/METRON-1986?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Nick Allen updated METRON-1986:
-------------------------------
Fix Version/s: Next + 1
> Batch Profiler Fails to Resolve Stats Stellar Functions
> -------------------------------------------------------
>
> Key: METRON-1986
> URL: https://issues.apache.org/jira/browse/METRON-1986
> Project: Metron
> Issue Type: Bug
> Reporter: Nick Allen
> Assignee: Nick Allen
> Priority: Major
> Fix For: Next + 1
>
> Time Spent: 1h
> Remaining Estimate: 0h
>
> The Batch Profiler does not seem to see the STATS_* functions. When I attempt to run a profile using any of those functions, it fails. It is able to 'see' a limited subset of functions, but not all those that I would expect.
> {code}
> [metron@nat-r7-dwys-metron-10 metron]$ cat config/zookeeper/batch-profiler.json
> {
> "profiles": [
> {
> "profile": "profile-using-stats",
> "foreach": "ip_src_addr",
> "update": { "s": "STATS_ADD(s, length)" },
> "result": "STATS_SUM(s)"
> }
> ],
> "timestampField":"timestamp"
> }
> {code}
> The error indicating that the `STATS_ADD` function cannot be resolved.
> {code}
> 19/01/28 16:17:38 ERROR DefaultProfileBuilder: Unable to apply message to profile: Bad 'update' expression: error='Unable to parse: STATS_ADD(s, length) due to: Unable to resolve function named 'STATS_ADD'. Valid functions are IS_EMAIL,IS_DATE,IS_DOMAIN,IS_URL,PROTOCOL_TO_NAME,IS_IP,DOMAIN_TYPOSQUAT,IS_INTEGER with relevant variables s=missing,length=missing', expr='STATS_ADD(s, length)', profile='get_mean_length', entity='192.168.66.1', variables-available='[java.lang.Object@19fdd93a, java.lang.Object@2b9545d2, java.lang.Object@6eb0a63a, java.lang.Object@62b7a53a, java.lang.Object@5c5c9b0d, java.lang.Object@6aada2b8, java.lang.Object@8decf48, java.lang.Object@d88dc3, java.lang.Object@66b5c5ae, java.lang.Object@5a8bc995, java.lang.Object@3db163af, java.lang.Object@27909f0f, java.lang.Object@5e1b4144, java.lang.Object@74478bd8, java.lang.Object@6881fbe0, java.lang.Object@126bb4ce, java.lang.Object@5053a836, java.lang.Object@512dfe04, java.lang.Object@51cb7ee2, java.lang.Object@5a3af410, java.lang.Object@15546cbe, java.lang.Object@5a58ec73, java.lang.Object@25d929f5, java.lang.Object@7b93967f, java.lang.Object@7eba9414, java.lang.Object@5191c77, java.lang.Object@6220b9cf, java.lang.Object@135e806c, java.lang.Object@7cf1a78, java.lang.Object@36a4d7de, java.lang.Object@d20a5dd, java.lang.Object@664a5335]' org.apache.metron.stellar.dsl.ParseException: Bad 'update' expression: error='Unable to parse: STATS_ADD(s, length) due to: Unable to resolve function named 'STATS_ADD'. Valid functions are IS_EMAIL,IS_DATE,IS_DOMAIN,IS_URL,PROTOCOL_TO_NAME,IS_IP,DOMAIN_TYPOSQUAT,IS_INTEGER with relevant variables s=missing,length=missing', expr='STATS_ADD(s, length)', profile='get_mean_length', entity='192.168.66.1', variables-available='[java.lang.Object@19fdd93a, java.lang.Object@2b9545d2, java.lang.Object@6eb0a63a, java.lang.Object@62b7a53a, java.lang.Object@5c5c9b0d, java.lang.Object@6aada2b8, java.lang.Object@8decf48, java.lang.Object@d88dc3, java.lang.Object@66b5c5ae, java.lang.Object@5a8bc995, java.lang.Object@3db163af, java.lang.Object@27909f0f, java.lang.Object@5e1b4144, java.lang.Object@74478bd8, java.lang.Object@6881fbe0, java.lang.Object@126bb4ce, java.lang.Object@5053a836, java.lang.Object@512dfe04, java.lang.Object@51cb7ee2, java.lang.Object@5a3af410, java.lang.Object@15546cbe, java.lang.Object@5a58ec73, java.lang.Object@25d929f5, java.lang.Object@7b93967f, java.lang.Object@7eba9414, java.lang.Object@5191c77, java.lang.Object@6220b9cf, java.lang.Object@135e806c, java.lang.Object@7cf1a78, java.lang.Object@36a4d7de, java.lang.Object@d20a5dd, java.lang.Object@664a5335]' at org.apache.metron.profiler.DefaultProfileBuilder.assign(DefaultProfileBuilder.java:303) at org.apache.metron.profiler.DefaultProfileBuilder.apply(DefaultProfileBuilder.java:141) at org.apache.metron.profiler.DefaultMessageDistributor.distribute(DefaultMessageDistributor.java:168) at org.apache.metron.profiler.spark.function.ProfileBuilderFunction.call(ProfileBuilderFunction.java:88) at org.apache.metron.profiler.spark.function.ProfileBuilderFunction.call(ProfileBuilderFunction.java:49) at org.apache.spark.sql.KeyValueGroupedDataset$$anonfun$mapGroups$1.apply(KeyValueGroupedDataset.scala:219) at org.apache.spark.sql.KeyValueGroupedDataset$$anonfun$mapGroups$1.apply(KeyValueGroupedDataset.scala:219) at org.apache.spark.sql.KeyValueGroupedDataset$$anonfun$1.apply(KeyValueGroupedDataset.scala:196) at org.apache.spark.sql.KeyValueGroupedDataset$$anonfun$1.apply(KeyValueGroupedDataset.scala:196) at org.apache.spark.sql.execution.MapGroupsExec$$anonfun$10$$anonfun$apply$4.apply(objects.scala:337) at org.apache.spark.sql.execution.MapGroupsExec$$anonfun$10$$anonfun$apply$4.apply(objects.scala:336) at scala.collection.Iterator$$anon$12.nextCur(Iterator.scala:434) at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:440) at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage2.processNext(Unknown Source) at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43) at org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$10$$anon$1.hasNext(WholeStageCodegenExec.scala:614) at org.apache.spark.sql.execution.columnar.InMemoryRelation$$anonfun$2$$anon$1.hasNext(InMemoryRelation.scala:149) at org.apache.spark.storage.memory.MemoryStore.putIteratorAsValues(MemoryStore.scala:216) at org.apache.spark.storage.BlockManager$$anonfun$doPutIterator$1.apply(BlockManager.scala:1092) at org.apache.spark.storage.BlockManager$$anonfun$doPutIterator$1.apply(BlockManager.scala:1083) at org.apache.spark.storage.BlockManager.doPut(BlockManager.scala:1018) at org.apache.spark.storage.BlockManager.doPutIterator(BlockManager.scala:1083) at org.apache.spark.storage.BlockManager.getOrElseUpdate(BlockManager.scala:809) at org.apache.spark.rdd.RDD.getOrCompute(RDD.scala:335) at org.apache.spark.rdd.RDD.iterator(RDD.scala:286) at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:324) at org.apache.spark.rdd.RDD.iterator(RDD.scala:288) at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:324) at org.apache.spark.rdd.RDD.iterator(RDD.scala:288) at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:324) at org.apache.spark.rdd.RDD.iterator(RDD.scala:288) at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:324) at org.apache.spark.rdd.RDD.iterator(RDD.scala:288) at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:96) at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:53) at org.apache.spark.scheduler.Task.run(Task.scala:109) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:345) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: org.apache.metron.stellar.dsl.ParseException: Unable to parse: STATS_ADD(s, length) due to: Unable to resolve function named 'STATS_ADD'. Valid functions are IS_EMAIL,IS_DATE,IS_DOMAIN,IS_URL,PROTOCOL_TO_NAME,IS_IP,DOMAIN_TYPOSQUAT,IS_INTEGER with relevant variables s=missing,length=missing at org.apache.metron.stellar.common.BaseStellarProcessor.createException(BaseStellarProcessor.java:173) at org.apache.metron.stellar.common.BaseStellarProcessor.parse(BaseStellarProcessor.java:154) at org.apache.metron.stellar.common.DefaultStellarStatefulExecutor.execute(DefaultStellarStatefulExecutor.java:160) at org.apache.metron.stellar.common.DefaultStellarStatefulExecutor.assign(DefaultStellarStatefulExecutor.java:95) at org.apache.metron.profiler.DefaultProfileBuilder.assign(DefaultProfileBuilder.java:291) ... 43 more Caused by: org.apache.metron.stellar.dsl.ParseException: Unable to resolve function named 'STATS_ADD'. Valid functions are IS_EMAIL,IS_DATE,IS_DOMAIN,IS_URL,PROTOCOL_TO_NAME,IS_IP,DOMAIN_TYPOSQUAT,IS_INTEGER at org.apache.metron.stellar.common.StellarCompiler.resolveFunction(StellarCompiler.java:696) at org.apache.metron.stellar.common.StellarCompiler.lambda$exitTransformationFunc$13(StellarCompiler.java:659) at org.apache.metron.stellar.common.StellarCompiler$Expression.apply(StellarCompiler.java:259) at org.apache.metron.stellar.common.BaseStellarProcessor.parse(BaseStellarProcessor.java:151) ... 46 more Caused by: java.lang.IllegalStateException: Unknown function: `STATS_ADD` at org.apache.metron.stellar.dsl.functions.resolver.BaseFunctionResolver.apply(BaseFunctionResolver.java:150) at org.apache.metron.stellar.dsl.functions.resolver.BaseFunctionResolver.apply(BaseFunctionResolver.java:48) at org.apache.metron.stellar.common.StellarCompiler.resolveFunction(StellarCompiler.java:691) ... 49 more
> {code}
> This is the case even though the Profiler jar contains the STATS functions like `STATS_ADD`.
> {code}
> [metron@nat-r7-dwys-metron-10 metron]$ jar -tvf $PROFILER_JAR | grep "StellarStatisticsFunctions\$Add" 2492 Wed Jan 09 23:13:10 UTC 2019 org/apache/metron/statistics/StellarStatisticsFunctions$Add.class }}
> {code}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)