You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@metron.apache.org by "Nick Allen (JIRA)" <ji...@apache.org> on 2018/11/30 21:10:00 UTC
[jira] [Updated] (METRON-1916) Stellar Classpath Function Resolver
Should Handle ClassNotFoundException
[ https://issues.apache.org/jira/browse/METRON-1916?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Nick Allen updated METRON-1916:
-------------------------------
Description:
Stellar can be executed in an environment where not all of the dependencies are available for each of the functions provided in Stellar Common. When this happens, the classpath resolver blows-up even if you do not use the function that is missing dependencies.
For example this exception is thrown when not all of the dependencies are available for the new REST function. I was only attempting to use STATS_INIT, not the REST function.
{code:java}
2018-11-30 15:47:00 DEBUG BaseFunctionResolver:165 - Resolving Stellar function in class RestFunctions.RestGet
2018-11-30 15:47:00 ERROR DefaultProfileBuilder:302 - Bad 'init' expression: error='Unable to parse: STATS_INIT() due to: org/apache/http/conn/HttpClientConnectionManager', expr='STATS_INIT()', profile='profile-with-stats', entity='global', variables-available='[adapter.threatinteladapter.end.ts, bro_timestamp, ip_dst_port, enrichmentsplitterbolt.splitter.end.ts, enrichmentsplitterbolt.splitter.begin.ts, adapter.hostfromjsonlistadapter.end.ts, adapter.geoadapter.begin.ts, uid, trans_depth, protocol, original_string, ip_dst_addr, threatinteljoinbolt.joiner.ts, host, enrichmentjoinbolt.joiner.ts, adapter.hostfromjsonlistadapter.begin.ts, threatintelsplitterbolt.splitter.begin.ts, ip_src_addr, user_agent, timestamp, method, request_body_len, uri, tags, source.type, adapter.geoadapter.end.ts, referrer, threatintelsplitterbolt.splitter.end.ts, adapter.threatinteladapter.begin.ts, ip_src_port, guid, response_body_len]'
org.apache.metron.stellar.dsl.ParseException: Unable to parse: STATS_INIT() due to: org/apache/http/conn/HttpClientConnectionManager
at org.apache.metron.stellar.common.BaseStellarProcessor.createException(BaseStellarProcessor.java:166)
at org.apache.metron.stellar.common.BaseStellarProcessor.parse(BaseStellarProcessor.java:154)
at org.apache.metron.stellar.common.DefaultStellarStatefulExecutor.execute(DefaultStellarStatefulExecutor.java:160)
at org.apache.metron.stellar.common.DefaultStellarStatefulExecutor.assign(DefaultStellarStatefulExecutor.java:95)
at org.apache.metron.profiler.DefaultProfileBuilder.assign(DefaultProfileBuilder.java:291)
at org.apache.metron.profiler.DefaultProfileBuilder.apply(DefaultProfileBuilder.java:136)
at org.apache.metron.profiler.DefaultMessageDistributor.distribute(DefaultMessageDistributor.java:177)
at org.apache.metron.profiler.storm.ProfileBuilderBolt.handleMessage(ProfileBuilderBolt.java:411)
at org.apache.metron.profiler.storm.ProfileBuilderBolt.execute(ProfileBuilderBolt.java:337)
at org.apache.storm.topology.WindowedBoltExecutor$1.onActivation(WindowedBoltExecutor.java:307)
at org.apache.storm.windowing.WindowManager.onTrigger(WindowManager.java:145)
at org.apache.storm.windowing.WatermarkTimeTriggerPolicy.handleWaterMarkEvent(WatermarkTimeTriggerPolicy.java:80)
at org.apache.storm.windowing.WatermarkTimeTriggerPolicy.track(WatermarkTimeTriggerPolicy.java:48)
at org.apache.storm.windowing.WindowManager.track(WindowManager.java:177)
at org.apache.storm.windowing.WindowManager.add(WindowManager.java:110)
at org.apache.storm.windowing.WaterMarkEventGenerator.run(WaterMarkEventGenerator.java:80)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.NoClassDefFoundError: org/apache/http/conn/HttpClientConnectionManager
at java.lang.Class.getDeclaredConstructors0(Native Method)
at java.lang.Class.privateGetDeclaredConstructors(Class.java:2671)
at java.lang.Class.getConstructor0(Class.java:3075)
at java.lang.Class.getConstructor(Class.java:1825)
at org.apache.metron.stellar.dsl.functions.resolver.BaseFunctionResolver.createFunction(BaseFunctionResolver.java:240)
at org.apache.metron.stellar.dsl.functions.resolver.BaseFunctionResolver.resolveFunction(BaseFunctionResolver.java:194)
at org.apache.metron.stellar.dsl.functions.resolver.BaseFunctionResolver.resolveFunctions(BaseFunctionResolver.java:166)
at com.google.common.base.Suppliers$MemoizingSupplier.get(Suppliers.java:109)
at org.apache.metron.stellar.dsl.functions.resolver.BaseFunctionResolver.apply(BaseFunctionResolver.java:149)
at org.apache.metron.stellar.dsl.functions.resolver.BaseFunctionResolver.apply(BaseFunctionResolver.java:49)
at org.apache.metron.stellar.common.StellarCompiler.resolveFunction(StellarCompiler.java:688)
at org.apache.metron.stellar.common.StellarCompiler.lambda$exitTransformationFunc$13(StellarCompiler.java:656)
at org.apache.metron.stellar.common.StellarCompiler$Expression.apply(StellarCompiler.java:259)
at org.apache.metron.stellar.common.BaseStellarProcessor.parse(BaseStellarProcessor.java:151)
... 21 more{code}
When this happens the Classpath Function Resolver should allow me to use whichever functions can be loaded correctly.
was:
Stellar can be executed in an environment where not all of the dependencies are available for each of the functions provided in Stellar Common. When this happens, the classpath resolver blows-up even if you do not use the function that is missing dependencies.
For example this exception is thrown when not all of the dependencies are available for the new REST function.
{code:java}
2018-11-30 15:58:59 ERROR BaseFunctionResolver:243 - Unable to load org.apache.metron.stellar.dsl.functions.RestFunctions$RestGet because org/apache/http/conn/HttpClientConnectionManager
java.lang.NoClassDefFoundError: org/apache/http/conn/HttpClientConnectionManager
at java.lang.Class.getDeclaredConstructors0(Native Method)
at java.lang.Class.privateGetDeclaredConstructors(Class.java:2671)
at java.lang.Class.getConstructor0(Class.java:3075)
at java.lang.Class.getConstructor(Class.java:1825)
at org.apache.metron.stellar.dsl.functions.resolver.BaseFunctionResolver.createFunction(BaseFunctionResolver.java:240)
at org.apache.metron.stellar.dsl.functions.resolver.BaseFunctionResolver.resolveFunction(BaseFunctionResolver.java:194)
at org.apache.metron.stellar.dsl.functions.resolver.BaseFunctionResolver.resolveFunctions(BaseFunctionResolver.java:166)
at com.google.common.base.Suppliers$MemoizingSupplier.get(Suppliers.java:109)
at org.apache.metron.stellar.dsl.functions.resolver.BaseFunctionResolver.apply(BaseFunctionResolver.java:149)
at org.apache.metron.stellar.dsl.functions.resolver.BaseFunctionResolver.apply(BaseFunctionResolver.java:49)
at org.apache.metron.stellar.common.StellarCompiler.resolveFunction(StellarCompiler.java:688)
at org.apache.metron.stellar.common.StellarCompiler.lambda$exitTransformationFunc$13(StellarCompiler.java:656)
at org.apache.metron.stellar.common.StellarCompiler$Expression.apply(StellarCompiler.java:259)
at org.apache.metron.stellar.common.BaseStellarProcessor.parse(BaseStellarProcessor.java:151)
at org.apache.metron.stellar.common.DefaultStellarStatefulExecutor.execute(DefaultStellarStatefulExecutor.java:160)
at org.apache.metron.stellar.common.DefaultStellarStatefulExecutor.assign(DefaultStellarStatefulExecutor.java:95)
at org.apache.metron.profiler.DefaultProfileBuilder.assign(DefaultProfileBuilder.java:291)
at org.apache.metron.profiler.DefaultProfileBuilder.apply(DefaultProfileBuilder.java:136)
at org.apache.metron.profiler.DefaultMessageDistributor.distribute(DefaultMessageDistributor.java:177)
at org.apache.metron.profiler.storm.ProfileBuilderBolt.handleMessage(ProfileBuilderBolt.java:411)
at org.apache.metron.profiler.storm.ProfileBuilderBolt.execute(ProfileBuilderBolt.java:337)
at org.apache.storm.topology.WindowedBoltExecutor$1.onActivation(WindowedBoltExecutor.java:307)
at org.apache.storm.windowing.WindowManager.onTrigger(WindowManager.java:145)
at org.apache.storm.windowing.WatermarkTimeTriggerPolicy.handleWaterMarkEvent(WatermarkTimeTriggerPolicy.java:80)
at org.apache.storm.windowing.WatermarkTimeTriggerPolicy.track(WatermarkTimeTriggerPolicy.java:48)
at org.apache.storm.windowing.WindowManager.track(WindowManager.java:177)
at org.apache.storm.windowing.WindowManager.add(WindowManager.java:110)
at org.apache.storm.windowing.WaterMarkEventGenerator.run(WaterMarkEventGenerator.java:80)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.ClassNotFoundException: org.apache.http.conn.HttpClientConnectionManager
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 35 more{code}
When this happens the Classpath Function Resolver should allow me to use whichever functions can be loaded correctly.
> Stellar Classpath Function Resolver Should Handle ClassNotFoundException
> ------------------------------------------------------------------------
>
> Key: METRON-1916
> URL: https://issues.apache.org/jira/browse/METRON-1916
> Project: Metron
> Issue Type: Improvement
> Reporter: Nick Allen
> Assignee: Nick Allen
> Priority: Major
>
> Stellar can be executed in an environment where not all of the dependencies are available for each of the functions provided in Stellar Common. When this happens, the classpath resolver blows-up even if you do not use the function that is missing dependencies.
> For example this exception is thrown when not all of the dependencies are available for the new REST function. I was only attempting to use STATS_INIT, not the REST function.
> {code:java}
> 2018-11-30 15:47:00 DEBUG BaseFunctionResolver:165 - Resolving Stellar function in class RestFunctions.RestGet
> 2018-11-30 15:47:00 ERROR DefaultProfileBuilder:302 - Bad 'init' expression: error='Unable to parse: STATS_INIT() due to: org/apache/http/conn/HttpClientConnectionManager', expr='STATS_INIT()', profile='profile-with-stats', entity='global', variables-available='[adapter.threatinteladapter.end.ts, bro_timestamp, ip_dst_port, enrichmentsplitterbolt.splitter.end.ts, enrichmentsplitterbolt.splitter.begin.ts, adapter.hostfromjsonlistadapter.end.ts, adapter.geoadapter.begin.ts, uid, trans_depth, protocol, original_string, ip_dst_addr, threatinteljoinbolt.joiner.ts, host, enrichmentjoinbolt.joiner.ts, adapter.hostfromjsonlistadapter.begin.ts, threatintelsplitterbolt.splitter.begin.ts, ip_src_addr, user_agent, timestamp, method, request_body_len, uri, tags, source.type, adapter.geoadapter.end.ts, referrer, threatintelsplitterbolt.splitter.end.ts, adapter.threatinteladapter.begin.ts, ip_src_port, guid, response_body_len]'
> org.apache.metron.stellar.dsl.ParseException: Unable to parse: STATS_INIT() due to: org/apache/http/conn/HttpClientConnectionManager
> at org.apache.metron.stellar.common.BaseStellarProcessor.createException(BaseStellarProcessor.java:166)
> at org.apache.metron.stellar.common.BaseStellarProcessor.parse(BaseStellarProcessor.java:154)
> at org.apache.metron.stellar.common.DefaultStellarStatefulExecutor.execute(DefaultStellarStatefulExecutor.java:160)
> at org.apache.metron.stellar.common.DefaultStellarStatefulExecutor.assign(DefaultStellarStatefulExecutor.java:95)
> at org.apache.metron.profiler.DefaultProfileBuilder.assign(DefaultProfileBuilder.java:291)
> at org.apache.metron.profiler.DefaultProfileBuilder.apply(DefaultProfileBuilder.java:136)
> at org.apache.metron.profiler.DefaultMessageDistributor.distribute(DefaultMessageDistributor.java:177)
> at org.apache.metron.profiler.storm.ProfileBuilderBolt.handleMessage(ProfileBuilderBolt.java:411)
> at org.apache.metron.profiler.storm.ProfileBuilderBolt.execute(ProfileBuilderBolt.java:337)
> at org.apache.storm.topology.WindowedBoltExecutor$1.onActivation(WindowedBoltExecutor.java:307)
> at org.apache.storm.windowing.WindowManager.onTrigger(WindowManager.java:145)
> at org.apache.storm.windowing.WatermarkTimeTriggerPolicy.handleWaterMarkEvent(WatermarkTimeTriggerPolicy.java:80)
> at org.apache.storm.windowing.WatermarkTimeTriggerPolicy.track(WatermarkTimeTriggerPolicy.java:48)
> at org.apache.storm.windowing.WindowManager.track(WindowManager.java:177)
> at org.apache.storm.windowing.WindowManager.add(WindowManager.java:110)
> at org.apache.storm.windowing.WaterMarkEventGenerator.run(WaterMarkEventGenerator.java:80)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.NoClassDefFoundError: org/apache/http/conn/HttpClientConnectionManager
> at java.lang.Class.getDeclaredConstructors0(Native Method)
> at java.lang.Class.privateGetDeclaredConstructors(Class.java:2671)
> at java.lang.Class.getConstructor0(Class.java:3075)
> at java.lang.Class.getConstructor(Class.java:1825)
> at org.apache.metron.stellar.dsl.functions.resolver.BaseFunctionResolver.createFunction(BaseFunctionResolver.java:240)
> at org.apache.metron.stellar.dsl.functions.resolver.BaseFunctionResolver.resolveFunction(BaseFunctionResolver.java:194)
> at org.apache.metron.stellar.dsl.functions.resolver.BaseFunctionResolver.resolveFunctions(BaseFunctionResolver.java:166)
> at com.google.common.base.Suppliers$MemoizingSupplier.get(Suppliers.java:109)
> at org.apache.metron.stellar.dsl.functions.resolver.BaseFunctionResolver.apply(BaseFunctionResolver.java:149)
> at org.apache.metron.stellar.dsl.functions.resolver.BaseFunctionResolver.apply(BaseFunctionResolver.java:49)
> at org.apache.metron.stellar.common.StellarCompiler.resolveFunction(StellarCompiler.java:688)
> at org.apache.metron.stellar.common.StellarCompiler.lambda$exitTransformationFunc$13(StellarCompiler.java:656)
> at org.apache.metron.stellar.common.StellarCompiler$Expression.apply(StellarCompiler.java:259)
> at org.apache.metron.stellar.common.BaseStellarProcessor.parse(BaseStellarProcessor.java:151)
> ... 21 more{code}
> When this happens the Classpath Function Resolver should allow me to use whichever functions can be loaded correctly.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)