You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@metron.apache.org by cestella <gi...@git.apache.org> on 2017/02/28 07:23:41 UTC
[GitHub] incubator-metron pull request #468: METRON-744: Allow Stellar functions to b...
GitHub user cestella opened a pull request:
https://github.com/apache/incubator-metron/pull/468
METRON-744: Allow Stellar functions to be loaded from HDFS
## Description
The benefit of Stellar is that adding new functionality is as simple as providing a Jar. This enables people who want to integrate with Metron to easy add enrichments or other functionality. The snag currently with this is that we provide a single jar, so all stellar functions that we have available must be dependencies of the main jar that drives the topology plus what local directories we can configure via the storm configs. This makes the process of adding 3rd party jars not as easy as it could be.
Adjust the the following to additionally load classes from a location in HDFS /apps/metron/stellar using something like accumulo ( https://accumulo.apache.org/blog/2014/05/03/accumulo-classloader.html)
* Profiler topology
* Parser topology
* Enrichment topology
* Enrichment Flat file loader
* Enrichment MR loader
One thing to note about this PR is that we shift the way that we load classes from the classpath. We now use [ClassIndex](https://github.com/atteo/classindex) instead of Reflections. This means that stellar classes are found by analyzing a file dropped in META-INF. This has a couple of advantages:
* Blazing fast function loading because we're iterating over every jar, not every class. So fast, there is no noticeable delay for me in starting the REPL.
* Reflections had some serious issues with non-default classloaders that we ran into
For the Stellar function author, this has no impact to you. You just use the annotation and make sure to depend on metron-common (provided dependency) and the file in META-INF is automatically added.
## Testing Plan
TODO
### For all changes:
- [x] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
- [x] Does your PR title start with METRON-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.
- [x] Has your PR been rebased against the latest commit within the target branch (typically master)?
### For code changes:
- [x] Have you included steps to reproduce the behavior or problem that is being changed or addressed?
- [ ] Have you included steps or a guide to how the change may be verified and tested manually?
- [x] Have you ensured that the full suite of tests and checks have been executed in the root incubating-metron folder via:
```
mvn -q clean integration-test install && build_utils/verify_licenses.sh
```
- [x] Have you written or updated unit tests and or integration tests to verify your changes?
- [x] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [x] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent?
### For documentation related changes:
- [x] Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? If not then run the following commands and the verify changes via site-book/target/site/index.html.
```
cd site-book
bin/generate-md.sh
mvn site:site
```
### Note:
Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible.
It is also recommened that [travis-ci](https://travis-ci.org) is set up for your personal repository such that your branches are built there before submitting a pull request.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/cestella/incubator-metron loader_test
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-metron/pull/468.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #468
----
commit 6da8baa421ac416eea6bb13717ae883cfd3d8937
Author: cstella <ce...@gmail.com>
Date: 2017-02-27T22:50:07Z
UPdating parser to use a custom classloader.
commit 269950caafa9919c113e04faf743381377f49a62
Author: cstella <ce...@gmail.com>
Date: 2017-02-28T04:56:13Z
Got VFS and everything integrated.
commit 7c1b1d9df842ffd1ebc0ca2a860471bde6b74f9d
Author: cstella <ce...@gmail.com>
Date: 2017-02-28T05:23:19Z
Updating poms.
commit 1fa5c40cfd7016c3cf44fcc1a508cbd96e0bc2ba
Author: cstella <ce...@gmail.com>
Date: 2017-02-28T06:33:33Z
Added integration test.
commit 0e6bc70fcb4ca49d551ec78b49fb57e9da31f210
Author: cstella <ce...@gmail.com>
Date: 2017-02-28T06:55:25Z
Updating to just depend on accumulo directly
commit b1d21740e197891de369ffcddb4a543e272cd695
Author: cstella <ce...@gmail.com>
Date: 2017-02-28T06:57:43Z
Whoops, forgot dependency.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #468: METRON-744: Allow Stellar functions to b...
Posted by cestella <gi...@git.apache.org>.
Github user cestella commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/468#discussion_r103552286
--- Diff: metron-analytics/metron-profiler/pom.xml ---
@@ -339,6 +339,7 @@
</excludes>
</artifactSet>
<transformers>
+ <transformer implementation="org.atteo.classindex.ClassIndexTransformer"/>
--- End diff --
That's only required for shaded jar so that the shading merges the indices. I need to add some docs around "adding new stellar functions" for sure.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #468: METRON-744: Allow Stellar functions to b...
Posted by cestella <gi...@git.apache.org>.
Github user cestella commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/468#discussion_r103722745
--- Diff: metron-platform/metron-common/src/main/java/org/apache/metron/common/dsl/functions/resolver/ClasspathFunctionResolver.java ---
@@ -65,24 +72,48 @@
*
*/
public class ClasspathFunctionResolver extends BaseFunctionResolver {
--- End diff --
The old one is much slower than this one and this one completely replaces it for the purposes that it was intended for (resolving stellar functions from within the Metron project). This isn't so much an extension as an expansion and a replacement.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #468: METRON-744: Allow Stellar functions to be loade...
Posted by cestella <gi...@git.apache.org>.
Github user cestella commented on the issue:
https://github.com/apache/incubator-metron/pull/468
Did you let it run for a bit? Also, there's a problem with enrichments currently, I'm about to submit a new commit to fix a couple bugs.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #468: METRON-744: Allow Stellar functions to be loade...
Posted by cestella <gi...@git.apache.org>.
Github user cestella commented on the issue:
https://github.com/apache/incubator-metron/pull/468
Ok, I finished the docs and fixed some unit tests that were grumpy. I think that a default hdfs directory really should wait for a followon JIRA and done in the MPack. I'd advise waiting until we are running our the MPack from ansible.
Let me know what you guys think.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #468: METRON-744: Allow Stellar functions to be loade...
Posted by cestella <gi...@git.apache.org>.
Github user cestella commented on the issue:
https://github.com/apache/incubator-metron/pull/468
@dlyle65535 Yeah, in this case, the file was physically not in HDFS though.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #468: METRON-744: Allow Stellar functions to b...
Posted by cestella <gi...@git.apache.org>.
Github user cestella commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/468#discussion_r103722138
--- Diff: metron-platform/metron-common/src/main/java/org/apache/metron/common/dsl/functions/resolver/ClasspathFunctionResolver.java ---
@@ -65,24 +72,48 @@
*
*/
public class ClasspathFunctionResolver extends BaseFunctionResolver {
-
- /**
- * The key for a global property that defines one or more regular expressions
- * that specify what should be included when searching for Stellar functions.
- */
- public static final String STELLAR_SEARCH_INCLUDES_KEY = "stellar.function.resolver.includes";
-
- /**
+ public enum Config {
+ STELLAR_VFS_PATHS("stellar.function.paths", ""),
+ /**
+ * The key for a global property that defines one or more regular expressions
+ * that specify what should be included when searching for Stellar functions.
+ */
+ STELLAR_SEARCH_INCLUDES_KEY("stellar.function.resolver.includes", ""),
--- End diff --
What would you like to rename it to? It seems properly named to me, but I'm open to other names, for sure. I do intend on documenting it, for sure.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #468: METRON-744: Allow Stellar functions to be loade...
Posted by merrimanr <gi...@git.apache.org>.
Github user merrimanr commented on the issue:
https://github.com/apache/incubator-metron/pull/468
I can't think of anything else
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #468: METRON-744: Allow Stellar functions to b...
Posted by cestella <gi...@git.apache.org>.
Github user cestella commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/468#discussion_r103560960
--- Diff: metron-platform/metron-common/src/main/java/org/apache/metron/common/dsl/functions/resolver/ClasspathFunctionResolver.java ---
@@ -65,24 +72,48 @@
*
*/
public class ClasspathFunctionResolver extends BaseFunctionResolver {
--- End diff --
I thought about that, but decided against it. This is in essence still a ClasspathFunctionResolver, just one that can take alternative classloaders. Without specifying the classloader, this functions just as it did before.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #468: METRON-744: Allow Stellar functions to be loade...
Posted by cestella <gi...@git.apache.org>.
Github user cestella commented on the issue:
https://github.com/apache/incubator-metron/pull/468
Correct, though that is entirely hidden from us by the ClassIndex library.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #468: METRON-744: Allow Stellar functions to be loade...
Posted by mmiklavc <gi...@git.apache.org>.
Github user mmiklavc commented on the issue:
https://github.com/apache/incubator-metron/pull/468
Actually, one more thought on the hdfs function path - maybe adding that to global.json by default is better left for when we add dynamic runtime classpath reloading.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #468: METRON-744: Allow Stellar functions to b...
Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/468#discussion_r103548147
--- Diff: metron-analytics/metron-profiler/pom.xml ---
@@ -339,6 +339,7 @@
</excludes>
</artifactSet>
<transformers>
+ <transformer implementation="org.atteo.classindex.ClassIndexTransformer"/>
--- End diff --
Is this what creates the index file that is added to the jar? Do we need this in every project or just the ones that contain Stellar functions?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #468: METRON-744: Allow Stellar functions to be loade...
Posted by mmiklavc <gi...@git.apache.org>.
Github user mmiklavc commented on the issue:
https://github.com/apache/incubator-metron/pull/468
+1 tested this in Vagrant quick-dev.
@cestella I'm good with waiting on the default dir. I was also able to run this through e2e with your latest version. I also ran into issues with vagrant up not loading the geo enrichment. I get the following exception. (I'll add a Jira.)
```
2017-03-01 14:59:24.634 o.a.m.e.a.g.GeoLiteDatabase [ERROR] [Metron] Unable to open new database file /apps/metron/geo/default/GeoLite2-City.mmdb.gz
java.io.FileNotFoundException: File does not exist: /apps/metron/geo/default/GeoLite2-City.mmdb.gz
at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:71)
at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:61)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInt(FSNamesystem.java:1860)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1831)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1744)
at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:693)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:373)
at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:640)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2313)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2309)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2307)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:1.8.0_77]
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:1.8.0_77]
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:1.8.0_77]
at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_77]
at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) ~[stormjar.jar:?]
at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:73) ~[stormjar.jar:?]
at org.apache.hadoop.hdfs.DFSClient.callGetBlockLocations(DFSClient.java:1242) ~[stormjar.jar:?]
at org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:1227) ~[stormjar.jar:?]
at org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:1215) ~[stormjar.jar:?]
at org.apache.hadoop.hdfs.DFSInputStream.fetchLocatedBlocksAndGetLastBlockLength(DFSInputStream.java:303) ~[stormjar.jar:?]
at org.apache.hadoop.hdfs.DFSInputStream.openInfo(DFSInputStream.java:269) ~[stormjar.jar:?]
at org.apache.hadoop.hdfs.DFSInputStream.<init>(DFSInputStream.java:261) ~[stormjar.jar:?]
at org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:1540) ~[stormjar.jar:?]
at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:304) ~[stormjar.jar:?]
at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:299) ~[stormjar.jar:?]
at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) ~[stormjar.jar:?]
at org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:312) ~[stormjar.jar:?]
at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:767) ~[stormjar.jar:?]
at org.apache.metron.enrichment.adapters.geo.GeoLiteDatabase.update(GeoLiteDatabase.java:93) [stormjar.jar:?]
at org.apache.metron.enrichment.bolt.ThreatIntelJoinBolt.prepare(ThreatIntelJoinBolt.java:68) [stormjar.jar:?]
at org.apache.metron.enrichment.bolt.JoinBolt.prepare(JoinBolt.java:87) [stormjar.jar:?]
at org.apache.storm.daemon.executor$fn__6571$fn__6584.invoke(executor.clj:798) [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
at org.apache.storm.util$async_loop$fn__554.invoke(util.clj:482) [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_77]
Caused by: org.apache.hadoop.ipc.RemoteException: File does not exist: /apps/metron/geo/default/GeoLite2-City.mmdb.gz
at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:71)
at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:61)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInt(FSNamesystem.java:1860)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1831)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1744)
at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:693)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:373)
at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:640)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2313)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2309)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2307)
at org.apache.hadoop.ipc.Client.call(Client.java:1476) ~[stormjar.jar:?]
at org.apache.hadoop.ipc.Client.call(Client.java:1407) ~[stormjar.jar:?]
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229) ~[stormjar.jar:?]
at com.sun.proxy.$Proxy46.getBlockLocations(Unknown Source) ~[?:?]
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getBlockLocations(ClientNamenodeProtocolTranslatorPB.java:255) ~[stormjar.jar:?]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_77]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_77]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_77]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_77]
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187) ~[stormjar.jar:?]
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102) ~[stormjar.jar:?]
at com.sun.proxy.$Proxy47.getBlockLocations(Unknown Source) ~[?:?]
at org.apache.hadoop.hdfs.DFSClient.callGetBlockLocations(DFSClient.java:1240) ~[stormjar.jar:?]
... 18 more
2017-03-01 14:59:24.636 o.a.s.util [ERROR] Async loop died!
java.lang.IllegalStateException: [Metron] Unable to update MaxMind database
at org.apache.metron.enrichment.adapters.geo.GeoLiteDatabase.update(GeoLiteDatabase.java:107) ~[stormjar.jar:?]
at org.apache.metron.enrichment.bolt.ThreatIntelJoinBolt.prepare(ThreatIntelJoinBolt.java:68) ~[stormjar.jar:?]
at org.apache.metron.enrichment.bolt.JoinBolt.prepare(JoinBolt.java:87) ~[stormjar.jar:?]
at org.apache.storm.daemon.executor$fn__6571$fn__6584.invoke(executor.clj:798) ~[storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
at org.apache.storm.util$async_loop$fn__554.invoke(util.clj:482) [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_77]
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #468: METRON-744: Allow Stellar functions to b...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/incubator-metron/pull/468
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #468: METRON-744: Allow Stellar functions to b...
Posted by ottobackwards <gi...@git.apache.org>.
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/468#discussion_r103708886
--- Diff: metron-platform/metron-common/src/main/java/org/apache/metron/common/dsl/functions/resolver/ClasspathFunctionResolver.java ---
@@ -65,24 +72,48 @@
*
*/
public class ClasspathFunctionResolver extends BaseFunctionResolver {
-
- /**
- * The key for a global property that defines one or more regular expressions
- * that specify what should be included when searching for Stellar functions.
- */
- public static final String STELLAR_SEARCH_INCLUDES_KEY = "stellar.function.resolver.includes";
-
- /**
+ public enum Config {
+ STELLAR_VFS_PATHS("stellar.function.paths", ""),
+ /**
+ * The key for a global property that defines one or more regular expressions
+ * that specify what should be included when searching for Stellar functions.
+ */
+ STELLAR_SEARCH_INCLUDES_KEY("stellar.function.resolver.includes", ""),
--- End diff --
If we leave it for that purpose, let's rename it and document it as such so it is clear.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #468: METRON-744: Allow Stellar functions to b...
Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/468#discussion_r103549252
--- Diff: metron-platform/metron-common/src/main/java/org/apache/metron/common/dsl/functions/resolver/ClasspathFunctionResolver.java ---
@@ -65,24 +72,48 @@
*
*/
public class ClasspathFunctionResolver extends BaseFunctionResolver {
-
- /**
- * The key for a global property that defines one or more regular expressions
- * that specify what should be included when searching for Stellar functions.
- */
- public static final String STELLAR_SEARCH_INCLUDES_KEY = "stellar.function.resolver.includes";
-
- /**
+ public enum Config {
+ STELLAR_VFS_PATHS("stellar.function.paths", ""),
+ /**
+ * The key for a global property that defines one or more regular expressions
+ * that specify what should be included when searching for Stellar functions.
+ */
+ STELLAR_SEARCH_INCLUDES_KEY("stellar.function.resolver.includes", ""),
--- End diff --
Do we still need this with the index-based approach? We had this here before to minimize how many packages we had to search because it was so time consuming. What purpose does it serve now?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #468: METRON-744: Allow Stellar functions to b...
Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/468#discussion_r103550096
--- Diff: metron-platform/metron-common/src/main/java/org/apache/metron/common/utils/ClassloaderUtil.java ---
@@ -0,0 +1,146 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.common.utils;
+
+import org.apache.accumulo.start.classloader.vfs.UniqueFileReplicator;
+import org.apache.commons.logging.Log;
+import org.apache.commons.vfs2.*;
+import org.apache.commons.vfs2.cache.SoftRefFilesCache;
+import org.apache.commons.vfs2.impl.DefaultFileSystemManager;
+import org.apache.commons.vfs2.impl.FileContentInfoFilenameFactory;
+import org.apache.commons.vfs2.impl.VFSClassLoader;
+import org.apache.commons.vfs2.provider.FileReplicator;
+import org.apache.commons.vfs2.provider.UriParser;
+import org.apache.commons.vfs2.provider.VfsComponent;
+import org.apache.commons.vfs2.provider.VfsComponentContext;
+import org.apache.commons.vfs2.provider.hdfs.HdfsFileProvider;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.*;
+
+public class ClassloaderUtil {
--- End diff --
Javadoc would be awesome. I don't know how I would use these functions. Seems like you put them here because you think there are opportunities for reuse.
Also, whitespace and formatting is a little screwy in this file.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #468: METRON-744: Allow Stellar functions to be loade...
Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on the issue:
https://github.com/apache/incubator-metron/pull/468
My understanding is that an index of the annotated classes is generated at compile-time and packaged in the jar file. When we perform function resolution, we are just searching the index in each jar file, rather than all the classes on the classpath. Is that correct?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #468: METRON-744: Allow Stellar functions to be loade...
Posted by cestella <gi...@git.apache.org>.
Github user cestella commented on the issue:
https://github.com/apache/incubator-metron/pull/468
Ok, in the aftermath of the API PR, I added support for properly setting up stellar to read dependencies from HDFS for the API. As such, I validated the API continues to work by:
* Installing the REST endpoint in vagrant as described in the `metron-rest` documentation
* Starting the service
* Running `curl -u user:password -X GET --header 'Accept: application/json' 'http://node1:8082/api/v1/stellar/list/functions'` to list the functions and validating that `NOW` is showed among them.
Here is the excerpt that I pasted from swagger:
```
{
"name": "NOW",
"description": "Right now!",
"params": [],
"returns": "Timestamp"
},
```
@merrimanr anything else you think I should validate?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #468: METRON-744: Allow Stellar functions to be loade...
Posted by ottobackwards <gi...@git.apache.org>.
Github user ottobackwards commented on the issue:
https://github.com/apache/incubator-metron/pull/468
Run the free up space script
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #468: METRON-744: Allow Stellar functions to be loade...
Posted by cestella <gi...@git.apache.org>.
Github user cestella commented on the issue:
https://github.com/apache/incubator-metron/pull/468
@nickwallen Yes, that's correct. I'd add one further modification to the `ClasspathFunctionResolver` modification, it enables you to specify a classloader and it replaces Reflections with ClassIndex.
The configuration of `ClasspathFunctionResolver` from the `STELLAR` context does link the two, but I'd like to abstract that a bit at some point (allow people to specify their classloader from the config, etc.).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #468: METRON-744: Allow Stellar functions to be loade...
Posted by mmiklavc <gi...@git.apache.org>.
Github user mmiklavc commented on the issue:
https://github.com/apache/incubator-metron/pull/468
@cestella What do you think about adding `"stellar.function.paths" : "hdfs://node1:8020/apps/metron/stellar/.*.jar"` to the global.json by default? This way the order of operations is unimportant for deployment - currently, I had to remove the dummy.json configs that include functions contained in custom-*.jar
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #468: METRON-744: Allow Stellar functions to b...
Posted by cestella <gi...@git.apache.org>.
Github user cestella commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/468#discussion_r103561312
--- Diff: metron-platform/metron-common/src/main/java/org/apache/metron/common/dsl/functions/resolver/ClasspathFunctionResolver.java ---
@@ -65,24 +72,48 @@
*
*/
public class ClasspathFunctionResolver extends BaseFunctionResolver {
-
- /**
- * The key for a global property that defines one or more regular expressions
- * that specify what should be included when searching for Stellar functions.
- */
- public static final String STELLAR_SEARCH_INCLUDES_KEY = "stellar.function.resolver.includes";
-
- /**
+ public enum Config {
+ STELLAR_VFS_PATHS("stellar.function.paths", ""),
+ /**
+ * The key for a global property that defines one or more regular expressions
+ * that specify what should be included when searching for Stellar functions.
+ */
+ STELLAR_SEARCH_INCLUDES_KEY("stellar.function.resolver.includes", ""),
--- End diff --
I'm open to taking it out, but didn't want to revert functionality if it had some use.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #468: METRON-744: Allow Stellar functions to b...
Posted by cestella <gi...@git.apache.org>.
Github user cestella commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/468#discussion_r103724085
--- Diff: metron-analytics/metron-profiler/pom.xml ---
@@ -339,6 +339,7 @@
</excludes>
</artifactSet>
<transformers>
+ <transformer implementation="org.atteo.classindex.ClassIndexTransformer"/>
--- End diff --
A compiler annotation. By virtue of having metron-common as a dependency and therefore ClassIndex as a transitive dependency, you get the compiler annotation run automatically since is in `META-INF/services` ([here](https://github.com/atteo/classindex/blob/master/classindex/src/main/resources/META-INF/services/javax.annotation.processing.Processor) ).
This is an aspect of `javac`, not maven, so it should cross various build mechanisms.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #468: METRON-744: Allow Stellar functions to b...
Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/468#discussion_r103549519
--- Diff: metron-platform/metron-common/src/main/java/org/apache/metron/common/dsl/functions/resolver/ClasspathFunctionResolver.java ---
@@ -65,24 +72,48 @@
*
*/
public class ClasspathFunctionResolver extends BaseFunctionResolver {
-
- /**
- * The key for a global property that defines one or more regular expressions
- * that specify what should be included when searching for Stellar functions.
- */
- public static final String STELLAR_SEARCH_INCLUDES_KEY = "stellar.function.resolver.includes";
-
- /**
+ public enum Config {
+ STELLAR_VFS_PATHS("stellar.function.paths", ""),
--- End diff --
I am not sure what STELLAR_VFS_PATHS property is for.
Whitespace is a little screwy around this neighborhood in the code too.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #468: METRON-744: Allow Stellar functions to b...
Posted by cestella <gi...@git.apache.org>.
Github user cestella commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/468#discussion_r103552339
--- Diff: metron-platform/metron-common/src/main/java/org/apache/metron/common/dsl/functions/resolver/ClasspathFunctionResolver.java ---
@@ -19,22 +19,29 @@
package org.apache.metron.common.dsl.functions.resolver;
import org.apache.commons.lang.StringUtils;
+import org.apache.commons.vfs2.FileObject;
+import org.apache.commons.vfs2.FileSystemException;
+import org.apache.commons.vfs2.impl.VFSClassLoader;
import org.apache.metron.common.dsl.Context;
+import org.apache.metron.common.dsl.Stellar;
import org.apache.metron.common.dsl.StellarFunction;
+import org.apache.metron.common.utils.ClassloaderUtil;
+import org.apache.metron.common.utils.ConversionUtils;
+import org.atteo.classindex.ClassIndex;
import org.reflections.Reflections;
--- End diff --
We are using the Reflections filters, but I guess I could remove them
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #468: METRON-744: Allow Stellar functions to be loade...
Posted by mmiklavc <gi...@git.apache.org>.
Github user mmiklavc commented on the issue:
https://github.com/apache/incubator-metron/pull/468
I was able to invoke the `NOW()` function, however I got "NaN" when I ran `STATS_MEAN(STATS_MERGE(PROFILE_GET('stat', 'global', PROFILE_WINDOW('from 3 minutes ago'))))`. I'm still working through debugging.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #468: METRON-744: Allow Stellar functions to be loade...
Posted by dlyle65535 <gi...@git.apache.org>.
Github user dlyle65535 commented on the issue:
https://github.com/apache/incubator-metron/pull/468
Oh, hey, that stack trace looks familiar. I think Justin fixed this here:
https://github.com/apache/incubator-metron/pull/461.
-D...
On Thu, Mar 2, 2017 at 2:08 PM, Michael Miklavcic <no...@github.com>
wrote:
> +1 tested this in Vagrant quick-dev.
>
> @cestella <https://github.com/cestella> I'm good with waiting on the
> default dir. I was also able to run this through e2e with your latest
> version. I also ran into issues with vagrant up not loading the geo
> enrichment. I get the following exception. (I'll add a Jira.)
>
> 2017-03-01 14:59:24.634 o.a.m.e.a.g.GeoLiteDatabase [ERROR] [Metron] Unable to open new database file /apps/metron/geo/default/GeoLite2-City.mmdb.gz
> java.io.FileNotFoundException: File does not exist: /apps/metron/geo/default/GeoLite2-City.mmdb.gz
> at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:71)
> at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:61)
> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInt(FSNamesystem.java:1860)
> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1831)
> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1744)
> at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:693)
> at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:373)
> at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
> at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:640)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2313)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2309)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2307)
>
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:1.8.0_77]
> at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:1.8.0_77]
> at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:1.8.0_77]
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_77]
> at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) ~[stormjar.jar:?]
> at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:73) ~[stormjar.jar:?]
> at org.apache.hadoop.hdfs.DFSClient.callGetBlockLocations(DFSClient.java:1242) ~[stormjar.jar:?]
> at org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:1227) ~[stormjar.jar:?]
> at org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:1215) ~[stormjar.jar:?]
> at org.apache.hadoop.hdfs.DFSInputStream.fetchLocatedBlocksAndGetLastBlockLength(DFSInputStream.java:303) ~[stormjar.jar:?]
> at org.apache.hadoop.hdfs.DFSInputStream.openInfo(DFSInputStream.java:269) ~[stormjar.jar:?]
> at org.apache.hadoop.hdfs.DFSInputStream.<init>(DFSInputStream.java:261) ~[stormjar.jar:?]
> at org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:1540) ~[stormjar.jar:?]
> at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:304) ~[stormjar.jar:?]
> at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:299) ~[stormjar.jar:?]
> at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) ~[stormjar.jar:?]
> at org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:312) ~[stormjar.jar:?]
> at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:767) ~[stormjar.jar:?]
> at org.apache.metron.enrichment.adapters.geo.GeoLiteDatabase.update(GeoLiteDatabase.java:93) [stormjar.jar:?]
> at org.apache.metron.enrichment.bolt.ThreatIntelJoinBolt.prepare(ThreatIntelJoinBolt.java:68) [stormjar.jar:?]
> at org.apache.metron.enrichment.bolt.JoinBolt.prepare(JoinBolt.java:87) [stormjar.jar:?]
> at org.apache.storm.daemon.executor$fn__6571$fn__6584.invoke(executor.clj:798) [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
> at org.apache.storm.util$async_loop$fn__554.invoke(util.clj:482) [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
> at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
> at java.lang.Thread.run(Thread.java:745) [?:1.8.0_77]
> Caused by: org.apache.hadoop.ipc.RemoteException: File does not exist: /apps/metron/geo/default/GeoLite2-City.mmdb.gz
> at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:71)
> at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:61)
> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInt(FSNamesystem.java:1860)
> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1831)
> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1744)
> at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:693)
> at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:373)
> at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
> at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:640)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2313)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2309)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2307)
>
> at org.apache.hadoop.ipc.Client.call(Client.java:1476) ~[stormjar.jar:?]
> at org.apache.hadoop.ipc.Client.call(Client.java:1407) ~[stormjar.jar:?]
> at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229) ~[stormjar.jar:?]
> at com.sun.proxy.$Proxy46.getBlockLocations(Unknown Source) ~[?:?]
> at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getBlockLocations(ClientNamenodeProtocolTranslatorPB.java:255) ~[stormjar.jar:?]
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_77]
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_77]
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_77]
> at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_77]
> at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187) ~[stormjar.jar:?]
> at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102) ~[stormjar.jar:?]
> at com.sun.proxy.$Proxy47.getBlockLocations(Unknown Source) ~[?:?]
> at org.apache.hadoop.hdfs.DFSClient.callGetBlockLocations(DFSClient.java:1240) ~[stormjar.jar:?]
> ... 18 more
> 2017-03-01 14:59:24.636 o.a.s.util [ERROR] Async loop died!
> java.lang.IllegalStateException: [Metron] Unable to update MaxMind database
> at org.apache.metron.enrichment.adapters.geo.GeoLiteDatabase.update(GeoLiteDatabase.java:107) ~[stormjar.jar:?]
> at org.apache.metron.enrichment.bolt.ThreatIntelJoinBolt.prepare(ThreatIntelJoinBolt.java:68) ~[stormjar.jar:?]
> at org.apache.metron.enrichment.bolt.JoinBolt.prepare(JoinBolt.java:87) ~[stormjar.jar:?]
> at org.apache.storm.daemon.executor$fn__6571$fn__6584.invoke(executor.clj:798) ~[storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
> at org.apache.storm.util$async_loop$fn__554.invoke(util.clj:482) [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
> at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
> at java.lang.Thread.run(Thread.java:745) [?:1.8.0_77]
>
> \u2014
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/apache/incubator-metron/pull/468#issuecomment-283748490>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/ADR6CURMGZB9a17Cr5OY6umzktBJ-2s6ks5rhxPKgaJpZM4MODXQ>
> .
>
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #468: METRON-744: Allow Stellar functions to b...
Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/468#discussion_r103551859
--- Diff: metron-analytics/metron-profiler/pom.xml ---
@@ -339,6 +339,7 @@
</excludes>
</artifactSet>
<transformers>
+ <transformer implementation="org.atteo.classindex.ClassIndexTransformer"/>
--- End diff --
If I create a jar with Stellar functions, but no index file, then the function discovery will not work, correct?
If so, when using my own project outside of Metron that contains Stellar functions, I would need to add this transformer in my POM also, right? If so, we should document how to do this somewhere, if you haven't already.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #468: METRON-744: Allow Stellar functions to b...
Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/468#discussion_r103549438
--- Diff: metron-platform/metron-common/src/main/java/org/apache/metron/common/dsl/functions/resolver/ClasspathFunctionResolver.java ---
@@ -65,24 +72,48 @@
*
*/
public class ClasspathFunctionResolver extends BaseFunctionResolver {
--- End diff --
Would it make sense to keep the existing `ClasspathFunctionResolver` and add the new and improved indexed-based one as another implementation of the `FunctionResolver` interface?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #468: METRON-744: Allow Stellar functions to b...
Posted by cestella <gi...@git.apache.org>.
Github user cestella commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/468#discussion_r103561250
--- Diff: metron-platform/metron-common/src/main/java/org/apache/metron/common/dsl/functions/resolver/ClasspathFunctionResolver.java ---
@@ -65,24 +72,48 @@
*
*/
public class ClasspathFunctionResolver extends BaseFunctionResolver {
-
- /**
- * The key for a global property that defines one or more regular expressions
- * that specify what should be included when searching for Stellar functions.
- */
- public static final String STELLAR_SEARCH_INCLUDES_KEY = "stellar.function.resolver.includes";
-
- /**
+ public enum Config {
+ STELLAR_VFS_PATHS("stellar.function.paths", ""),
+ /**
+ * The key for a global property that defines one or more regular expressions
+ * that specify what should be included when searching for Stellar functions.
+ */
+ STELLAR_SEARCH_INCLUDES_KEY("stellar.function.resolver.includes", ""),
--- End diff --
I decided to leave it in. Some organizations may way to whitelist the stellar functions that they use in production to just those from `org.apache.metron` and `com.my.org` for instance.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
Re: [GitHub] incubator-metron issue #468: METRON-744: Allow Stellar
functions to be loade...
Posted by Casey Stella <ce...@gmail.com>.
I did not see anything like that in the ansible logs, but I guess I can't
be sure.
On Thu, Mar 2, 2017 at 8:49 AM, David Lyle <dl...@gmail.com> wrote:
> I have not. Did HDFS die while quick-dev was coming up?
>
> -D...
>
> On Wed, Mar 1, 2017 at 9:15 PM, cestella <gi...@git.apache.org> wrote:
>
> > Github user cestella commented on the issue:
> >
> > https://github.com/apache/incubator-metron/pull/468
> >
> > I ran this through from the beginning to the end without issue to
> make
> > sure the stuff I did in the last few hours didn't affect the test.
> >
> > The one issue, and this is off-topic, is that for some reason I
> cannot
> > get the quick-dev `vagrant up` to stage the geo enrichment database in
> HDFS
> > and, as a result, I get an enrichment exception and data doesn't flow
> > through. Does anyone else have this happen to them?
> >
> >
> >
> > ---
> > If your project is set up for it, you can reply to this email and have
> your
> > reply appear on GitHub as well. If your project does not have this
> feature
> > enabled and wishes so, or if the feature is enabled but not working,
> please
> > contact infrastructure at infrastructure@apache.org or file a JIRA
> ticket
> > with INFRA.
> > ---
> >
>
Re: [GitHub] incubator-metron issue #468: METRON-744: Allow Stellar
functions to be loade...
Posted by David Lyle <dl...@gmail.com>.
I have not. Did HDFS die while quick-dev was coming up?
-D...
On Wed, Mar 1, 2017 at 9:15 PM, cestella <gi...@git.apache.org> wrote:
> Github user cestella commented on the issue:
>
> https://github.com/apache/incubator-metron/pull/468
>
> I ran this through from the beginning to the end without issue to make
> sure the stuff I did in the last few hours didn't affect the test.
>
> The one issue, and this is off-topic, is that for some reason I cannot
> get the quick-dev `vagrant up` to stage the geo enrichment database in HDFS
> and, as a result, I get an enrichment exception and data doesn't flow
> through. Does anyone else have this happen to them?
>
>
>
> ---
> If your project is set up for it, you can reply to this email and have your
> reply appear on GitHub as well. If your project does not have this feature
> enabled and wishes so, or if the feature is enabled but not working, please
> contact infrastructure at infrastructure@apache.org or file a JIRA ticket
> with INFRA.
> ---
>
[GitHub] incubator-metron issue #468: METRON-744: Allow Stellar functions to be loade...
Posted by cestella <gi...@git.apache.org>.
Github user cestella commented on the issue:
https://github.com/apache/incubator-metron/pull/468
I ran this through from the beginning to the end without issue to make sure the stuff I did in the last few hours didn't affect the test.
The one issue, and this is off-topic, is that for some reason I cannot get the quick-dev `vagrant up` to stage the geo enrichment database in HDFS and, as a result, I get an enrichment exception and data doesn't flow through. Does anyone else have this happen to them?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #468: METRON-744: Allow Stellar functions to be loade...
Posted by cestella <gi...@git.apache.org>.
Github user cestella commented on the issue:
https://github.com/apache/incubator-metron/pull/468
Well, yeah, I'm going to add some docs today around the new stellar config. Not sure what kind of diagram I could create, but I'm going to describe:
* The stellar configs in the global config
* How to add your own stellar function to Metron with step-by-step walkthrough
I'll get that done this morning and you guys can take a look on this PR and let me know what I'm missing.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #468: METRON-744: Allow Stellar functions to b...
Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/468#discussion_r103548884
--- Diff: metron-platform/metron-common/src/main/java/org/apache/metron/common/dsl/functions/resolver/ClasspathFunctionResolver.java ---
@@ -19,22 +19,29 @@
package org.apache.metron.common.dsl.functions.resolver;
import org.apache.commons.lang.StringUtils;
+import org.apache.commons.vfs2.FileObject;
+import org.apache.commons.vfs2.FileSystemException;
+import org.apache.commons.vfs2.impl.VFSClassLoader;
import org.apache.metron.common.dsl.Context;
+import org.apache.metron.common.dsl.Stellar;
import org.apache.metron.common.dsl.StellarFunction;
+import org.apache.metron.common.utils.ClassloaderUtil;
+import org.apache.metron.common.utils.ConversionUtils;
+import org.atteo.classindex.ClassIndex;
import org.reflections.Reflections;
--- End diff --
Do we need to drop the `org.reflections.Reflections` import along with the dependency in the pom?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #468: METRON-744: Allow Stellar functions to be loade...
Posted by mmiklavc <gi...@git.apache.org>.
Github user mmiklavc commented on the issue:
https://github.com/apache/incubator-metron/pull/468
I left it run for about 15 minutes. There's data in Kafka, just nothing in the profiler hbase table when I scan. Will update.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #468: METRON-744: Allow Stellar functions to b...
Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/468#discussion_r103729198
--- Diff: metron-analytics/metron-profiler/pom.xml ---
@@ -339,6 +339,7 @@
</excludes>
</artifactSet>
<transformers>
+ <transformer implementation="org.atteo.classindex.ClassIndexTransformer"/>
--- End diff --
Very nice... https://www.jcp.org/en/jsr/detail?id=269
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #468: METRON-744: Allow Stellar functions to b...
Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/468#discussion_r103715008
--- Diff: metron-platform/metron-common/src/main/java/org/apache/metron/common/dsl/functions/resolver/ClasspathFunctionResolver.java ---
@@ -65,24 +72,48 @@
*
*/
public class ClasspathFunctionResolver extends BaseFunctionResolver {
--- End diff --
If there is never case where we might need to use the older mechanism, then I'd be onboard.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #468: METRON-744: Allow Stellar functions to be loade...
Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on the issue:
https://github.com/apache/incubator-metron/pull/468
One thing, I didn't wrap my head around before is there there are really two, independent parts to this PR.
I am not against these co-existing in the same PR, but I want to make sure everyone understands what we have to review here.
(1) The updated `ClasspathFunctionResolver` that uses the class index mechanism over a runtime search.
(2) The VFS stuff in `VFSClassloaderUtil` that allows us to access remote jars.
Correct me, if I am mistaken.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #468: METRON-744: Allow Stellar functions to be loade...
Posted by cestella <gi...@git.apache.org>.
Github user cestella commented on the issue:
https://github.com/apache/incubator-metron/pull/468
By the way, the following gist shows the pom and stellar function used in this acceptance test script. Notice nothing is going on in the gist that would make the author of the function know about the classpath indexing:
https://gist.github.com/cestella/7aea46f1a2031c11afc9f1eba9210c86
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #468: METRON-744: Allow Stellar functions to b...
Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/468#discussion_r103684910
--- Diff: metron-analytics/metron-profiler/pom.xml ---
@@ -339,6 +339,7 @@
</excludes>
</artifactSet>
<transformers>
+ <transformer implementation="org.atteo.classindex.ClassIndexTransformer"/>
--- End diff --
So what generates the class index?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #468: METRON-744: Allow Stellar functions to be loade...
Posted by cestella <gi...@git.apache.org>.
Github user cestella commented on the issue:
https://github.com/apache/incubator-metron/pull/468
I fixed a ton of bugs here in the last commit, so if things were failing for you before now, it's expected.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---