You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@metron.apache.org by cestella <gi...@git.apache.org> on 2016/08/15 12:48:26 UTC
[GitHub] incubator-metron pull request #210: METRON-366: Add MODEL_APPLY to Stellar
GitHub user cestella opened a pull request:
https://github.com/apache/incubator-metron/pull/210
METRON-366: Add MODEL_APPLY to Stellar
The preferred method of applying models should be via stellar integration. This should be added as a function and made available as a FieldTransformation and as part of Threat Triage.
Testing instructions pending...
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/cestella/incubator-metron METRON-366
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-metron/pull/210.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #210
----
commit 3888a19989d4fd58c54bc9dd84d7a5ac8ddbc7ee
Author: cstella <ce...@gmail.com>
Date: 2016-08-15T12:42:57Z
METRON-366: Add MODEL_APPLY to Stellar
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #210: METRON-366: Add MODEL_APPLY to Stellar
Posted by cestella <gi...@git.apache.org>.
Github user cestella commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/210#discussion_r75541801
--- Diff: metron-analytics/metron-maas-common/src/main/java/org/apache/metron/maas/discovery/ServiceDiscoverer.java ---
@@ -143,14 +167,59 @@ public void unregisterByContainer(String containerId) {
}
}
+ public void blacklist(ModelEndpoint endpoint) {
+ blacklist(toUrl(endpoint.getEndpoint().getUrl()));
+ }
+
+ public void blacklist(URL url) {
+ rwLock.writeLock().lock();
+ blacklist.put(url, true);
+ rwLock.writeLock().unlock();
+ }
+
+ public ModelEndpoint getEndpoint(String modelName) {
+ String version = null;
+ rwLock.readLock().lock();
+ version = modelToCurrentVersion.get(modelName);
+ rwLock.readLock().unlock();
+ if(version == null) {
+ throw new IllegalStateException("Unable to find version for " + modelName);
+ }
+ return getEndpoint(modelName, version);
+ }
+
+ private static URL toUrl(String url) {
+ try {
+ return new URL(url);
+ } catch (MalformedURLException e) {
+ throw new IllegalStateException("Endpoint does not refer to an actual URL");
+ }
+ }
+
+ public ModelEndpoint getEndpoint(String modelName, String modelVersion) {
+ return getEndpoint(new Model(modelName, modelVersion));
+ }
public ModelEndpoint getEndpoint(Model model) {
rwLock.readLock().lock();
try {
List<ModelEndpoint> endpoints = state.get(model);
ModelEndpoint ret = null;
if(endpoints != null) {
- int i = ThreadLocalRandom.current().nextInt(endpoints.size());
- ret = endpoints.get(i);
+ for(int j = 0;j < 10;++j) {
+ int i = ThreadLocalRandom.current().nextInt(endpoints.size());
+ ret = endpoints.get(i);
+ try {
+ if (blacklist.asMap().containsKey(toUrl(ret.getEndpoint().getUrl()))) {
+ continue;
+ }
+ else {
+ return ret;
+ }
+ }
+ catch(IllegalStateException ise) {
+
+ }
--- End diff --
Totally agreed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #210: METRON-366: Add MODEL_APPLY to Stellar
Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/210#discussion_r75541694
--- Diff: metron-analytics/metron-maas-common/src/main/java/org/apache/metron/maas/discovery/ServiceDiscoverer.java ---
@@ -143,14 +167,59 @@ public void unregisterByContainer(String containerId) {
}
}
+ public void blacklist(ModelEndpoint endpoint) {
+ blacklist(toUrl(endpoint.getEndpoint().getUrl()));
+ }
+
+ public void blacklist(URL url) {
+ rwLock.writeLock().lock();
+ blacklist.put(url, true);
+ rwLock.writeLock().unlock();
+ }
+
+ public ModelEndpoint getEndpoint(String modelName) {
+ String version = null;
+ rwLock.readLock().lock();
+ version = modelToCurrentVersion.get(modelName);
+ rwLock.readLock().unlock();
+ if(version == null) {
+ throw new IllegalStateException("Unable to find version for " + modelName);
+ }
+ return getEndpoint(modelName, version);
+ }
+
+ private static URL toUrl(String url) {
+ try {
+ return new URL(url);
+ } catch (MalformedURLException e) {
+ throw new IllegalStateException("Endpoint does not refer to an actual URL");
+ }
+ }
+
+ public ModelEndpoint getEndpoint(String modelName, String modelVersion) {
+ return getEndpoint(new Model(modelName, modelVersion));
+ }
public ModelEndpoint getEndpoint(Model model) {
rwLock.readLock().lock();
try {
List<ModelEndpoint> endpoints = state.get(model);
ModelEndpoint ret = null;
if(endpoints != null) {
- int i = ThreadLocalRandom.current().nextInt(endpoints.size());
- ret = endpoints.get(i);
+ for(int j = 0;j < 10;++j) {
+ int i = ThreadLocalRandom.current().nextInt(endpoints.size());
+ ret = endpoints.get(i);
+ try {
+ if (blacklist.asMap().containsKey(toUrl(ret.getEndpoint().getUrl()))) {
+ continue;
+ }
+ else {
+ return ret;
+ }
+ }
+ catch(IllegalStateException ise) {
+
+ }
--- End diff --
My $.02 would be to throw that in a comment in the empty block. Empty catch blocks are always fishy.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #210: METRON-366: Add MODEL_APPLY to Stellar
Posted by cestella <gi...@git.apache.org>.
Github user cestella commented on the issue:
https://github.com/apache/incubator-metron/pull/210
14 minutes?! Woah! @dlyle65535 is my hero.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #210: METRON-366: Add MODEL_APPLY to Stellar
Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/210#discussion_r75536649
--- Diff: metron-analytics/metron-maas-common/src/main/java/org/apache/metron/maas/discovery/ServiceDiscoverer.java ---
@@ -143,14 +167,59 @@ public void unregisterByContainer(String containerId) {
}
}
+ public void blacklist(ModelEndpoint endpoint) {
+ blacklist(toUrl(endpoint.getEndpoint().getUrl()));
+ }
+
+ public void blacklist(URL url) {
+ rwLock.writeLock().lock();
+ blacklist.put(url, true);
+ rwLock.writeLock().unlock();
+ }
+
+ public ModelEndpoint getEndpoint(String modelName) {
+ String version = null;
+ rwLock.readLock().lock();
+ version = modelToCurrentVersion.get(modelName);
+ rwLock.readLock().unlock();
+ if(version == null) {
+ throw new IllegalStateException("Unable to find version for " + modelName);
+ }
+ return getEndpoint(modelName, version);
+ }
+
+ private static URL toUrl(String url) {
+ try {
+ return new URL(url);
+ } catch (MalformedURLException e) {
+ throw new IllegalStateException("Endpoint does not refer to an actual URL");
+ }
+ }
+
+ public ModelEndpoint getEndpoint(String modelName, String modelVersion) {
+ return getEndpoint(new Model(modelName, modelVersion));
+ }
public ModelEndpoint getEndpoint(Model model) {
rwLock.readLock().lock();
try {
List<ModelEndpoint> endpoints = state.get(model);
ModelEndpoint ret = null;
if(endpoints != null) {
- int i = ThreadLocalRandom.current().nextInt(endpoints.size());
- ret = endpoints.get(i);
+ for(int j = 0;j < 10;++j) {
+ int i = ThreadLocalRandom.current().nextInt(endpoints.size());
+ ret = endpoints.get(i);
+ try {
+ if (blacklist.asMap().containsKey(toUrl(ret.getEndpoint().getUrl()))) {
+ continue;
+ }
+ else {
+ return ret;
+ }
+ }
+ catch(IllegalStateException ise) {
+
+ }
--- End diff --
Do we intend to just eat the exception?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #210: METRON-366: Add MODEL_APPLY to Stellar
Posted by dlyle65535 <gi...@git.apache.org>.
Github user dlyle65535 commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/210#discussion_r75533955
--- Diff: metron-analytics/metron-maas-service/src/main/java/org/apache/metron/maas/submit/ModelSubmission.java ---
@@ -203,7 +203,7 @@ else if(mode.equalsIgnoreCase("LIST")) {
String modelTitle = "Model " + kv.getKey().getName() + " @ " + kv.getKey().getVersion();
System.out.println(modelTitle);
for(ModelEndpoint endpoint : kv.getValue()){
- System.out.println("\t" + endpoint.getContainerId() + " at " + endpoint.getUrl());
+ System.out.println(endpoint);
}
--- End diff --
Got it, makes sense to me.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #210: METRON-366: Add MODEL_APPLY to Stellar
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/incubator-metron/pull/210
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #210: METRON-366: Add MODEL_APPLY to Stellar
Posted by dlyle65535 <gi...@git.apache.org>.
Github user dlyle65535 commented on the issue:
https://github.com/apache/incubator-metron/pull/210
+1, worked like a champ!
I'm going to run it up one more time with some additional skip tags. Since we're close on memory headroom, it doesn't make sense to me to take time to install a bunch of stuff I'll just have to shut down. If I have any success, I'll put the commands here.
I'm also using the new quick-dev image (vagrant box update).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #210: METRON-366: Add MODEL_APPLY to Stellar
Posted by merrimanr <gi...@git.apache.org>.
Github user merrimanr commented on the issue:
https://github.com/apache/incubator-metron/pull/210
+1 by inspection. Will try to run it up on full-dev later today or this weekend. Nice job!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #210: METRON-366: Add MODEL_APPLY to Stellar
Posted by dlyle65535 <gi...@git.apache.org>.
Github user dlyle65535 commented on the issue:
https://github.com/apache/incubator-metron/pull/210
Also successfully tested doing the following:
1) Rather than run.sh, I ran:
``` vagrant --ansible-tags="hdp-deploy,metron" --ansible-skip-tags="solr,sensors,start,report,monit" up ```
2) Started enrichment and indexing topologies:
```
/usr/metron/0.2.0BETA/bin/start_enrichment_topology.sh
/usr/metron/0.2.0BETA/bin/start_elasticsearch_topology.sh
```
3) Followed your instructions
This dropped the startup time to just over 14 minutes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #210: METRON-366: Add MODEL_APPLY to Stellar
Posted by cestella <gi...@git.apache.org>.
Github user cestella commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/210#discussion_r75539635
--- Diff: metron-analytics/metron-maas-common/src/main/java/org/apache/metron/maas/discovery/ServiceDiscoverer.java ---
@@ -40,8 +48,16 @@
private ServiceDiscovery<ModelEndpoint> serviceDiscovery;
private Map<Model, List<ModelEndpoint>> state = new HashMap<>();
private Map<String, ServiceInstance<ModelEndpoint>> containerToEndpoint = new HashMap<>();
+ private Map<String, String> modelToCurrentVersion = new HashMap<>();
+ private Cache<URL, Boolean> blacklist;
public ServiceDiscoverer(CuratorFramework client, String root) {
+ blacklist = CacheBuilder.newBuilder()
+ .concurrencyLevel(4)
+ .weakKeys()
+ .expireAfterWrite(10, TimeUnit.MINUTES)
+ .build();
+
JsonInstanceSerializer<ModelEndpoint> serializer = new JsonInstanceSerializer<>(ModelEndpoint.class);
serviceDiscovery = ServiceDiscoveryBuilder.builder(ModelEndpoint.class)
--- End diff --
We should create `root` if it does not exist:
```
try {
Stat exists = client.checkExists().forPath(root);
if(exists == null) {
client.create().forPath(root);
}
} catch (Exception e) {
LOG.error("Unable to create path: " + e.getMessage(), e);
}
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #210: METRON-366: Add MODEL_APPLY to Stellar
Posted by cestella <gi...@git.apache.org>.
Github user cestella commented on the issue:
https://github.com/apache/incubator-metron/pull/210
Please note that it is non-optimal to only be able to reference models from the beginning (i.e. parsers) and end (i.e. threat triage) of the pipeline. As a follow-on, I'll be adding an enrichment adapter which can be called from the enrichment or threat triage phase and perform arbitrary stellar statement transformations. This should fill in the gap and allow the user to apply their models anywhere in the pipeline.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #210: METRON-366: Add MODEL_APPLY to Stellar
Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/210#discussion_r75533140
--- Diff: metron-analytics/metron-maas-common/src/main/java/org/apache/metron/maas/discovery/ServiceDiscoverer.java ---
@@ -93,6 +116,7 @@ private void updateState() {
}
}
rwLock.writeLock().lock();
+ this.modelToCurrentVersion = modelToVersion;
this.state = state;
this.containerToEndpoint = containerToEndpoint;
rwLock.writeLock().unlock();
--- End diff --
Shouldn't the unlock be in a finally block?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #210: METRON-366: Add MODEL_APPLY to Stellar
Posted by cestella <gi...@git.apache.org>.
Github user cestella commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/210#discussion_r75534840
--- Diff: metron-analytics/metron-maas-common/src/main/java/org/apache/metron/maas/discovery/ServiceDiscoverer.java ---
@@ -76,12 +92,19 @@ public void resetState() {
private void updateState() {
Map<Model, List<ModelEndpoint>> state = new HashMap<>();
+ Map<String, String> modelToVersion = new HashMap<>();
Map<String, ServiceInstance<ModelEndpoint>> containerToEndpoint = new HashMap<>();
try {
for(String name : serviceDiscovery.queryForNames()) {
for(ServiceInstance<ModelEndpoint> endpoint: serviceDiscovery.queryForInstances(name)) {
ModelEndpoint ep = endpoint.getPayload();
LOG.info("Found model endpoint " + ep);
+ String currentVersion = modelToVersion.getOrDefault(ep.getName(), ep.getVersion());
+ modelToVersion.put( ep.getName()
+ , currentVersion.compareTo(ep.getVersion()) < 0
+ ? ep.getVersion()
+ : currentVersion
+ );
--- End diff --
It's hard for the author to make code easier to read that he did not intentionally make hard to read..after all, I don't find my own code hard to read. :)
That being said, that particular block of code is intended to update a map with the current version of the model which has been discovered. If you request a model without a version, an instance of the most recent version will be returned. I can, perhaps, use another variable and put a comment.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #210: METRON-366: Add MODEL_APPLY to Stellar
Posted by cestella <gi...@git.apache.org>.
Github user cestella commented on the issue:
https://github.com/apache/incubator-metron/pull/210
happy to add javadoc to that package. To answer your question, Model and ModelEndpoint are different in the sense that Model is a reference to the model. The ModelEndpoint is a reference to where the model is currently being served. You generally search for a Model and are returned a set of ModelEndpoints. If you do not specify a version in the search, you will get ModelEndpoints of multiple versions of the same model.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #210: METRON-366: Add MODEL_APPLY to Stellar
Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on the issue:
https://github.com/apache/incubator-metron/pull/210
Do you have documented anywhere the model classes in `org.apache.metron.maas`? These classes seem to represent core abstractions.
A few questions come to mind that some simple javadoc might help with. But one example... A `Model` has a version and a `ModelEndpoint` has a version. Why do they both have versions? Wouldn't they evolve together and so have the same version?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #210: METRON-366: Add MODEL_APPLY to Stellar
Posted by dlyle65535 <gi...@git.apache.org>.
Github user dlyle65535 commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/210#discussion_r75532270
--- Diff: metron-analytics/metron-maas-service/src/main/java/org/apache/metron/maas/submit/ModelSubmission.java ---
@@ -203,7 +203,7 @@ else if(mode.equalsIgnoreCase("LIST")) {
String modelTitle = "Model " + kv.getKey().getName() + " @ " + kv.getKey().getVersion();
System.out.println(modelTitle);
for(ModelEndpoint endpoint : kv.getValue()){
- System.out.println("\t" + endpoint.getContainerId() + " at " + endpoint.getUrl());
+ System.out.println(endpoint);
}
--- End diff --
Not a today thing, but do you think this would be better off as a logger?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #210: METRON-366: Add MODEL_APPLY to Stellar
Posted by cestella <gi...@git.apache.org>.
Github user cestella commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/210#discussion_r75537367
--- Diff: metron-analytics/metron-maas-common/src/main/java/org/apache/metron/maas/discovery/ServiceDiscoverer.java ---
@@ -143,14 +167,59 @@ public void unregisterByContainer(String containerId) {
}
}
+ public void blacklist(ModelEndpoint endpoint) {
+ blacklist(toUrl(endpoint.getEndpoint().getUrl()));
+ }
+
+ public void blacklist(URL url) {
+ rwLock.writeLock().lock();
+ blacklist.put(url, true);
+ rwLock.writeLock().unlock();
+ }
+
+ public ModelEndpoint getEndpoint(String modelName) {
+ String version = null;
+ rwLock.readLock().lock();
+ version = modelToCurrentVersion.get(modelName);
+ rwLock.readLock().unlock();
+ if(version == null) {
+ throw new IllegalStateException("Unable to find version for " + modelName);
+ }
+ return getEndpoint(modelName, version);
+ }
+
+ private static URL toUrl(String url) {
+ try {
+ return new URL(url);
+ } catch (MalformedURLException e) {
+ throw new IllegalStateException("Endpoint does not refer to an actual URL");
+ }
+ }
+
+ public ModelEndpoint getEndpoint(String modelName, String modelVersion) {
+ return getEndpoint(new Model(modelName, modelVersion));
+ }
public ModelEndpoint getEndpoint(Model model) {
rwLock.readLock().lock();
try {
List<ModelEndpoint> endpoints = state.get(model);
ModelEndpoint ret = null;
if(endpoints != null) {
- int i = ThreadLocalRandom.current().nextInt(endpoints.size());
- ret = endpoints.get(i);
+ for(int j = 0;j < 10;++j) {
+ int i = ThreadLocalRandom.current().nextInt(endpoints.size());
+ ret = endpoints.get(i);
+ try {
+ if (blacklist.asMap().containsKey(toUrl(ret.getEndpoint().getUrl()))) {
+ continue;
+ }
+ else {
+ return ret;
+ }
+ }
+ catch(IllegalStateException ise) {
+
+ }
--- End diff --
Yes it is trying 10 times to get an endpoint. If an exception happens on an attempt then we move on. Frankly this is an excess of caution since we parse the URLs on the server side before they go into zookeeper, so they are valid.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #210: METRON-366: Add MODEL_APPLY to Stellar
Posted by cestella <gi...@git.apache.org>.
Github user cestella commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/210#discussion_r75533349
--- Diff: metron-analytics/metron-maas-common/src/main/java/org/apache/metron/maas/discovery/ServiceDiscoverer.java ---
@@ -93,6 +116,7 @@ private void updateState() {
}
}
rwLock.writeLock().lock();
+ this.modelToCurrentVersion = modelToVersion;
this.state = state;
this.containerToEndpoint = containerToEndpoint;
rwLock.writeLock().unlock();
--- End diff --
Yes, very good catch.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #210: METRON-366: Add MODEL_APPLY to Stellar
Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/210#discussion_r75529006
--- Diff: metron-analytics/metron-maas-common/src/main/java/org/apache/metron/maas/discovery/ServiceDiscoverer.java ---
@@ -40,8 +48,16 @@
private ServiceDiscovery<ModelEndpoint> serviceDiscovery;
private Map<Model, List<ModelEndpoint>> state = new HashMap<>();
private Map<String, ServiceInstance<ModelEndpoint>> containerToEndpoint = new HashMap<>();
+ private Map<String, String> modelToCurrentVersion = new HashMap<>();
+ private Cache<URL, Boolean> blacklist;
public ServiceDiscoverer(CuratorFramework client, String root) {
+ blacklist = CacheBuilder.newBuilder()
+ .concurrencyLevel(4)
+ .weakKeys()
+ .expireAfterWrite(10, TimeUnit.MINUTES)
+ .build();
--- End diff --
Should we parameterize these settings?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #210: METRON-366: Add MODEL_APPLY to Stellar
Posted by cestella <gi...@git.apache.org>.
Github user cestella commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/210#discussion_r75533231
--- Diff: metron-analytics/metron-maas-service/src/main/java/org/apache/metron/maas/submit/ModelSubmission.java ---
@@ -203,7 +203,7 @@ else if(mode.equalsIgnoreCase("LIST")) {
String modelTitle = "Model " + kv.getKey().getName() + " @ " + kv.getKey().getVersion();
System.out.println(modelTitle);
for(ModelEndpoint endpoint : kv.getValue()){
- System.out.println("\t" + endpoint.getContainerId() + " at " + endpoint.getUrl());
+ System.out.println(endpoint);
}
--- End diff --
Actually, this the LIST operation is intended to output the list of endpoints returned. What I think I *will* do is make some of the logging debug level because it's getting quite chatty in practice and redundant.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #210: METRON-366: Add MODEL_APPLY to Stellar
Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/210#discussion_r75534093
--- Diff: metron-analytics/metron-maas-common/src/main/java/org/apache/metron/maas/discovery/ServiceDiscoverer.java ---
@@ -76,12 +92,19 @@ public void resetState() {
private void updateState() {
Map<Model, List<ModelEndpoint>> state = new HashMap<>();
+ Map<String, String> modelToVersion = new HashMap<>();
Map<String, ServiceInstance<ModelEndpoint>> containerToEndpoint = new HashMap<>();
try {
for(String name : serviceDiscovery.queryForNames()) {
for(ServiceInstance<ModelEndpoint> endpoint: serviceDiscovery.queryForInstances(name)) {
ModelEndpoint ep = endpoint.getPayload();
LOG.info("Found model endpoint " + ep);
+ String currentVersion = modelToVersion.getOrDefault(ep.getName(), ep.getVersion());
+ modelToVersion.put( ep.getName()
+ , currentVersion.compareTo(ep.getVersion()) < 0
+ ? ep.getVersion()
+ : currentVersion
+ );
--- End diff --
I am not sure that I totally understand what the intention here is. I've had to stare at this code block for like 10 minutes. Is there no way to make this an easier read? Just my opinion; maybe I'm slow.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #210: METRON-366: Add MODEL_APPLY to Stellar
Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/210#discussion_r75536146
--- Diff: metron-analytics/metron-maas-common/src/main/java/org/apache/metron/maas/discovery/ServiceDiscoverer.java ---
@@ -93,6 +116,7 @@ private void updateState() {
}
}
rwLock.writeLock().lock();
+ this.modelToCurrentVersion = modelToVersion;
this.state = state;
this.containerToEndpoint = containerToEndpoint;
rwLock.writeLock().unlock();
--- End diff --
There are a few other places where locks are used without try/finally. For example in blacklist(...) and getEndpoint(...). May need to hit those up too.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #210: METRON-366: Add MODEL_APPLY to Stellar
Posted by cestella <gi...@git.apache.org>.
Github user cestella commented on the issue:
https://github.com/apache/incubator-metron/pull/210
Testing Instructions
## Free Up Space on SNV
First, let's free up some headroom on SNV. If you are running this on a multinode cluster, you would not have to do this.
* Kill monit via `service monit stop`
* Kill tcpreplay via `for i in $(ps -ef | grep tcpreplay | awk '{print $2}');do kill -9 $i;done`
* Kill existing parser topologies via
* `storm kill snort`
* `storm kill bro`
* Kill flume via `for i in $(ps -ef | grep flume | awk '{print $2}');do kill -9 $i;done`
* Kill yaf via `for i in $(ps -ef | grep yaf | awk '{print $2}');do kill -9 $i;done`
* Kill bro via `for i in $(ps -ef | grep bro | awk '{print $2}');do kill -9 $i;done`
## Install Prerequisites and Mock DGA Service
Now let's install some prerequisites:
* Flask via `yum install python-flask`
* Jinja2 via `yum install yum install python-jinja2`
Now that we have flask and jinja, we can create a mock DGA service to deploy with MaaS:
* Download the files in [this](https://gist.github.com/cestella/cba10aff0f970078a4c2c8cade3a4d1a) gist into the `/root/mock_dga` directory
* Make `rest.sh` executable via `chmod +x /root/mock_dga/rest.sh`
This service will treat `yahoo.com` and `amazon.com` as legit and everything else as malicious. The contract is that the REST service exposes an endpoint `/apply` and returns back JSON maps with a single key `is_malicious` which can be `malicious` or `legit`.
## Deploy Mock DGA Service via MaaS
Now let's start MaaS and deploy the Mock DGA Service:
* Start MaaS via `/usr/metron/0.2.0BETA/bin/maas_service.sh -zq node1:2181`
* Start one instance of the mock DGA model with 512M of memory via `/usr/metron/0.2.0BETA/bin/maas_deploy.sh -zq node1:2181 -lmp /root/mock_dga -hmp /user/root/models -mo ADD -m 512 -n dga -v 1.0 -ni 1`
* As a sanity check:
* Ensure that the model is running via `/usr/metron/0.2.0BETA/bin/maas_deploy.sh -zq node1:2181 -mo LIST`. You should see `Model dga @ 1.0` be displayed and under that a url such as (but not exactly) `http://node1:36161`
* Try to hit the model via curl: `curl 'http://localhost:36161/apply?host=caseystella.com'` and ensure that it returns a JSON map indicating the domain is malicious.
## Adjust Field Transformations for Squid to Call Model
TBD
## Adjust Threat Intel Triage to Adjust Risk Based on Model
TBD
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---