You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@metron.apache.org by cestella <gi...@git.apache.org> on 2016/05/06 22:59:19 UTC
[GitHub] incubator-metron pull request: METRON-141: The ability to do threa...
GitHub user cestella opened a pull request:
https://github.com/apache/incubator-metron/pull/108
METRON-141: The ability to do threat triage
We have the ability to mark messages as part of the enrichment topology as threat alerts, but we have no ability to prioritize those alerts.
We should allow for the prioritization of messages that have some threat intelligence alert via a scoring mechanism.
The general idea is for each message from each sensor:
1. Map threat conditions as defined by a lightweight query language to threat levels (a number).
2. More than one condition may match a given message, so allow for the choice of aggregation functions.
3. If the aggregated score is positive, then add a field (`threat.triage.level`) with the level to the JSON message.
The above configuration is done at the per-sensor level.
As part of this PR, the following were added out of necessity:
* A query language DSL which is used in defining the conditions which map to threat levels. Docs at [here](https://github.com/cestella/incubator-metron/tree/METRON-141/metron-platform/metron-common#query-language)
* The addition of a `triageConfig` section to the sensor enrichment config. Description at [here](https://github.com/cestella/incubator-metron/tree/METRON-141/metron-platform/metron-common#the-threatintel-configuration)
* Actual documentation of our enrichment configuration. Documentation at [here](https://github.com/cestella/incubator-metron/tree/METRON-141/metron-platform/metron-common#enrichment-configuration)
* A rewrite of the `zk_load_configs.sh` utility to support both pushing and pulling configs (the threat triage rules are part of the configs and until we have a UI, we need to add them). Docs at [here](https://github.com/cestella/incubator-metron/tree/METRON-141/metron-platform/metron-common#management-utility)
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/cestella/incubator-metron METRON-141
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-metron/pull/108.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #108
----
commit cfdffd48d6a5ea7c228c4b66f23c16cc55691f79
Author: cstella <ce...@gmail.com>
Date: 2016-05-06T22:46:02Z
METRON-141: The ability to do threat triage
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request: METRON-141: The ability to do threa...
Posted by cestella <gi...@git.apache.org>.
Github user cestella commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/108#discussion_r62583229
--- Diff: metron-platform/metron-common/src/test/java/org/apache/metron/common/cli/ConfigurationsUtilsTest.java ---
@@ -74,7 +74,7 @@ public void test() throws Exception {
}
- @Test
+ /*@Test
--- End diff --
Whoops, absolutely right! Will do.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request: METRON-141: The ability to do threa...
Posted by merrimanr <gi...@git.apache.org>.
Github user merrimanr commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/108#discussion_r62572042
--- Diff: metron-platform/metron-common/src/test/java/org/apache/metron/common/cli/ConfigurationsUtilsTest.java ---
@@ -74,7 +74,7 @@ public void test() throws Exception {
}
- @Test
+ /*@Test
--- End diff --
Should we remove this instead of commenting out?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request: METRON-141: The ability to do threa...
Posted by dlyle65535 <gi...@git.apache.org>.
Github user dlyle65535 commented on the pull request:
https://github.com/apache/incubator-metron/pull/108#issuecomment-218166930
+1 this is really great!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request: METRON-141: The ability to do threa...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/incubator-metron/pull/108
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request: METRON-141: The ability to do threa...
Posted by merrimanr <gi...@git.apache.org>.
Github user merrimanr commented on the pull request:
https://github.com/apache/incubator-metron/pull/108#issuecomment-217990228
+1
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request: METRON-141: The ability to do threa...
Posted by cestella <gi...@git.apache.org>.
Github user cestella commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/108#discussion_r62583706
--- Diff: metron-platform/metron-enrichment/src/main/java/org/apache/metron/enrichment/bolt/EnrichmentJoinBolt.java ---
@@ -84,7 +88,7 @@ public JSONObject joinMessages(Map<String, JSONObject> streamMessageMap) {
if(sourceType != null) {
SensorEnrichmentConfig config = configurations.getSensorEnrichmentConfig(sourceType);
if (config != null) {
- return config.getEnrichmentFieldMap();
+ return config.getEnrichment().getFieldMap();
--- End diff --
I think that the enrichment variable is initialized, but a null check is never a bad idea.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request: METRON-141: The ability to do threa...
Posted by merrimanr <gi...@git.apache.org>.
Github user merrimanr commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/108#discussion_r62572579
--- Diff: metron-platform/metron-enrichment/src/main/java/org/apache/metron/enrichment/bolt/EnrichmentJoinBolt.java ---
@@ -84,7 +88,7 @@ public JSONObject joinMessages(Map<String, JSONObject> streamMessageMap) {
if(sourceType != null) {
SensorEnrichmentConfig config = configurations.getSensorEnrichmentConfig(sourceType);
if (config != null) {
- return config.getEnrichmentFieldMap();
+ return config.getEnrichment().getFieldMap();
--- End diff --
Does this introduce the possibility of a null pointer exception?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---