You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@metron.apache.org by mm...@apache.org on 2016/12/05 20:38:36 UTC
incubator-metron git commit: METRON-607: Enrichment doc improvement
and test cleanup (mmiklavc) closes apache/incubator-metron#386
Repository: incubator-metron
Updated Branches:
refs/heads/master 670f50ae3 -> e46539a98
METRON-607: Enrichment doc improvement and test cleanup (mmiklavc) closes apache/incubator-metron#386
Project: http://git-wip-us.apache.org/repos/asf/incubator-metron/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-metron/commit/e46539a9
Tree: http://git-wip-us.apache.org/repos/asf/incubator-metron/tree/e46539a9
Diff: http://git-wip-us.apache.org/repos/asf/incubator-metron/diff/e46539a9
Branch: refs/heads/master
Commit: e46539a9802930e99eafbba7c74680e0fb48eaa3
Parents: 670f50a
Author: mmiklavc <mi...@gmail.com>
Authored: Mon Dec 5 13:38:18 2016 -0700
Committer: Michael Miklavcic <mi...@gmail.com>
Committed: Mon Dec 5 13:38:18 2016 -0700
----------------------------------------------------------------------
metron-platform/metron-enrichment/README.md | 14 ++++----
.../integration/EnrichmentIntegrationTest.java | 38 +++++++-------------
2 files changed, 21 insertions(+), 31 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/e46539a9/metron-platform/metron-enrichment/README.md
----------------------------------------------------------------------
diff --git a/metron-platform/metron-enrichment/README.md b/metron-platform/metron-enrichment/README.md
index 9527c8e..9f96f3e 100644
--- a/metron-platform/metron-enrichment/README.md
+++ b/metron-platform/metron-enrichment/README.md
@@ -48,8 +48,8 @@ The configuration is a complex JSON object with the following top level fields:
| Field | Description | Example |
|------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|------------------------------------------------------------------|
-| `fieldToTypeMap` | In the case of a simple HBase enrichment (i.e. a key/value lookup), the mapping between fields and the enrichment types associated with those fields must be known. This enrichment type is used as part of the HBase key. | `"fieldToTypeMap" : { "ip_src_addr" : [ "asset_enrichment" ] }` |
-| `fieldMap` | The map of enrichment bolts names to configuration handlers which know how to split the message up. The simplest of which is just a list of fields. More complex examples would be the stellar enrichment which provides stellar statements. Each field is sent to the enrichment referenced in the key. | `"fieldMap": {"hbaseEnrichment": ["ip_src_addr","ip_dst_addr"]}` |
+| `fieldToTypeMap` | In the case of a simple HBase enrichment (i.e. a key/value lookup), the mapping between fields and the enrichment types associated with those fields must be known. This enrichment type is used as part of the HBase key. Note: applies to hbaseEnrichment only. | `"fieldToTypeMap" : { "ip_src_addr" : [ "asset_enrichment" ] }` |
+| `fieldMap` | The map of enrichment bolts names to configuration handlers which know how to split the message up. The simplest of which is just a list of fields. More complex examples would be the stellar enrichment which provides stellar statements. Each field listed in the array arg is sent to the enrichment referenced in the key. Cardinality of fields to enrichments is many-to-many. | `"fieldMap": {"hbaseEnrichment": ["ip_src_addr","ip_dst_addr"]}` |
| `config` | The general configuration for the enrichment | `"config": {"typeToColumnFamily": { "asset_enrichment" : "cf" } }` |
The `config` map is intended to house enrichment specific configuration.
@@ -73,7 +73,7 @@ The `fieldMap`contents are of interest because they contain the routing and conf
]
}
```
-For the `geo`, `host` and `hbaseEnrichment`, this is sufficient. However, more complex enrichments may contain their own configuration. Currently, the `stellar` enrichment requires a more complex configuration, such as:
+Based on this sample config, both ip_src_addr and ip_dst_addr will go to the `geo`, `host`, and `hbaseEnrichment` adapter bolts. For the `geo`, `host` and `hbaseEnrichment`, this is sufficient. However, more complex enrichments may contain their own configuration. Currently, the `stellar` enrichment requires a more complex configuration, such as:
```
"fieldMap": {
...
@@ -106,14 +106,14 @@ The other way in which the stellar enrichment is somewhat more complex is in how
}
}
```
-We have a group called `numeric` whose stellar statements will be executed sequentially. In parallel to that, we have the group of stellar statements under the group `text` executing. The intent here is to allow you to not force higher latency operations to be done sequentially.
+We have a group called `numeric` whose stellar statements will be executed sequentially. In parallel to that, we have the group of stellar statements under the group `text` executing. The intent here is to allow you to not force higher latency operations to be done sequentially. You can use any name for your groupings you like. Be aware that the configuration is a map and duplicate configuration keys' values are not combined, so the duplicate configuration value will be overwritten.
###The `threatIntel` Configuration
| Field | Description | Example |
|------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------|
-| `fieldToTypeMap` | In the case of a simple HBase threat intel enrichment (i.e. a key/value lookup), the mapping between fields and the enrichment types associated with those fields must be known. This enrichment type is used as part of the HBase key. | `"fieldToTypeMap" : { "ip_src_addr" : [ "malicious_ips" ] }` |
-| `fieldMap` | The map of threat intel enrichment bolts names to fields in the JSON messages. Each field is sent to the threat intel enrichment bolt referenced in the key. | `"fieldMap": {"hbaseThreatIntel": ["ip_src_addr","ip_dst_addr"]}` |
+| `fieldToTypeMap` | In the case of a simple HBase threat intel enrichment (i.e. a key/value lookup), the mapping between fields and the enrichment types associated with those fields must be known. This enrichment type is used as part of the HBase key. Note: applies to hbaseThreatIntel only. | `"fieldToTypeMap" : { "ip_src_addr" : [ "malicious_ips" ] }` |
+| `fieldMap` | The map of threat intel enrichment bolts names to fields in the JSON messages. Each field is sent to the threat intel enrichment bolt referenced in the key. Each field listed in the array arg is sent to the enrichment referenced in the key. Cardinality of fields to enrichments is many-to-many. | `"fieldMap": {"hbaseThreatIntel": ["ip_src_addr","ip_dst_addr"]}` |
| `triageConfig` | The configuration of the threat triage scorer. In the situation where a threat is detected, a score is assigned to the message and embedded in the indexed message. | `"riskLevelRules" : { "IN_SUBNET(ip_dst_addr, '192.168.0.0/24')" : 10 }` |
| `config` | The general configuration for the Threat Intel | `"config": {"typeToColumnFamily": { "malicious_ips","cf" } }` |
@@ -190,6 +190,8 @@ An example configuration for the YAF sensor is as follows:
}
```
+ThreatIntel alert levels are emitted as a new field "threat.triage.level." So for the example above, an incoming message that trips the `ip_src_addr` rule will have a new field threat.triage.level=10.
+
# Example Enrichment via Stellar
Let's walk through doing a simple enrichment using Stellar on your cluster using the Squid topology.
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/e46539a9/metron-platform/metron-enrichment/src/test/java/org/apache/metron/enrichment/integration/EnrichmentIntegrationTest.java
----------------------------------------------------------------------
diff --git a/metron-platform/metron-enrichment/src/test/java/org/apache/metron/enrichment/integration/EnrichmentIntegrationTest.java b/metron-platform/metron-enrichment/src/test/java/org/apache/metron/enrichment/integration/EnrichmentIntegrationTest.java
index a0199a1..d9c1dec 100644
--- a/metron-platform/metron-enrichment/src/test/java/org/apache/metron/enrichment/integration/EnrichmentIntegrationTest.java
+++ b/metron-platform/metron-enrichment/src/test/java/org/apache/metron/enrichment/integration/EnrichmentIntegrationTest.java
@@ -18,33 +18,32 @@
package org.apache.metron.enrichment.integration;
import com.fasterxml.jackson.core.type.TypeReference;
-import com.google.common.base.*;
-
+import com.google.common.base.Joiner;
+import com.google.common.base.Predicate;
+import com.google.common.base.Predicates;
+import com.google.common.base.Splitter;
import com.google.common.collect.Iterables;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.hbase.client.HTableInterface;
import org.apache.metron.TestConstants;
import org.apache.metron.common.Constants;
-import org.apache.metron.common.configuration.EnrichmentConfigurations;
+import org.apache.metron.common.utils.JSONUtils;
import org.apache.metron.enrichment.bolt.ErrorEnrichmentBolt;
+import org.apache.metron.enrichment.converter.EnrichmentHelper;
+import org.apache.metron.enrichment.converter.EnrichmentKey;
+import org.apache.metron.enrichment.converter.EnrichmentValue;
+import org.apache.metron.enrichment.integration.components.ConfigUploadComponent;
+import org.apache.metron.enrichment.integration.mock.MockGeoAdapter;
+import org.apache.metron.enrichment.lookup.LookupKV;
import org.apache.metron.enrichment.lookup.accesstracker.PersistentBloomTrackerCreator;
import org.apache.metron.enrichment.stellar.SimpleHBaseEnrichmentFunctions;
import org.apache.metron.hbase.TableProvider;
-import org.apache.metron.enrichment.converter.EnrichmentKey;
-import org.apache.metron.enrichment.converter.EnrichmentValue;
-import org.apache.metron.enrichment.converter.EnrichmentHelper;
import org.apache.metron.integration.*;
-import org.apache.metron.enrichment.integration.components.ConfigUploadComponent;
+import org.apache.metron.integration.components.FluxTopologyComponent;
import org.apache.metron.integration.components.KafkaComponent;
import org.apache.metron.integration.components.ZKServerComponent;
import org.apache.metron.integration.utils.TestUtils;
-import org.apache.metron.integration.components.FluxTopologyComponent;
-import org.apache.metron.enrichment.integration.mock.MockGeoAdapter;
import org.apache.metron.test.mock.MockHTable;
-import org.apache.metron.enrichment.lookup.LookupKV;
-
-import org.apache.metron.enrichment.integration.utils.SampleUtil;
-import org.apache.metron.common.utils.JSONUtils;
import org.junit.Assert;
import org.junit.Test;
@@ -52,13 +51,7 @@ import javax.annotation.Nullable;
import java.io.File;
import java.io.IOException;
import java.io.Serializable;
-import java.util.ArrayList;
-import java.util.HashMap;
-import java.util.HashSet;
-import java.util.List;
-import java.util.Map;
-import java.util.Properties;
-import java.util.Set;
+import java.util.*;
public class EnrichmentIntegrationTest extends BaseIntegrationTest {
private static final String SRC_IP = "ip_src_addr";
@@ -68,11 +61,8 @@ public class EnrichmentIntegrationTest extends BaseIntegrationTest {
private static final Map<String, Object> PLAYFUL_ENRICHMENT = new HashMap<String, Object>() {{
put("orientation", "north");
}};
- protected String testSensorType = "test";
- protected String hdfsDir = "target/enrichmentIntegrationTest/hdfs";
protected String fluxPath = "../metron-enrichment/src/main/flux/enrichment/test.yaml";
protected String sampleParsedPath = TestConstants.SAMPLE_DATA_PARSED_PATH + "TestExampleParsed";
- private String sampleIndexedPath = TestConstants.SAMPLE_DATA_INDEXED_PATH + "TestIndexed";
public static class Provider implements TableProvider, Serializable {
MockHTable.Provider provider = new MockHTable.Provider();
@@ -84,8 +74,6 @@ public class EnrichmentIntegrationTest extends BaseIntegrationTest {
@Test
public void test() throws Exception {
- final EnrichmentConfigurations configurations = SampleUtil.getSampleEnrichmentConfigs();
- final String dateFormat = "yyyy.MM.dd.HH";
final List<byte[]> inputMessages = TestUtils.readSampleData(sampleParsedPath);
final String cf = "cf";
final String trackerHBaseTableName = "tracker";