You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@metron.apache.org by cestella <gi...@git.apache.org> on 2016/04/01 16:40:28 UTC
[GitHub] incubator-metron pull request: METRON-93: Generalize the HBase thr...
GitHub user cestella opened a pull request:
https://github.com/apache/incubator-metron/pull/66
METRON-93: Generalize the HBase threat intel infrastructure to support enrichments
As it stands, the threat intel infrastructure is awkward. Namely, different threat intelligence sources must be pushed into separate hbase tables (malicious_ips separate form malicious_hosts, for instance). We'd rather have one table where the type is brought into the rowkey. Since this infrastructure is generalized, also add a simple hbase enrichment adapter.
Furthermore, the configuration for a new enrichment should be added to zookeeper as part of the data load.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/cestella/incubator-metron MET-129
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-metron/pull/66.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #66
----
commit 3e55257e092c0c5651d7b3b8c69d0d7ccd568fc7
Author: cstella <ce...@gmail.com>
Date: 2016-03-21T18:50:38Z
Refactoring extractor framework to not use ThreatIntel anymore
commit 225ab51498c9bb978246a0ce1449d0dc97bbed97
Author: cstella <ce...@gmail.com>
Date: 2016-03-21T20:43:19Z
Updating dataloads to not use threatintelkey/value anymore.
commit 72b3b0ec9814902c21774a7b0f486ee5af19d97a
Author: cstella <ce...@gmail.com>
Date: 2016-03-22T14:21:30Z
Refactoring out the threat intel adapter stuff.
commit 6e4501d101ac41b407867c85ec76de0d31bc34b2
Author: cstella <ce...@gmail.com>
Date: 2016-03-23T18:13:24Z
Updating test yaml.
commit 72eea03e35412f8720c0612776ced4709cca9697
Author: cstella <ce...@gmail.com>
Date: 2016-03-24T15:44:51Z
Updating other adapters to use the new scheme.
commit b75c38a6343164aa050dba713f14322f861d6d78
Author: cstella <ce...@gmail.com>
Date: 2016-03-24T16:21:22Z
Updating configuredbolt to not load configs unnecessarily.
commit 998f69e9d20396a55ccfeec07238b6b22fa386cf
Author: cstella <ce...@gmail.com>
Date: 2016-03-24T19:07:39Z
updating
commit 3233e517b128de24dbfa5c2e9e58dcf32c1c876d
Author: cstella <ce...@gmail.com>
Date: 2016-03-25T14:07:04Z
fixing utilities to update zookeeper.
commit 15bfed855815931fd1ed068c47506b5a039a5bfe
Author: cstella <ce...@gmail.com>
Date: 2016-03-28T17:32:12Z
updated ansible
commit 0c58f2c32891e251c354ab23b09cc3db29ece817
Author: cstella <ce...@gmail.com>
Date: 2016-03-28T17:36:48Z
Merge branch 'master' into MET-129
commit 19e3d94b5b2ee08498eee5b76d8e1479caa1c3ed
Author: cstella <ce...@gmail.com>
Date: 2016-03-28T18:57:42Z
Whoops.
commit 00beb07eee99fc18e1d6f10530fa1c848889c3dc
Author: cstella <ce...@gmail.com>
Date: 2016-03-29T16:58:20Z
Added test.
commit 2551af972e026add7eb4fecc5506907841456625
Author: cstella <ce...@gmail.com>
Date: 2016-03-29T18:16:13Z
Unfolding the data in hbase before it goes into the index.
commit 0b0a62c2eca8ee1d8a72a6602e46c75200923ac0
Author: cstella <ce...@gmail.com>
Date: 2016-03-30T14:20:06Z
Updating bolt.
commit 79dcbfff208aefd2bbfac09482dcaa393db39171
Author: cstella <ce...@gmail.com>
Date: 2016-03-30T21:02:52Z
Updating to support multiget, etc.
commit d079c5d6df3248c68d1dd5304e1e257f7899a2c3
Author: cstella <ce...@gmail.com>
Date: 2016-03-31T19:01:24Z
updating ansible.
commit 520cbf7350fdc43d09e769530d343d7aad5ee815
Author: cstella <ce...@gmail.com>
Date: 2016-04-01T14:07:29Z
adding shell script
commit b4fc71e0a18ba9f2b897c3b01e16093903ed100a
Author: cstella <ce...@gmail.com>
Date: 2016-04-01T14:37:49Z
Merging from master
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request: METRON-93: Generalize the HBase thr...
Posted by cestella <gi...@git.apache.org>.
Github user cestella commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/66#discussion_r58729427
--- Diff: metron-streaming/Metron-Common/pom.xml ---
@@ -34,8 +34,11 @@
<name>Metron Kraken Repository</name>
<url>https://raw.github.com/opensoc/kraken/mvn-repo</url>
</repository>
+
</repositories>
<dependencies>
+
+
--- End diff --
the newlines? Nope, I'll remove them.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request: METRON-93: Generalize the HBase thr...
Posted by cestella <gi...@git.apache.org>.
Github user cestella commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/66#discussion_r58729524
--- Diff: metron-streaming/Metron-Common/src/main/java/org/apache/metron/bolt/BulkMessageWriterBolt.java ---
@@ -81,8 +81,7 @@ public void execute(Tuple tuple) {
sensorMessageMap.put(sensorType, messageList);
} else {
try {
- String esType = sensorType + "_doc";
- bulkMessageWriter.write(esType, configurations, tupleList, messageList);
+ bulkMessageWriter.write(sensorType, configurations, tupleList, messageList);
--- End diff --
Yeah, I moved the _doc modification down into the ES writer rather than having it at the bolt level. Should still work with the UI.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request: METRON-93: Generalize the HBase thr...
Posted by merrimanr <gi...@git.apache.org>.
Github user merrimanr commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/66#discussion_r58374345
--- Diff: metron-streaming/Metron-Common/src/test/java/org/apache/metron/enrichment/EnrichmentConfigTest.java ---
@@ -0,0 +1,253 @@
+package org.apache.metron.enrichment;
+
+import org.apache.metron.Constants;
+import org.apache.metron.domain.SourceConfig;
+import org.apache.metron.utils.JSONUtils;
+import org.junit.Assert;
+import org.junit.Test;
+
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+public class EnrichmentConfigTest {
+ @Test
+ public void testThreatIntel() throws Exception {
+ /*
+ {
+ "index": "bro",
--- End diff --
We should probably remove the commented blocks.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request: METRON-93: Generalize the HBase thr...
Posted by cestella <gi...@git.apache.org>.
Github user cestella commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/66#discussion_r58373817
--- Diff: metron-streaming/Metron-Common/src/main/java/org/apache/metron/enrichment/EnrichmentConfig.java ---
@@ -0,0 +1,203 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.metron.enrichment;
+
+import com.google.common.base.Joiner;
+import org.apache.curator.framework.CuratorFramework;
+import org.apache.metron.Constants;
+import org.apache.metron.domain.SourceConfig;
+import org.apache.metron.domain.SourceConfigUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.*;
+
+public class EnrichmentConfig {
+ public static enum Type {
+ THREAT_INTEL
+ ,ENRICHMENT
+ }
+
+ protected static final Logger _LOG = LoggerFactory.getLogger(EnrichmentConfig.class);
+ public static class FieldList {
+ Type type;
+ Map<String, List<String>> fieldToEnrichmentTypes;
+
+ public Type getType() {
+ return type;
+ }
+
+ public void setType(Type type) {
+ this.type = type;
+ }
+
+ public Map<String, List<String>> getFieldToEnrichmentTypes() {
+ return fieldToEnrichmentTypes;
+ }
+
+ public void setFieldToEnrichmentTypes(Map<String, List<String>> fieldToEnrichmentTypes) {
+ this.fieldToEnrichmentTypes = fieldToEnrichmentTypes;
+ }
+ }
+ public String zkQuorum;
+ public Map<String, FieldList> sensorToFieldList;
+
+ public String getZkQuorum() {
+ return zkQuorum;
+ }
+
+ public void setZkQuorum(String zkQuorum) {
+ this.zkQuorum = zkQuorum;
+ }
+
+ public Map<String, FieldList> getSensorToFieldList() {
+ return sensorToFieldList;
+ }
+
+ public void setSensorToFieldList(Map<String, FieldList> sensorToFieldList) {
+ this.sensorToFieldList = sensorToFieldList;
+ }
+
+ public void updateSensorConfigs( ) throws Exception {
+ CuratorFramework client = SourceConfigUtils.getClient(getZkQuorum());
+ try {
+ client.start();
+ updateSensorConfigs(new ZKSourceConfigHandler(client), sensorToFieldList);
+ }
+ finally {
+ client.close();
+ }
+ }
+
+ public static interface SourceConfigHandler {
+ SourceConfig readConfig(String sensor) throws Exception;
+ void persistConfig(String sensor, SourceConfig config) throws Exception;
+ }
+
+ public static class ZKSourceConfigHandler implements SourceConfigHandler {
+ CuratorFramework client;
+ public ZKSourceConfigHandler(CuratorFramework client) {
+ this.client = client;
+ }
+ @Override
+ public SourceConfig readConfig(String sensor) throws Exception {
+ return SourceConfigUtils.readConfigFromZookeeper(client, sensor);
+ }
+
+ @Override
+ public void persistConfig(String sensor, SourceConfig config) throws Exception {
+ SourceConfigUtils.writeToZookeeper(client, sensor, config.toJSON().getBytes());
+ }
+ }
+
+ public static void updateSensorConfigs( SourceConfigHandler scHandler
+ , Map<String, FieldList> sensorToFieldList
+ ) throws Exception
+ {
+ Map<String, SourceConfig> sourceConfigsChanged = new HashMap<>();
+ for (Map.Entry<String, FieldList> kv : sensorToFieldList.entrySet()) {
+ SourceConfig config = sourceConfigsChanged.get(kv.getKey());
+ if(config == null) {
+ config = scHandler.readConfig(kv.getKey());
+ if(_LOG.isDebugEnabled()) {
+ _LOG.debug(config.toJSON());
+ }
+ }
+ Map<String, List<String> > fieldMap = null;
+ Map<String, List<String> > fieldToTypeMap = null;
+ List<String> fieldList = null;
+ if(kv.getValue().type == Type.THREAT_INTEL) {
+ fieldMap = config.getThreatIntelFieldMap();
+ if(fieldMap!= null) {
+ fieldList = fieldMap.get(Constants.SIMPLE_HBASE_THREAT_INTEL);
+ }
+ if(fieldList == null) {
+ fieldList = new ArrayList<>();
+ fieldMap.put(Constants.SIMPLE_HBASE_THREAT_INTEL, fieldList);
+ }
+ fieldToTypeMap = config.getFieldToThreatIntelTypeMap();
+ if(fieldToTypeMap == null) {
+ fieldToTypeMap = new HashMap<>();
+ config.setFieldToThreatIntelTypeMap(fieldToTypeMap);
+ }
+ }
+ else if(kv.getValue().type == Type.ENRICHMENT) {
+ fieldMap = config.getEnrichmentFieldMap();
+ if(fieldMap!= null) {
+ fieldList = fieldMap.get(Constants.SIMPLE_HBASE_ENRICHMENT);
+ }
+ if(fieldList == null) {
+ fieldList = new ArrayList<>();
+ fieldMap.put(Constants.SIMPLE_HBASE_ENRICHMENT, fieldList);
+ }
+ fieldToTypeMap = config.getFieldToEnrichmentTypeMap();
+ if(fieldToTypeMap == null) {
+ fieldToTypeMap = new HashMap<>();
+ config.setFieldToEnrichmentTypeMap(fieldToTypeMap);
+ }
+ }
+ if(fieldToTypeMap == null || fieldMap == null) {
+ _LOG.debug("fieldToTypeMap is null or fieldMap is null, so skipping");
+ continue;
+ }
+ //Add the additional fields to the field list associated with the hbase adapter
+ {
--- End diff --
Just as logical separation between handling parts of this method handling the additional enrichment types and the part above handling the additional fields that might be added.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request: METRON-93: Generalize the HBase thr...
Posted by cestella <gi...@git.apache.org>.
Github user cestella commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/66#discussion_r58375098
--- Diff: metron-streaming/Metron-Common/src/test/java/org/apache/metron/enrichment/EnrichmentConfigTest.java ---
@@ -0,0 +1,253 @@
+package org.apache.metron.enrichment;
+
+import org.apache.metron.Constants;
+import org.apache.metron.domain.SourceConfig;
+import org.apache.metron.utils.JSONUtils;
+import org.junit.Assert;
+import org.junit.Test;
+
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+public class EnrichmentConfigTest {
+ @Test
+ public void testThreatIntel() throws Exception {
+ /*
+ {
+ "index": "bro",
--- End diff --
Normally I'd agree, but those comments are there because multiline strings in java are lacking, so if changes to the JSON structure happen, the changes generally happen to the comment and then the string is replaced, letting the IDE handle escaping quotes. One thing that we could consider is incorporating something like https://github.com/benelog/multiline, which i've seen in other projects for these kinds of situations.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request: METRON-93: Generalize the HBase thr...
Posted by dlyle65535 <gi...@git.apache.org>.
Github user dlyle65535 commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/66#discussion_r58729091
--- Diff: metron-streaming/Metron-Common/pom.xml ---
@@ -34,8 +34,11 @@
<name>Metron Kraken Repository</name>
<url>https://raw.github.com/opensoc/kraken/mvn-repo</url>
</repository>
+
</repositories>
<dependencies>
+
+
--- End diff --
Did you mean to add these?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request: METRON-93: Generalize the HBase thr...
Posted by cestella <gi...@git.apache.org>.
Github user cestella commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/66#discussion_r58468314
--- Diff: metron-streaming/Metron-Common/src/test/java/org/apache/metron/enrichment/EnrichmentConfigTest.java ---
@@ -0,0 +1,253 @@
+package org.apache.metron.enrichment;
+
+import org.apache.metron.Constants;
+import org.apache.metron.domain.SourceConfig;
+import org.apache.metron.utils.JSONUtils;
+import org.junit.Assert;
+import org.junit.Test;
+
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+public class EnrichmentConfigTest {
+ @Test
+ public void testThreatIntel() throws Exception {
+ /*
+ {
+ "index": "bro",
--- End diff --
Ok, fair enough, I'm adding the multiline string utility for the unit tests and moving the config JSON to there, so we don't have to maintain them in 2 different places. This is a similar pattern as they use in DataFu for their unit tests.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request: METRON-93: Generalize the HBase thr...
Posted by dlyle65535 <gi...@git.apache.org>.
Github user dlyle65535 commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/66#discussion_r58236854
--- Diff: metron-streaming/Metron-DataLoads/src/main/java/org/apache/metron/dataloads/hbase/mr/PrunerMapper.java ---
@@ -48,7 +48,7 @@ public void setup(Context context) throws IOException
}
try {
tracker = AccessTrackerUtil.INSTANCE.loadAll(AccessTrackerUtil.INSTANCE.loadAll(table, atCF, atName, timestamp));
- } catch (Exception e) {
+ } catch (Throwable e) {
--- End diff --
Was this changed to deliberately catch subclasses of Error?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request: METRON-93: Generalize the HBase thr...
Posted by dlyle65535 <gi...@git.apache.org>.
Github user dlyle65535 commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/66#discussion_r58729170
--- Diff: metron-streaming/Metron-Common/src/main/java/org/apache/metron/bolt/BulkMessageWriterBolt.java ---
@@ -81,8 +81,7 @@ public void execute(Tuple tuple) {
sensorMessageMap.put(sensorType, messageList);
} else {
try {
- String esType = sensorType + "_doc";
- bulkMessageWriter.write(esType, configurations, tupleList, messageList);
+ bulkMessageWriter.write(sensorType, configurations, tupleList, messageList);
--- End diff --
This is going to break the ui query.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request: METRON-93: Generalize the HBase thr...
Posted by dlyle65535 <gi...@git.apache.org>.
Github user dlyle65535 commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/66#discussion_r58855860
--- Diff: deployment/roles/metron_streaming/tasks/main.yml ---
@@ -109,8 +111,10 @@
- { regexp: "bolt.hbase.table.name=", line: "bolt.hbase.table.name={{ pcap_hbase_table }}" }
- { regexp: "threat.intel.tracker.table=", line: "threat.intel.tracker.table={{ tracker_hbase_table }}" }
- { regexp: "threat.intel.tracker.cf=", line: "threat.intel.tracker.cf=t" }
- - { regexp: "threat.intel.ip.table=", line: "threat.intel.ip.table={{ threatintel_ip_hbase_table }}" }
- - { regexp: "threat.intel.ip.cf=", line: "threat.intel.ip.cf=t" }
+ - { regexp: "threat.intel.simple.hbase.table=", line: "threat.intel.simple.hbase.table={{ threatintel_hbase_table }}" }
+ - { regexp: "threat.intel.simple.hbase.cf=", line: "threat.intel.simple.hbase.cf=t" }
+ - { regexp: "enrichment.simple.hbase.table=", line: "enrichment.simple.hbase.table={{ enrichment_hbase_table }}" }
+ - { regexp: "enrichment.simple.hbase.cf=", line: "enrichment.simple.hbase.cf=t" }
- { regexp: "mysql.ip=", line: "mysql.ip={{ groups.mysql[0] }}" }
- { regexp: "mysql.password=", line: "mysql.password={{ mysql_root_password }}" }
- { regexp: "index.hdfs.output=", line: "index.hdfs.output={{ metron_hdfs_output_dir }}/enrichment/indexed" }
--- End diff --
Sorry for the late comment- I just noticed. Outside of being duplicate files outside of the name, I think there needs to be a spout.kafka.topic.yaf and spout.kafka.topic.snort in the above. Right now it defaults okay, but it cannot be overridden.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request: METRON-93: Generalize the HBase thr...
Posted by dlyle65535 <gi...@git.apache.org>.
Github user dlyle65535 commented on the pull request:
https://github.com/apache/incubator-metron/pull/66#issuecomment-206789678
+1
Ran this up last night on ec2 and let it soak. Looks good!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request: METRON-93: Generalize the HBase thr...
Posted by dlyle65535 <gi...@git.apache.org>.
Github user dlyle65535 commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/66#discussion_r58729981
--- Diff: metron-streaming/Metron-Common/src/main/java/org/apache/metron/bolt/BulkMessageWriterBolt.java ---
@@ -81,8 +81,7 @@ public void execute(Tuple tuple) {
sensorMessageMap.put(sensorType, messageList);
} else {
try {
- String esType = sensorType + "_doc";
- bulkMessageWriter.write(esType, configurations, tupleList, messageList);
+ bulkMessageWriter.write(sensorType, configurations, tupleList, messageList);
--- End diff --
Disregard. I see it below. :)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request: METRON-93: Generalize the HBase thr...
Posted by merrimanr <gi...@git.apache.org>.
Github user merrimanr commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/66#discussion_r58373473
--- Diff: metron-streaming/Metron-Common/src/main/java/org/apache/metron/enrichment/EnrichmentConfig.java ---
@@ -0,0 +1,203 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.metron.enrichment;
+
+import com.google.common.base.Joiner;
+import org.apache.curator.framework.CuratorFramework;
+import org.apache.metron.Constants;
+import org.apache.metron.domain.SourceConfig;
+import org.apache.metron.domain.SourceConfigUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.*;
+
+public class EnrichmentConfig {
+ public static enum Type {
+ THREAT_INTEL
+ ,ENRICHMENT
+ }
+
+ protected static final Logger _LOG = LoggerFactory.getLogger(EnrichmentConfig.class);
+ public static class FieldList {
+ Type type;
+ Map<String, List<String>> fieldToEnrichmentTypes;
+
+ public Type getType() {
+ return type;
+ }
+
+ public void setType(Type type) {
+ this.type = type;
+ }
+
+ public Map<String, List<String>> getFieldToEnrichmentTypes() {
+ return fieldToEnrichmentTypes;
+ }
+
+ public void setFieldToEnrichmentTypes(Map<String, List<String>> fieldToEnrichmentTypes) {
+ this.fieldToEnrichmentTypes = fieldToEnrichmentTypes;
+ }
+ }
+ public String zkQuorum;
+ public Map<String, FieldList> sensorToFieldList;
+
+ public String getZkQuorum() {
+ return zkQuorum;
+ }
+
+ public void setZkQuorum(String zkQuorum) {
+ this.zkQuorum = zkQuorum;
+ }
+
+ public Map<String, FieldList> getSensorToFieldList() {
+ return sensorToFieldList;
+ }
+
+ public void setSensorToFieldList(Map<String, FieldList> sensorToFieldList) {
+ this.sensorToFieldList = sensorToFieldList;
+ }
+
+ public void updateSensorConfigs( ) throws Exception {
+ CuratorFramework client = SourceConfigUtils.getClient(getZkQuorum());
+ try {
+ client.start();
+ updateSensorConfigs(new ZKSourceConfigHandler(client), sensorToFieldList);
+ }
+ finally {
+ client.close();
+ }
+ }
+
+ public static interface SourceConfigHandler {
+ SourceConfig readConfig(String sensor) throws Exception;
+ void persistConfig(String sensor, SourceConfig config) throws Exception;
+ }
+
+ public static class ZKSourceConfigHandler implements SourceConfigHandler {
+ CuratorFramework client;
+ public ZKSourceConfigHandler(CuratorFramework client) {
+ this.client = client;
+ }
+ @Override
+ public SourceConfig readConfig(String sensor) throws Exception {
+ return SourceConfigUtils.readConfigFromZookeeper(client, sensor);
+ }
+
+ @Override
+ public void persistConfig(String sensor, SourceConfig config) throws Exception {
+ SourceConfigUtils.writeToZookeeper(client, sensor, config.toJSON().getBytes());
+ }
+ }
+
+ public static void updateSensorConfigs( SourceConfigHandler scHandler
+ , Map<String, FieldList> sensorToFieldList
+ ) throws Exception
+ {
+ Map<String, SourceConfig> sourceConfigsChanged = new HashMap<>();
+ for (Map.Entry<String, FieldList> kv : sensorToFieldList.entrySet()) {
+ SourceConfig config = sourceConfigsChanged.get(kv.getKey());
+ if(config == null) {
+ config = scHandler.readConfig(kv.getKey());
+ if(_LOG.isDebugEnabled()) {
+ _LOG.debug(config.toJSON());
+ }
+ }
+ Map<String, List<String> > fieldMap = null;
+ Map<String, List<String> > fieldToTypeMap = null;
+ List<String> fieldList = null;
+ if(kv.getValue().type == Type.THREAT_INTEL) {
+ fieldMap = config.getThreatIntelFieldMap();
+ if(fieldMap!= null) {
+ fieldList = fieldMap.get(Constants.SIMPLE_HBASE_THREAT_INTEL);
+ }
+ if(fieldList == null) {
+ fieldList = new ArrayList<>();
+ fieldMap.put(Constants.SIMPLE_HBASE_THREAT_INTEL, fieldList);
+ }
+ fieldToTypeMap = config.getFieldToThreatIntelTypeMap();
+ if(fieldToTypeMap == null) {
+ fieldToTypeMap = new HashMap<>();
+ config.setFieldToThreatIntelTypeMap(fieldToTypeMap);
+ }
+ }
+ else if(kv.getValue().type == Type.ENRICHMENT) {
+ fieldMap = config.getEnrichmentFieldMap();
+ if(fieldMap!= null) {
+ fieldList = fieldMap.get(Constants.SIMPLE_HBASE_ENRICHMENT);
+ }
+ if(fieldList == null) {
+ fieldList = new ArrayList<>();
+ fieldMap.put(Constants.SIMPLE_HBASE_ENRICHMENT, fieldList);
+ }
+ fieldToTypeMap = config.getFieldToEnrichmentTypeMap();
+ if(fieldToTypeMap == null) {
+ fieldToTypeMap = new HashMap<>();
+ config.setFieldToEnrichmentTypeMap(fieldToTypeMap);
+ }
+ }
+ if(fieldToTypeMap == null || fieldMap == null) {
+ _LOG.debug("fieldToTypeMap is null or fieldMap is null, so skipping");
+ continue;
+ }
+ //Add the additional fields to the field list associated with the hbase adapter
+ {
--- End diff --
Are these inline braces necessary?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request: METRON-93: Generalize the HBase thr...
Posted by cestella <gi...@git.apache.org>.
Github user cestella commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/66#discussion_r58862697
--- Diff: deployment/roles/metron_streaming/tasks/main.yml ---
@@ -109,8 +111,10 @@
- { regexp: "bolt.hbase.table.name=", line: "bolt.hbase.table.name={{ pcap_hbase_table }}" }
- { regexp: "threat.intel.tracker.table=", line: "threat.intel.tracker.table={{ tracker_hbase_table }}" }
- { regexp: "threat.intel.tracker.cf=", line: "threat.intel.tracker.cf=t" }
- - { regexp: "threat.intel.ip.table=", line: "threat.intel.ip.table={{ threatintel_ip_hbase_table }}" }
- - { regexp: "threat.intel.ip.cf=", line: "threat.intel.ip.cf=t" }
+ - { regexp: "threat.intel.simple.hbase.table=", line: "threat.intel.simple.hbase.table={{ threatintel_hbase_table }}" }
+ - { regexp: "threat.intel.simple.hbase.cf=", line: "threat.intel.simple.hbase.cf=t" }
+ - { regexp: "enrichment.simple.hbase.table=", line: "enrichment.simple.hbase.table={{ enrichment_hbase_table }}" }
+ - { regexp: "enrichment.simple.hbase.cf=", line: "enrichment.simple.hbase.cf=t" }
- { regexp: "mysql.ip=", line: "mysql.ip={{ groups.mysql[0] }}" }
- { regexp: "mysql.password=", line: "mysql.password={{ mysql_root_password }}" }
- { regexp: "index.hdfs.output=", line: "index.hdfs.output={{ metron_hdfs_output_dir }}/enrichment/indexed" }
--- End diff --
I think the problem is deeper than this. We need separate properties files for the sensor topologies than we have for the enrichment topologies. As it stands, we're using the same config for everything and that's just confusing. I'm creating https://issues.apache.org/jira/browse/METRON-99 to capture it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request: METRON-93: Generalize the HBase thr...
Posted by dlyle65535 <gi...@git.apache.org>.
Github user dlyle65535 commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/66#discussion_r58461859
--- Diff: metron-streaming/Metron-Common/src/test/java/org/apache/metron/enrichment/EnrichmentConfigTest.java ---
@@ -0,0 +1,253 @@
+package org.apache.metron.enrichment;
+
+import org.apache.metron.Constants;
+import org.apache.metron.domain.SourceConfig;
+import org.apache.metron.utils.JSONUtils;
+import org.junit.Assert;
+import org.junit.Test;
+
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+public class EnrichmentConfigTest {
+ @Test
+ public void testThreatIntel() throws Exception {
+ /*
+ {
+ "index": "bro",
--- End diff --
I'd get rid of them too. I suspect they'll quickly get out of sync and become misleading. Maybe go ahead and add in multiline or just use a resource file?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request: METRON-93: Generalize the HBase thr...
Posted by cestella <gi...@git.apache.org>.
Github user cestella commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/66#discussion_r58237558
--- Diff: metron-streaming/Metron-DataLoads/src/main/java/org/apache/metron/dataloads/hbase/mr/PrunerMapper.java ---
@@ -48,7 +48,7 @@ public void setup(Context context) throws IOException
}
try {
tracker = AccessTrackerUtil.INSTANCE.loadAll(AccessTrackerUtil.INSTANCE.loadAll(table, atCF, atName, timestamp));
- } catch (Exception e) {
+ } catch (Throwable e) {
--- End diff --
This is just a style thing. It's going to throw a runtime exception
regardless and kill the task, but I wanted it to be clear in the logs that
it was due to not bing able to load the access tracker stuff (as opposed to
an OOM exception).
On Fri, Apr 1, 2016 at 1:18 PM, David Lyle <no...@github.com> wrote:
> In
> metron-streaming/Metron-DataLoads/src/main/java/org/apache/metron/dataloads/hbase/mr/PrunerMapper.java
> <https://github.com/apache/incubator-metron/pull/66#discussion_r58236854>:
>
> > @@ -48,7 +48,7 @@ public void setup(Context context) throws IOException
> > }
> > try {
> > tracker = AccessTrackerUtil.INSTANCE.loadAll(AccessTrackerUtil.INSTANCE.loadAll(table, atCF, atName, timestamp));
> > - } catch (Exception e) {
> > + } catch (Throwable e) {
>
> Was this changed to deliberately catch subclasses of Error?
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly or view it on GitHub
> <https://github.com/apache/incubator-metron/pull/66/files/b4fc71e0a18ba9f2b897c3b01e16093903ed100a#r58236854>
>
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request: METRON-93: Generalize the HBase thr...
Posted by cestella <gi...@git.apache.org>.
Github user cestella closed the pull request at:
https://github.com/apache/incubator-metron/pull/66
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---