You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@metron.apache.org by cestella <gi...@git.apache.org> on 2016/05/23 20:52:54 UTC
[GitHub] incubator-metron pull request: METRON-183 Allow the simple hbase e...
GitHub user cestella opened a pull request:
https://github.com/apache/incubator-metron/pull/131
METRON-183 Allow the simple hbase enrichment adapter and simple threat intel adapter to use multiple column families
Allow the simple hbase enrichment adapter and simple threat intel adapter to use multiple column families. As it stands, the table and column family used are configured at topology submission time rather than via zookeeper. It makes sense to allow different enrichment types to be associated with different column families so that column family specific configuration, such as retention policies, can be used per enrichment type.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/cestella/incubator-metron cf_per_et
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-metron/pull/131.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #131
----
commit 00a0b13a5b307be39aad45031c1673de9a2f5ea4
Author: cstella <ce...@gmail.com>
Date: 2016-05-23T17:50:40Z
Added the ability to support multiple column families in the simple hbase enrichment and threat intel.
commit adc3765fcfeeee84bf24d58ac8a4e5e20851ffae
Author: cstella <ce...@gmail.com>
Date: 2016-05-23T20:46:35Z
Updating documentation.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request: METRON-183 Allow the simple hbase e...
Posted by james-sirota <gi...@git.apache.org>.
Github user james-sirota commented on the pull request:
https://github.com/apache/incubator-metron/pull/131#issuecomment-222400667
+1. I was able to put the enrichments into different column families and have the message enriched appropriately
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request: METRON-183 Allow the simple hbase e...
Posted by james-sirota <gi...@git.apache.org>.
Github user james-sirota commented on the pull request:
https://github.com/apache/incubator-metron/pull/131#issuecomment-222347914
The docs should probably be moved out of common to the enrichment topology
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request: METRON-183 Allow the simple hbase e...
Posted by james-sirota <gi...@git.apache.org>.
Github user james-sirota commented on the pull request:
https://github.com/apache/incubator-metron/pull/131#issuecomment-222348213
What am I supposed to do to validate this? I see there is now a CF field in the enrichment and threat intel loaders and it bulk loads HBase with that CF. Is that it?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request: METRON-183 Allow the simple hbase e...
Posted by cestella <gi...@git.apache.org>.
Github user cestella commented on the pull request:
https://github.com/apache/incubator-metron/pull/131#issuecomment-222363251
The way to validate this is to
* Create a second column family on the enrichment HBase table, say `cf1`
* Push some enrichment data into the table in that column family. Something similar to the malicious IP's from the blog post. I'd suggest calling the enrichment type `malicious_ip`
* Modify the configs for one of the topologies you want to test in `/usr/metron/0.1BETA/config/zookeeper/enrichment` to add a `config` section under `threatIntel` marking that `malicious_ip` should come from column family `cf1` like the following
```
{
"index": "bro",
"batchSize": 5,
"threatIntel": {
"fieldMap": {
"hbaseThreatIntel" : [ "ip_dst_addr" ]
},
"fieldToTypeMap": {
"ip_dst_addr" : [ "malicious_ip" ]
},
"config" : {
"typeToColumnFamily" : {
"malicious_ip" : "cf1"
}
}
}
}
```
* Run some data through and ensure that enrichments still exist.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request: METRON-183 Allow the simple hbase e...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/incubator-metron/pull/131
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request: METRON-183 Allow the simple hbase e...
Posted by cestella <gi...@git.apache.org>.
Github user cestella commented on the pull request:
https://github.com/apache/incubator-metron/pull/131#issuecomment-222363326
As it stands, we have the docs for the configs with where the configurations live. I think you are absolutely right that we need to move them closer to the place where they are used, rather than where the configuration objects are defined.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request: METRON-183 Allow the simple hbase e...
Posted by merrimanr <gi...@git.apache.org>.
Github user merrimanr commented on the pull request:
https://github.com/apache/incubator-metron/pull/131#issuecomment-222154269
+1
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---