You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@metron.apache.org by cestella <gi...@git.apache.org> on 2017/01/26 21:57:02 UTC
[GitHub] incubator-metron issue #426: METRON-675: Make Threat Triage rules able to be...
Github user cestella commented on the issue:
https://github.com/apache/incubator-metron/pull/426
Testing Instructions beyond the normal smoke test (i.e. letting data
flow through to the indices and checking them).
## Preliminaries
It is helpful to install the elasticsearch head plugin:
* `/usr/share/elasticsearch/bin/plugin install mobz/elasticsearch-head`
Also, set an environment variable to indicate `METRON_HOME`:
* `export METRON_HOME=/usr/metron/0.3.0`
## Adjust configs to Bro
We will adjust the bro topology to have a couple of threat triage rules:
* Edit `$METRON_HOME/config/zookeeper/enrichment/bro.json` as follows:
```
{
"enrichment" : {
"fieldMap": {
"geo": ["ip_dst_addr", "ip_src_addr"],
"host": ["host"]
}
},
"threatIntel": {
"fieldMap": {
"hbaseThreatIntel": ["ip_src_addr", "ip_dst_addr"],
"stellar" : {
"config" : {
"is_alert" : "ip_dst_port == 80 or ip_dst_port == 5353"
}
}
},
"fieldToTypeMap": {
"ip_src_addr" : ["malicious_ip"],
"ip_dst_addr" : ["malicious_ip"]
},
"triageConfig" : {
"riskLevelRules" : [
{
"name" : "web",
"comment" : "Bump risk if web connection",
"rule" : "ip_dst_port == 80",
"score" : 10
},
{
"name" : "dns",
"comment" : "Bump risk if dns connection",
"rule" : "ip_dst_port == 5353",
"score" : 20
}
],
"aggregator" : "MAX"
}
}
}
```
This should create 2 rules:
* set the triage level to 20 if the destination port is a DNS port
* set the triage level to 10 if the destination port is a web port
Ensure via the elasticsearch head plugin that the following is true:
* All bro messages with `is_alert == true` and `ip_dst_port == 5353` have a `threat:triage:level` of 20
* All bro messages with `is_alert == true` and `threat:triage:level == 20` have a `ip_dst_port` of 20
* All bro messages with `is_alert == true` and `ip_dst_port == 80` have a `threat:triage:level` of 10
* All bro messages with `is_alert == true` and `threat:triage:level == 10` have a `ip_dst_port` of 10
### Test Case: Stellar Management Functions
* Upload the management functions via `scp metron-platform/metron-management/target/metron-management-0.3.0.jar root@node1:/usr/metron/0.3.0/lib`
* Create a file with the following contents named `~/script.stellar`
```
# First we get the squid enrichment config from zookeeper.
# If it is not there, which it is not by default, a suitable default
# config will be specified.
squid_enrichment_config := CONFIG_GET('ENRICHMENT', 'squid')
# We should not have any threat triage rules
THREAT_TRIAGE_PRINT(squid_enrichment_config)
# Just for illustration, we can create a threat alert if the country of the domain registered
# is non-US, then we can make an alert. To do that, we need to create an is_alert field on the message.
#
# I know that maps get folded into the message, so that whois_info enrichment is going to create a few fields:
# * domain mapped to whois_info.domain
# * registrar mapped to whois_info.registrar
# * home_country mapped to whois_info.home_country
# * owner mapped to whois_info.owner
whois_info.home_country := 'US'
# Now with this, we can create a rule or two to triage these alerts.
# This means associating a rule as described by a stellar expression that returns true or false with a score
# Also associated with this ruleset is an aggregation function, the default of which is MAX.
# Now we can make a couple rules:
# * If the message is an alert and from a non-us whois source, we can set the level to 10
# * If the message is an alert and non-local, we can set the level to 20
# * If the message is an alert and both non-local and non-us, then we can set the level to 50
# If multiple rules hit, then we should take the max (default behavior)
non_us := whois_info.home_country != 'US'
is_local := IN_SUBNET( if IS_IP(ip_src_addr) then ip_src_addr else NULL, '192.168.0.0/21')
is_both := whois_info.home_country != 'US' && IN_SUBNET( if IS_IP(ip_src_addr) then ip_src_addr else NULL, '192.168.0.0/21')
rules := [ { 'name' : 'is non-us', 'rule' : SHELL_GET_EXPRESSION('non_us'), 'score' : 10 } , { 'name' : 'is local', 'rule' : SHELL_GET_EXPRESSION('is_local'), 'score' : 20 } , { 'name' : 'both non-us and local', 'comment' : 'union of both rules.', 'rule' : SHELL_GET_EXPRESSION('is_both'), 'score' : 50 } ]
# Now that we have our rules staged, we can add them to our config.
squid_enrichment_config_new := THREAT_TRIAGE_ADD( squid_enrichment_config_new, rules )
# Pretty Print the rules
THREAT_TRIAGE_PRINT(squid_enrichment_config_new)
# Now just print the raw config to make sure it jives
squid_enrichment_config_new
# Now that we have admired it, we can remove the rules
squid_enrichment_config_new := THREAT_TRIAGE_REMOVE( squid_enrichment_config_new, [ SHELL_GET_EXPRESSION('non_us') , SHELL_GET_EXPRESSION('is_local') , SHELL_GET_EXPRESSION('is_both') ] )
THREAT_TRIAGE_PRINT(squid_enrichment_config_new)
```
* Execute the script via `cat script.stellar | /usr/metron/0.3.0/bin/stellar -z node1 -na` You should see the following output:
```
Stellar, Go!
Please note that functions are loading lazily in the background and will be unavailable until loaded fully.
{es.clustername=metron, es.ip=node1, es.port=9300, es.date.format=yyyy.MM.dd.HH}
[Stellar]>>> # First we get the squid enrichment config from zookeeper.
[Stellar]>>> # If it is not there, which it is not by default, a suitable default
[Stellar]>>> # config will be specified.
[Stellar]>>> squid_enrichment_config := CONFIG_GET('ENRICHMENT', 'squid')
Functions loaded, you may refer to functions now...
[Stellar]>>> # We should not have any threat triage rules
[Stellar]>>> THREAT_TRIAGE_PRINT(squid_enrichment_config)
\u2554\u2550\u2550\u2550\u2550\u2550\u2550\u2564\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2564\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2564\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2557
\u2551 Name \u2502 Comment \u2502 Triage Rule \u2502 Score \u2551
\u2560\u2550\u2550\u2550\u2550\u2550\u2550\u2567\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2567\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2567\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2563
\u2551 (empty) \u2551
\u255a\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u255d
[Stellar]>>> # Just for illustration, we can create a threat alert if the country of the domain registered
[Stellar]>>> # is non-US, then we can make an alert. To do that, we need to create an is_alert field on the message.
[Stellar]>>> #
[Stellar]>>> # I know that maps get folded into the message, so that whois_info enrichment is going to create a few fields:
[Stellar]>>> # * domain mapped to whois_info.domain
[Stellar]>>> # * registrar mapped to whois_info.registrar
[Stellar]>>> # * home_country mapped to whois_info.home_country
[Stellar]>>> # * owner mapped to whois_info.owner
[Stellar]>>> whois_info.home_country := 'US'
[Stellar]>>> # Now with this, we can create a rule or two to triage these alerts.
[Stellar]>>> # This means associating a rule as described by a stellar expression that returns true or false with a score
[Stellar]>>> # Also associated with this ruleset is an aggregation function, the default of which is MAX.
[Stellar]>>> # Now we can make a couple rules:
[Stellar]>>> # * If the message is an alert and from a non-us whois source, we can set the level to 10
[Stellar]>>> # * If the message is an alert and non-local, we can set the level to 20
[Stellar]>>> # * If the message is an alert and both non-local and non-us, then we can set the level to 50
[Stellar]>>> # If multiple rules hit, then we should take the max (default behavior)
[Stellar]>>> non_us := whois_info.home_country != 'US'
[Stellar]>>> is_local := IN_SUBNET( if IS_IP(ip_src_addr) then ip_src_addr else NULL, '192.168.0.0/21')
[Stellar]>>> is_both := whois_info.home_country != 'US' && IN_SUBNET( if IS_IP(ip_src_addr) then ip_src_addr else NULL, '192.168.0.0/21')
[Stellar]>>> rules := [ { 'name' : 'is non-us', 'rule' : SHELL_GET_EXPRESSION('non_us'), 'score' : 10 } , { 'name' : 'is local', 'rule' : SHELL_GET_EXPRESSION('is_local '), 'score' : 20 } , { 'name' : 'both non-us and local', 'comment' : 'union of both rules.', 'rule' : SHELL_GET_EXPRESSION('is_both'), 'score' : 50 } ]
[Stellar]>>> # Now that we have our rules staged, we can add them to our config.
[Stellar]>>> squid_enrichment_config_new := THREAT_TRIAGE_ADD( squid_enrichment_config_new, rules )
[Stellar]>>> # Pretty Print the rules
[Stellar]>>> THREAT_TRIAGE_PRINT(squid_enrichment_config_new)
\u2554\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2564\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2564\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2564\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2557
\u2551 Name \u2502 Comment \u2502 Triage Rule \u2502 Score \u2551
\u2560\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u256a\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u256a\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u256a\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2563
\u2551 is non-us \u2502 \u2502 whois_info.home_country != 'US' \u2502 10 \u2551
\u255f\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u253c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u253c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u253c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2562
\u2551 is local \u2502 \u2502 IN_SUBNET( if IS_IP(ip_src_addr) then ip_src_addr else NULL, '192.168.0.0/21') \u2502 20 \u2551
\u255f\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u253c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u253c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u253c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2562
\u2551 both non-us and local \u2502 union of both rules. \u2502 whois_info.home_country != 'US' && IN_SUBNET( if IS_IP(ip_src_addr) then ip_src_addr else NULL, '192.168.0.0/21') \u2502 50 \u2551
\u255a\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2567\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2567\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2567\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u255d
Aggregation: MAX
[Stellar]>>> # Now just print the raw config to make sure it jives
[Stellar]>>> squid_enrichment_config_new
{
"enrichment" : {
"fieldMap" : { },
"fieldToTypeMap" : { },
"config" : { }
},
"threatIntel" : {
"fieldMap" : { },
"fieldToTypeMap" : { },
"config" : { },
"triageConfig" : {
"riskLevelRules" : [ {
"name" : "is non-us",
"rule" : "whois_info.home_country != 'US'",
"score" : 10.0
}, {
"name" : "is local",
"rule" : "IN_SUBNET( if IS_IP(ip_src_addr) then ip_src_addr else NULL, '192.168.0.0/21')",
"score" : 20.0
}, {
"name" : "both non-us and local",
"comment" : "union of both rules.",
"rule" : "whois_info.home_country != 'US' && IN_SUBNET( if IS_IP(ip_src_addr) then ip_src_addr else NULL, '192.168.0.0/21')",
"score" : 50.0
} ],
"aggregator" : "MAX",
"aggregationConfig" : { }
}
},
"configuration" : { }
}
[Stellar]>>> # Now that we have admired it, we can remove the rules
[Stellar]>>> squid_enrichment_config_new := THREAT_TRIAGE_REMOVE( squid_enrichment_config_new, [ SHELL_GET_EXPRESSION('non_us') , SHELL_GET_EXPRESSION('is_local') , SHE LL_GET_EXPRESSION('is_both') ] )
[Stellar]>>> THREAT_TRIAGE_PRINT(squid_enrichment_config_new)
\u2554\u2550\u2550\u2550\u2550\u2550\u2550\u2564\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2564\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2564\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2557
\u2551 Name \u2502 Comment \u2502 Triage Rule \u2502 Score \u2551
\u2560\u2550\u2550\u2550\u2550\u2550\u2550\u2567\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2567\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2567\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2563
\u2551 (empty) \u2551
\u255a\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u255d
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---