You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@metron.apache.org by "ASF GitHub Bot (JIRA)" <ji...@apache.org> on 2017/02/03 19:39:51 UTC
[jira] [Commented] (METRON-686) Record Rule Set that Fired During
Threat Triage
[ https://issues.apache.org/jira/browse/METRON-686?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15852008#comment-15852008 ]
ASF GitHub Bot commented on METRON-686:
---------------------------------------
GitHub user nickwallen opened a pull request:
https://github.com/apache/incubator-metron/pull/438
METRON-686 Record Rule Set that Fired During Threat Triage
The goal of threat triage is to prioritize the alerts that pose the greatest threat and thus need urgent attention. To perform threat triage, a set of rules are applied to each message. Each rule has a predicate to determine if the rule applies or not. The threat score from each applied rule is aggregated into a single threat triage score that can be used to prioritize high risk threats.
### Changes
* All of the threat triage rules that were applied to a message during threat triage are captured and recorded.
* A new 'reason' field was added to the threat triage rules to allow a rule author to provide contextual information to assist in actioning a threat.
### Implementation
The `ThreatTriageProcessor` attempts to apply each `RiskLevelRule` to a message being triaged. If the rule's predicate returns true, the rule is applied to the message resulting in a `RuleScore`. The `RuleScore` represents the portion of the threat score contributed by a single rule. Each of the `RuleScore`s are aggregated into an overall `ThreatScore`, which captures the outcome of the threat triage process. This `ThreatScore` is then attached to the message.
### Example
Previously the result of threat triage was a single threat score represented as a double.
```
"threat.triage.level": 10.0
```
The result of threat triage now provides greater information about the triage process. The rule set...
```
"triageConfig" : {
"riskLevelRules" : [ {
"name" : "Abnormal DNS Port",
"rule" : "source.type == \"bro\" and protocol == \"dns\" and ip_dst_port != 53",
"score" : 10.0,
"reason" : "FORMAT(\"Abnormal DNS Port: expected: 53, found: %s:%d\", ip_dst_addr, ip_dst_port)"
} ],
"aggregator" : "MAX",
"aggregationConfig" : { }
}
```
produces a result like...
```
"threat.triage.level":{
"score":10.0,
"rules":[
{
"name":"Abnormal DNS Port",
"comment":null
"score":10.0,
"reason":"Abnormal DNS Port: expected: 53, found: 224.0.0.251:5353",
}
]
}
```
### Testing
Setup a Metron environment with the default Bro sensor running. The following commands can be executed in the Stellar REPL.
```
# fetch the current config
conf := CONFIG_GET("ENRICHMENT", "bro")
# triage all bro telemetry
is_alert := source.type == 'bro'
conf := ENRICHMENT_STELLAR_TRANSFORM_ADD(conf, 'ENRICHMENT', SHELL_VARS2MAP('is_alert'))
# create a rule that looks for DNS running on non-standard ports
rule := source.type == "bro" and protocol == "dns" and ip_dst_port != 53
triage := { "name":"Abnormal DNS Port", "rule": SHELL_GET_EXPRESSION('rule'), "reason": SHELL_GET_EXPRESSION('reason'), "score":"10" }
conf := THREAT_TRIAGE_ADD(conf, [triage])
# save the configuration
CONFIG_PUT("ENRICHMENT", conf, "bro")
```
Monitor the messages being pushed to the 'indexing' topic in Kafka. You should expect to see a message where the 'threat.triage.level' is like the example shown above.
Printing the threat triage rule in the REPL is going to look something like this.
```
> THREAT_TRIAGE_PRINT(conf)
╔═══════════════════╤═════════╤══════════════════════════════════════════════════════════════════╤═══════╤═══════════════════════════════════════════════════════════════════════════════════╗
║ Name │ Comment │ Triage Rule │ Score │ Reason ║
╠═══════════════════╪═════════╪══════════════════════════════════════════════════════════════════╪═══════╪═══════════════════════════════════════════════════════════════════════════════════╣
║ Abnormal DNS Port │ │ source.type == "bro" and protocol == "dns" and ip_dst_port != 53 │ 10 │ FORMAT("Abnormal DNS Port: expected: 53, found: %s:%d", ip_dst_addr, ip_dst_port) ║
╚═══════════════════╧═════════╧══════════════════════════════════════════════════════════════════╧═══════╧═══════════════════════════════════════════════════════════════════════════════════╝
```
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/nickwallen/incubator-metron METRON-686
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-metron/pull/438.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #438
----
commit fe455187ff1f928edc43a6bfa509408695783a54
Author: Nick Allen <ni...@nickallen.org>
Date: 2017-02-02T23:29:07Z
METRON-686 Record Rule Set that Fired During Threat Triage
----
> Record Rule Set that Fired During Threat Triage
> -----------------------------------------------
>
> Key: METRON-686
> URL: https://issues.apache.org/jira/browse/METRON-686
> Project: Metron
> Issue Type: Improvement
> Reporter: Nick Allen
> Assignee: Nick Allen
>
> h3. Problem
> There is little transparency into the Threat Triage process itself. When Threat Triage runs, all I get is a score. I don't know how that score was arrived at, which rules were triggered, and the specific values that caused a rule to trigger.
> More specifically, there is no way to generate a message that looks like "The host 'powned.svr.bank.com' has '230' inbound flows, exceeding the threshold of '202'". This makes it difficult for an analyst to action the alert.
> h3. Proposed Solution
> To improve the transparency of the Threat Triage process, I am proposing these enhancements.
> (1) Threat Triage should attach to each message all of the rules that fired in addition to the total calculated threat triage score.
> (2) Threat Triage should allow a custom message to be generated for each rule. The custom message would allow for some form of string interpolation so that I can add specific values from each message to the generated alert. We could allow this in one or both of the new fields that Casey just added, name and comment.
> (3) The specific method of string interpolation will be implemented under a separate issue.
> h3. Example
> (1) In this example, we have a telemetry message with a field called 'value' that we need to monitor. In Enrichment, I calculate some sort of value threshold, over which an alert should be generated.
> (2) In Threat Triage, I use the calculated value threshold to alert on any message that has a value exceeding this threshold.
> (3) I can embed values from the message, like the hostname, value, and value threshold, into the alert produced by Threat Triage. Notice that I am using {noformat}${this}{noformat} for string interpolation, but it could be any syntax that we choose.
> {code}
> "triageConfig" : {
> "riskLevelRules" : [
> {
> "name" : "Abnormal Value",
> "comment" : "For ${hostname}; the value ${value} exceeds threshold of ${value_threshold}",
> "rule" : "value > value_threshold",
> "score" : 10
> }
> ],
> "aggregator" : "MAX"
> }
> {code}
> (4) The Threat Triage process today would add only the total calculated score.
> {code}
> "threat.triage.level": 10.0
> {code}
> With this proposal, Threat Triage would add the following to the message.
> Notice how each of the {noformat}${variables}{noformat} have been replaced with the actual values extracted from the message. This allows for more contextual information to action the alert.
> {code}
> "threat.triage": {
> "score": 10.0,
> "rules": [
> {
> "name": "Abnormal Value",
> "comment" : "For 10.0.0.1; the value 101 exceeds threshold of 42",
> "score" : 10
> }
> ]
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)