You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@metron.apache.org by nickwallen <gi...@git.apache.org> on 2017/09/05 21:47:33 UTC
[GitHub] metron pull request #733: Metron 1156
GitHub user nickwallen opened a pull request:
https://github.com/apache/metron/pull/733
Metron 1156
Troubleshooting issues when programming against a live stream of data is difficult. It would be useful to have a means to run the entire threat triage process within the REPL before deploying your rule set on a Metron cluster. This creates a set of functions to allow simulation of Threat Triage inside of the Stellar REPL. This is useful for creating new triage rules, debugging existing triage rules, and to iterate quickly when testing rule sets.
### Changes
Created the following new functions.
* `THREAT_TRIAGE_INIT`
* `THREAT_TRIAGE_SCORE`
* `THREAT_TRIAGE_CONFIG`
Modified these functions to preserve existing functionality, along with enhancing them to work with the newly created functions.
* `THREAT_TRIAGE_PRINT`
* `THREAT_TRIAGE_ADD`
* `THREAT_TRIAGE_REMOVE`
* `THREAT_TRIAGE_SET_AGGREGATOR`
### Testing
The following shows how the new functions can be used.
1. Create a threat triage engine.
```
[Stellar]>>> t := THREAT_TRIAGE_INIT()
[Stellar]>>> t
ThreatTriage{0 rule(s)}
```
1. Add a few triage rules.
```
[Stellar]>>> THREAT_TRIAGE_ADD(t, {"name":"rule1", "rule":"value>10", "score":10})
{
"enrichment" : {
"fieldMap" : { },
"fieldToTypeMap" : { },
"config" : { }
},
"threatIntel" : {
"fieldMap" : { },
"fieldToTypeMap" : { },
"config" : { },
"triageConfig" : {
"riskLevelRules" : [ {
"name" : "rule1",
"rule" : "value>10",
"score" : 10.0
} ],
"aggregator" : "MAX",
"aggregationConfig" : { }
}
},
"configuration" : { }
}
```
```
[Stellar]>>> THREAT_TRIAGE_ADD(t, {"name":"rule2", "rule":"value>20", "score":20})
{
"enrichment" : {
"fieldMap" : { },
"fieldToTypeMap" : { },
"config" : { }
},
"threatIntel" : {
"fieldMap" : { },
"fieldToTypeMap" : { },
"config" : { },
"triageConfig" : {
"riskLevelRules" : [ {
"name" : "rule1",
"rule" : "value>10",
"score" : 10.0
}, {
"name" : "rule2",
"rule" : "value>20",
"score" : 20.0
} ],
"aggregator" : "MAX",
"aggregationConfig" : { }
}
},
"configuration" : { }
}
```
```
[Stellar]>>> THREAT_TRIAGE_ADD(t, {"name":"rule3", "rule":"value>30", "score":30})
{
"enrichment" : {
"fieldMap" : { },
"fieldToTypeMap" : { },
"config" : { }
},
"threatIntel" : {
"fieldMap" : { },
"fieldToTypeMap" : { },
"config" : { },
"triageConfig" : {
"riskLevelRules" : [ {
"name" : "rule1",
"rule" : "value>10",
"score" : 10.0
}, {
"name" : "rule2",
"rule" : "value>20",
"score" : 20.0
}, {
"name" : "rule3",
"rule" : "value>30",
"score" : 30.0
} ],
"aggregator" : "MAX",
"aggregationConfig" : { }
}
},
"configuration" : { }
}
```
1. Review the rules that you have created.
```
[Stellar]>>> THREAT_TRIAGE_PRINT(t)
╔═══════╤═════════╤═════════════╤═══════╤════════╗
║ Name │ Comment │ Triage Rule │ Score │ Reason ║
╠═══════╪═════════╪═════════════╪═══════╪════════╣
║ rule1 │ │ value>10 │ 10 │ ║
╟───────┼─────────┼─────────────┼───────┼────────╢
║ rule2 │ │ value>20 │ 20 │ ║
╟───────┼─────────┼─────────────┼───────┼────────╢
║ rule3 │ │ value>30 │ 30 │ ║
╚═══════╧═════════╧═════════════╧═══════╧════════╝
```
1. Create a few test messages to simulate your telemetry.
```
[Stellar]>>> msg1 := "{ \"value\":22 }"
[Stellar]>>> msg1
{ "value":22 }
```
```
[Stellar]>>> msg2 := "{ \"value\":44 }"
[Stellar]>>> msg2
{ "value":44 }
```
1. Score a message based on the rules that have been defined. The result allows you to see the total score, the aggregator, along with details about each rule that fired.
```
[Stellar]>>> THREAT_TRIAGE_SCORE( msg1, t)
{score=20.0, aggregator=MAX, rules=[{score=10.0, name=rule1, rule=value>10}, {score=20.0, name=rule2, rule=value>20}]}
```
```
[Stellar]>>> THREAT_TRIAGE_SCORE( msg2, t)
{score=30.0, aggregator=MAX, rules=[{score=10.0, name=rule1, rule=value>10}, {score=20.0, name=rule2, rule=value>20}, {score=30.0, name=rule3, rule=value>30}]}
```
1. Now that I know the rule set has worked well, I can extract the configuration and push it into my live, Metron cluster.
```
[Stellar]>>> conf := THREAT_TRIAGE_CONFIG( t)
[Stellar]>>> conf
{
"enrichment" : {
"fieldMap" : { },
"fieldToTypeMap" : { },
"config" : { }
},
"threatIntel" : {
"fieldMap" : { },
"fieldToTypeMap" : { },
"config" : { },
"triageConfig" : {
"riskLevelRules" : [ {
"name" : "rule1",
"rule" : "value>10",
"score" : 10.0
}, {
"name" : "rule2",
"rule" : "value>20",
"score" : 20.0
}, {
"name" : "rule3",
"rule" : "value>30",
"score" : 30.0
}],
"aggregator" : "MAX",
"aggregationConfig" : { }
}
},
"configuration" : { }
}
```
```
[Stellar]>>> CONFIG_PUT("ENRICHMENT", conf, "bro")
```
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/nickwallen/metron METRON-1156
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/metron/pull/733.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #733
----
commit 5ed6bcbe9c312f0132f21bec44cc109b1e9fe111
Author: Nick Allen <ni...@nickallen.org>
Date: 2017-09-05T02:29:41Z
METRON-1156 Simulate Triage Rules in the Stellar REPL
commit 3fb4accc9175a2920aa6aa9ebf0d6f96e4d30c7c
Author: Nick Allen <ni...@nickallen.org>
Date: 2017-09-05T21:46:03Z
Removed unnecessary import
----
---
[GitHub] metron issue #733: METRON-1156 Simulate Triage Rules in the Stellar REPL
Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on the issue:
https://github.com/apache/metron/pull/733
Bump.
This is useful for debugging your triage rules or even as a way to introduce new users to how threat triage works. Similar in function and form to the Profile Debugger.
---
[GitHub] metron issue #733: METRON-1156 Simulate Triage Rules in the Stellar REPL
Posted by cestella <gi...@git.apache.org>.
Github user cestella commented on the issue:
https://github.com/apache/metron/pull/733
+1 this is useful work; thanks @nickwallen
---
[GitHub] metron pull request #733: METRON-1156 Simulate Triage Rules in the Stellar R...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/metron/pull/733
---