You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@metron.apache.org by mm...@apache.org on 2019/11/25 16:37:53 UTC
[metron] branch master updated: METRON-2326 Unable to Call
ENRICHMENT_GET from Threat Triage Rule Reason Field (nickwallen via
mmiklavc) closes apache/metron#1570
This is an automated email from the ASF dual-hosted git repository.
mmiklavcic pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/metron.git
The following commit(s) were added to refs/heads/master by this push:
new 657028b METRON-2326 Unable to Call ENRICHMENT_GET from Threat Triage Rule Reason Field (nickwallen via mmiklavc) closes apache/metron#1570
657028b is described below
commit 657028bc7f69fb7b4d54fae22b3c3fcb5d88b61b
Author: nickwallen <ni...@nickallen.org>
AuthorDate: Mon Nov 25 09:37:27 2019 -0700
METRON-2326 Unable to Call ENRICHMENT_GET from Threat Triage Rule Reason Field (nickwallen via mmiklavc) closes apache/metron#1570
---
.../enrichment/threatintel/ThreatTriageConfig.java | 14 +---
.../metron-enrichment-common/README.md | 98 ++++++++++++++++++----
2 files changed, 82 insertions(+), 30 deletions(-)
diff --git a/metron-platform/metron-common/src/main/java/org/apache/metron/common/configuration/enrichment/threatintel/ThreatTriageConfig.java b/metron-platform/metron-common/src/main/java/org/apache/metron/common/configuration/enrichment/threatintel/ThreatTriageConfig.java
index f363f76..0582a70 100644
--- a/metron-platform/metron-common/src/main/java/org/apache/metron/common/configuration/enrichment/threatintel/ThreatTriageConfig.java
+++ b/metron-platform/metron-common/src/main/java/org/apache/metron/common/configuration/enrichment/threatintel/ThreatTriageConfig.java
@@ -20,8 +20,6 @@ package org.apache.metron.common.configuration.enrichment.threatintel;
import com.google.common.base.Joiner;
import org.apache.metron.common.aggregator.Aggregators;
-import org.apache.metron.stellar.common.StellarPredicateProcessor;
-import org.apache.metron.stellar.common.StellarProcessor;
import java.util.ArrayList;
import java.util.HashMap;
@@ -49,8 +47,6 @@ public class ThreatTriageConfig {
public void setRiskLevelRules(List<RiskLevelRule> riskLevelRules) {
List<RiskLevelRule> rules = new ArrayList<>();
Set<String> ruleIndex = new HashSet<>();
- StellarPredicateProcessor predicateProcessor = new StellarPredicateProcessor();
- StellarProcessor processor = new StellarProcessor();
for(RiskLevelRule rule : riskLevelRules) {
if(rule.getRule() == null || rule.getScoreExpression() == null) {
@@ -58,17 +54,9 @@ public class ThreatTriageConfig {
}
if(ruleIndex.contains(rule.getRule())) {
continue;
- }
- else {
+ } else {
ruleIndex.add(rule.getRule());
}
-
- // validate the fields which are expected to be valid Stellar expressions
- predicateProcessor.validate(rule.getRule());
- if(rule.getReason() != null) {
- processor.validate(rule.getReason());
- }
-
rules.add(rule);
}
this.riskLevelRules = rules;
diff --git a/metron-platform/metron-enrichment/metron-enrichment-common/README.md b/metron-platform/metron-enrichment/metron-enrichment-common/README.md
index c6e6efc..530be4a 100644
--- a/metron-platform/metron-enrichment/metron-enrichment-common/README.md
+++ b/metron-platform/metron-enrichment/metron-enrichment-common/README.md
@@ -217,26 +217,89 @@ The `triageConfig` field is also a complex field and it bears some description:
| `riskLevelRules` | This is a list of rules (represented as Stellar expressions) associated with scores with optional names and comments | see below|
| `aggregator` | An aggregation function that takes all non-zero scores representing the matching queries from `riskLevelRules` and aggregates them into a single score. | `"MAX"` |
-A risk level rule is of the following format:
-* `name` : The name of the threat triage rule
-* `comment` : A comment describing the rule
-* `rule` : The rule, represented as a Stellar statement
-* `score` : The score attributed to the rule. Can be either numeric or a Stellar expression. The expression has access to all fields with the message being triaged.
-* `reason` : Reason the rule tripped. Can be represented as a Stellar statement
-
-An example of a rule is as follows:
+
+#### Risk Level Rules
+
+A message is triaged by applying a set of risk scoring rules. These rules are used to calculate an overall threat score that can be used to prioritize threats. For each message a rule may either apply and attribute to the overall risk score or the rule may be ignored. A set of rules might look like the following.
+
```
- "riskLevelRules" : [
- {
- "name" : "is internal",
- "comment" : "determines if the destination is internal.",
- "rule" : "IN_SUBNET(ip_dst_addr, '192.168.0.0/24')",
- "score" : 10,
- "reason" : "FORMAT('%s is internal', ip_dst_addr)"
- }
- ]
+"riskLevelRules" : [
+ {
+ "name" : "Destination IP is internal",
+ "comment" : "Determines if the destination IP is on the internal network.",
+ "rule" : "IN_SUBNET(ip_dst_addr, '192.168.0.0/24')",
+ "score" : 10,
+ "reason" : "FORMAT('%s is an internal IP', ip_dst_addr)"
+ },
+ {
+ "name" : "Originates outside of the United States",
+ "comments": "External to US, but lesser risk applies to North America.",
+ "rule" : "geo.country != 'US'",
+ "score" : "if geo.country in ['MX','CA'] then 10 else 200",
+ "reason" : "FORMAT('%s originates from %s', ip_dst_addr, geo.country)"
+ }
+]
```
+A risk level rule can contain the following fields.
+* [name](#name)
+* [comment](#comment)
+* [rule](#rule)
+* [score](#score)
+* [reason](#reason)
+
+##### name
+
+The name of the threat triage rule.
+
+* This is an optional field.
+* This is expected to be a simple string.
+
+##### comment
+
+A comment describing the threat triage rule.
+
+* This is an optional field.
+* This is expected to be a simple string.
+
+##### rule
+
+A Stellar expression that determines whether this Risk Level Rule applies to a given message. If the [rule](#rule) expression returns true, the [score](#score) will be aggregated into the message's overall threat score.
+
+* This is a required field.
+* This is expected to be a valid Stellar expression.
+* The expression can refer to any field within the message.
+* The expression must return a boolean.
+* Any non-boolean value returned by the expression is treated as an error.
+* See [Best Practices](#best-practices) when using Stellar expression in Risk Level Rules.
+
+##### score
+
+The [score](#score) that is aggregated into a message's overall threat score.
+
+* This is a required field.
+* This is expected to be a valid Stellar expression.
+* This can be a numeric value, as any numeric value is also a valid Stellar expression.
+* The expression can refer to any field within the message.
+* Any non-numeric value returned by the expression is treated as an error.
+* See [Best Practices](#best-practices) when using Stellar expression in Risk Level Rules.
+
+##### reason
+
+Provides a reason for why the Risk Level Rule was applied. This allows additional context to be retrieved from the message. This is intended to enable SOC operators to better address the threat.
+
+* This is an optional field.
+* This is expected to be a valid Stellar expression.
+* The expression should return a string or a value that can be converted to a string.
+* The expression can refer to any field within the message.
+* See [Best Practices](#best-practices) when using Stellar expression in Risk Level Rules.
+
+##### Best Practices
+
+For best performance, the Stellar expressions contained within a Risk Level Rule should avoid I/O intensive operations like making queries to external platforms like HBase. Instead, the query should be performed as an Enrichment (see [Stellar Enrichment Configuration](#stellar_enrichment_configuration)) and the value should be stored within the message. This value can then be directly referenced by the field name when defining the [rule](#rule), [score](#score), or [reason](#reason) fiel [...]
+
+#### Threat Score Aggregation
+
The supported aggregation functions are:
* `MAX` : The max of all of the associated values for matching queries
* `MIN` : The min of all of the associated values for matching queries
@@ -302,3 +365,4 @@ An example configuration for the YAF sensor is as follows:
```
ThreatIntel alert levels are emitted as a new field "threat.triage.level." So for the example above, an incoming message that trips the `ip_src_addr` rule will have a new field threat.triage.level=10.
+