You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@metron.apache.org by mmiklavc <gi...@git.apache.org> on 2016/11/07 15:01:07 UTC
[GitHub] incubator-metron pull request #346: METRON-554: Require proper error handlin...
GitHub user mmiklavc opened a pull request:
https://github.com/apache/incubator-metron/pull/346
METRON-554: Require proper error handling when invalid input is fed to Threat triage rules
This completes https://issues.apache.org/jira/browse/METRON-554
Non-boolean-returning functions used as predicates in the triage rules will cause generic exceptions like the following:
```
java.lang.ClassCastException: Cannot cast java.lang.String to java.lang.Boolean
at java.lang.Class.cast(Class.java:3369) ~[?:1.8.0_60]
at org.apache.metron.common.stellar.BaseStellarProcessor.parse(BaseStellarProcessor.java:58) ~[stormjar.jar:?]
at org.apache.metron.common.stellar.StellarPredicateProcessor.parse(StellarPredicateProcessor.java:53) ~[stormjar.jar:?]
at org.apache.metron.threatintel.triage.ThreatTriageProcessor.apply(ThreatTriageProcessor.java:58) ~[stormjar.jar:?]
```
This fix makes it clear where the problem is.
**Testing**
Edit the bro enrichment config in $METRON_HOME/config/zookeeper/enrichments/bro.json
Add a predicate to the riskLevelRules that does not return a boolean value, e.g. `"TO_UPPER(protocol)" : 0.92"` as shown in the example below.
```
{
"index": "bro",
"batchSize": 5,
"enrichment" : {
"fieldMap": {
"geo": ["ip_dst_addr", "ip_src_addr"],
"host": ["host"]
}
},
"threatIntel": {
"fieldMap": {
"hbaseThreatIntel": ["ip_src_addr", "ip_dst_addr"]
},
"fieldToTypeMap": {
"ip_src_addr" : ["malicious_ip"],
"ip_dst_addr" : ["malicious_ip"]
},
"triageConfig" : {
"riskLevelRules" : {
"exists(ip_dst_addr)" : 0.10,
"TO_UPPER(protocol) == 'HTTP'" : 0.91,
"TO_UPPER(protocol)" : 0.92,
"exists(ip_dst_port)" : 0.20,
"exists(ip_src_port)" : 0.30000000000
},
"aggregator" : "MAX",
"aggregationConfig":
{
"NEGATIVE_VALUES_TRUMP_CONF" : "false"
}
}
}
}
```
Load the new configuration in zookeeper doing the following:
```
$METRON_HOME/bin/zk_load_configs.sh -z node1:2181 -m PUSH -i $METRON_HOME/config/zookeeper/
```
The configuration should push out to the enrichment topology. You will need to wait a bit for new bro messages to percolate through the system. Verify the new, more specific error message in the storm worker logs, an example of which is shown below.
example path for enrichment - /var/log/storm/workers-artifacts/enrichment-7-1478449668/6700/worker.log
Revised Storm worker error message:
```
2016-11-06 16:47:56.325 o.a.m.e.b.JoinBolt [ERROR] [Metron] Unable to join messages: {"adapter.threatinteladapter.end.ts":"1478450876296","adapter.threatinteladapter.begin.ts":"1478450876296","source.type":"bro"}
java.lang.IllegalArgumentException: The rule 'TO_UPPER(protocol)' does not return a boolean value.
at org.apache.metron.common.stellar.StellarPredicateProcessor.parse(StellarPredicateProcessor.java:55) ~[stormjar.jar:?]
at org.apache.metron.threatintel.triage.ThreatTriageProcessor.apply(ThreatTriageProcessor.java:58) ~[stormjar.jar:?]
at org.apache.metron.enrichment.bolt.ThreatIntelJoinBolt.joinMessages(ThreatIntelJoinBolt.java:133) ~[stormjar.jar:?]
at org.apache.metron.enrichment.bolt.ThreatIntelJoinBolt.joinMessages(ThreatIntelJoinBolt.java:38) ~[stormjar.jar:?]
at org.apache.metron.enrichment.bolt.JoinBolt.execute(JoinBolt.java:113) [stormjar.jar:?]
at org.apache.storm.daemon.executor$fn__6571$tuple_action_fn__6573.invoke(executor.clj:734) [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
at org.apache.storm.daemon.executor$mk_task_receiver$fn__6492.invoke(executor.clj:466) [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
at org.apache.storm.disruptor$clojure_handler$reify__6005.onEvent(disruptor.clj:40) [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
at org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:451) [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
at org.apache.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:430) [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
at org.apache.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:73) [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
at org.apache.storm.daemon.executor$fn__6571$fn__6584$fn__6637.invoke(executor.clj:853) [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
at org.apache.storm.util$async_loop$fn__554.invoke(util.clj:484) [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_77]
2016-11-06 16:47:56.326 o.a.s.d.executor [ERROR]
java.lang.IllegalArgumentException: The rule 'TO_UPPER(protocol)' does not return a boolean value.
at org.apache.metron.common.stellar.StellarPredicateProcessor.parse(StellarPredicateProcessor.java:55) ~[stormjar.jar:?]
at org.apache.metron.threatintel.triage.ThreatTriageProcessor.apply(ThreatTriageProcessor.java:58) ~[stormjar.jar:?]
at org.apache.metron.enrichment.bolt.ThreatIntelJoinBolt.joinMessages(ThreatIntelJoinBolt.java:133) ~[stormjar.jar:?]
at org.apache.metron.enrichment.bolt.ThreatIntelJoinBolt.joinMessages(ThreatIntelJoinBolt.java:38) ~[stormjar.jar:?]
at org.apache.metron.enrichment.bolt.JoinBolt.execute(JoinBolt.java:113) [stormjar.jar:?]
at org.apache.storm.daemon.executor$fn__6571$tuple_action_fn__6573.invoke(executor.clj:734) [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
at org.apache.storm.daemon.executor$mk_task_receiver$fn__6492.invoke(executor.clj:466) [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
at org.apache.storm.disruptor$clojure_handler$reify__6005.onEvent(disruptor.clj:40) [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
at org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:451) [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
at org.apache.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:430) [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
at org.apache.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:73) [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
at org.apache.storm.daemon.executor$fn__6571$fn__6584$fn__6637.invoke(executor.clj:853) [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
at org.apache.storm.util$async_loop$fn__554.invoke(util.clj:484) [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_77]
```
I went with an IllegalArgumentException because, while it's a runtime exception, this is more of a configuration error.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/mmiklavc/incubator-metron METRON-554
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-metron/pull/346.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #346
----
commit fc331b30f18473278884d58714ec3b3451bb2467
Author: Michael Miklavcic <mi...@gmail.com>
Date: 2016-11-06T16:17:32Z
Throw meaningful exception when predicate return value fails to cast to boolean
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #346: METRON-554: Require proper error handlin...
Posted by cestella <gi...@git.apache.org>.
Github user cestella commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/346#discussion_r86792656
--- Diff: metron-platform/metron-common/src/main/java/org/apache/metron/common/stellar/StellarPredicateProcessor.java ---
@@ -51,6 +48,11 @@ public Boolean parse( String rule
if(rule == null || isEmpty(rule.trim())) {
return true;
}
- return super.parse(rule, variableResolver, functionResolver, context);
+ try {
+ return super.parse(rule, variableResolver, functionResolver, context);
+ } catch (ClassCastException e) {
+ // predicate must return boolean
+ throw new IllegalArgumentException(String.format("The rule '%s' does not return a boolean value.", rule));
--- End diff --
Can you pass in `e` as an argument to the IAE so the user can tell the type that was returned. Alternatively, you could add e.getMessage() to the IAE message.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #346: METRON-554: Require proper error handlin...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/incubator-metron/pull/346
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #346: METRON-554: Require proper error handling when ...
Posted by cestella <gi...@git.apache.org>.
Github user cestella commented on the issue:
https://github.com/apache/incubator-metron/pull/346
Can you please assign this to yourself in JIRA and put it in the 0.2.2BETA release?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---