You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@metron.apache.org by "ASF GitHub Bot (JIRA)" <ji...@apache.org> on 2016/11/07 15:01:07 UTC
[jira] [Commented] (METRON-554) Require proper error handling when
invalid input is fed to Threat triage rules
[ https://issues.apache.org/jira/browse/METRON-554?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15644410#comment-15644410 ]
ASF GitHub Bot commented on METRON-554:
---------------------------------------
GitHub user mmiklavc opened a pull request:
https://github.com/apache/incubator-metron/pull/346
METRON-554: Require proper error handling when invalid input is fed to Threat triage rules
This completes https://issues.apache.org/jira/browse/METRON-554
Non-boolean-returning functions used as predicates in the triage rules will cause generic exceptions like the following:
```
java.lang.ClassCastException: Cannot cast java.lang.String to java.lang.Boolean
at java.lang.Class.cast(Class.java:3369) ~[?:1.8.0_60]
at org.apache.metron.common.stellar.BaseStellarProcessor.parse(BaseStellarProcessor.java:58) ~[stormjar.jar:?]
at org.apache.metron.common.stellar.StellarPredicateProcessor.parse(StellarPredicateProcessor.java:53) ~[stormjar.jar:?]
at org.apache.metron.threatintel.triage.ThreatTriageProcessor.apply(ThreatTriageProcessor.java:58) ~[stormjar.jar:?]
```
This fix makes it clear where the problem is.
**Testing**
Edit the bro enrichment config in $METRON_HOME/config/zookeeper/enrichments/bro.json
Add a predicate to the riskLevelRules that does not return a boolean value, e.g. `"TO_UPPER(protocol)" : 0.92"` as shown in the example below.
```
{
"index": "bro",
"batchSize": 5,
"enrichment" : {
"fieldMap": {
"geo": ["ip_dst_addr", "ip_src_addr"],
"host": ["host"]
}
},
"threatIntel": {
"fieldMap": {
"hbaseThreatIntel": ["ip_src_addr", "ip_dst_addr"]
},
"fieldToTypeMap": {
"ip_src_addr" : ["malicious_ip"],
"ip_dst_addr" : ["malicious_ip"]
},
"triageConfig" : {
"riskLevelRules" : {
"exists(ip_dst_addr)" : 0.10,
"TO_UPPER(protocol) == 'HTTP'" : 0.91,
"TO_UPPER(protocol)" : 0.92,
"exists(ip_dst_port)" : 0.20,
"exists(ip_src_port)" : 0.30000000000
},
"aggregator" : "MAX",
"aggregationConfig":
{
"NEGATIVE_VALUES_TRUMP_CONF" : "false"
}
}
}
}
```
Load the new configuration in zookeeper doing the following:
```
$METRON_HOME/bin/zk_load_configs.sh -z node1:2181 -m PUSH -i $METRON_HOME/config/zookeeper/
```
The configuration should push out to the enrichment topology. You will need to wait a bit for new bro messages to percolate through the system. Verify the new, more specific error message in the storm worker logs, an example of which is shown below.
example path for enrichment - /var/log/storm/workers-artifacts/enrichment-7-1478449668/6700/worker.log
Revised Storm worker error message:
```
2016-11-06 16:47:56.325 o.a.m.e.b.JoinBolt [ERROR] [Metron] Unable to join messages: {"adapter.threatinteladapter.end.ts":"1478450876296","adapter.threatinteladapter.begin.ts":"1478450876296","source.type":"bro"}
java.lang.IllegalArgumentException: The rule 'TO_UPPER(protocol)' does not return a boolean value.
at org.apache.metron.common.stellar.StellarPredicateProcessor.parse(StellarPredicateProcessor.java:55) ~[stormjar.jar:?]
at org.apache.metron.threatintel.triage.ThreatTriageProcessor.apply(ThreatTriageProcessor.java:58) ~[stormjar.jar:?]
at org.apache.metron.enrichment.bolt.ThreatIntelJoinBolt.joinMessages(ThreatIntelJoinBolt.java:133) ~[stormjar.jar:?]
at org.apache.metron.enrichment.bolt.ThreatIntelJoinBolt.joinMessages(ThreatIntelJoinBolt.java:38) ~[stormjar.jar:?]
at org.apache.metron.enrichment.bolt.JoinBolt.execute(JoinBolt.java:113) [stormjar.jar:?]
at org.apache.storm.daemon.executor$fn__6571$tuple_action_fn__6573.invoke(executor.clj:734) [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
at org.apache.storm.daemon.executor$mk_task_receiver$fn__6492.invoke(executor.clj:466) [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
at org.apache.storm.disruptor$clojure_handler$reify__6005.onEvent(disruptor.clj:40) [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
at org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:451) [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
at org.apache.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:430) [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
at org.apache.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:73) [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
at org.apache.storm.daemon.executor$fn__6571$fn__6584$fn__6637.invoke(executor.clj:853) [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
at org.apache.storm.util$async_loop$fn__554.invoke(util.clj:484) [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_77]
2016-11-06 16:47:56.326 o.a.s.d.executor [ERROR]
java.lang.IllegalArgumentException: The rule 'TO_UPPER(protocol)' does not return a boolean value.
at org.apache.metron.common.stellar.StellarPredicateProcessor.parse(StellarPredicateProcessor.java:55) ~[stormjar.jar:?]
at org.apache.metron.threatintel.triage.ThreatTriageProcessor.apply(ThreatTriageProcessor.java:58) ~[stormjar.jar:?]
at org.apache.metron.enrichment.bolt.ThreatIntelJoinBolt.joinMessages(ThreatIntelJoinBolt.java:133) ~[stormjar.jar:?]
at org.apache.metron.enrichment.bolt.ThreatIntelJoinBolt.joinMessages(ThreatIntelJoinBolt.java:38) ~[stormjar.jar:?]
at org.apache.metron.enrichment.bolt.JoinBolt.execute(JoinBolt.java:113) [stormjar.jar:?]
at org.apache.storm.daemon.executor$fn__6571$tuple_action_fn__6573.invoke(executor.clj:734) [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
at org.apache.storm.daemon.executor$mk_task_receiver$fn__6492.invoke(executor.clj:466) [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
at org.apache.storm.disruptor$clojure_handler$reify__6005.onEvent(disruptor.clj:40) [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
at org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:451) [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
at org.apache.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:430) [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
at org.apache.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:73) [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
at org.apache.storm.daemon.executor$fn__6571$fn__6584$fn__6637.invoke(executor.clj:853) [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
at org.apache.storm.util$async_loop$fn__554.invoke(util.clj:484) [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_77]
```
I went with an IllegalArgumentException because, while it's a runtime exception, this is more of a configuration error.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/mmiklavc/incubator-metron METRON-554
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-metron/pull/346.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #346
----
commit fc331b30f18473278884d58714ec3b3451bb2467
Author: Michael Miklavcic <mi...@gmail.com>
Date: 2016-11-06T16:17:32Z
Throw meaningful exception when predicate return value fails to cast to boolean
----
> Require proper error handling when invalid input is fed to Threat triage rules
> ------------------------------------------------------------------------------
>
> Key: METRON-554
> URL: https://issues.apache.org/jira/browse/METRON-554
> Project: Metron
> Issue Type: Improvement
> Reporter: Michael Miklavcic
> Assignee: Michael Miklavcic
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)