You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@metron.apache.org by mm...@apache.org on 2018/07/11 01:33:04 UTC
[48/50] [abbrv] metron git commit: METRON-1644: Support parser
chaining closes apache/incubator-metron#1084
http://git-wip-us.apache.org/repos/asf/metron/blob/cbdaee17/use-cases/parser_chaining/README.md
----------------------------------------------------------------------
diff --git a/use-cases/parser_chaining/README.md b/use-cases/parser_chaining/README.md
new file mode 100644
index 0000000..26fd333
--- /dev/null
+++ b/use-cases/parser_chaining/README.md
@@ -0,0 +1,235 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+# Problem Statement
+
+Aggregating many different types sensors into a single data source (e.g.
+syslog) and ingesting that aggregate sensor into Metron is a common pattern. It
+is not obvious precisely how to manage these types of aggregate sensors
+as they require two-pass parsing. This document will walk through an
+example of supporting this kind of multi-pass ingest.
+
+Multi-pass parser involves the following requirements:
+* The enveloping parser (e.g. the aggregation format such as syslog or
+ plain CSV) may contain metadata which should be ingested along with the data.
+* The enveloping sensor contains many different sensor types
+
+# High Level Solution
+
+
+
+At a high level, we continue to maintain the architectural invariant of
+a 1-1 relationship between logical sensors and storm topologies.
+Eventually this relationship may become more complex, but at the moment
+the approach is to construct a routing parser which will have two
+responsibilities:
+* Parse the envelope (e.g. syslog data) and extract any metadata fields
+ from the envelope to pass along
+* Route the unfolded data to the appropriate kafka topic associated with
+ the enveloped sensor data
+
+Because the data emitted from the routing parser is just like any data
+emitted from any other parser, in that it is a JSON blob like any
+data emitted from any parser, we will need to adjust the downstream
+parsers to extract the enveloped data from the JSON blob and treat it as
+the data to parse.
+
+# Example
+
+## Preliminaries
+
+We assume that the following environment variables are set:
+* `METRON_HOME` - the home directory for metron
+* `ZOOKEEPER` - The zookeeper quorum (comma separated with port specified: e.g. `node1:2181` for full-dev)
+* `BROKERLIST` - The Kafka broker list (comma separated with port specified: e.g. `node1:6667` for full-dev)
+* `ES_HOST` - The elasticsearch master (and port) e.g. `node1:9200` for full-dev.
+
+Before editing configurations, be sure to pull the configs from zookeeper locally via
+```
+$METRON_HOME/bin/zk_load_configs.sh --mode PULL -z $ZOOKEEPER -o $METRON_HOME/config/zookeeper/ -f
+```
+
+
+## The Scenario
+
+Consider the following situation, we have some logs from a Cisco PIX
+device that we would like to ingest. The format is syslog, but multiple
+scenarios exist in the same log file. Specificaly, let's consider the
+sample logs
+[here](http://www.monitorware.com/en/logsamples/cisco-pix-61(2).php).
+
+The log lines in general have the following components:
+* A timestamp
+* A message type tag
+* The message payload that is dependent upon the tag
+
+Let's consider two types of messages that we'd like to parse:
+* Tag `6-302*` which are connection creation and teardown messages e.g. `Built UDP connection for faddr 198.207.223.240/53337 gaddr 10.0.0.187/53 laddr 192.168.0.2/53`
+* Tag `5-304*` which are URL access events e.g. `192.168.0.2 Accessed URL 66.102.9.99:/`
+
+A couple things are apparent from this:
+* The formats we care about are easy to represent in grok, but are very
+ different and logically represent very different sensors.
+* The syslog loglines output by this device has many types of events that I do not care
+ about (yet).
+
+We will proceed to create 3 separate parsers:
+* A `pix_syslog_router` parser which will:
+ * Parse the timestamp field
+ * Parse the payload into a field called `data`
+ * Parse the tag into a field called `pix_type`
+ * Route the enveloped messages to the appropriate kafka topic
+ based on the tag
+* A `cisco-6-302` and `cisco-5-304` parser which will append to the existing fields from
+ the `pix_syslog_router` the sensor specific fields based on the tag type.
+
+## Cisco PIX Grok Patterns
+In order to assist in these parsers, we're going to accumulate some grok
+expressions which will help us deal with these various parsers.
+
+* Open a file `~/cisco_patterns` and place the following in there
+```
+CISCO_ACTION Built|Teardown|Deny|Denied|denied|requested|permitted|denied by ACL|discarded|est-allowed|Dropping|created|deleted
+CISCO_REASON Duplicate TCP SYN|Failed to locate egress interface|Invalid transport field|No matching connection|DNS Response|DNS Query|(?:%{WORD}\s*)*
+CISCO_DIRECTION Inbound|inbound|Outbound|outbound
+CISCOFW302020_302021 %{CISCO_ACTION:action}(?:%{CISCO_DIRECTION:direction})? %{WORD:protocol} connection %{GREEDYDATA:ignore} faddr %{IP:ip_dst_addr}/%{INT:icmp_seq_num}(?:\(%{DATA:fwuser}\))? gaddr %{IP:ip_src_xlated}/%{INT:icmp_code_xlated} laddr %{IP:ip_src_addr}/%{INT:icmp_code}( \(%{DATA:user}\))?
+ACCESSED %{URIHOST:ip_src_addr} Accessed URL %{IP:ip_dst_addr}:%{URIPATHPARAM:uri_path}
+CISCO_PIX %{GREEDYDATA:timestamp}: %PIX-%{NOTSPACE:pix_type}: %{GREEDYDATA:data}
+```
+* Place this pattern in HDFS at `/tmp/cisco_patterns` via `hadoop fs -put ~/cisco_patterns /tmp`
+ * NOTE: In production, we'd have more battle hardened patterns as well as place them in a more sensible location.
+
+## The `pix_syslog_router` Parser
+
+* Create the `pix_syslog_router` kafka topic via:
+```
+/usr/hdp/current/kafka-broker/bin/kafka-topics.sh --zookeeper $ZOOKEEPER --create --topic pix_syslog_router --partitions 1 --replication-factor 1
+```
+* Create the `pix_syslog_router` parser by opening `$METRON_HOME/config/zookeeper/parsers/pix_syslog_router.json` and placing the following:
+```
+{
+ "parserClassName" : "org.apache.metron.parsers.GrokParser"
+ ,"sensorTopic" : "pix_syslog_router"
+ , "parserConfig": {
+ "grokPath": "/tmp/cisco_patterns",
+ "batchSize" : 1,
+ "patternLabel": "CISCO_PIX",
+ "timestampField": "timestamp",
+ "timeFields" : [ "timestamp" ],
+ "dateFormat" : "MMM dd yyyy HH:mm:ss",
+ "kafka.topicField" : "logical_source_type"
+ }
+ ,"fieldTransformations" : [
+ {
+ "transformation" : "REGEX_SELECT"
+ ,"input" : "pix_type"
+ ,"output" : "logical_source_type"
+ ,"config" : {
+ "cisco-6-302" : "^6-302.*",
+ "cisco-5-304" : "^5-304.*"
+ }
+ }
+ ]
+}
+```
+A couple of things to note about this config:
+* In the `parserConfig` section, note that we are specifying `kafka.topicField` is `logical_source_field`. This specifies that the parser will send messages to the topic specified in the `logical_source_type` field. If the field does not exist, then the message is not sent.
+* The `REGEX_SELECT` field transformation sets the `logical_source_type` field based on the value in the `pix_type` field, which recall is our tag. This will enable us to route the broad category of cisco firewall messages along to the specific parser.
+
+
+## The `cisco-6-302` Parser
+
+* Create the `cisco-6-302` kafka topic via:
+```
+/usr/hdp/current/kafka-broker/bin/kafka-topics.sh --zookeeper $ZOOKEEPER --create --topic cisco-6-302 --partitions 1 --replication-factor 1
+```
+* Create the `cisco-6-302` parser by opening `$METRON_HOME/config/zookeeper/parsers/cisco-6-302.json` and placing the following:
+```
+{
+ "parserClassName" : "org.apache.metron.parsers.GrokParser"
+ ,"sensorTopic" : "cisco-6-302"
+ ,"rawMessageStrategy" : "ENVELOPE"
+ ,"rawMessageStrategyConfig" : {
+ "messageField" : "data",
+ "metadataPrefix" : ""
+ }
+ , "parserConfig": {
+ "grokPath": "/tmp/cisco_patterns",
+ "batchSize" : 1,
+ "patternLabel": "CISCOFW302020_302021"
+ }
+}
+```
+
+Note a couple of things:
+* We are specifying the `rawMessageStrategy` to be `ENVELOPE` to indicate that it is not a straight data feed, but rather it's enveloped in a JSON map (i.e. the output of the `pix_syslog_router)
+* Because this is enveloped, we must specify the field which contains the actual raw data by setting `messageField` in `rawMessageStrategyConfig`
+* You may be wondering why we specify `metadataPrefix` to be empty string. We want some of the fields in the enveloped message to be merged in without prefix. Most specifically, we want the `timestamp` field. By default, the prefix is `metron.metadata`.
+
+## The `cisco-5-304` Parser
+
+* Create the `cisco-5-304` kafka topic via:
+```
+/usr/hdp/current/kafka-broker/bin/kafka-topics.sh --zookeeper $ZOOKEEPER --create --topic cisco-5-304 --partitions 1 --replication-factor 1
+```
+* Create the `cisco-5-304` parser by opening `$METRON_HOME/config/zookeeper/parsers/cisco-5-304.json` and placing the following:
+```
+{
+ "parserClassName" : "org.apache.metron.parsers.GrokParser"
+ ,"sensorTopic" : "cisco-5-304"
+ ,"rawMessageStrategy" : "ENVELOPE"
+ ,"rawMessageStrategyConfig" : {
+ "messageField" : "data",
+ "metadataPrefix" : ""
+ }
+ , "parserConfig": {
+ "grokPath": "/tmp/cisco_patterns",
+ "batchSize" : 1,
+ "patternLabel": "ACCESSED"
+ }
+}
+```
+
+Mostly the same comments from the previous parser apply here; we are just using a different pattern label.
+
+# Start the Parsers
+Now we should start the parsers
+* Push the configs that we've created for the 3 parsers:
+```
+$METRON_HOME/bin/zk_load_configs.sh --mode PUSH -z $ZOOKEEPER -i $METRON_HOME/config/zookeeper/
+```
+* Start the `cisco-6-302` parser via
+```
+$METRON_HOME/bin/start_parser_topology.sh -k $BROKERLIST -z $ZOOKEEPER -s cisco-6-302
+```
+* Start the `cisco-5-304` parser via
+```
+$METRON_HOME/bin/start_parser_topology.sh -k $BROKERLIST -z $ZOOKEEPER -s cisco-5-304
+```
+* Start the `pix_syslog_router` parser via
+```
+$METRON_HOME/bin/start_parser_topology.sh -k $BROKERLIST -z $ZOOKEEPER -s pix_syslog_router
+```
+
+# Send Data
+* Create a file called `~/data.log` with the sample syslog loglines [here](http://www.monitorware.com/en/logsamples/cisco-pix-61(2).php).
+* Send the data in via kafka console producer
+```
+cat ~/data.log | /usr/hdp/current/kafka-broker/bin/kafka-console-producer.sh --broker-list $BROKERLIST --topic pix_syslog_router
+```
+
+You should see indices created for the `cisco-5-304` and `cisco-6-302` data with appropriate fields created for each type.
http://git-wip-us.apache.org/repos/asf/metron/blob/cbdaee17/use-cases/parser_chaining/message_routing_high_level.svg
----------------------------------------------------------------------
diff --git a/use-cases/parser_chaining/message_routing_high_level.svg b/use-cases/parser_chaining/message_routing_high_level.svg
new file mode 100644
index 0000000..1d80766
--- /dev/null
+++ b/use-cases/parser_chaining/message_routing_high_level.svg
@@ -0,0 +1,14 @@
+<!--
+ Licensed to the Apache Software
+ Foundation (ASF) under one or more contributor license agreements. See the
+ NOTICE file distributed with this work for additional information regarding
+ copyright ownership. The ASF licenses this file to You under the Apache License,
+ Version 2.0 (the "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software distributed
+ under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES
+ OR CONDITIONS OF ANY KIND, either express or implied. See the License for
+ the specific language governing permissions and limitations under the License.
+ -->
+<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN" "http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd">
+<svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" width="749px" height="474px" version="1.1" content="<mxfile userAgent="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.62 Safari/537.36" version="8.7.6" editor="www.draw.io" type="device"><diagram id="35f3a7a3-d1c2-8c18-e1fc-d490fc6a3b68" name="Page-1">7ZrNcpswEICfxsdmABmDj7GTtjNNp5n60PYogwBNZOQRcmz36bsY8Sfs2HUJdhNyyEgrWKTdj12tzABNF5tPAi+jr9wnbGAZ/maA7gaWNR4i+J8Ktplg6NqZIBTUz0RmKZjR30QJDSVdUZ8ktQsl50zSZV3o8TgmnqzJsBB8Xb8s4Kz+1CUOSUMw8zBrSn9QX0aZ1LVGpfwzoWGUP9kcjbOROfaeQsFXsXrewELB7i8bXuBcl1poEmGfrysidD9AU8G5zFqLzZSw1LS52bL7Ph4YLeYtSCxPuUFN+xmzlVr6d76SNA5B+IhFQoSap9zmtllHVJLZEntpfw3uH6BJJBcMeiY0cbLMPBLQDYGnTHbWSFu7YYbnhE0KK0054wKGYh6DukkiBX8iuRCMZ5Ox56NiJHcG2GkSUMa02wMeSwWTOUrnwmgYQ8cDa8BK1NMfeUIl5bWBZyIkBf8/aBfMuZR8UbngVqmUPF23sh6Mkc1BD5iFX+F1IXxBpNjCJfkNjkJBvSqmrfrrCnhKFFWYy
2VYoR4Wmkt3Q0N5/AAuRsP9t2EoSIglAfGMxAmY1zK+4OAJN0AAdpdp09syCh5OzXiEjXmGwsO8EBQgfFtJ0EKUvGfmRWbs48iMXgsZs4HMQTACRja3aTSGVZPYV807j+EkoV4djapnyqDZdA9YSGx/Qs/IO7/Szo2ddjdU/lT60nY5kk2R+I2or1kclsFXwtOykcQiJLISMJuOqVje3mP5XCYIw5I+12exzx3qCY+cwvzKWGHUHY8czaPZ7NVd1bCvK9IIQpamKFtyQ9GOjmLZpwFjNYCZQYagARBQCTF9sukkcKC637vMNaOeg6vlALkdguC8nwxy0WyBDFRz8nh8Y5+XLpBzTFOL+cLt8bgIHpbdGh97VLUISLNm7RaQtwzB0PwHCLScMn61PaXVLEL27SX6svW6dh16mOiwbM1jxHvIKsWJaaVszaqwS0WZ4fDAgcXfxhh7pBH0innGQicFmb5guUTBUhygd1CwWMM+2/yHyDQ2pUaH6cbukXkDyBRb2i6QedN178V2HlreGBpn1ze25RxT1eLm49JF7rugwTp7H6qx0FDUHgn5lCsk3MeCetGC7GbU55DryiGOpTFmdpdD0JX+OOu0VOVWK1pk7HdDN5FkZGp55fxzM8dwjqlqMZpc+hzkIAUFOx+MG8N0tLTjurngkQgKC07f8h1mp9NTOyNB104UQs6ZPJlHklyLNDWPSK4i3pgvb1Na5Cj/uuGKONK/J3PR2ZHJ1ZBsCyTolp84ZpeXn5Gi+z8=</diagram></mxfile>"><defs/><g transform="translate(0.5,0.5)"><rect x="128" y="101" width="80" height="80" rx="12" ry="12" fill="none" stroke="#5e9cd3" stroke-width="2" pointer-events="none"/><g transform="translate(12
9.5,188.5)"><switch><foreignObject style="overflow:visible;" pointer-events="all" width="75" height="36" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: inline-block; font-size: 16px; font-family: Helvetica; color: rgb(0, 0, 0); line-height: 1.2; vertical-align: top; width: 75px; white-space: normal; word-wrap: normal; text-align: center;"><div xmlns="http://www.w3.org/1999/xhtml" style="display:inline-block;text-align:inherit;text-decoration:inherit;">Routing Parser</div></div></foreignObject><text x="38" y="26" fill="#000000" text-anchor="middle" font-size="16px" font-family="Helvetica">Routing Parser</text></switch></g><path d="M 8 117 C 8 133 68 133 68 117" fill="none" stroke="#5e9cd3" stroke-width="2" stroke-miterlimit="10" pointer-events="none"/><path d="M 8 117 C 8 95.67 68 95.67 68 117 L 68 165 C 68 186.33 8 186.33 8 165 Z" fill="none" stroke="#5e9cd3" stroke-width="2" stroke-miterlimit="10" pointe
r-events="none"/><g transform="translate(-0.5,188.5)"><switch><foreignObject style="overflow:visible;" pointer-events="all" width="75" height="55" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: inline-block; font-size: 16px; font-family: Helvetica; color: rgb(0, 0, 0); line-height: 1.2; vertical-align: top; width: 75px; white-space: normal; word-wrap: normal; text-align: center;"><div xmlns="http://www.w3.org/1999/xhtml" style="display:inline-block;text-align:inherit;text-decoration:inherit;">Aggregate Sensor Kafka</div></div></foreignObject><text x="38" y="36" fill="#000000" text-anchor="middle" font-size="16px" font-family="Helvetica">Aggregate Sensor Kafka</text></switch></g><path d="M 68.5 146 L 68.5 136 L 108.5 136 L 108.5 125.5 L 127.5 141 L 108.5 156.5 L 108.5 146 Z" fill="#ffffff" stroke="#000000" stroke-linejoin="round" stroke-miterlimit="10" pointer-events="none"/><rect x="488" y="1" width="80"
height="80" rx="12" ry="12" fill="none" stroke="#5e9cd3" stroke-width="2" pointer-events="none"/><g transform="translate(489.5,88.5)"><switch><foreignObject style="overflow:visible;" pointer-events="all" width="75" height="55" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: inline-block; font-size: 16px; font-family: Helvetica; color: rgb(0, 0, 0); line-height: 1.2; vertical-align: top; width: 75px; white-space: normal; word-wrap: normal; text-align: center;"><div xmlns="http://www.w3.org/1999/xhtml" style="display:inline-block;text-align:inherit;text-decoration:inherit;">Specific Sensor Parser</div></div></foreignObject><text x="38" y="36" fill="#000000" text-anchor="middle" font-size="16px" font-family="Helvetica">Specific Sensor Parser</text></switch></g><rect x="488" y="331" width="80" height="80" rx="12" ry="12" fill="none" stroke="#5e9cd3" stroke-width="2" pointer-events="none"/><g transform="transla
te(489.5,418.5)"><switch><foreignObject style="overflow:visible;" pointer-events="all" width="75" height="55" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: inline-block; font-size: 16px; font-family: Helvetica; color: rgb(0, 0, 0); line-height: 1.2; vertical-align: top; width: 75px; white-space: normal; word-wrap: normal; text-align: center;"><div xmlns="http://www.w3.org/1999/xhtml" style="display:inline-block;text-align:inherit;text-decoration:inherit;">Specific Sensor Parser</div></div></foreignObject><text x="38" y="36" fill="#000000" text-anchor="middle" font-size="16px" font-family="Helvetica">Specific Sensor Parser</text></switch></g><path d="M 211.37 144.73 L 205.44 136.68 L 312.33 58.03 L 306.11 49.57 L 330.6 50.8 L 324.48 74.54 L 318.26 66.08 Z" fill="#ffffff" stroke="#000000" stroke-linejoin="round" stroke-miterlimit="10" pointer-events="none"/><path d="M 205.98 145.6 L 210.9 136.89 L 316.48 1
96.55 L 321.65 187.41 L 330.56 210.25 L 306.4 214.4 L 311.56 205.26 Z" fill="#ffffff" stroke="#000000" stroke-linejoin="round" stroke-miterlimit="10" pointer-events="none"/><path d="M 203.97 144 L 212.54 138.86 L 322.25 321.71 L 331.26 316.3 L 327.74 340.57 L 304.68 332.25 L 313.68 326.85 Z" fill="#ffffff" stroke="#000000" stroke-linejoin="round" stroke-miterlimit="10" pointer-events="none"/><path d="M 331 17 C 331 33 391 33 391 17" fill="none" stroke="#5e9cd3" stroke-width="2" stroke-miterlimit="10" pointer-events="none"/><path d="M 331 17 C 331 -4.33 391 -4.33 391 17 L 391 65 C 391 86.33 331 86.33 331 65 Z" fill="none" stroke="#5e9cd3" stroke-width="2" stroke-miterlimit="10" pointer-events="none"/><g transform="translate(331.5,88.5)"><switch><foreignObject style="overflow:visible;" pointer-events="all" width="57" height="55" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: inline-block; font-size: 16px; f
ont-family: Helvetica; color: rgb(0, 0, 0); line-height: 1.2; vertical-align: top; width: 57px; white-space: normal; word-wrap: normal; text-align: center;"><div xmlns="http://www.w3.org/1999/xhtml" style="display:inline-block;text-align:inherit;text-decoration:inherit;">Specific Sensor Kafka</div></div></foreignObject><text x="29" y="36" fill="#000000" text-anchor="middle" font-size="16px" font-family="Helvetica">[Not supported by viewer]</text></switch></g><path d="M 391.5 46 L 391.5 36 L 468.5 36 L 468.5 25.5 L 487.5 41 L 468.5 56.5 L 468.5 46 Z" fill="#ffffff" stroke="#000000" stroke-linejoin="round" stroke-miterlimit="10" pointer-events="none"/><rect x="488" y="171" width="80" height="80" rx="12" ry="12" fill="none" stroke="#5e9cd3" stroke-width="2" pointer-events="none"/><g transform="translate(489.5,258.5)"><switch><foreignObject style="overflow:visible;" pointer-events="all" width="75" height="55" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns
="http://www.w3.org/1999/xhtml" style="display: inline-block; font-size: 16px; font-family: Helvetica; color: rgb(0, 0, 0); line-height: 1.2; vertical-align: top; width: 75px; white-space: normal; word-wrap: normal; text-align: center;"><div xmlns="http://www.w3.org/1999/xhtml" style="display:inline-block;text-align:inherit;text-decoration:inherit;">Specific Sensor Parser</div></div></foreignObject><text x="38" y="36" fill="#000000" text-anchor="middle" font-size="16px" font-family="Helvetica">Specific Sensor Parser</text></switch></g><path d="M 331 167 C 331 183 391 183 391 167" fill="none" stroke="#5e9cd3" stroke-width="2" stroke-miterlimit="10" pointer-events="none"/><path d="M 331 167 C 331 145.67 391 145.67 391 167 L 391 215 C 391 236.33 331 236.33 331 215 Z" fill="none" stroke="#5e9cd3" stroke-width="2" stroke-miterlimit="10" pointer-events="none"/><g transform="translate(331.5,238.5)"><switch><foreignObject style="overflow:visible;" pointer-events="all" width="57" height="55"
requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: inline-block; font-size: 16px; font-family: Helvetica; color: rgb(0, 0, 0); line-height: 1.2; vertical-align: top; width: 57px; white-space: normal; word-wrap: normal; text-align: center;"><div xmlns="http://www.w3.org/1999/xhtml" style="display:inline-block;text-align:inherit;text-decoration:inherit;">Specific Sensor Kafka</div></div></foreignObject><text x="29" y="36" fill="#000000" text-anchor="middle" font-size="16px" font-family="Helvetica">[Not supported by viewer]</text></switch></g><path d="M 331 337 C 331 353 391 353 391 337" fill="none" stroke="#5e9cd3" stroke-width="2" stroke-miterlimit="10" pointer-events="none"/><path d="M 331 337 C 331 315.67 391 315.67 391 337 L 391 385 C 391 406.33 331 406.33 331 385 Z" fill="none" stroke="#5e9cd3" stroke-width="2" stroke-miterlimit="10" pointer-events="none"/><g transform="translate(331.5,408.5)"><switch><fo
reignObject style="overflow:visible;" pointer-events="all" width="57" height="55" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: inline-block; font-size: 16px; font-family: Helvetica; color: rgb(0, 0, 0); line-height: 1.2; vertical-align: top; width: 57px; white-space: normal; word-wrap: normal; text-align: center;"><div xmlns="http://www.w3.org/1999/xhtml" style="display:inline-block;text-align:inherit;text-decoration:inherit;">Specific Sensor Kafka</div></div></foreignObject><text x="29" y="36" fill="#000000" text-anchor="middle" font-size="16px" font-family="Helvetica">[Not supported by viewer]</text></switch></g><path d="M 388.5 365.5 L 388.5 355.5 L 465.5 355.5 L 465.5 345 L 484.5 360.5 L 465.5 376 L 465.5 365.5 Z" fill="#ffffff" stroke="#000000" stroke-linejoin="round" stroke-miterlimit="10" pointer-events="none"/><path d="M 388.5 206 L 388.5 196 L 465.5 196 L 465.5 185.5 L 484.5 201 L 465.5 216.5 L
465.5 206 Z" fill="#ffffff" stroke="#000000" stroke-linejoin="round" stroke-miterlimit="10" pointer-events="none"/><path d="M 678 177 C 678 193 738 193 738 177" fill="none" stroke="#5e9cd3" stroke-width="2" stroke-miterlimit="10" pointer-events="none"/><path d="M 678 177 C 678 155.67 738 155.67 738 177 L 738 225 C 738 246.33 678 246.33 678 225 Z" fill="none" stroke="#5e9cd3" stroke-width="2" stroke-miterlimit="10" pointer-events="none"/><g transform="translate(666.5,248.5)"><switch><foreignObject style="overflow:visible;" pointer-events="all" width="82" height="36" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: inline-block; font-size: 16px; font-family: Helvetica; color: rgb(0, 0, 0); line-height: 1.2; vertical-align: top; width: 82px; white-space: normal; word-wrap: normal; text-align: center;"><div xmlns="http://www.w3.org/1999/xhtml" style="display:inline-block;text-align:inherit;text-decoration:inhe
rit;">Enrichment Kafka</div></div></foreignObject><text x="41" y="26" fill="#000000" text-anchor="middle" font-size="16px" font-family="Helvetica">Enrichment Kafka</text></switch></g><path d="M 572.36 373.01 L 564.23 367.18 L 662.58 229.94 L 654.04 223.82 L 677.71 217.41 L 679.24 241.88 L 670.71 235.76 Z" fill="#ffffff" stroke="#000000" stroke-linejoin="round" stroke-miterlimit="10" pointer-events="none"/><path d="M 568.64 215.98 L 568.36 205.99 L 657.37 203.54 L 657.08 193.04 L 676.5 208.01 L 657.93 224.03 L 657.64 213.53 Z" fill="#ffffff" stroke="#000000" stroke-linejoin="round" stroke-miterlimit="10" pointer-events="none"/><path d="M 564.48 44.59 L 572.16 38.18 L 669.36 154.82 L 677.42 148.1 L 677.68 172.62 L 653.61 167.94 L 661.68 161.22 Z" fill="#ffffff" stroke="#000000" stroke-linejoin="round" stroke-miterlimit="10" pointer-events="none"/></g></svg>
http://git-wip-us.apache.org/repos/asf/metron/blob/cbdaee17/use-cases/parser_chaining/message_routing_high_level.xml
----------------------------------------------------------------------
diff --git a/use-cases/parser_chaining/message_routing_high_level.xml b/use-cases/parser_chaining/message_routing_high_level.xml
new file mode 100644
index 0000000..127fb3c
--- /dev/null
+++ b/use-cases/parser_chaining/message_routing_high_level.xml
@@ -0,0 +1,14 @@
+<!--
+ Licensed to the Apache Software
+ Foundation (ASF) under one or more contributor license agreements. See the
+ NOTICE file distributed with this work for additional information regarding
+ copyright ownership. The ASF licenses this file to You under the Apache License,
+ Version 2.0 (the "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software distributed
+ under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES
+ OR CONDITIONS OF ANY KIND, either express or implied. See the License for
+ the specific language governing permissions and limitations under the License.
+ -->
+<!-- This is a draw.io diagram. You can load it from http://www.draw.io -->
+<mxfile userAgent="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.62 Safari/537.36" version="8.7.6" editor="www.draw.io" type="device"><diagram id="35f3a7a3-d1c2-8c18-e1fc-d490fc6a3b68" name="Page-1">7ZrNctowEICfhmMztoQxHANJ25mm00w5tD0KW9iaCIuRRYA+fddY/pMhUOoYmjiHjLSy19Lu512tcA9PFptPkizDr8KnvIcsf9PDdz2ERn0M/xPBNhU4aJQKAsn8VGQXgin7TbXQ0tIV82lcuVAJwRVbVoWeiCLqqYqMSCnW1cvmglefuiQBrQmmHuF16Q/mqzCVDtGgkH+mLAizJ9sDvb4Z8Z4CKVaRfl4P4fnuLx1ekEyXXmgcEl+sSyJ838MTKYRKW4vNhPLEtJnZ0vs+HhjN5y1ppE65QU/7mfCVXvp3sVIsCkD4SGRMpZ6n2ma2WYdM0emSeEl/De7v4XGoFhx6NjRJvEw9MmcbCk8Z76yRtHbDnMwoH+dWmgguJAxFIgJ141hJ8UQzIRjPoSPPx/lI5gyw03jOODdun4tIaZjsQTIXzoIIOh5YA1ain/4oYqaYqAw8U6kY+P/BuGAmlBKL0gW3WqUSybq19WCMbg56wM79Cq8LFQuq5BYuyW5wNQr6VbEd3V+XwNOisMRcJiMa9SDXXLgbGtrjB3Cxau6/DQJJA6IoiKc0isG8yPpC5k+kBgKwu0ya3pYz8HBixiNszFIUHma5IAfh20qBFqrlHTMvMuMcR2bwWsjYNWQOgjHndHObRGNYNY183bzzOIlj5lXRKHumCJp194CF5PYn9Kys8yvp3DhJd8PUT60vaRcj6RSpX4v6hsVhGWIlPSMbKSIDqkoBs+6Y
kuWdPZbPZJJyothzdRb73KGf8CgYzK+IFVbV8dg1PJrOXt9VDvumIoMgjAxF6ZJrinZ05Ms+DRhUA2YKGYLNgYBSiOmSTSuBA1f93mauGXQcXC0HeNgiCO77ySAXzRbYwhUnj0awkrPSBXaPaWowXww7PC6CB3Ia42OPqgYBqdes7QLyliHo2/8AgZFTRq+2p0T1ImTfXqIrW69r12GGiRbL1ixGvIeskp+YlsrWtAq7VJTp9w8cWPxtjHEGBkGvmGcQPinIdAXLJQqW/AC9hYIF9bts8x8iU9uUWi2mG6dD5g0gk29p20DmTde9F9t5GHmjb51d3zjIPaaqwc3HpYvcd0EDOnsfarBQU9QcCdmUSyTcR5J54YLuZtTlkOvKIS4yGLPbyyH4Sn+cdRuqcssVLbb2u6GdSDKwjbxy/rmZa7nHVDUYTS59DnKQgpydD9aNZbtG2hkOM8EjlQwWnLzlO8xOp6dyRoKvnSiM3TN5so8kuQZpqh+RXEW8sV/epjTIUfZ1wxVxZH5PNsRnR6ahgWRTIEG3+MQxvbz4jBTf/wE=</diagram></mxfile>