You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@metron.apache.org by ma...@apache.org on 2017/09/09 04:54:51 UTC
svn commit: r21542 [16/23] - in /dev/metron/0.4.1-RC4: ./ site-book/
site-book/css/ site-book/images/ site-book/images/logos/
site-book/images/profiles/ site-book/img/ site-book/js/
site-book/metron-analytics/ site-book/metron-analytics/metron-maas-ser...
Added: dev/metron/0.4.1-RC4/site-book/metron-platform/Performance-tuning-guide.html
==============================================================================
--- dev/metron/0.4.1-RC4/site-book/metron-platform/Performance-tuning-guide.html (added)
+++ dev/metron/0.4.1-RC4/site-book/metron-platform/Performance-tuning-guide.html Sat Sep 9 04:54:51 2017
@@ -0,0 +1,677 @@
+<!DOCTYPE html>
+<!--
+ | Generated by Apache Maven Doxia at 2017-09-08
+ | Rendered using Apache Maven Fluido Skin 1.3.0
+-->
+<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
+ <head>
+ <meta charset="UTF-8" />
+ <meta name="viewport" content="width=device-width, initial-scale=1.0" />
+ <meta name="Date-Revision-yyyymmdd" content="20170908" />
+ <meta http-equiv="Content-Language" content="en" />
+ <title>Metron – Metron Performance Tuning Guide</title>
+ <link rel="stylesheet" href="../css/apache-maven-fluido-1.3.0.min.css" />
+ <link rel="stylesheet" href="../css/site.css" />
+ <link rel="stylesheet" href="../css/print.css" media="print" />
+
+
+ <script type="text/javascript" src="../js/apache-maven-fluido-1.3.0.min.js"></script>
+
+
+
+<script type="text/javascript">$( document ).ready( function() { $( '.carousel' ).carousel( { interval: 3500 } ) } );</script>
+
+ </head>
+ <body class="topBarDisabled">
+
+
+
+
+ <div class="container-fluid">
+ <div id="banner">
+ <div class="pull-left">
+ <a href="http://metron.apache.org/" id="bannerLeft">
+ <img src="../images/metron-logo.png" alt="Apache Metron" width="148px" height="48px"/>
+ </a>
+ </div>
+ <div class="pull-right"> </div>
+ <div class="clear"><hr/></div>
+ </div>
+
+ <div id="breadcrumbs">
+ <ul class="breadcrumb">
+
+
+ <li class="">
+ <a href="http://www.apache.org" class="externalLink" title="Apache">
+ Apache</a>
+ </li>
+ <li class="divider ">/</li>
+ <li class="">
+ <a href="http://metron.apache.org/" class="externalLink" title="Metron">
+ Metron</a>
+ </li>
+ <li class="divider ">/</li>
+ <li class="">
+ <a href="../index.html" title="Documentation">
+ Documentation</a>
+ </li>
+ <li class="divider ">/</li>
+ <li class="">Metron Performance Tuning Guide</li>
+
+
+
+ <li id="publishDate" class="pull-right">Last Published: 2017-09-08</li> <li class="divider pull-right">|</li>
+ <li id="projectVersion" class="pull-right">Version: 0.4.1</li>
+
+ </ul>
+ </div>
+
+
+ <div class="row-fluid">
+ <div id="leftColumn" class="span3">
+ <div class="well sidebar-nav">
+
+
+ <ul class="nav nav-list">
+ <li class="nav-header">User Documentation</li>
+
+ <li>
+
+ <a href="../index.html" title="Metron">
+ <i class="icon-chevron-down"></i>
+ Metron</a>
+ <ul class="nav nav-list">
+
+ <li>
+
+ <a href="../Upgrading.html" title="Upgrading">
+ <i class="none"></i>
+ Upgrading</a>
+ </li>
+
+ <li>
+
+ <a href="../metron-analytics/index.html" title="Analytics">
+ <i class="icon-chevron-right"></i>
+ Analytics</a>
+ </li>
+
+ <li>
+
+ <a href="../metron-contrib/metron-docker/index.html" title="Docker">
+ <i class="none"></i>
+ Docker</a>
+ </li>
+
+ <li>
+
+ <a href="../metron-deployment/index.html" title="Deployment">
+ <i class="icon-chevron-right"></i>
+ Deployment</a>
+ </li>
+
+ <li>
+
+ <a href="../metron-interface/metron-alerts/index.html" title="Alerts">
+ <i class="none"></i>
+ Alerts</a>
+ </li>
+
+ <li>
+
+ <a href="../metron-interface/metron-config/index.html" title="Config">
+ <i class="none"></i>
+ Config</a>
+ </li>
+
+ <li>
+
+ <a href="../metron-interface/metron-rest/index.html" title="Rest">
+ <i class="none"></i>
+ Rest</a>
+ </li>
+
+ <li>
+
+ <a href="../metron-platform/index.html" title="Platform">
+ <i class="icon-chevron-down"></i>
+ Platform</a>
+ <ul class="nav nav-list">
+
+ <li class="active">
+
+ <a href="#"><i class="none"></i>Performance-tuning-guide</a>
+ </li>
+
+ <li>
+
+ <a href="../metron-platform/metron-api/index.html" title="Api">
+ <i class="none"></i>
+ Api</a>
+ </li>
+
+ <li>
+
+ <a href="../metron-platform/metron-common/index.html" title="Common">
+ <i class="none"></i>
+ Common</a>
+ </li>
+
+ <li>
+
+ <a href="../metron-platform/metron-data-management/index.html" title="Data-management">
+ <i class="none"></i>
+ Data-management</a>
+ </li>
+
+ <li>
+
+ <a href="../metron-platform/metron-enrichment/index.html" title="Enrichment">
+ <i class="none"></i>
+ Enrichment</a>
+ </li>
+
+ <li>
+
+ <a href="../metron-platform/metron-indexing/index.html" title="Indexing">
+ <i class="none"></i>
+ Indexing</a>
+ </li>
+
+ <li>
+
+ <a href="../metron-platform/metron-management/index.html" title="Management">
+ <i class="none"></i>
+ Management</a>
+ </li>
+
+ <li>
+
+ <a href="../metron-platform/metron-parsers/index.html" title="Parsers">
+ <i class="icon-chevron-right"></i>
+ Parsers</a>
+ </li>
+
+ <li>
+
+ <a href="../metron-platform/metron-pcap-backend/index.html" title="Pcap-backend">
+ <i class="none"></i>
+ Pcap-backend</a>
+ </li>
+
+ <li>
+
+ <a href="../metron-platform/metron-writer/index.html" title="Writer">
+ <i class="none"></i>
+ Writer</a>
+ </li>
+ </ul>
+ </li>
+
+ <li>
+
+ <a href="../metron-sensors/index.html" title="Sensors">
+ <i class="icon-chevron-right"></i>
+ Sensors</a>
+ </li>
+
+ <li>
+
+ <a href="../metron-stellar/stellar-common/index.html" title="Stellar-common">
+ <i class="icon-chevron-right"></i>
+ Stellar-common</a>
+ </li>
+
+ <li>
+
+ <a href="../use-cases/index.html" title="Use-cases">
+ <i class="icon-chevron-right"></i>
+ Use-cases</a>
+ </li>
+ </ul>
+ </li>
+ </ul>
+
+
+
+ <hr class="divider" />
+
+ <div id="poweredBy">
+ <div class="clear"></div>
+ <div class="clear"></div>
+ <div class="clear"></div>
+ <a href="http://maven.apache.org/" title="Built by Maven" class="poweredBy">
+ <img class="builtBy" alt="Built by Maven" src="../images/logos/maven-feather.png" />
+ </a>
+ </div>
+ </div>
+ </div>
+
+
+ <div id="bodyColumn" class="span9" >
+
+ <h1>Metron Performance Tuning Guide</h1>
+<p><a name="Metron_Performance_Tuning_Guide"></a></p>
+<div class="section">
+<h2><a name="Overview"></a>Overview</h2>
+<p>This document provides guidance from our experiences tuning the Apache Metron Storm topologies for maximum performance. You’ll find suggestions for optimum configurations under a 1 gbps load along with some guidance around the tooling we used to monitor and assess our throughput.</p>
+<p>In the simplest terms, Metron is a streaming architecture created on top of Kafka and three main types of Storm topologies: parsers, enrichment, and indexing. Each parser has it’s own topology and there is also a highly performant, specialized spout-only topology for streaming PCAP data to HDFS. We found that the architecture can be tuned almost exclusively through using a few primary Storm and Kafka parameters along with a few Metron-specific options. You can think of the data flow as being similar to water flowing through a pipe, and the majority of these options assist in tweaking the various pipe widths in the system.</p></div>
+<div class="section">
+<h2><a name="General_Tuning_Suggestions"></a>General Tuning Suggestions</h2>
+<p>Note that there is currently no method for specifying the number of tasks from the number of executors in Flux topologies (enrichment, indexing). By default, the number of tasks will equal the number of executors. Logically, setting the number of tasks equal to the number of executors is sensible. Storm enforces num executors <= num tasks. The reason you might set the number of tasks higher than the number of executors is for future performance tuning and rebalancing without the need to bring down your topologies. The number of tasks is fixed at topology startup time whereas the number of executors can be increased up to a maximum value equal to the number of tasks.</p>
+<p>When configuring Storm Kafka spouts, we found that the default values for poll.timeout.ms, offset.commit.period.ms, and max.uncommitted.offsets worked well in nearly all cases. As a general rule, it was optimal to set spout parallelism equal to the number of partitions used in your Kafka topic. Any greater parallelism will leave you with idle consumers since Kafka limits the max number of consumers to the number of partitions. This is important because Kafka has certain ordering guarantees for message delivery per partition that would not be possible if more than one consumer in a given consumer group were able to read from that partition.</p></div>
+<div class="section">
+<h2><a name="Component_Tuning_Levers"></a>Component Tuning Levers</h2>
+
+<ul>
+
+<li>Kafka
+
+<ul>
+
+<li>Number partitions</li>
+ </ul></li>
+
+<li>Storm
+
+<ul>
+
+<li>Kafka spout
+
+<ul>
+
+<li>Polling frequency</li>
+
+<li>Polling timeouts</li>
+
+<li>Offset commit period</li>
+
+<li>Max uncommitted offsets</li>
+ </ul></li>
+
+<li>Number workers (OS processes)</li>
+
+<li>Number executors (threads in a process)</li>
+
+<li>Number ackers</li>
+
+<li>Max spout pending</li>
+
+<li>Spout and bolt parallelism</li>
+ </ul></li>
+
+<li>HDFS
+
+<ul>
+
+<li>Replication factor</li>
+ </ul></li>
+</ul>
+<div class="section">
+<h3><a name="Kafka_Tuning"></a>Kafka Tuning</h3>
+<p>The main lever you’re going to work with when tuning Kafka throughput will be the number of partitions. A handy method for deciding how many partitions to use is to first calculate the throughput for a single producer (p) and a single consumer (c), and then use that with the desired throughput (t) to roughly estimate the number of partitions to use. You would want at least max(t/p, t/c) partitions to attain the desired throughput. See <a class="externalLink" href="https://www.confluent.io/blog/how-to-choose-the-number-of-topicspartitions-in-a-kafka-cluster/">https://www.confluent.io/blog/how-to-choose-the-number-of-topicspartitions-in-a-kafka-cluster/</a> for more details.</p></div>
+<div class="section">
+<h3><a name="Storm_Tuning"></a>Storm Tuning</h3>
+<p>There are quite a few options you will be confronted with when tuning your Storm topologies and this is largely trial and error. As a general rule of thumb, we recommend starting with the defaults and smaller numbers in terms of parallelism while iteratively working up until the desired performance is achieved. You will find the offset lag tool indispensable while verifying your settings.</p>
+<p>We won’t go into a full discussion about Storm’s architecture - see references section for more info - but there are some general rules of thumb that should be followed. It’s first important to understand the ways you can impact parallelism in a Storm topology.</p>
+
+<ul>
+
+<li>num tasks</li>
+
+<li>num executors (parallelism hint)</li>
+
+<li>num workers</li>
+</ul>
+<p>Tasks are instances of a given spout or bolt, executors are threads in a process, and workers are jvm processes. You’ll want the number of tasks as a multiple of the number of executors, the number of executors as multiple of the number of workers, and the number of workers as a multiple of the number of machines. The main reason for this approach is that it will give a uniform distribution of work to each machine and jvm process. More often than not, your number of tasks will be equal to the number of executors, which is the default in Storm. Flux does not actually provide a way to independently set number of tasks, so for enrichments and indexing which use Flux, num tasks will always equal num executors.</p>
+<p>You can change the number of workers via the property <tt>topology.workers</tt></p>
+<p><b>Other Storm Settings</b></p>
+
+<div class="source">
+<div class="source">
+<pre>topology.max.spout.pending
+</pre></div></div>
+<p>This is the maximum number of tuples that can be in flight (ie, not yet acked) at any given time within your topology. You set this as a form of backpressure to ensure you don’t flood your topology.</p>
+
+<div class="source">
+<div class="source">
+<pre>topology.ackers.executors
+</pre></div></div>
+<p>This specifies how many threads should be dedicated to tuple acking. We found that setting this equal to the number of partitions in your inbound Kafka topic worked well.</p>
+<p><b>spout-config.json</b></p>
+
+<div class="source">
+<div class="source">
+<pre>{
+ ...
+ "spout.pollTimeoutMs" : 200,
+ "spout.maxUncommittedOffsets" : 10000000,
+ "spout.offsetCommitPeriodMs" : 30000
+}
+</pre></div></div>
+<p>These are the spout recommended defaults from Storm and are currently the defaults provided in the Kafka spout itself. In fact, if you find the recommended defaults work fine for you, then you can omit these settings altogether.</p></div></div>
+<div class="section">
+<h2><a name="Use_Case_Specific_Tuning_Suggestions"></a>Use Case Specific Tuning Suggestions</h2>
+<p>The below discussion outlines a specific tuning exercise we went through for driving 1 Gbps of traffic through a Metron cluster running with 4 Kafka brokers and 4 Storm Supervisors.</p>
+<p>General machine specs</p>
+
+<ul>
+
+<li>10 Gb network cards</li>
+
+<li>256 GB memory</li>
+
+<li>12 disks</li>
+
+<li>32 cores</li>
+</ul>
+<div class="section">
+<h3><a name="Performance_Monitoring_Tools"></a>Performance Monitoring Tools</h3>
+<p>Before we get to tuning our cluster, it helps to describe what we might actually want to monitor as well as any potential pain points. Prior to switching over to the new Storm Kafka client, which leverages the new Kafka consumer API under the hood, offsets were stored in Zookeeper. While the broker hosts are still stored in Zookeeper, this is no longer true for the offsets which are now stored in Kafka itself. This is a configurable option, and you may switch back to Zookeeper if you choose, but Metron is currently using the new defaults. With this in mind, there are some useful tools that come with Storm and Kafka that we can use to monitor our topologies.</p>
+<div class="section">
+<h4><a name="Tooling"></a>Tooling</h4>
+<p>Kafka</p>
+
+<ul>
+
+<li>consumer group offset lag viewer</li>
+
+<li>There is a GUI tool to make creating, modifying, and generally managing your Kafka topics a bit easier - see <a class="externalLink" href="https://github.com/yahoo/kafka-manager">https://github.com/yahoo/kafka-manager</a></li>
+
+<li>console consumer - useful for quickly verifying topic contents</li>
+</ul>
+<p>Storm</p>
+
+<ul>
+
+<li>Storm UI - <a class="externalLink" href="http://www.malinga.me/reading-and-understanding-the-storm-ui-storm-ui-explained/">http://www.malinga.me/reading-and-understanding-the-storm-ui-storm-ui-explained/</a></li>
+</ul></div>
+<div class="section">
+<h4><a name="Example_-_Viewing_Kafka_Offset_Lags"></a>Example - Viewing Kafka Offset Lags</h4>
+<p>First we need to setup some environment variables</p>
+
+<div class="source">
+<div class="source">
+<pre>export BROKERLIST=<your broker comma-delimated list of host:ports>
+export ZOOKEEPER=<your zookeeper comma-delimated list of host:ports>
+export KAFKA_HOME=<kafka home dir>
+export METRON_HOME=<your metron home>
+export HDP_HOME=<your HDP home>
+</pre></div></div>
+<p>If you have Kerberos enabled, setup the security protocol</p>
+
+<div class="source">
+<div class="source">
+<pre>$ cat /tmp/consumergroup.config
+security.protocol=SASL_PLAINTEXT
+</pre></div></div>
+<p>Now run the following command for a running topology’s consumer group. In this example we are using enrichments.</p>
+
+<div class="source">
+<div class="source">
+<pre>${KAFKA_HOME}/bin/kafka-consumer-groups.sh \
+ --command-config=/tmp/consumergroup.config \
+ --describe \
+ --group enrichments \
+ --bootstrap-server $BROKERLIST \
+ --new-consumer
+</pre></div></div>
+<p>This will return a table with the following output depicting offsets for all partitions and consumers associated with the specified consumer group:</p>
+
+<div class="source">
+<div class="source">
+<pre>GROUP TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG OWNER
+enrichments enrichments 9 29746066 29746067 1 consumer-2_/xxx.xxx.xxx.xxx
+enrichments enrichments 3 29754325 29754326 1 consumer-1_/xxx.xxx.xxx.xxx
+enrichments enrichments 43 29754331 29754332 1 consumer-6_/xxx.xxx.xxx.xxx
+...
+</pre></div></div>
+<p><i>Note</i>: You won’t see any output until a topology is actually running because the consumer groups only exist while consumers in the spouts are up and running.</p>
+<p>The primary column we’re concerned with paying attention to is the LAG column, which is the current delta calculation between the current and end offset for the partition. This tells us how close we are to keeping up with incoming data. And, as we found through multiple trials, whether there are any problems with specific consumers getting stuck.</p>
+<p>Taking this one step further, it’s probably more useful if we can watch the offsets and lags change over time. In order to do this we’ll add a “watch” command and set the refresh rate to 10 seconds.</p>
+
+<div class="source">
+<div class="source">
+<pre>watch -n 10 -d ${KAFKA_HOME}/bin/kafka-consumer-groups.sh \
+ --command-config=/tmp/consumergroup.config \
+ --describe \
+ --group enrichments \
+ --bootstrap-server $BROKERLIST \
+ --new-consumer
+</pre></div></div>
+<p>Every 10 seconds the command will re-run and the screen will be refreshed with new information. The most useful bit is that the watch command will highlight the differences from the current output and the last output screens.</p></div></div>
+<div class="section">
+<h3><a name="Parser_Tuning"></a>Parser Tuning</h3>
+<p>We’ll be using the bro sensor in this example. Note that the parsers and PCAP use a builder utility, as opposed to enrichments and indexing, which use Flux.</p>
+<p>We started with a single partition for the inbound Kafka topics and eventually worked our way up to 48. And We’re using the following pending value, as shown below. The default is ‘null’ which would result in no limit.</p>
+<p><b>storm-bro.config</b></p>
+
+<div class="source">
+<div class="source">
+<pre>{
+ ...
+ "topology.max.spout.pending" : 2000
+ ...
+}
+</pre></div></div>
+<p>And the following default spout settings. Again, this can be ommitted entirely since we are using the defaults.</p>
+<p><b>spout-bro.config</b></p>
+
+<div class="source">
+<div class="source">
+<pre>{
+ ...
+ "spout.pollTimeoutMs" : 200,
+ "spout.maxUncommittedOffsets" : 10000000,
+ "spout.offsetCommitPeriodMs" : 30000
+}
+</pre></div></div>
+<p>And we ran our bro parser topology with the following options. We did not need to fully match the number of Kafka partitions with our parallelism in this case, though you could certainly do so if necessary. Notice that we only needed 1 worker.</p>
+
+<div class="source">
+<div class="source">
+<pre>/usr/metron/0.4.0/bin/start_parser_topology.sh -k $BROKERLIST -z $ZOOKEEPER -s bro -ksp SASL_PLAINTEXT
+ -ot enrichments
+ -e ~metron/.storm/storm-bro.config \
+ -esc ~/.storm/spout-bro.config \
+ -sp 24 \
+ -snt 24 \
+ -nw 1 \
+ -pnt 24 \
+ -pp 24 \
+</pre></div></div>
+<p>From the usage docs, here are the options we’ve used. The full reference can be found here - <a class="externalLink" href="https://github.com/apache/metron/blob/master/metron-platform/metron-parsers/README.md">https://github.com/apache/metron/blob/master/metron-platform/metron-parsers/README.md</a></p>
+
+<div class="source">
+<div class="source">
+<pre>-e,--extra_topology_options <JSON_FILE> Extra options in the form
+ of a JSON file with a map
+ for content.
+-esc,--extra_kafka_spout_config <JSON_FILE> Extra spout config options
+ in the form of a JSON file
+ with a map for content.
+ Possible keys are:
+ retryDelayMaxMs,retryDelay
+ Multiplier,retryInitialDel
+ ayMs,stateUpdateIntervalMs
+ ,bufferSizeBytes,fetchMaxW
+ ait,fetchSizeBytes,maxOffs
+ etBehind,metricsTimeBucket
+ SizeInSecs,socketTimeoutMs
+-sp,--spout_p <SPOUT_PARALLELISM_HINT> Spout Parallelism Hint
+-snt,--spout_num_tasks <NUM_TASKS> Spout Num Tasks
+-nw,--num_workers <NUM_WORKERS> Number of Workers
+-pnt,--parser_num_tasks <NUM_TASKS> Parser Num Tasks
+-pp,--parser_p <PARALLELISM_HINT> Parser Parallelism Hint
+</pre></div></div></div>
+<div class="section">
+<h3><a name="Enrichment_Tuning"></a>Enrichment Tuning</h3>
+<p>We landed on the same number of partitions for enrichemnt and indexing as we did for bro - 48.</p>
+<p>For configuring Storm, there is a flux file and properties file that we modified. Here are the settings we changed for bro in Flux. Note that the main Metron-specific option we’ve changed to accomodate the desired rate of data throughput is max cache size in the join bolts. More information on Flux can be found here - <a class="externalLink" href="http://storm.apache.org/releases/1.0.1/flux.html">http://storm.apache.org/releases/1.0.1/flux.html</a></p>
+<p><b>General storm settings</b></p>
+
+<div class="source">
+<div class="source">
+<pre>topology.workers: 8
+topology.acker.executors: 48
+topology.max.spout.pending: 2000
+</pre></div></div>
+<p><b>Spout and Bolt Settings</b></p>
+
+<div class="source">
+<div class="source">
+<pre>kafkaSpout
+ parallelism=48
+ session.timeout.ms=29999
+ enable.auto.commit=false
+ setPollTimeoutMs=200
+ setMaxUncommittedOffsets=10000000
+ setOffsetCommitPeriodMs=30000
+enrichmentSplitBolt
+ parallelism=4
+enrichmentJoinBolt
+ parallelism=8
+ withMaxCacheSize=200000
+ withMaxTimeRetain=10
+threatIntelSplitBolt
+ parallelism=4
+threatIntelJoinBolt
+ parallelism=4
+ withMaxCacheSize=200000
+ withMaxTimeRetain=10
+outputBolt
+ parallelism=48
+</pre></div></div></div>
+<div class="section">
+<h3><a name="Indexing_HDFS_Tuning"></a>Indexing (HDFS) Tuning</h3>
+<p>There are 48 partitions set for the indexing partition, per the enrichment exercise above.</p>
+<p>These are the batch size settings for the bro index</p>
+
+<div class="source">
+<div class="source">
+<pre>cat ${METRON_HOME}/config/zookeeper/indexing/bro.json
+{
+ "hdfs" : {
+ "index": "bro",
+ "batchSize": 50,
+ "enabled" : true
+ }...
+}
+</pre></div></div>
+<p>And here are the settings we used for the indexing topology</p>
+<p><b>General storm settings</b></p>
+
+<div class="source">
+<div class="source">
+<pre>topology.workers: 4
+topology.acker.executors: 24
+topology.max.spout.pending: 2000
+</pre></div></div>
+<p><b>Spout and Bolt Settings</b></p>
+
+<div class="source">
+<div class="source">
+<pre>hdfsSyncPolicy
+ org.apache.storm.hdfs.bolt.sync.CountSyncPolicy
+ constructor arg=100000
+hdfsRotationPolicy
+ bolt.hdfs.rotation.policy.units=DAYS
+ bolt.hdfs.rotation.policy.count=1
+kafkaSpout
+ parallelism: 24
+ session.timeout.ms=29999
+ enable.auto.commit=false
+ setPollTimeoutMs=200
+ setMaxUncommittedOffsets=10000000
+ setOffsetCommitPeriodMs=30000
+hdfsIndexingBolt
+ parallelism: 24
+</pre></div></div></div>
+<div class="section">
+<h3><a name="PCAP_Tuning"></a>PCAP Tuning</h3>
+<p>PCAP is a specialized topology that is a Spout-only topology. Both Kafka topic consumption and HDFS writing is done within a spout to avoid the additional network hop required if using an additional bolt.</p>
+<p><b>General Storm topology properties</b></p>
+
+<div class="source">
+<div class="source">
+<pre>topology.workers=16
+topology.ackers.executors: 0
+</pre></div></div>
+<p><b>Spout and Bolt properties</b></p>
+
+<div class="source">
+<div class="source">
+<pre>kafkaSpout
+ parallelism: 128
+ poll.timeout.ms=100
+ offset.commit.period.ms=30000
+ session.timeout.ms=39000
+ max.uncommitted.offsets=200000000
+ max.poll.interval.ms=10
+ max.poll.records=200000
+ receive.buffer.bytes=431072
+ max.partition.fetch.bytes=10000000
+ enable.auto.commit=false
+ setMaxUncommittedOffsets=20000000
+ setOffsetCommitPeriodMs=30000
+
+writerConfig
+ withNumPackets=1265625
+ withMaxTimeMS=0
+ withReplicationFactor=1
+ withSyncEvery=80000
+ withHDFSConfig
+ io.file.buffer.size=1000000
+ dfs.blocksize=1073741824
+</pre></div></div></div></div>
+<div class="section">
+<h2><a name="Issues"></a>Issues</h2>
+<p><b>Error</b></p>
+
+<div class="source">
+<div class="source">
+<pre>org.apache.kafka.clients.consumer.CommitFailedException: Commit cannot be completed since the group has already rebalanced and assigned
+the partitions to another member. This means that the time between subsequent calls to poll() was longer than the configured session.timeout.ms,
+which typically implies that the poll loop is spending too much time message processing. You can address this either by increasing the
+session timeout or by reducing the maximum size of batches returned in poll() with max.poll.records
+</pre></div></div>
+<p><b>Suggestions</b></p>
+<p>This implies that the spout hasn’t been given enough time between polls before committing the offsets. In other words, the amount of time taken to process the messages is greater than the timeout window. In order to fix this, you can improve message throughput by modifying the options outlined above, increasing the poll timeout, or both.</p></div>
+<div class="section">
+<h2><a name="Reference"></a>Reference</h2>
+
+<ul>
+
+<li><a class="externalLink" href="http://storm.apache.org/releases/1.0.1/flux.html">http://storm.apache.org/releases/1.0.1/flux.html</a></li>
+
+<li><a class="externalLink" href="https://stackoverflow.com/questions/17257448/what-is-the-task-in-storm-parallelism">https://stackoverflow.com/questions/17257448/what-is-the-task-in-storm-parallelism</a></li>
+
+<li><a class="externalLink" href="http://storm.apache.org/releases/current/Understanding-the-parallelism-of-a-Storm-topology.html">http://storm.apache.org/releases/current/Understanding-the-parallelism-of-a-Storm-topology.html</a></li>
+
+<li><a class="externalLink" href="http://www.malinga.me/reading-and-understanding-the-storm-ui-storm-ui-explained/">http://www.malinga.me/reading-and-understanding-the-storm-ui-storm-ui-explained/</a></li>
+
+<li><a class="externalLink" href="https://www.confluent.io/blog/how-to-choose-the-number-of-topicspartitions-in-a-kafka-cluster/">https://www.confluent.io/blog/how-to-choose-the-number-of-topicspartitions-in-a-kafka-cluster/</a></li>
+
+<li><a class="externalLink" href="https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.1/bk_storm-component-guide/content/storm-kafkaspout-perf.html">https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.1/bk_storm-component-guide/content/storm-kafkaspout-perf.html</a></li>
+</ul></div>
+ </div>
+ </div>
+ </div>
+
+ <hr/>
+
+ <footer>
+ <div class="container-fluid">
+ <div class="row span12">Copyright © 2017
+ <a href="https://www.apache.org">The Apache Software Foundation</a>.
+ All Rights Reserved.
+
+ </div>
+
+
+
+ </div>
+ </footer>
+ </body>
+</html>
Added: dev/metron/0.4.1-RC4/site-book/metron-platform/index.html
==============================================================================
--- dev/metron/0.4.1-RC4/site-book/metron-platform/index.html (added)
+++ dev/metron/0.4.1-RC4/site-book/metron-platform/index.html Sat Sep 9 04:54:51 2017
@@ -0,0 +1,296 @@
+<!DOCTYPE html>
+<!--
+ | Generated by Apache Maven Doxia at 2017-09-08
+ | Rendered using Apache Maven Fluido Skin 1.3.0
+-->
+<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
+ <head>
+ <meta charset="UTF-8" />
+ <meta name="viewport" content="width=device-width, initial-scale=1.0" />
+ <meta name="Date-Revision-yyyymmdd" content="20170908" />
+ <meta http-equiv="Content-Language" content="en" />
+ <title>Metron – Current Build</title>
+ <link rel="stylesheet" href="../css/apache-maven-fluido-1.3.0.min.css" />
+ <link rel="stylesheet" href="../css/site.css" />
+ <link rel="stylesheet" href="../css/print.css" media="print" />
+
+
+ <script type="text/javascript" src="../js/apache-maven-fluido-1.3.0.min.js"></script>
+
+
+
+<script type="text/javascript">$( document ).ready( function() { $( '.carousel' ).carousel( { interval: 3500 } ) } );</script>
+
+ </head>
+ <body class="topBarDisabled">
+
+
+
+
+ <div class="container-fluid">
+ <div id="banner">
+ <div class="pull-left">
+ <a href="http://metron.apache.org/" id="bannerLeft">
+ <img src="../images/metron-logo.png" alt="Apache Metron" width="148px" height="48px"/>
+ </a>
+ </div>
+ <div class="pull-right"> </div>
+ <div class="clear"><hr/></div>
+ </div>
+
+ <div id="breadcrumbs">
+ <ul class="breadcrumb">
+
+
+ <li class="">
+ <a href="http://www.apache.org" class="externalLink" title="Apache">
+ Apache</a>
+ </li>
+ <li class="divider ">/</li>
+ <li class="">
+ <a href="http://metron.apache.org/" class="externalLink" title="Metron">
+ Metron</a>
+ </li>
+ <li class="divider ">/</li>
+ <li class="">
+ <a href="../index.html" title="Documentation">
+ Documentation</a>
+ </li>
+ <li class="divider ">/</li>
+ <li class="">Current Build</li>
+
+
+
+ <li id="publishDate" class="pull-right">Last Published: 2017-09-08</li> <li class="divider pull-right">|</li>
+ <li id="projectVersion" class="pull-right">Version: 0.4.1</li>
+
+ </ul>
+ </div>
+
+
+ <div class="row-fluid">
+ <div id="leftColumn" class="span3">
+ <div class="well sidebar-nav">
+
+
+ <ul class="nav nav-list">
+ <li class="nav-header">User Documentation</li>
+
+ <li>
+
+ <a href="../index.html" title="Metron">
+ <i class="icon-chevron-down"></i>
+ Metron</a>
+ <ul class="nav nav-list">
+
+ <li>
+
+ <a href="../Upgrading.html" title="Upgrading">
+ <i class="none"></i>
+ Upgrading</a>
+ </li>
+
+ <li>
+
+ <a href="../metron-analytics/index.html" title="Analytics">
+ <i class="icon-chevron-right"></i>
+ Analytics</a>
+ </li>
+
+ <li>
+
+ <a href="../metron-contrib/metron-docker/index.html" title="Docker">
+ <i class="none"></i>
+ Docker</a>
+ </li>
+
+ <li>
+
+ <a href="../metron-deployment/index.html" title="Deployment">
+ <i class="icon-chevron-right"></i>
+ Deployment</a>
+ </li>
+
+ <li>
+
+ <a href="../metron-interface/metron-alerts/index.html" title="Alerts">
+ <i class="none"></i>
+ Alerts</a>
+ </li>
+
+ <li>
+
+ <a href="../metron-interface/metron-config/index.html" title="Config">
+ <i class="none"></i>
+ Config</a>
+ </li>
+
+ <li>
+
+ <a href="../metron-interface/metron-rest/index.html" title="Rest">
+ <i class="none"></i>
+ Rest</a>
+ </li>
+
+ <li class="active">
+
+ <a href="#"><i class="icon-chevron-down"></i>Platform</a>
+ <ul class="nav nav-list">
+
+ <li>
+
+ <a href="../metron-platform/Performance-tuning-guide.html" title="Performance-tuning-guide">
+ <i class="none"></i>
+ Performance-tuning-guide</a>
+ </li>
+
+ <li>
+
+ <a href="../metron-platform/metron-api/index.html" title="Api">
+ <i class="none"></i>
+ Api</a>
+ </li>
+
+ <li>
+
+ <a href="../metron-platform/metron-common/index.html" title="Common">
+ <i class="none"></i>
+ Common</a>
+ </li>
+
+ <li>
+
+ <a href="../metron-platform/metron-data-management/index.html" title="Data-management">
+ <i class="none"></i>
+ Data-management</a>
+ </li>
+
+ <li>
+
+ <a href="../metron-platform/metron-enrichment/index.html" title="Enrichment">
+ <i class="none"></i>
+ Enrichment</a>
+ </li>
+
+ <li>
+
+ <a href="../metron-platform/metron-indexing/index.html" title="Indexing">
+ <i class="none"></i>
+ Indexing</a>
+ </li>
+
+ <li>
+
+ <a href="../metron-platform/metron-management/index.html" title="Management">
+ <i class="none"></i>
+ Management</a>
+ </li>
+
+ <li>
+
+ <a href="../metron-platform/metron-parsers/index.html" title="Parsers">
+ <i class="icon-chevron-right"></i>
+ Parsers</a>
+ </li>
+
+ <li>
+
+ <a href="../metron-platform/metron-pcap-backend/index.html" title="Pcap-backend">
+ <i class="none"></i>
+ Pcap-backend</a>
+ </li>
+
+ <li>
+
+ <a href="../metron-platform/metron-writer/index.html" title="Writer">
+ <i class="none"></i>
+ Writer</a>
+ </li>
+ </ul>
+ </li>
+
+ <li>
+
+ <a href="../metron-sensors/index.html" title="Sensors">
+ <i class="icon-chevron-right"></i>
+ Sensors</a>
+ </li>
+
+ <li>
+
+ <a href="../metron-stellar/stellar-common/index.html" title="Stellar-common">
+ <i class="icon-chevron-right"></i>
+ Stellar-common</a>
+ </li>
+
+ <li>
+
+ <a href="../use-cases/index.html" title="Use-cases">
+ <i class="icon-chevron-right"></i>
+ Use-cases</a>
+ </li>
+ </ul>
+ </li>
+ </ul>
+
+
+
+ <hr class="divider" />
+
+ <div id="poweredBy">
+ <div class="clear"></div>
+ <div class="clear"></div>
+ <div class="clear"></div>
+ <a href="http://maven.apache.org/" title="Built by Maven" class="poweredBy">
+ <img class="builtBy" alt="Built by Maven" src="../images/logos/maven-feather.png" />
+ </a>
+ </div>
+ </div>
+ </div>
+
+
+ <div id="bodyColumn" class="span9" >
+
+ <!-- Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to You under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License. --><h1>Current Build</h1>
+<p><a name="Current_Build"></a></p>
+<p>The latest build of metron-platform is 0.4.1.</p>
+<p>We are still in the process of merging/porting additional features from our production code base into this open source release. This release will be followed by a number of additional beta releases until the port is complete. We will also work on getting additional documentation and user/developer guides to the community as soon as we can. At this time we offer no support for the beta software, but will try to respond to requests as promptly as we can.</p>
+<p><a name="metron-platform"></a></p>
+<h1>metron-platform</h1>
+<p>Extensible set of Storm topologies and topology attributes for streaming, enriching, indexing, and storing telemetry in Hadoop. General information on Metron is available at <a class="externalLink" href="https://metron.apache.org/">https://metron.apache.org/</a></p>
+<p><a name="Documentation"></a></p>
+<h1>Documentation</h1>
+<p>Please see documentation within each individual module for description and usage instructions. Sample topologies are provided under Metron_Topologies to get you started with the framework. We pre-assume knowledge of Hadoop, Storm, Kafka, and HBase.</p>
+ </div>
+ </div>
+ </div>
+
+ <hr/>
+
+ <footer>
+ <div class="container-fluid">
+ <div class="row span12">Copyright © 2017
+ <a href="https://www.apache.org">The Apache Software Foundation</a>.
+ All Rights Reserved.
+
+ </div>
+
+
+
+ </div>
+ </footer>
+ </body>
+</html>
Added: dev/metron/0.4.1-RC4/site-book/metron-platform/metron-api/index.html
==============================================================================
--- dev/metron/0.4.1-RC4/site-book/metron-platform/metron-api/index.html (added)
+++ dev/metron/0.4.1-RC4/site-book/metron-platform/metron-api/index.html Sat Sep 9 04:54:51 2017
@@ -0,0 +1,332 @@
+<!DOCTYPE html>
+<!--
+ | Generated by Apache Maven Doxia at 2017-09-08
+ | Rendered using Apache Maven Fluido Skin 1.3.0
+-->
+<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
+ <head>
+ <meta charset="UTF-8" />
+ <meta name="viewport" content="width=device-width, initial-scale=1.0" />
+ <meta name="Date-Revision-yyyymmdd" content="20170908" />
+ <meta http-equiv="Content-Language" content="en" />
+ <title>Metron – Metron PCAP Service</title>
+ <link rel="stylesheet" href="../../css/apache-maven-fluido-1.3.0.min.css" />
+ <link rel="stylesheet" href="../../css/site.css" />
+ <link rel="stylesheet" href="../../css/print.css" media="print" />
+
+
+ <script type="text/javascript" src="../../js/apache-maven-fluido-1.3.0.min.js"></script>
+
+
+
+<script type="text/javascript">$( document ).ready( function() { $( '.carousel' ).carousel( { interval: 3500 } ) } );</script>
+
+ </head>
+ <body class="topBarDisabled">
+
+
+
+
+ <div class="container-fluid">
+ <div id="banner">
+ <div class="pull-left">
+ <a href="http://metron.apache.org/" id="bannerLeft">
+ <img src="../../images/metron-logo.png" alt="Apache Metron" width="148px" height="48px"/>
+ </a>
+ </div>
+ <div class="pull-right"> </div>
+ <div class="clear"><hr/></div>
+ </div>
+
+ <div id="breadcrumbs">
+ <ul class="breadcrumb">
+
+
+ <li class="">
+ <a href="http://www.apache.org" class="externalLink" title="Apache">
+ Apache</a>
+ </li>
+ <li class="divider ">/</li>
+ <li class="">
+ <a href="http://metron.apache.org/" class="externalLink" title="Metron">
+ Metron</a>
+ </li>
+ <li class="divider ">/</li>
+ <li class="">
+ <a href="../../index.html" title="Documentation">
+ Documentation</a>
+ </li>
+ <li class="divider ">/</li>
+ <li class="">Metron PCAP Service</li>
+
+
+
+ <li id="publishDate" class="pull-right">Last Published: 2017-09-08</li> <li class="divider pull-right">|</li>
+ <li id="projectVersion" class="pull-right">Version: 0.4.1</li>
+
+ </ul>
+ </div>
+
+
+ <div class="row-fluid">
+ <div id="leftColumn" class="span3">
+ <div class="well sidebar-nav">
+
+
+ <ul class="nav nav-list">
+ <li class="nav-header">User Documentation</li>
+
+ <li>
+
+ <a href="../../index.html" title="Metron">
+ <i class="icon-chevron-down"></i>
+ Metron</a>
+ <ul class="nav nav-list">
+
+ <li>
+
+ <a href="../../Upgrading.html" title="Upgrading">
+ <i class="none"></i>
+ Upgrading</a>
+ </li>
+
+ <li>
+
+ <a href="../../metron-analytics/index.html" title="Analytics">
+ <i class="icon-chevron-right"></i>
+ Analytics</a>
+ </li>
+
+ <li>
+
+ <a href="../../metron-contrib/metron-docker/index.html" title="Docker">
+ <i class="none"></i>
+ Docker</a>
+ </li>
+
+ <li>
+
+ <a href="../../metron-deployment/index.html" title="Deployment">
+ <i class="icon-chevron-right"></i>
+ Deployment</a>
+ </li>
+
+ <li>
+
+ <a href="../../metron-interface/metron-alerts/index.html" title="Alerts">
+ <i class="none"></i>
+ Alerts</a>
+ </li>
+
+ <li>
+
+ <a href="../../metron-interface/metron-config/index.html" title="Config">
+ <i class="none"></i>
+ Config</a>
+ </li>
+
+ <li>
+
+ <a href="../../metron-interface/metron-rest/index.html" title="Rest">
+ <i class="none"></i>
+ Rest</a>
+ </li>
+
+ <li>
+
+ <a href="../../metron-platform/index.html" title="Platform">
+ <i class="icon-chevron-down"></i>
+ Platform</a>
+ <ul class="nav nav-list">
+
+ <li>
+
+ <a href="../../metron-platform/Performance-tuning-guide.html" title="Performance-tuning-guide">
+ <i class="none"></i>
+ Performance-tuning-guide</a>
+ </li>
+
+ <li class="active">
+
+ <a href="#"><i class="none"></i>Api</a>
+ </li>
+
+ <li>
+
+ <a href="../../metron-platform/metron-common/index.html" title="Common">
+ <i class="none"></i>
+ Common</a>
+ </li>
+
+ <li>
+
+ <a href="../../metron-platform/metron-data-management/index.html" title="Data-management">
+ <i class="none"></i>
+ Data-management</a>
+ </li>
+
+ <li>
+
+ <a href="../../metron-platform/metron-enrichment/index.html" title="Enrichment">
+ <i class="none"></i>
+ Enrichment</a>
+ </li>
+
+ <li>
+
+ <a href="../../metron-platform/metron-indexing/index.html" title="Indexing">
+ <i class="none"></i>
+ Indexing</a>
+ </li>
+
+ <li>
+
+ <a href="../../metron-platform/metron-management/index.html" title="Management">
+ <i class="none"></i>
+ Management</a>
+ </li>
+
+ <li>
+
+ <a href="../../metron-platform/metron-parsers/index.html" title="Parsers">
+ <i class="icon-chevron-right"></i>
+ Parsers</a>
+ </li>
+
+ <li>
+
+ <a href="../../metron-platform/metron-pcap-backend/index.html" title="Pcap-backend">
+ <i class="none"></i>
+ Pcap-backend</a>
+ </li>
+
+ <li>
+
+ <a href="../../metron-platform/metron-writer/index.html" title="Writer">
+ <i class="none"></i>
+ Writer</a>
+ </li>
+ </ul>
+ </li>
+
+ <li>
+
+ <a href="../../metron-sensors/index.html" title="Sensors">
+ <i class="icon-chevron-right"></i>
+ Sensors</a>
+ </li>
+
+ <li>
+
+ <a href="../../metron-stellar/stellar-common/index.html" title="Stellar-common">
+ <i class="icon-chevron-right"></i>
+ Stellar-common</a>
+ </li>
+
+ <li>
+
+ <a href="../../use-cases/index.html" title="Use-cases">
+ <i class="icon-chevron-right"></i>
+ Use-cases</a>
+ </li>
+ </ul>
+ </li>
+ </ul>
+
+
+
+ <hr class="divider" />
+
+ <div id="poweredBy">
+ <div class="clear"></div>
+ <div class="clear"></div>
+ <div class="clear"></div>
+ <a href="http://maven.apache.org/" title="Built by Maven" class="poweredBy">
+ <img class="builtBy" alt="Built by Maven" src="../../images/logos/maven-feather.png" />
+ </a>
+ </div>
+ </div>
+ </div>
+
+
+ <div id="bodyColumn" class="span9" >
+
+ <h1>Metron PCAP Service</h1>
+<p><a name="Metron_PCAP_Service"></a></p>
+<p>The purpose of the Metron PCAP service is to provide a middle tier to negotiate retrieving packet capture data which flows into Metron. This packet data is of a form which <tt>libpcap</tt> based tools can read.</p>
+<div class="section">
+<h2><a name="Starting_the_Service"></a>Starting the Service</h2>
+<p>You can start the service either via the init.d script installed, <tt>/etc/init.d/pcapservice</tt> or directly via the <tt>yarn jar</tt> command: <tt>yarn jar $METRON_HOME/lib/metron-api-$METRON_VERSION.jar org.apache.metron.pcapservice.rest.PcapService -port $SERVICE_PORT -query_hdfs_path $QUERY_PATH -pcap_hdfs_path $PCAP_PATH</tt></p>
+<p>where</p>
+
+<ul>
+
+<li><tt>METRON_HOME</tt> is the location of the metron installation</li>
+
+<li><tt>METRON_VERSION</tt> is the version of the metron installation</li>
+
+<li><tt>SERVICE_PORT</tt> is the port to bind the REST service to.</li>
+
+<li><tt>QUERY_PATH</tt> is the temporary location to store query results. They are deleted after the service reads them.</li>
+
+<li><tt>PCAP_PATH</tt> is the path to the packet data on HDFS</li>
+</ul></div>
+<div class="section">
+<h2><a name="The_pcapGettergetPcapsByIdentifiers_endpoint"></a>The <tt>/pcapGetter/getPcapsByIdentifiers</tt> endpoint</h2>
+<p>This endpoint takes the following query parameters and returns the subset of packets matching this query:</p>
+
+<ul>
+
+<li><tt>srcIp</tt> : The source IP to match on</li>
+
+<li><tt>srcPort</tt> : The source port to match on</li>
+
+<li><tt>dstIp</tt> : The destination IP to match on</li>
+
+<li><tt>dstPort</tt> : The destination port to match on</li>
+
+<li><tt>startTime</tt> : The start time in milliseconds</li>
+
+<li><tt>endTime</tt> : The end time in milliseconds</li>
+
+<li><tt>numReducers</tt> : Specify the number of reducers to use when executing the mapreduce job</li>
+
+<li><tt>includeReverseTraffic</tt> : Indicates if filter should check swapped src/dest addresses and IPs</li>
+</ul></div>
+<div class="section">
+<h2><a name="The_pcapGettergetPcapsByQuery_endpoint"></a>The <tt>/pcapGetter/getPcapsByQuery</tt> endpoint</h2>
+<p>This endpoint takes the following query parameters and returns the subset of packets matching this query. This endpoint exposes Stellar querying capabilities:</p>
+
+<ul>
+
+<li><tt>query</tt> : The Stellar query to execute</li>
+
+<li><tt>startTime</tt> : The start time in milliseconds</li>
+
+<li><tt>endTime</tt> : The end time in milliseconds</li>
+
+<li><tt>numReducers</tt> : Specify the number of reducers to use when executing the mapreduce job</li>
+</ul>
+<p>Example: <tt>curl -XGET "http://node1:8081/pcapGetter/getPcapsByQuery?query=ip_src_addr+==+'192.168.66.121'+and+ip_src_port+==+'60500'&startTime=1476936000000"</tt></p>
+<p>All of these parameters are optional. In the case of a missing parameter, it is treated as a wildcard.</p>
+<p>Unlike the CLI tool, there is no paging mechanism. The REST API will stream back data as a single file.</p></div>
+ </div>
+ </div>
+ </div>
+
+ <hr/>
+
+ <footer>
+ <div class="container-fluid">
+ <div class="row span12">Copyright © 2017
+ <a href="https://www.apache.org">The Apache Software Foundation</a>.
+ All Rights Reserved.
+
+ </div>
+
+
+
+ </div>
+ </footer>
+ </body>
+</html>
Added: dev/metron/0.4.1-RC4/site-book/metron-platform/metron-common/index.html
==============================================================================
--- dev/metron/0.4.1-RC4/site-book/metron-platform/metron-common/index.html (added)
+++ dev/metron/0.4.1-RC4/site-book/metron-platform/metron-common/index.html Sat Sep 9 04:54:51 2017
@@ -0,0 +1,522 @@
+<!DOCTYPE html>
+<!--
+ | Generated by Apache Maven Doxia at 2017-09-08
+ | Rendered using Apache Maven Fluido Skin 1.3.0
+-->
+<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
+ <head>
+ <meta charset="UTF-8" />
+ <meta name="viewport" content="width=device-width, initial-scale=1.0" />
+ <meta name="Date-Revision-yyyymmdd" content="20170908" />
+ <meta http-equiv="Content-Language" content="en" />
+ <title>Metron – Contents</title>
+ <link rel="stylesheet" href="../../css/apache-maven-fluido-1.3.0.min.css" />
+ <link rel="stylesheet" href="../../css/site.css" />
+ <link rel="stylesheet" href="../../css/print.css" media="print" />
+
+
+ <script type="text/javascript" src="../../js/apache-maven-fluido-1.3.0.min.js"></script>
+
+
+
+<script type="text/javascript">$( document ).ready( function() { $( '.carousel' ).carousel( { interval: 3500 } ) } );</script>
+
+ </head>
+ <body class="topBarDisabled">
+
+
+
+
+ <div class="container-fluid">
+ <div id="banner">
+ <div class="pull-left">
+ <a href="http://metron.apache.org/" id="bannerLeft">
+ <img src="../../images/metron-logo.png" alt="Apache Metron" width="148px" height="48px"/>
+ </a>
+ </div>
+ <div class="pull-right"> </div>
+ <div class="clear"><hr/></div>
+ </div>
+
+ <div id="breadcrumbs">
+ <ul class="breadcrumb">
+
+
+ <li class="">
+ <a href="http://www.apache.org" class="externalLink" title="Apache">
+ Apache</a>
+ </li>
+ <li class="divider ">/</li>
+ <li class="">
+ <a href="http://metron.apache.org/" class="externalLink" title="Metron">
+ Metron</a>
+ </li>
+ <li class="divider ">/</li>
+ <li class="">
+ <a href="../../index.html" title="Documentation">
+ Documentation</a>
+ </li>
+ <li class="divider ">/</li>
+ <li class="">Contents</li>
+
+
+
+ <li id="publishDate" class="pull-right">Last Published: 2017-09-08</li> <li class="divider pull-right">|</li>
+ <li id="projectVersion" class="pull-right">Version: 0.4.1</li>
+
+ </ul>
+ </div>
+
+
+ <div class="row-fluid">
+ <div id="leftColumn" class="span3">
+ <div class="well sidebar-nav">
+
+
+ <ul class="nav nav-list">
+ <li class="nav-header">User Documentation</li>
+
+ <li>
+
+ <a href="../../index.html" title="Metron">
+ <i class="icon-chevron-down"></i>
+ Metron</a>
+ <ul class="nav nav-list">
+
+ <li>
+
+ <a href="../../Upgrading.html" title="Upgrading">
+ <i class="none"></i>
+ Upgrading</a>
+ </li>
+
+ <li>
+
+ <a href="../../metron-analytics/index.html" title="Analytics">
+ <i class="icon-chevron-right"></i>
+ Analytics</a>
+ </li>
+
+ <li>
+
+ <a href="../../metron-contrib/metron-docker/index.html" title="Docker">
+ <i class="none"></i>
+ Docker</a>
+ </li>
+
+ <li>
+
+ <a href="../../metron-deployment/index.html" title="Deployment">
+ <i class="icon-chevron-right"></i>
+ Deployment</a>
+ </li>
+
+ <li>
+
+ <a href="../../metron-interface/metron-alerts/index.html" title="Alerts">
+ <i class="none"></i>
+ Alerts</a>
+ </li>
+
+ <li>
+
+ <a href="../../metron-interface/metron-config/index.html" title="Config">
+ <i class="none"></i>
+ Config</a>
+ </li>
+
+ <li>
+
+ <a href="../../metron-interface/metron-rest/index.html" title="Rest">
+ <i class="none"></i>
+ Rest</a>
+ </li>
+
+ <li>
+
+ <a href="../../metron-platform/index.html" title="Platform">
+ <i class="icon-chevron-down"></i>
+ Platform</a>
+ <ul class="nav nav-list">
+
+ <li>
+
+ <a href="../../metron-platform/Performance-tuning-guide.html" title="Performance-tuning-guide">
+ <i class="none"></i>
+ Performance-tuning-guide</a>
+ </li>
+
+ <li>
+
+ <a href="../../metron-platform/metron-api/index.html" title="Api">
+ <i class="none"></i>
+ Api</a>
+ </li>
+
+ <li class="active">
+
+ <a href="#"><i class="none"></i>Common</a>
+ </li>
+
+ <li>
+
+ <a href="../../metron-platform/metron-data-management/index.html" title="Data-management">
+ <i class="none"></i>
+ Data-management</a>
+ </li>
+
+ <li>
+
+ <a href="../../metron-platform/metron-enrichment/index.html" title="Enrichment">
+ <i class="none"></i>
+ Enrichment</a>
+ </li>
+
+ <li>
+
+ <a href="../../metron-platform/metron-indexing/index.html" title="Indexing">
+ <i class="none"></i>
+ Indexing</a>
+ </li>
+
+ <li>
+
+ <a href="../../metron-platform/metron-management/index.html" title="Management">
+ <i class="none"></i>
+ Management</a>
+ </li>
+
+ <li>
+
+ <a href="../../metron-platform/metron-parsers/index.html" title="Parsers">
+ <i class="icon-chevron-right"></i>
+ Parsers</a>
+ </li>
+
+ <li>
+
+ <a href="../../metron-platform/metron-pcap-backend/index.html" title="Pcap-backend">
+ <i class="none"></i>
+ Pcap-backend</a>
+ </li>
+
+ <li>
+
+ <a href="../../metron-platform/metron-writer/index.html" title="Writer">
+ <i class="none"></i>
+ Writer</a>
+ </li>
+ </ul>
+ </li>
+
+ <li>
+
+ <a href="../../metron-sensors/index.html" title="Sensors">
+ <i class="icon-chevron-right"></i>
+ Sensors</a>
+ </li>
+
+ <li>
+
+ <a href="../../metron-stellar/stellar-common/index.html" title="Stellar-common">
+ <i class="icon-chevron-right"></i>
+ Stellar-common</a>
+ </li>
+
+ <li>
+
+ <a href="../../use-cases/index.html" title="Use-cases">
+ <i class="icon-chevron-right"></i>
+ Use-cases</a>
+ </li>
+ </ul>
+ </li>
+ </ul>
+
+
+
+ <hr class="divider" />
+
+ <div id="poweredBy">
+ <div class="clear"></div>
+ <div class="clear"></div>
+ <div class="clear"></div>
+ <a href="http://maven.apache.org/" title="Built by Maven" class="poweredBy">
+ <img class="builtBy" alt="Built by Maven" src="../../images/logos/maven-feather.png" />
+ </a>
+ </div>
+ </div>
+ </div>
+
+
+ <div id="bodyColumn" class="span9" >
+
+ <h1>Contents</h1>
+<p><a name="Contents"></a></p>
+
+<ul>
+
+<li><a href="#Stellar_Language">Stellar Language</a></li>
+
+<li><a href="#Global_Configuration">Global Configuration</a></li>
+
+<li><a href="#Validation_Framework">Validation Framework</a></li>
+
+<li><a href="#Management_Utility">Management Utility</a></li>
+
+<li><a href="topology-errors/index.html">Topology Errors</a></li>
+
+<li><a href="#Performance_Logging">Performance Logging</a></li>
+</ul>
+<p><a name="Stellar_Language"></a></p>
+<h1>Stellar Language</h1>
+<p>For a variety of components (threat intelligence triage and field transformations) we have the need to do simple computation and transformation using the data from messages as variables.<br />For those purposes, there exists a simple, scaled down DSL created to do simple computation and transformation.</p>
+<p>The query language supports the following:</p>
+
+<ul>
+
+<li>Referencing fields in the enriched JSON</li>
+
+<li>String literals are quoted with either <tt>'</tt> or <tt>"</tt>, and support escaping for <tt>'</tt>, <tt>"</tt>, <tt>\t</tt>, <tt>\r</tt>, <tt>\n</tt>, and backslash</li>
+
+<li>Simple boolean operations: <tt>and</tt>, <tt>not</tt>, <tt>or</tt>
+
+<ul>
+
+<li>Boolean expressions are short-circuited (e.g. <tt>true or FUNC()</tt> would never execute <tt>FUNC</tt>)</li>
+ </ul></li>
+
+<li>Simple arithmetic operations: <tt>*</tt>, <tt>/</tt>, <tt>+</tt>, <tt>-</tt> on real numbers or integers</li>
+
+<li>Simple comparison operations <tt><</tt>, <tt>></tt>, <tt><=</tt>, <tt>>=</tt></li>
+
+<li>Simple equality comparison operations <tt>==</tt>, <tt>!=</tt></li>
+
+<li>if/then/else comparisons (i.e. <tt>if var1 < 10 then 'less than 10' else '10 or more'</tt>)</li>
+
+<li>Determining whether a field exists (via <tt>exists</tt>)</li>
+
+<li>An <tt>in</tt> operator that works like the <tt>in</tt> in Python</li>
+
+<li>The ability to have parenthesis to make order of operations explicit</li>
+
+<li>User defined functions, including Lambda expressions</li>
+</ul>
+<p>For documentation of Stellar, please see the <a href="../../metron-stellar/stellar-common/index.html">Stellar README</a>.</p>
+<p><a name="Global_Configuration"></a></p>
+<h1>Global Configuration</h1>
+<p>The format of the global enrichment is a JSON String to Object map. This is intended for configuration which is non sensor specific configuration.</p>
+<p>This configuration is stored in zookeeper, but looks something like</p>
+
+<div class="source">
+<div class="source">
+<pre>{
+ "es.clustername": "metron",
+ "es.ip": "node1",
+ "es.port": "9300",
+ "es.date.format": "yyyy.MM.dd.HH",
+ "parser.error.topic": "indexing"
+ "fieldValidations" : [
+ {
+ "input" : [ "ip_src_addr", "ip_dst_addr" ],
+ "validation" : "IP",
+ "config" : {
+ "type" : "IPV4"
+ }
+ }
+ ]
+}
+</pre></div></div>
+<p><a name="Validation_Framework"></a></p>
+<h1>Validation Framework</h1>
+<p>Inside of the global configuration, there is a validation framework in place that enables the validation that messages coming from all parsers are valid. This is done in the form of validation plugins where assertions about fields or whole messages can be made. </p>
+<p>The format for this is a <tt>fieldValidations</tt> field inside of global config. This is associated with an array of field validation objects structured like so:</p>
+
+<ul>
+
+<li><tt>input</tt> : An array of input fields or a single field. If this is omitted, then the whole messages is passed to the validator.</li>
+
+<li><tt>config</tt> : A String to Object map for validation configuration. This is optional if the validation function requires no configuration.</li>
+
+<li><tt>validation</tt> : The validation function to be used. This is one of
+
+<ul>
+
+<li><tt>STELLAR</tt> : Execute a Stellar Language statement. Expects the query string in the <tt>condition</tt> field of the config.</li>
+
+<li><tt>IP</tt> : Validates that the input fields are an IP address. By default, if no configuration is set, it assumes <tt>IPV4</tt>, but you can specify the type by passing in the config by passing in <tt>type</tt> with either <tt>IPV6</tt> or <tt>IPV4</tt> or by passing in a list [<tt>IPV4</tt>,<tt>IPV6</tt>] in which case the input(s) will be validated against both.</li>
+
+<li><tt>DOMAIN</tt> : Validates that the fields are all domains.</li>
+
+<li><tt>EMAIL</tt> : Validates that the fields are all email addresses</li>
+
+<li><tt>URL</tt> : Validates that the fields are all URLs</li>
+
+<li><tt>DATE</tt> : Validates that the fields are a date. Expects <tt>format</tt> in the config.</li>
+
+<li><tt>INTEGER</tt> : Validates that the fields are an integer. String representation of an integer is allowed.</li>
+
+<li><tt>REGEX_MATCH</tt> : Validates that the fields match a regex. Expects <tt>pattern</tt> in the config.</li>
+
+<li><tt>NOT_EMPTY</tt> : Validates that the fields exist and are not empty (after trimming.)</li>
+ </ul></li>
+</ul>
+<p><a name="Management_Utility"></a></p>
+<h1>Management Utility</h1>
+<p>Configurations should be stored on disk in the following structure starting at <tt>$BASE_DIR</tt>:</p>
+
+<ul>
+
+<li>global.json : The global config</li>
+
+<li><tt>sensors</tt> : The subdirectory containing sensor enrichment configuration JSON (e.g. <tt>snort.json</tt>, <tt>bro.json</tt>)</li>
+</ul>
+<p>By default, this directory as deployed by the ansible infrastructure is at <tt>$METRON_HOME/config/zookeeper</tt></p>
+<p>While the configs are stored on disk, they must be loaded into Zookeeper to be used. To this end, there is a utility program to assist in this called <tt>$METRON_HOME/bin/zk_load_config.sh</tt></p>
+<p>This has the following options:</p>
+
+<div class="source">
+<div class="source">
+<pre> -f,--force Force operation
+ -h,--help Generate Help screen
+ -i,--input_dir <DIR> The input directory containing
+ the configuration files named
+ like "$source.json"
+ -m,--mode <MODE> The mode of operation: DUMP,
+ PULL, PUSH
+ -o,--output_dir <DIR> The output directory which will
+ store the JSON configuration
+ from Zookeeper
+ -z,--zk_quorum <host:port,[host:port]*> Zookeeper Quorum URL
+ (zk1:port,zk2:port,...)
+</pre></div></div>
+<p>Usage examples:</p>
+
+<ul>
+
+<li>To dump the existing configs from zookeeper on the singlenode vagrant machine: <tt>$METRON_HOME/bin/zk_load_configs.sh -z node1:2181 -m DUMP</tt></li>
+
+<li>To push the configs into zookeeper on the singlenode vagrant machine: <tt>$METRON_HOME/bin/zk_load_configs.sh -z node1:2181 -m PUSH -i $METRON_HOME/config/zookeeper</tt></li>
+
+<li>To pull the configs from zookeeper to the singlenode vagrant machine disk: <tt>$METRON_HOME/bin/zk_load_configs.sh -z node1:2181 -m PULL -o $METRON_HOME/config/zookeeper -f</tt></li>
+</ul>
+<p><a name="Topology_Errors"></a></p>
+<h1>Topology Errors</h1>
+<p>Errors generated in Metron topologies are transformed into JSON format and follow this structure:</p>
+
+<div class="source">
+<div class="source">
+<pre>{
+ "exception": "java.lang.IllegalStateException: Unable to parse Message: ...",
+ "failed_sensor_type": "bro",
+ "stack": "java.lang.IllegalStateException: Unable to parse Message: ...",
+ "hostname": "node1",
+ "source:type": "error",
+ "raw_message": "{\"http\": {\"ts\":1488809627.000000.31915,\"uid\":\"C9JpSd2vFAWo3mXKz1\", ...",
+ "error_hash": "f7baf053f2d3c801a01d196f40f3468e87eea81788b2567423030100865c5061",
+ "error_type": "parser_error",
+ "message": "Unable to parse Message: {\"http\": {\"ts\":1488809627.000000.31915,\"uid\":\"C9JpSd2vFAWo3mXKz1\", ...",
+ "timestamp": 1488809630698
+}
+</pre></div></div>
+<p>Each topology can be configured to send error messages to a specific Kafka topic. The parser topologies retrieve this setting from the the <tt>parser.error.topic</tt> setting in the global config:</p>
+
+<div class="source">
+<div class="source">
+<pre>{
+ "es.clustername": "metron",
+ "es.ip": "node1",
+ "es.port": "9300",
+ "es.date.format": "yyyy.MM.dd.HH",
+ "parser.error.topic": "indexing"
+}
+</pre></div></div>
+<p>Error topics for enrichment and threat intel errors are passed into the enrichment topology as flux properties named <tt>enrichment.error.topic</tt> and <tt>threat.intel.error.topic</tt>. These properties can be found in <tt>$METRON_HOME/config/enrichment.properties</tt>.</p>
+<p>The error topic for indexing errors is passed into the indexing topology as a flux property named <tt>index.error.topic</tt>. This property can be found in either <tt>$METRON_HOME/config/elasticsearch.properties</tt> or <tt>$METRON_HOME/config/solr.properties</tt> depending on the search engine selected.</p>
+<p>By default all error messages are sent to the <tt>indexing</tt> topic so that they are indexed and archived, just like other messages. The indexing config for error messages can be found at <tt>$METRON_HOME/config/zookeeper/indexing/error.json</tt>.</p>
+<p><a name="Performance_Logging"></a></p>
+<h1>Performance Logging</h1>
+<p>The PerformanceLogger class provides functionality that enables developers to debug performance issues. Basic usage looks like the following:</p>
+
+<div class="source">
+<div class="source">
+<pre>// create a simple inner performance class to use for logger instantiation
+public static class Perf {}
+// instantiation
+PerformnanceLogger perfLog = new PerformanceLogger(() -> getConfigurations().getGlobalConfig(), Perf.class.getName());
+// marking a start time
+perfLog.mark("mark1");
+// ...do some high performance stuff...
+// log the elapsed time
+perfLog.log("mark1", "My high performance stuff is very performant");
+// log no additional message, just the basics
+perfLog.log("mark1");
+</pre></div></div>
+<p>The logger maintains a Map<String, Long> of named markers that correspond to start times. Calling mark() performs a put on the underlying timing store. Output includes the mark name, elapsed time in nanoseconds, as well as any custom messaging you provide. A sample log would look like the following:</p>
+
+<div class="source">
+<div class="source">
+<pre>[DEBUG] markName=execute,time(ns)=121411,message=key=7a8dbe44-4cb9-4db2-9d04-7632f543b56c, elapsed time to run execute
+</pre></div></div>
+<p><b>Configuration</b></p>
+<p>The first argument to the logger is a java.util.function.Supplier<Map<String, Object>>. The offers flexibility in being able to provide multiple configuration “suppliers” depending on your individual usage requirements. The example above, taken from org.apache.metron.enrichment.bolt.GenericEnrichmentBolt, leverages the global config to dymanically provide configuration from Zookeeper. Any updates to the global config via Zookeeper are reflected live at runtime. Currently, the PerformanceLogger supports the following options:</p>
+
+<table border="0" class="table table-striped">
+ <thead>
+
+<tr class="a">
+
+<th>Property Name </th>
+
+<th>Type </th>
+
+<th>Valid Values </th>
+ </tr>
+ </thead>
+ <tbody>
+
+<tr class="b">
+
+<td>performance.logging.percent.records </td>
+
+<td>Integer </td>
+
+<td>0-100 </td>
+ </tr>
+ </tbody>
+</table>
+<p><b>Other Usage Details</b></p>
+<p>You can also provide your own format String and provide arguments that will be used when formatting that String. This code avoids expensive String concatenation by only formatting when debugging is enabled. For more complex arguments, e.g. JSON serialization, we expose an isDebugEnabled() method.</p>
+
+<div class="source">
+<div class="source">
+<pre>// log with format String and single argument
+perfLog.log("join-message", "key={}, elapsed time to join messages", key);
+
+// check if debugging is enabled for the performance logger to avoid more expensive operations
+if (perfLog.isDebugEnabled()) {
+ perfLog.log("join-message", "key={}, elapsed time to join messages, message={}", key, rawMessage.toJSONString());
+}
+</pre></div></div>
+<p><b>Side Effects</b></p>
+<p>Calling the mark() method multiple times simply resets the start time to the current nano time. Calling log() with a non-existent mark name will log 0 ns elapsed time with a warning indicating that log has been invoked for a mark name that does not exist. The class is not thread-safe and makes no attempt at keeping multiple threads from modifying the same markers.</p>
+ </div>
+ </div>
+ </div>
+
+ <hr/>
+
+ <footer>
+ <div class="container-fluid">
+ <div class="row span12">Copyright © 2017
+ <a href="https://www.apache.org">The Apache Software Foundation</a>.
+ All Rights Reserved.
+
+ </div>
+
+
+
+ </div>
+ </footer>
+ </body>
+</html>