You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@metron.apache.org by ma...@apache.org on 2018/01/04 20:31:22 UTC
[1/7] metron git commit: METRON-1388 update public web site to point
at 0.4.2 new release (mattf-horton) closes apache/metron#887
Repository: metron
Updated Branches:
refs/heads/master 01c26a77b -> 3381b853d
http://git-wip-us.apache.org/repos/asf/metron/blob/3381b853/site/current-book/use-cases/forensic_clustering/index.html
----------------------------------------------------------------------
diff --git a/site/current-book/use-cases/forensic_clustering/index.html b/site/current-book/use-cases/forensic_clustering/index.html
new file mode 100644
index 0000000..73a087e
--- /dev/null
+++ b/site/current-book/use-cases/forensic_clustering/index.html
@@ -0,0 +1,726 @@
+<!DOCTYPE html>
+<!--
+ | Generated by Apache Maven Doxia at 2018-01-03
+ | Rendered using Apache Maven Fluido Skin 1.3.0
+-->
+<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
+ <head>
+ <meta charset="UTF-8" />
+ <meta name="viewport" content="width=device-width, initial-scale=1.0" />
+ <meta name="Date-Revision-yyyymmdd" content="20180103" />
+ <meta http-equiv="Content-Language" content="en" />
+ <title>Metron – Problem Statement</title>
+ <link rel="stylesheet" href="../../css/apache-maven-fluido-1.3.0.min.css" />
+ <link rel="stylesheet" href="../../css/site.css" />
+ <link rel="stylesheet" href="../../css/print.css" media="print" />
+
+
+ <script type="text/javascript" src="../../js/apache-maven-fluido-1.3.0.min.js"></script>
+
+
+
+<script type="text/javascript">$( document ).ready( function() { $( '.carousel' ).carousel( { interval: 3500 } ) } );</script>
+
+ </head>
+ <body class="topBarDisabled">
+
+
+
+
+ <div class="container-fluid">
+ <div id="banner">
+ <div class="pull-left">
+ <a href="http://metron.apache.org/" id="bannerLeft">
+ <img src="../../images/metron-logo.png" alt="Apache Metron" width="148px" height="48px"/>
+ </a>
+ </div>
+ <div class="pull-right"> </div>
+ <div class="clear"><hr/></div>
+ </div>
+
+ <div id="breadcrumbs">
+ <ul class="breadcrumb">
+
+
+ <li class="">
+ <a href="http://www.apache.org" class="externalLink" title="Apache">
+ Apache</a>
+ </li>
+ <li class="divider ">/</li>
+ <li class="">
+ <a href="http://metron.apache.org/" class="externalLink" title="Metron">
+ Metron</a>
+ </li>
+ <li class="divider ">/</li>
+ <li class="">
+ <a href="../../index.html" title="Documentation">
+ Documentation</a>
+ </li>
+ <li class="divider ">/</li>
+ <li class="">Problem Statement</li>
+
+
+
+ <li id="publishDate" class="pull-right">Last Published: 2018-01-03</li> <li class="divider pull-right">|</li>
+ <li id="projectVersion" class="pull-right">Version: 0.4.2</li>
+
+ </ul>
+ </div>
+
+
+ <div class="row-fluid">
+ <div id="leftColumn" class="span3">
+ <div class="well sidebar-nav">
+
+
+ <ul class="nav nav-list">
+ <li class="nav-header">User Documentation</li>
+
+ <li>
+
+ <a href="../../index.html" title="Metron">
+ <i class="icon-chevron-down"></i>
+ Metron</a>
+ <ul class="nav nav-list">
+
+ <li>
+
+ <a href="../../Upgrading.html" title="Upgrading">
+ <i class="none"></i>
+ Upgrading</a>
+ </li>
+
+ <li>
+
+ <a href="../../metron-analytics/index.html" title="Analytics">
+ <i class="icon-chevron-right"></i>
+ Analytics</a>
+ </li>
+
+ <li>
+
+ <a href="../../metron-contrib/metron-docker/index.html" title="Docker">
+ <i class="none"></i>
+ Docker</a>
+ </li>
+
+ <li>
+
+ <a href="../../metron-deployment/index.html" title="Deployment">
+ <i class="icon-chevron-right"></i>
+ Deployment</a>
+ </li>
+
+ <li>
+
+ <a href="../../metron-interface/metron-alerts/index.html" title="Alerts">
+ <i class="none"></i>
+ Alerts</a>
+ </li>
+
+ <li>
+
+ <a href="../../metron-interface/metron-config/index.html" title="Config">
+ <i class="none"></i>
+ Config</a>
+ </li>
+
+ <li>
+
+ <a href="../../metron-interface/metron-rest/index.html" title="Rest">
+ <i class="none"></i>
+ Rest</a>
+ </li>
+
+ <li>
+
+ <a href="../../metron-platform/index.html" title="Platform">
+ <i class="icon-chevron-right"></i>
+ Platform</a>
+ </li>
+
+ <li>
+
+ <a href="../../metron-sensors/index.html" title="Sensors">
+ <i class="icon-chevron-right"></i>
+ Sensors</a>
+ </li>
+
+ <li>
+
+ <a href="../../metron-stellar/stellar-3rd-party-example/index.html" title="Stellar-3rd-party-example">
+ <i class="none"></i>
+ Stellar-3rd-party-example</a>
+ </li>
+
+ <li>
+
+ <a href="../../metron-stellar/stellar-common/index.html" title="Stellar-common">
+ <i class="icon-chevron-right"></i>
+ Stellar-common</a>
+ </li>
+
+ <li>
+
+ <a href="../../use-cases/index.html" title="Use-cases">
+ <i class="icon-chevron-down"></i>
+ Use-cases</a>
+ <ul class="nav nav-list">
+
+ <li class="active">
+
+ <a href="#"><i class="none"></i>Forensic_clustering</a>
+ </li>
+
+ <li>
+
+ <a href="../../use-cases/geographic_login_outliers/index.html" title="Geographic_login_outliers">
+ <i class="none"></i>
+ Geographic_login_outliers</a>
+ </li>
+ </ul>
+ </li>
+ </ul>
+ </li>
+ </ul>
+
+
+
+ <hr class="divider" />
+
+ <div id="poweredBy">
+ <div class="clear"></div>
+ <div class="clear"></div>
+ <div class="clear"></div>
+ <a href="http://maven.apache.org/" title="Built by Maven" class="poweredBy">
+ <img class="builtBy" alt="Built by Maven" src="../../images/logos/maven-feather.png" />
+ </a>
+ </div>
+ </div>
+ </div>
+
+
+ <div id="bodyColumn" class="span9" >
+
+ <h1>Problem Statement</h1>
+<p><a name="Problem_Statement"></a></p>
+<p>Having a forensic hash, such as <a class="externalLink" href="https://github.com/trendmicro/tlsh">TLSH</a>, is a useful tool in cybersecurity. In short, the notion is that semantically similar documents should hash to a value which also similar. Contrast this with your standard cryptographic hashes, such as SHA and MD, where small deviations in the input data will yield large deviations in the hashes.</p>
+<p>The traditional use-case is to hash input documents or binaries and compare against a known blacklist of malicious hashes. A sufficiently similar hash will indicate a match. This will avoid malicious parties fuzzing input data to avoid detection.</p>
+<p>While this is interesting, it still requires metric-space searches in a blacklist. I envisioned a slightly more interesting streaming use-case of on-the-fly clustering of data. While the TLSH hashes created do not necessarily hash to precisely the same value on similar documents, more traditional non-forensic hashes <i>do</i> collide when sufficiently similar. Namely, the Hamming distance <a class="externalLink" href="https://en.wikipedia.org/wiki/Locality-sensitive_hashing#Bit_sampling_for_Hamming_distance">LSH</a> applied to the TLSH hash would give us a way to bin semantic hashes such that similar hashes (by hamming distance) have the same hash.</p>
+<p>Inspired by a good <a class="externalLink" href="https://github.com/fluenda/dataworks_summit_iot_botnet/blob/master/dws-fucs-lopresto.pdf">talk</a> by Andy LoPresto and Andre Fucs de Miranda from Apache NiFi, we will proceed to take logs from the Cowrie honeypot and compute TLSH hashes and semantic bins so that users can easily find similarly malicious activity to known threats in logs.</p>
+<p>Consider the following excerpts from the Cowrie logs the authors above have shared:</p>
+
+<div class="source">
+<div class="source">
+<pre>{
+ "eventid": "cowrie.command.success"
+, "timestamp": "2017-09-18T11:45:25.028091Z"
+, "message": "Command found: /bin/busybox LSUCT"
+, "system": "CowrieTelnetTransport,787,121.237.129.163"
+, "isError": 0
+, "src_ip": "121.237.129.163"
+, "session": "21caf72c6358"
+, "input": "/bin/busybox LSUCT"
+, "sensor": "a927e8b28666"
+}
+</pre></div></div>
+<p>and</p>
+
+<div class="source">
+<div class="source">
+<pre>{
+ "eventid": "cowrie.command.success"
+, "timestamp": "2017-09-17T04:06:39.673206Z"
+, "message": "Command found: /bin/busybox XUSRH"
+, "system": "CowrieTelnetTransport,93,94.51.110.74"
+, "isError": 0
+, "src_ip": "94.51.110.74"
+, "session": "4c047bbc016c"
+, "input": "/bin/busybox XUSRH"
+, "sensor": "a927e8b28666"
+}
+</pre></div></div>
+<p>You will note the <tt>/bin/busybox</tt> call with a random selection afterwards.<br />Excerpting from an analysis of an IOT exploit <a class="externalLink" href="https://isc.sans.edu/diary/21543">here</a>:</p>
+
+<div class="source">
+<div class="source">
+<pre>The use of the command "busybox ECCHI" appears to have two functions.
+First of all, cowrie, and more "complete" Linux distrubtions then
+commonly found on DVRs will respond with a help screen if a wrong module
+is used. So this way, "ECCHI" can be used to detect honeypots and
+irrelevant systems if the reply isn't simply "ECCHI: applet not found".
+Secondly, the command is used as a market to indicate that the prior
+command finished. Later, the attacker adds "/bin/busybox ECCHI" at the
+end of each line, following the actual command to be executed.
+</pre></div></div>
+<p>We have a few options at our disposal:</p>
+
+<ul>
+
+<li>If we were merely filtering and alerting on the execution of <tt>/bin/busybox</tt> we would include false positives.</li>
+
+<li>If we looked at <tt>/bin/busybox XUSRH</tt>, we’d miss many attempts with a <i>different</i> value as <tt>XUSRH</tt> is able to be swapped out for another random sequence to foil overly strict rules.</li>
+
+<li>If we looked for <tt>/bin/busybox *</tt> then we’d capture this scenario well, but it’d be nice to be able to not be specific to detecting the <tt>/bin/busybox</tt> style of exploits.</li>
+</ul>
+<p>Indeed, this is precisely what semantic hashing and binning allows us, the ability to group by semantic similarity without being too specific about what we mean of as “semantic” or “similar”. We want to cast a wide net, but not pull back every fish in the sea.</p>
+<p>For this demonstration, we will </p>
+
+<ul>
+
+<li>ingest some 400 cowrie records</li>
+
+<li>tag records from an IP blacklist for known malicious actors</li>
+
+<li>use the alerts UI to investigate and find similar attacks.</li>
+</ul>
+<div class="section">
+<h2><a name="Preliminaries"></a>Preliminaries</h2>
+<p>We assume that the following environment variables are set:</p>
+
+<ul>
+
+<li><tt>METRON_HOME</tt> - the home directory for metron</li>
+
+<li><tt>ZOOKEEPER</tt> - The zookeeper quorum (comma separated with port specified: e.g. <tt>node1:2181</tt> for full-dev)</li>
+
+<li><tt>BROKERLIST</tt> - The Kafka broker list (comma separated with port specified: e.g. <tt>node1:6667</tt> for full-dev)</li>
+
+<li><tt>ES_HOST</tt> - The elasticsearch master (and port) e.g. <tt>node1:9200</tt> for full-dev.</li>
+</ul>
+<p>Also, this does not assume that you are using a kerberized cluster. If you are, then the parser start command will adjust slightly to include the security protocol.</p>
+<p>Before editing configurations, be sure to pull the configs from zookeeper locally via</p>
+
+<div class="source">
+<div class="source">
+<pre>$METRON_HOME/bin/zk_load_configs.sh --mode PULL -z $ZOOKEEPER -o $METRON_HOME/config/zookeeper/ -f
+</pre></div></div></div>
+<div class="section">
+<h2><a name="Setting_up_the_Data"></a>Setting up the Data</h2>
+<p>First we must set up the cowrie log data in our cluster’s access node.</p>
+
+<ul>
+
+<li>Download the data from the github repository for the talk mentioned above <a class="externalLink" href="https://github.com/fluenda/dataworks_summit_iot_botnet/blob/master/180424243034750.tar.gz">here</a>. Ensure that’s moved into your home directory on the metron node.</li>
+
+<li>Create a directory called <tt>cowrie</tt> in ~ and untar the tarball into that directory via:</li>
+</ul>
+
+<div class="source">
+<div class="source">
+<pre>mkdir ~/cowrie
+cd ~/cowrie
+tar xzvf ~/180424243034750.tar.gz
+</pre></div></div></div>
+<div class="section">
+<h2><a name="Configuring_the_Parser"></a>Configuring the Parser</h2>
+<p>The Cowrie data is coming in as simple JSON blobs, so it’s easy to parse. We really just need to adjust the timestamp and a few fields and we have valid data.</p>
+
+<ul>
+
+<li>Create <tt>$METRON_HOME/config/zookeeper/parsers/cowrie.json</tt> with the following content:</li>
+</ul>
+
+<div class="source">
+<div class="source">
+<pre>{
+ "parserClassName":"org.apache.metron.parsers.json.JSONMapParser",
+ "sensorTopic":"cowrie",
+ "fieldTransformations" : [
+ {
+ "transformation" : "STELLAR"
+ ,"output" : [ "timestamp"]
+ ,"config" : {
+ "timestamp" : "TO_EPOCH_TIMESTAMP( timestamp, 'yyyy-MM-dd\\'T\\'HH:mm:ss.SSS')"
+ }
+ }
+ ]
+
+}
+
+</pre></div></div>
+<p>Before we start, we will want to install ES mappings so ES knows how to interpret our fields:</p>
+
+<div class="source">
+<div class="source">
+<pre>curl -XPUT 'http://$ES_HOST/cowrie*/_mapping/cowrie_doc' -d '
+{
+ "properties" : {
+ "adapter:stellaradapter:begin:ts" : {
+ "type" : "string"
+ },
+ "adapter:stellaradapter:end:ts" : {
+ "type" : "string"
+ },
+ "blacklisted" : {
+ "type" : "boolean"
+ },
+ "compCS" : {
+ "type" : "string"
+ },
+ "data" : {
+ "type" : "string"
+ },
+ "dst_ip" : {
+ "type" : "string"
+ },
+ "dst_port" : {
+ "type" : "long"
+ },
+ "duration" : {
+ "type" : "double"
+ },
+ "encCS" : {
+ "type" : "string"
+ },
+ "enrichmentjoinbolt:joiner:ts" : {
+ "type" : "string"
+ },
+ "enrichmentsplitterbolt:splitter:begin:ts" : {
+ "type" : "string"
+ },
+ "enrichmentsplitterbolt:splitter:end:ts" : {
+ "type" : "string"
+ },
+ "eventid" : {
+ "type" : "string"
+ },
+ "guid" : {
+ "type" : "string"
+ },
+ "input" : {
+ "type" : "string"
+ },
+ "isError" : {
+ "type" : "long"
+ },
+ "is_alert" : {
+ "type" : "string"
+ },
+ "kexAlgs" : {
+ "type" : "string"
+ },
+ "keyAlgs" : {
+ "type" : "string"
+ },
+ "macCS" : {
+ "type" : "string"
+ },
+ "message" : {
+ "type" : "string"
+ },
+ "original_string" : {
+ "type" : "string"
+ },
+ "password" : {
+ "type" : "string"
+ },
+ "sensor" : {
+ "type" : "string"
+ },
+ "session" : {
+ "type" : "string"
+ },
+ "similarity_bin" : {
+ "type" : "string"
+ },
+ "size" : {
+ "type" : "long"
+ },
+ "source:type" : {
+ "type" : "string"
+ },
+ "src_ip" : {
+ "type" : "string"
+ },
+ "src_port" : {
+ "type" : "long"
+ },
+ "system" : {
+ "type" : "string"
+ },
+ "threat:triage:rules:0:comment" : {
+ "type" : "string"
+ },
+ "threat:triage:rules:0:name" : {
+ "type" : "string"
+ },
+ "threat:triage:rules:0:reason" : {
+ "type" : "string"
+ },
+ "threat:triage:rules:0:score" : {
+ "type" : "long"
+ },
+ "threat:triage:score" : {
+ "type" : "double"
+ },
+ "threatinteljoinbolt:joiner:ts" : {
+ "type" : "string"
+ },
+ "threatintelsplitterbolt:splitter:begin:ts" : {
+ "type" : "string"
+ },
+ "threatintelsplitterbolt:splitter:end:ts" : {
+ "type" : "string"
+ },
+ "timestamp" : {
+ "type" : "long"
+ },
+ "tlsh" : {
+ "type" : "string"
+ },
+ "ttylog" : {
+ "type" : "string"
+ },
+ "username" : {
+ "type" : "string"
+ },
+ "version" : {
+ "type" : "string"
+ },
+ "alert" : {
+ "type" : "nested"
+ }
+ }
+}
+'
+</pre></div></div>
+
+<ul>
+
+<li>Create the <tt>cowrie</tt> kafka topic via:</li>
+</ul>
+
+<div class="source">
+<div class="source">
+<pre>/usr/hdp/current/kafka-broker/bin/kafka-topics.sh --zookeeper $ZOOKEEPER --create --topic cowrie --partitions 1 --replication-factor 1
+</pre></div></div></div>
+<div class="section">
+<h2><a name="Import_the_Blacklist"></a>Import the Blacklist</h2>
+<p>Here, to build out a scenario, we will assume that we have a blacklist of known malicious hosts. For our purposes, we’ll choose one particular host IP to be malicious.</p>
+
+<ul>
+
+<li>Create <tt>~/blacklist.csv</tt> to contain the following:</li>
+</ul>
+
+<div class="source">
+<div class="source">
+<pre>94.51.110.74
+</pre></div></div>
+
+<ul>
+
+<li>Create <tt>~/blacklist_extractor.json</tt> to contain the following:</li>
+</ul>
+
+<div class="source">
+<div class="source">
+<pre>{
+ "config" : {
+ "columns" : {
+ "ip" : 0
+ },
+ "indicator_column" : "ip",
+ "type" : "blacklist",
+ "separator" : ","
+ },
+ "extractor" : "CSV"
+}
+</pre></div></div>
+
+<ul>
+
+<li>Import the data <tt>$METRON_HOME/bin/flatfile_loader.sh -i ~/blacklist.csv -t threatintel -c t -e ~/blacklist_extractor.json</tt></li>
+</ul>
+<p>This will create a new enrichment type “blacklist” with a single entry “94.51.110.74”.</p></div>
+<div class="section">
+<h2><a name="Configure_Enrichments"></a>Configure Enrichments</h2>
+<p>We will want to do the following:</p>
+
+<ul>
+
+<li>Add enrichments to faciliate binning
+
+<ul>
+
+<li>Construct what we consider to be a sufficient representation of the thing we want to cluster. For our purposes, this is centered around the input command, so that would be:
+
+<ul>
+
+<li>The <tt>message</tt> field</li>
+
+<li>The <tt>input</tt> field</li>
+
+<li>The <tt>isError</tt> field</li>
+ </ul></li>
+
+<li>Compute the TLSH hash of this representation, called <tt>tlsh</tt></li>
+
+<li>Compute the locality sensitive hash of the TLSH hash suitable for binning, called <tt>similarity_bin</tt></li>
+ </ul></li>
+
+<li>Set up the threat intelligence to use the blacklist
+
+<ul>
+
+<li>Set an alert if the message is from an IP address in the threat intelligence blacklist.</li>
+
+<li>Score blacklisted messages with <tt>10</tt>. In production, this would be more complex.</li>
+ </ul></li>
+</ul>
+<p>Now, we can create the enrichments thusly by creating <tt>$METRON_HOME/config/zookeeper/enrichments/cowrie.json</tt> with the following content:</p>
+
+<div class="source">
+<div class="source">
+<pre>{
+ "enrichment": {
+ "fieldMap": {
+ "stellar" : {
+ "config" : [
+ "characteristic_rep := JOIN([ 'message', exists(message)?message:'', 'input', exists(input)?input:'', 'isError', exists(isError)?isError:''], '|')",
+ "forensic_hashes := HASH(characteristic_rep, 'tlsh', { 'hashes' : 16, 'bucketSize' : 128 })",
+ "similarity_bin := MAP_GET('tlsh_bin', forensic_hashes)",
+ "tlsh := MAP_GET('tlsh', forensic_hashes)",
+ "forensic_hashes := null",
+ "characteristic_rep := null"
+ ]
+ }
+ }
+ ,"fieldToTypeMap": { }
+ },
+ "threatIntel": {
+ "fieldMap": {
+ "stellar" : {
+ "config" : [
+ "blacklisted := ENRICHMENT_EXISTS( 'blacklist', src_ip, 'threatintel', 't')",
+ "is_alert := (exists(is_alert) && is_alert) || blacklisted"
+ ]
+ }
+
+ },
+ "fieldToTypeMap": { },
+ "triageConfig" : {
+ "riskLevelRules" : [
+ {
+ "name" : "Blacklisted Host",
+ "comment" : "Determine if a host is blacklisted",
+ "rule" : "blacklisted != null && blacklisted",
+ "score" : 10,
+ "reason" : "FORMAT('IP %s is blacklisted', src_ip)"
+ }
+ ],
+ "aggregator" : "MAX"
+ }
+ }
+}
+</pre></div></div>
+<div class="section">
+<h3><a name="A_Note_About_Similarity_Hashes_and_TLSH"></a>A Note About Similarity Hashes and TLSH</h3>
+<p>Notice that we have specified a number of hash functions of <tt>16</tt> when constructing the similarity bin.<br />I arrived at that by trial and error, which is not always tenable, frankly. What is more sensible is likely to construct <i>multiple</i> similarity bins of size <tt>8</tt>, <tt>16</tt>, <tt>32</tt> at minimum.</p>
+
+<ul>
+
+<li>The smaller the number of hashes, the more loose the notion of similarity (more possibly dissimilar things would get grouped together).</li>
+
+<li>The larger the number of hashes, the more strict (similar things may not be grouped together).</li>
+</ul></div></div>
+<div class="section">
+<h2><a name="Create_the_Data_Loader"></a>Create the Data Loader</h2>
+<p>We want to pull a snapshot of the cowrie logs, so create <tt>~/load_data.sh</tt> with the following content:</p>
+
+<div class="source">
+<div class="source">
+<pre>COWRIE_HOME=~/cowrie
+for i in cowrie.1626302-1636522.json cowrie.16879981-16892488.json cowrie.21312194-21331475.json cowrie.698260-710913.json cowrie.762933-772239.json cowrie.929866-939552.json cowrie.1246880-1248235.json cowrie.19285959-19295444.json cowrie.16542668-16581213.json cowrie.5849832-5871517.json cowrie.6607473-6609163.json;do
+ echo $i
+ cat $COWRIE_HOME/$i | /usr/hdp/current/kafka-broker/bin/kafka-console-producer.sh --broker-list node1:6667 --topic cowrie
+ sleep 2
+done
+</pre></div></div>
+
+<ul>
+
+<li>Set the <tt>+x</tt> bit on the executable via:</li>
+</ul>
+
+<div class="source">
+<div class="source">
+<pre>chmod +x ~/load_data.sh
+</pre></div></div></div>
+<div class="section">
+<h2><a name="Execute_Demonstration"></a>Execute Demonstration</h2>
+<p>From here, we’ve set up our configuration and can push the configs:</p>
+
+<ul>
+
+<li>Push the configs to zookeeper via</li>
+</ul>
+
+<div class="source">
+<div class="source">
+<pre>$METRON_HOME/bin/zk_load_configs.sh --mode PUSH -z $ZOOKEEPER -i $METRON_HOME/config/zookeeper/
+</pre></div></div>
+
+<ul>
+
+<li>Start the parser via:</li>
+</ul>
+
+<div class="source">
+<div class="source">
+<pre>$METRON_HOME/bin/start_parser_topology.sh -k $BROKERLIST -z $ZOOKEEPER -s cowrie
+</pre></div></div>
+
+<ul>
+
+<li>Push cowrie data into the <tt>cowrie</tt> topic via</li>
+</ul>
+
+<div class="source">
+<div class="source">
+<pre>~/load_data.sh
+</pre></div></div>
+<p>Once this data is loaded, we can use the Alerts UI, starting from known malicious actors, to find others doing similar things.</p>
+
+<ul>
+
+<li>
+<p>First we can look at the alerts directly and find an instance of our <tt>/bin/busybox</tt> activity: <img src="find_alerts.png" alt="Alerts" /></p></li>
+
+<li>
+<p>We can now pivot and look for instances of messages with the same <tt>semantic_hash</tt> but who are <i>not</i> alerts: <img src="clustered.png" alt="Pivot" /></p></li>
+</ul>
+<p>As you can see, we have found a few more malicious actors:</p>
+
+<ul>
+
+<li>177.239.192.172</li>
+
+<li>180.110.69.182</li>
+
+<li>177.238.236.21</li>
+
+<li>94.78.80.45</li>
+</ul>
+<p>Now we can look at <i>other</i> things that they’re doing to build and refine our definition of what an alert is without resorting to hard-coding of rules. Note that nothing in our enrichments actually used the string <tt>busybox</tt>, so this is a more general purpose way of navigating similar things.</p></div>
+ </div>
+ </div>
+ </div>
+
+ <hr/>
+
+ <footer>
+ <div class="container-fluid">
+ <div class="row span12">Copyright © 2018
+ <a href="https://www.apache.org">The Apache Software Foundation</a>.
+ All Rights Reserved.
+
+ </div>
+
+
+
+ </div>
+ </footer>
+ </body>
+</html>
http://git-wip-us.apache.org/repos/asf/metron/blob/3381b853/site/current-book/use-cases/geographic_login_outliers/index.html
----------------------------------------------------------------------
diff --git a/site/current-book/use-cases/geographic_login_outliers/index.html b/site/current-book/use-cases/geographic_login_outliers/index.html
index f1c3cff..4d0ff74 100644
--- a/site/current-book/use-cases/geographic_login_outliers/index.html
+++ b/site/current-book/use-cases/geographic_login_outliers/index.html
@@ -1,13 +1,13 @@
<!DOCTYPE html>
<!--
- | Generated by Apache Maven Doxia at 2017-09-15
+ | Generated by Apache Maven Doxia at 2018-01-03
| Rendered using Apache Maven Fluido Skin 1.3.0
-->
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
- <meta name="Date-Revision-yyyymmdd" content="20170915" />
+ <meta name="Date-Revision-yyyymmdd" content="20180103" />
<meta http-equiv="Content-Language" content="en" />
<title>Metron – Problem Statement</title>
<link rel="stylesheet" href="../../css/apache-maven-fluido-1.3.0.min.css" />
@@ -61,8 +61,8 @@
- <li id="publishDate" class="pull-right">Last Published: 2017-09-15</li> <li class="divider pull-right">|</li>
- <li id="projectVersion" class="pull-right">Version: 0.4.1</li>
+ <li id="publishDate" class="pull-right">Last Published: 2018-01-03</li> <li class="divider pull-right">|</li>
+ <li id="projectVersion" class="pull-right">Version: 0.4.2</li>
</ul>
</div>
@@ -103,7 +103,7 @@
<i class="none"></i>
Docker</a>
</li>
-
+
<li>
<a href="../../metron-deployment/index.html" title="Deployment">
@@ -131,20 +131,27 @@
<i class="none"></i>
Rest</a>
</li>
-
+
<li>
<a href="../../metron-platform/index.html" title="Platform">
<i class="icon-chevron-right"></i>
Platform</a>
</li>
-
+
<li>
<a href="../../metron-sensors/index.html" title="Sensors">
<i class="icon-chevron-right"></i>
Sensors</a>
</li>
+
+ <li>
+
+ <a href="../../metron-stellar/stellar-3rd-party-example/index.html" title="Stellar-3rd-party-example">
+ <i class="none"></i>
+ Stellar-3rd-party-example</a>
+ </li>
<li>
@@ -152,7 +159,7 @@
<i class="icon-chevron-right"></i>
Stellar-common</a>
</li>
-
+
<li>
<a href="../../use-cases/index.html" title="Use-cases">
@@ -160,6 +167,13 @@
Use-cases</a>
<ul class="nav nav-list">
+ <li>
+
+ <a href="../../use-cases/forensic_clustering/index.html" title="Forensic_clustering">
+ <i class="none"></i>
+ Forensic_clustering</a>
+ </li>
+
<li class="active">
<a href="#"><i class="none"></i>Geographic_login_outliers</a>
@@ -443,7 +457,7 @@ if __name__ == '__main__':
<div class="source">
<div class="source">
-<pre>$METRON_HOME/bin/zk_load_configs.sh --mode PUSH -z node1:2181 -i $METRON_HOME/config/zookeeper/
+<pre>$METRON_HOME/bin/zk_load_configs.sh --mode PUSH -z $ZOOKEEPER -i $METRON_HOME/config/zookeeper/
</pre></div></div>
<ul>
@@ -463,7 +477,8 @@ if __name__ == '__main__':
<div class="source">
<div class="source">
-<pre>python ~/gen_data.py | /usr/hdp/current/kafka-broker/bin/kafka-console-producer.sh --broker-list node1:6667 --topic auth
+<pre>python ~/gen_data.py |
+/usr/hdp/current/kafka-broker/bin/kafka-console-producer.sh --broker-list $BROKERLIST --topic auth
</pre></div></div>
<ul>
@@ -518,7 +533,7 @@ if __name__ == '__main__':
<footer>
<div class="container-fluid">
- <div class="row span12">Copyright © 2017
+ <div class="row span12">Copyright © 2018
<a href="https://www.apache.org">The Apache Software Foundation</a>.
All Rights Reserved.
http://git-wip-us.apache.org/repos/asf/metron/blob/3381b853/site/current-book/use-cases/index.html
----------------------------------------------------------------------
diff --git a/site/current-book/use-cases/index.html b/site/current-book/use-cases/index.html
index 7db5f99..e8c6b28 100644
--- a/site/current-book/use-cases/index.html
+++ b/site/current-book/use-cases/index.html
@@ -1,13 +1,13 @@
<!DOCTYPE html>
<!--
- | Generated by Apache Maven Doxia at 2017-09-15
+ | Generated by Apache Maven Doxia at 2018-01-03
| Rendered using Apache Maven Fluido Skin 1.3.0
-->
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
- <meta name="Date-Revision-yyyymmdd" content="20170915" />
+ <meta name="Date-Revision-yyyymmdd" content="20180103" />
<meta http-equiv="Content-Language" content="en" />
<title>Metron – Worked Examples</title>
<link rel="stylesheet" href="../css/apache-maven-fluido-1.3.0.min.css" />
@@ -61,8 +61,8 @@
- <li id="publishDate" class="pull-right">Last Published: 2017-09-15</li> <li class="divider pull-right">|</li>
- <li id="projectVersion" class="pull-right">Version: 0.4.1</li>
+ <li id="publishDate" class="pull-right">Last Published: 2018-01-03</li> <li class="divider pull-right">|</li>
+ <li id="projectVersion" class="pull-right">Version: 0.4.2</li>
</ul>
</div>
@@ -103,7 +103,7 @@
<i class="none"></i>
Docker</a>
</li>
-
+
<li>
<a href="../metron-deployment/index.html" title="Deployment">
@@ -131,20 +131,27 @@
<i class="none"></i>
Rest</a>
</li>
-
+
<li>
<a href="../metron-platform/index.html" title="Platform">
<i class="icon-chevron-right"></i>
Platform</a>
</li>
-
+
<li>
<a href="../metron-sensors/index.html" title="Sensors">
<i class="icon-chevron-right"></i>
Sensors</a>
</li>
+
+ <li>
+
+ <a href="../metron-stellar/stellar-3rd-party-example/index.html" title="Stellar-3rd-party-example">
+ <i class="none"></i>
+ Stellar-3rd-party-example</a>
+ </li>
<li>
@@ -152,7 +159,7 @@
<i class="icon-chevron-right"></i>
Stellar-common</a>
</li>
-
+
<li class="active">
<a href="#"><i class="icon-chevron-down"></i>Use-cases</a>
@@ -160,6 +167,13 @@
<li>
+ <a href="../use-cases/forensic_clustering/index.html" title="Forensic_clustering">
+ <i class="none"></i>
+ Forensic_clustering</a>
+ </li>
+
+ <li>
+
<a href="../use-cases/geographic_login_outliers/index.html" title="Geographic_login_outliers">
<i class="none"></i>
Geographic_login_outliers</a>
@@ -199,7 +213,7 @@
<footer>
<div class="container-fluid">
- <div class="row span12">Copyright © 2017
+ <div class="row span12">Copyright © 2018
<a href="https://www.apache.org">The Apache Software Foundation</a>.
All Rights Reserved.
http://git-wip-us.apache.org/repos/asf/metron/blob/3381b853/site/documentation/index.md
----------------------------------------------------------------------
diff --git a/site/documentation/index.md b/site/documentation/index.md
index 723e548..6b38ed6 100644
--- a/site/documentation/index.md
+++ b/site/documentation/index.md
@@ -74,24 +74,24 @@ title: Apache Metron Documentation
<section class="events-section info-block darken" id="releases">
<div class="wrapper downloads">
<div class="text-center">
- <h4>Current Release: 0.4.1</h4>
+ <h4>Current Release: 0.4.2</h4>
</div>
<div class="content-960 hover-btn text-center">
<ul>
<li>
- <a href="http://www.apache.org/dyn/closer.cgi/metron/0.4.1/apache-metron-0.4.1.tar.gz" target="new">
- apache-metron-0.4.1.tar.gz
+ <a href="http://www.apache.org/dyn/closer.cgi/metron/0.4.2/apache-metron-0.4.2.tar.gz" target="new">
+ apache-metron-0.4.2.tar.gz
</a>
- [ <a href="https://dist.apache.org/repos/dist/release/metron/0.4.1/apache-metron-0.4.1.tar.gz.asc" target="new">
+ [ <a href="https://dist.apache.org/repos/dist/release/metron/0.4.2/apache-metron-0.4.2.tar.gz.asc" target="new">
PGP
</a>]
- [ <a href="https://dist.apache.org/repos/dist/release/metron/0.4.1/apache-metron-0.4.1.tar.gz.sha1" target="new">
+ [ <a href="https://dist.apache.org/repos/dist/release/metron/0.4.2/apache-metron-0.4.2.tar.gz.sha1" target="new">
SHA
</a>]
- [ <a href="https://dist.apache.org/repos/dist/release/metron/0.4.1/apache-metron-0.4.1.tar.gz.sha512" target="new">
+ [ <a href="https://dist.apache.org/repos/dist/release/metron/0.4.2/apache-metron-0.4.2.tar.gz.sha512" target="new">
SHA512
</a>]
- [ <a href="https://dist.apache.org/repos/dist/release/metron/0.4.1/apache-metron-0.4.1.tar.gz.md5" target="new">
+ [ <a href="https://dist.apache.org/repos/dist/release/metron/0.4.2/apache-metron-0.4.2.tar.gz.md5" target="new">
MD5
</a>]
[ <a href="/current-book/index.html" target="new"> Book </a>]
[6/7] metron git commit: METRON-1388 update public web site to point
at 0.4.2 new release (mattf-horton) closes apache/metron#887
Posted by ma...@apache.org.
http://git-wip-us.apache.org/repos/asf/metron/blob/3381b853/site/current-book/metron-deployment/Kerberos-manual-setup.html
----------------------------------------------------------------------
diff --git a/site/current-book/metron-deployment/Kerberos-manual-setup.html b/site/current-book/metron-deployment/Kerberos-manual-setup.html
index c07156d..1c1ed0a 100644
--- a/site/current-book/metron-deployment/Kerberos-manual-setup.html
+++ b/site/current-book/metron-deployment/Kerberos-manual-setup.html
@@ -1,13 +1,13 @@
<!DOCTYPE html>
<!--
- | Generated by Apache Maven Doxia at 2017-09-15
+ | Generated by Apache Maven Doxia at 2018-01-03
| Rendered using Apache Maven Fluido Skin 1.3.0
-->
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
- <meta name="Date-Revision-yyyymmdd" content="20170915" />
+ <meta name="Date-Revision-yyyymmdd" content="20180103" />
<meta http-equiv="Content-Language" content="en" />
<title>Metron – Kerberos Setup</title>
<link rel="stylesheet" href="../css/apache-maven-fluido-1.3.0.min.css" />
@@ -61,8 +61,8 @@
- <li id="publishDate" class="pull-right">Last Published: 2017-09-15</li> <li class="divider pull-right">|</li>
- <li id="projectVersion" class="pull-right">Version: 0.4.1</li>
+ <li id="publishDate" class="pull-right">Last Published: 2018-01-03</li> <li class="divider pull-right">|</li>
+ <li id="projectVersion" class="pull-right">Version: 0.4.2</li>
</ul>
</div>
@@ -103,7 +103,7 @@
<i class="none"></i>
Docker</a>
</li>
-
+
<li>
<a href="../metron-deployment/index.html" title="Deployment">
@@ -171,7 +171,7 @@
<i class="icon-chevron-right"></i>
Roles</a>
</li>
-
+
<li>
<a href="../metron-deployment/vagrant/index.html" title="Vagrant">
@@ -201,20 +201,27 @@
<i class="none"></i>
Rest</a>
</li>
-
+
<li>
<a href="../metron-platform/index.html" title="Platform">
<i class="icon-chevron-right"></i>
Platform</a>
</li>
-
+
<li>
<a href="../metron-sensors/index.html" title="Sensors">
<i class="icon-chevron-right"></i>
Sensors</a>
</li>
+
+ <li>
+
+ <a href="../metron-stellar/stellar-3rd-party-example/index.html" title="Stellar-3rd-party-example">
+ <i class="none"></i>
+ Stellar-3rd-party-example</a>
+ </li>
<li>
@@ -222,7 +229,7 @@
<i class="icon-chevron-right"></i>
Stellar-common</a>
</li>
-
+
<li>
<a href="../use-cases/index.html" title="Use-cases">
@@ -252,7 +259,7 @@
<div id="bodyColumn" class="span9" >
<h1>Kerberos Setup</h1>
-<p>This document provides instructions for kerberizing Metron’s Vagrant-based development environments; “Quick Dev” and “Full Dev”. These instructions do not cover the Ambari MPack or sensors. General Kerberization notes can be found in the metron-deployment <a href="../index.html">README.md</a>.</p>
+<p>This document provides instructions for kerberizing Metron’s Vagrant-based development environments. These instructions do not cover the Ambari MPack or sensors. General Kerberization notes can be found in the metron-deployment <a href="../index.html">README.md</a>.</p>
<ul>
@@ -282,7 +289,7 @@
<ol style="list-style-type: decimal">
<li>
-<p>Deploy a Vagrant development environment; either <a href="vagrant/full-dev-platform/index.html">Full Dev</a> or <a href="vagrant/quick-dev-platform/index.html">Quick Dev</a>.</p></li>
+<p>Deploy the <a href="vagrant/full-dev-platform/index.html">development environment.</a>.</p></li>
<li>
<p>Export the following environment variables. These need to be set for the remainder of the instructions. Replace <tt>node1</tt> with the appropriate hosts, if you are running Metron anywhere other than Vagrant.</p>
@@ -297,7 +304,7 @@ export ELASTICSEARCH=node1:9200
export BROKERLIST=node1:6667
export HDP_HOME="/usr/hdp/current"
export KAFKA_HOME="${HDP_HOME}/kafka-broker"
-export METRON_VERSION="0.4.1"
+export METRON_VERSION="0.4.2"
export METRON_HOME="/usr/metron/${METRON_VERSION}"
</pre></div></div></li>
@@ -343,16 +350,18 @@ sudo -u hdfs hdfs dfs -chmod 770 /user/metron
</pre></div></div></li>
<li>
-<p>Define the host, <tt>node1</tt>, as the KDC.</p>
+<p>Define the current host as the KDC.</p>
<div class="source">
<div class="source">
-<pre>sed -i 's/kerberos.example.com/node1/g' /etc/krb5.conf
+<pre>KDC=`hostname`
+sed -i.orig 's/kerberos.example.com/'"$KDC"'/g' /etc/krb5.conf
cp -f /etc/krb5.conf /var/lib/ambari-server/resources/scripts
</pre></div></div></li>
<li>
-<p>Ensure the KDC can issue renewable tickets. This can be necessary on a real cluster, but should not be on full-dev. In /var/kerberos/krb5kdc/kdc.conf ensure the following is in the realm section</p>
+<p>Ensure that the KDC can issue renewable tickets. This may be necessary on a real cluster, but should not be on <a href="vagrant/full-dev-platform/index.html">Full Dev</a>.</p>
+<p>Edit <tt>/var/kerberos/krb5kdc/kdc.conf</tt> and ensure the following is added to the <tt>realm</tt> section</p>
<div class="source">
<div class="source">
@@ -360,49 +369,86 @@ cp -f /etc/krb5.conf /var/lib/ambari-server/resources/scripts
</pre></div></div></li>
<li>
-<p>Do not copy/paste this full set of commands as the <tt>kdb5_util</tt> command will not run as expected. Run the commands individually to ensure they all execute. This step takes a moment. It creates the kerberos database.</p>
+<p>Create the KDC principal database. You will be prompted for a password. This step takes a moment.</p>
<div class="source">
<div class="source">
<pre>kdb5_util create -s
-/etc/rc.d/init.d/krb5kdc start
-chkconfig krb5kdc on
-/etc/rc.d/init.d/kadmin start
+</pre></div></div></li>
+
+<li>
+<p>Start the KDC and ensure that it starts on boot.</p>
+
+<div class="source">
+<div class="source">
+<pre>/etc/rc.d/init.d/krb5kdc start
+chkconfig krb5kdc on
+</pre></div></div></li>
+
+<li>
+<p>Start the Kerberos Admin service and ensure that it starts on boot. </p>
+
+<div class="source">
+<div class="source">
+<pre>/etc/rc.d/init.d/kadmin start
chkconfig kadmin on
</pre></div></div></li>
<li>
-<p>Setup the <tt>admin</tt> and <tt>metron</tt> principals. You’ll <tt>kinit</tt> as the <tt>metron</tt> principal when running topologies. Make sure to remember the passwords.</p>
+<p>Setup the <tt>admin</tt> principal. You will be prompted for a password; do not forget it.</p>
<div class="source">
<div class="source">
<pre>kadmin.local -q "addprinc admin/admin"
-kadmin.local -q "addprinc metron"
+</pre></div></div></li>
+
+<li>
+<p>Setup the <tt>metron</tt> principal. You will <tt>kinit</tt> as the <tt>metron</tt> principal when running topologies. You will be prompted for a password; do not forget it.</p>
+
+<div class="source">
+<div class="source">
+<pre>kadmin.local -q "addprinc metron"
</pre></div></div></li>
</ol></div>
<div class="section">
<h2><a name="Verify_KDC"></a>Verify KDC</h2>
-<p>Ticket renewal is by default disallowed in many linux distributions. If the KDC cannot issue renewable tickets, an error will be thrown when starting Metron’s Storm topologies:</p>
+<ol style="list-style-type: decimal">
+
+<li>
+<p>Ticket renewal is disallowed by default in many Linux distributions. If the KDC cannot issue renewable tickets, an error will be thrown when starting Metron’s Storm topologies:</p>
+
<div class="source">
<div class="source">
-<pre>Exception in thread "main" java.lang.RuntimeException: java.lang.RuntimeException: The TGT found is not renewable
-</pre></div></div>
-<p>Ensure the Metron keytab is renewable. Look for the ‘R’ flag from the following command</p>
-
+<pre>Exception in thread "main" java.lang.RuntimeException:
+java.lang.RuntimeException: The TGT found is not renewable
+</pre></div></div></li>
+
+<li>
+<p>Ensure the Metron keytab is renewable. Look for the ‘R’ flag in the output of the following command.</p>
+
<div class="source">
<div class="source">
<pre>klist -f
</pre></div></div>
-<p>If the ‘R’ flags are present, you may skip to next section.</p>
-<p>If the ‘R’ flags are absent, you will need to follow the below steps: If the KDC is already setup, then editing max_life and max_renewable_life in <tt>/var/kerberos/krb5kdc/kdc.conf</tt>, and restarting kadmin and krb5kdc services will not change the policies for existing users. </p>
-<p>You need to set the renew lifetime for existing users and krbtgt realm. Modify the appropriate principals to allow renewable tickets using the following commands. Adjust the parameters to match your desired KDC parameters:</p>
-
+
+<ul>
+
+<li>If the ‘R’ flags are present, you may skip to next section.</li>
+
+<li>If the ‘R’ flags are absent, you will need to follow the below steps:</li>
+ </ul></li>
+
+<li>
+<p>If the KDC is already setup, then editing <tt>max_life</tt> and <tt>max_renewable_life</tt> in <tt>/var/kerberos/krb5kdc/kdc.conf</tt>, then restarting <tt>kadmin</tt> and <tt>krb5kdc</tt> services will not change the policies for existing users.</p>
+<p>You need to set the renew lifetime for existing users and the <tt>krbtgt</tt> realm. Modify the appropriate principals to allow renewable tickets using the following commands. Adjust the parameters to match your desired KDC parameters:</p>
+
<div class="source">
<div class="source">
<pre>kadmin.local -q "modprinc -maxlife 1days -maxrenewlife 7days +allow_renewable krbtgt/EXAMPLE.COM@EXAMPLE.COM"
kadmin.local -q "modprinc -maxlife 1days -maxrenewlife 7days +allow_renewable metron@EXAMPLE.COM"
-</pre></div></div></div>
+</pre></div></div></li>
+</ol></div>
<div class="section">
<h2><a name="Enable_Kerberos"></a>Enable Kerberos</h2>
@@ -846,7 +892,7 @@ done
<footer>
<div class="container-fluid">
- <div class="row span12">Copyright © 2017
+ <div class="row span12">Copyright © 2018
<a href="https://www.apache.org">The Apache Software Foundation</a>.
All Rights Reserved.
http://git-wip-us.apache.org/repos/asf/metron/blob/3381b853/site/current-book/metron-deployment/amazon-ec2/index.html
----------------------------------------------------------------------
diff --git a/site/current-book/metron-deployment/amazon-ec2/index.html b/site/current-book/metron-deployment/amazon-ec2/index.html
index e81f4f5..4e7e771 100644
--- a/site/current-book/metron-deployment/amazon-ec2/index.html
+++ b/site/current-book/metron-deployment/amazon-ec2/index.html
@@ -1,13 +1,13 @@
<!DOCTYPE html>
<!--
- | Generated by Apache Maven Doxia at 2017-09-15
+ | Generated by Apache Maven Doxia at 2018-01-03
| Rendered using Apache Maven Fluido Skin 1.3.0
-->
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
- <meta name="Date-Revision-yyyymmdd" content="20170915" />
+ <meta name="Date-Revision-yyyymmdd" content="20180103" />
<meta http-equiv="Content-Language" content="en" />
<title>Metron – Apache Metron on Amazon EC2</title>
<link rel="stylesheet" href="../../css/apache-maven-fluido-1.3.0.min.css" />
@@ -61,8 +61,8 @@
- <li id="publishDate" class="pull-right">Last Published: 2017-09-15</li> <li class="divider pull-right">|</li>
- <li id="projectVersion" class="pull-right">Version: 0.4.1</li>
+ <li id="publishDate" class="pull-right">Last Published: 2018-01-03</li> <li class="divider pull-right">|</li>
+ <li id="projectVersion" class="pull-right">Version: 0.4.2</li>
</ul>
</div>
@@ -103,7 +103,7 @@
<i class="none"></i>
Docker</a>
</li>
-
+
<li>
<a href="../../metron-deployment/index.html" title="Deployment">
@@ -171,7 +171,7 @@
<i class="icon-chevron-right"></i>
Roles</a>
</li>
-
+
<li>
<a href="../../metron-deployment/vagrant/index.html" title="Vagrant">
@@ -201,20 +201,27 @@
<i class="none"></i>
Rest</a>
</li>
-
+
<li>
<a href="../../metron-platform/index.html" title="Platform">
<i class="icon-chevron-right"></i>
Platform</a>
</li>
-
+
<li>
<a href="../../metron-sensors/index.html" title="Sensors">
<i class="icon-chevron-right"></i>
Sensors</a>
</li>
+
+ <li>
+
+ <a href="../../metron-stellar/stellar-3rd-party-example/index.html" title="Stellar-3rd-party-example">
+ <i class="none"></i>
+ Stellar-3rd-party-example</a>
+ </li>
<li>
@@ -222,7 +229,7 @@
<i class="icon-chevron-right"></i>
Stellar-common</a>
</li>
-
+
<li>
<a href="../../use-cases/index.html" title="Use-cases">
@@ -582,7 +589,7 @@ fatal: [ec2-52-26-113-221.us-west-2.compute.amazonaws.com]: UNREACHABLE! => {
<footer>
<div class="container-fluid">
- <div class="row span12">Copyright © 2017
+ <div class="row span12">Copyright © 2018
<a href="https://www.apache.org">The Apache Software Foundation</a>.
All Rights Reserved.
http://git-wip-us.apache.org/repos/asf/metron/blob/3381b853/site/current-book/metron-deployment/index.html
----------------------------------------------------------------------
diff --git a/site/current-book/metron-deployment/index.html b/site/current-book/metron-deployment/index.html
index d1071d7..72723a6 100644
--- a/site/current-book/metron-deployment/index.html
+++ b/site/current-book/metron-deployment/index.html
@@ -1,13 +1,13 @@
<!DOCTYPE html>
<!--
- | Generated by Apache Maven Doxia at 2017-09-15
+ | Generated by Apache Maven Doxia at 2018-01-03
| Rendered using Apache Maven Fluido Skin 1.3.0
-->
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
- <meta name="Date-Revision-yyyymmdd" content="20170915" />
+ <meta name="Date-Revision-yyyymmdd" content="20180103" />
<meta http-equiv="Content-Language" content="en" />
<title>Metron – Overview</title>
<link rel="stylesheet" href="../css/apache-maven-fluido-1.3.0.min.css" />
@@ -61,8 +61,8 @@
- <li id="publishDate" class="pull-right">Last Published: 2017-09-15</li> <li class="divider pull-right">|</li>
- <li id="projectVersion" class="pull-right">Version: 0.4.1</li>
+ <li id="publishDate" class="pull-right">Last Published: 2018-01-03</li> <li class="divider pull-right">|</li>
+ <li id="projectVersion" class="pull-right">Version: 0.4.2</li>
</ul>
</div>
@@ -103,7 +103,7 @@
<i class="none"></i>
Docker</a>
</li>
-
+
<li class="active">
<a href="#"><i class="icon-chevron-down"></i>Deployment</a>
@@ -171,7 +171,7 @@
<i class="icon-chevron-right"></i>
Roles</a>
</li>
-
+
<li>
<a href="../metron-deployment/vagrant/index.html" title="Vagrant">
@@ -201,20 +201,27 @@
<i class="none"></i>
Rest</a>
</li>
-
+
<li>
<a href="../metron-platform/index.html" title="Platform">
<i class="icon-chevron-right"></i>
Platform</a>
</li>
-
+
<li>
<a href="../metron-sensors/index.html" title="Sensors">
<i class="icon-chevron-right"></i>
Sensors</a>
</li>
+
+ <li>
+
+ <a href="../metron-stellar/stellar-3rd-party-example/index.html" title="Stellar-3rd-party-example">
+ <i class="none"></i>
+ Stellar-3rd-party-example</a>
+ </li>
<li>
@@ -222,7 +229,7 @@
<i class="icon-chevron-right"></i>
Stellar-common</a>
</li>
-
+
<li>
<a href="../use-cases/index.html" title="Use-cases">
@@ -306,7 +313,7 @@
<p>Currently, the playbooks supports building a local development cluster running on one node or deploying to a 10 node cluster on AWS EC2.</p></div>
<div class="section">
<h2><a name="Vagrant"></a>Vagrant</h2>
-<p>There are current two Vagrant modes, full-dev and quick-dev. Full-dev installs the entire Ambari/Metron stack. This is useful in testing out changes to the installation procedure. Quick-dev re-installs the core Metron Services (Parsing, Enrichment, and Indexing)on a pre-built instance. Use quick-dev for testing out changes to core Metron services.</p>
+<p>There is a development environment based on Vagrant that is referred to as “Full Dev”. This installs the entire Ambari/Metron stack. This is useful in testing out changes to the installation procedure.</p>
<div class="section">
<h3><a name="Prerequsities"></a>Prerequsities</h3>
@@ -318,10 +325,7 @@
</ul></div>
<div class="section">
<h3><a name="Full-Dev"></a>Full-Dev</h3>
-<p>Navigate to <tt>metron/metron-deployment/vagrant/full-dev-platform</tt> and run <tt>vagrant up</tt>.</p></div>
-<div class="section">
-<h3><a name="Quick-Dev"></a>Quick-Dev</h3>
-<p>Navigate to <tt>metron/metron-deployment/vagrant/quick-dev-platform</tt> and run <tt>vagrant up</tt>.</p></div></div>
+<p>Navigate to <tt>metron/metron-deployment/vagrant/full-dev-platform</tt> and run <tt>vagrant up</tt>.</p></div></div>
<div class="section">
<h2><a name="Ambari_Management_Pack"></a>Ambari Management Pack</h2>
<p>An Ambari Management Pack can be built in order to make the Metron service available on top of an existing stack, rather than needing a direct stack update.</p>
@@ -348,7 +352,7 @@
<li>A cluster managed by Ambari 2.4.2+</li>
-<li>Metron RPMs available on the cluster in the /localrepo directory. See <a href="#RPM">RPM</a> for further information.</li>
+<li>Metron RPMs available on the cluster in the /localrepo directory. See <a href="#RPMs">RPMs</a> for further information.</li>
<li><a class="externalLink" href="https://nodejs.org/en/download/package-manager/">Node.js</a> repository installed on the Management UI host</li>
</ul></div>
@@ -467,7 +471,7 @@ mvn clean package -Pbuild-rpms
<div class="section">
<h2><a name="Kibana_Dashboards"></a>Kibana Dashboards</h2>
<p>The dashboards installed by the Kibana custom action are managed by the dashboard.p file. This file is created by exporting existing dashboards from a running Kibana instance.</p>
-<p>To create a new version of the file, make any necessary changes to Kibana (e.g. on quick-dev), and export with the appropriate script.</p>
+<p>To create a new version of the file, make any necessary changes to Kibana (e.g. on full-dev), and export with the appropriate script.</p>
<div class="source">
<div class="source">
@@ -522,7 +526,7 @@ packaging/ambari/metron-mpack/src/main/resources/common-services/KIBANA/4.5.1/pa
<footer>
<div class="container-fluid">
- <div class="row span12">Copyright © 2017
+ <div class="row span12">Copyright © 2018
<a href="https://www.apache.org">The Apache Software Foundation</a>.
All Rights Reserved.
http://git-wip-us.apache.org/repos/asf/metron/blob/3381b853/site/current-book/metron-deployment/other-examples/index.html
----------------------------------------------------------------------
diff --git a/site/current-book/metron-deployment/other-examples/index.html b/site/current-book/metron-deployment/other-examples/index.html
index 7dd9888..3cf67e2 100644
--- a/site/current-book/metron-deployment/other-examples/index.html
+++ b/site/current-book/metron-deployment/other-examples/index.html
@@ -1,13 +1,13 @@
<!DOCTYPE html>
<!--
- | Generated by Apache Maven Doxia at 2017-09-15
+ | Generated by Apache Maven Doxia at 2018-01-03
| Rendered using Apache Maven Fluido Skin 1.3.0
-->
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
- <meta name="Date-Revision-yyyymmdd" content="20170915" />
+ <meta name="Date-Revision-yyyymmdd" content="20180103" />
<meta http-equiv="Content-Language" content="en" />
<title>Metron – Other Example Deployments</title>
<link rel="stylesheet" href="../../css/apache-maven-fluido-1.3.0.min.css" />
@@ -61,8 +61,8 @@
- <li id="publishDate" class="pull-right">Last Published: 2017-09-15</li> <li class="divider pull-right">|</li>
- <li id="projectVersion" class="pull-right">Version: 0.4.1</li>
+ <li id="publishDate" class="pull-right">Last Published: 2018-01-03</li> <li class="divider pull-right">|</li>
+ <li id="projectVersion" class="pull-right">Version: 0.4.2</li>
</ul>
</div>
@@ -103,7 +103,7 @@
<i class="none"></i>
Docker</a>
</li>
-
+
<li>
<a href="../../metron-deployment/index.html" title="Deployment">
@@ -180,7 +180,7 @@
<i class="icon-chevron-right"></i>
Roles</a>
</li>
-
+
<li>
<a href="../../metron-deployment/vagrant/index.html" title="Vagrant">
@@ -210,20 +210,27 @@
<i class="none"></i>
Rest</a>
</li>
-
+
<li>
<a href="../../metron-platform/index.html" title="Platform">
<i class="icon-chevron-right"></i>
Platform</a>
</li>
-
+
<li>
<a href="../../metron-sensors/index.html" title="Sensors">
<i class="icon-chevron-right"></i>
Sensors</a>
</li>
+
+ <li>
+
+ <a href="../../metron-stellar/stellar-3rd-party-example/index.html" title="Stellar-3rd-party-example">
+ <i class="none"></i>
+ Stellar-3rd-party-example</a>
+ </li>
<li>
@@ -231,7 +238,7 @@
<i class="icon-chevron-right"></i>
Stellar-common</a>
</li>
-
+
<li>
<a href="../../use-cases/index.html" title="Use-cases">
@@ -276,7 +283,7 @@
<footer>
<div class="container-fluid">
- <div class="row span12">Copyright © 2017
+ <div class="row span12">Copyright © 2018
<a href="https://www.apache.org">The Apache Software Foundation</a>.
All Rights Reserved.
http://git-wip-us.apache.org/repos/asf/metron/blob/3381b853/site/current-book/metron-deployment/other-examples/manual-install/Manual_Install_CentOS6.html
----------------------------------------------------------------------
diff --git a/site/current-book/metron-deployment/other-examples/manual-install/Manual_Install_CentOS6.html b/site/current-book/metron-deployment/other-examples/manual-install/Manual_Install_CentOS6.html
index 62128e1..eeadfe6 100644
--- a/site/current-book/metron-deployment/other-examples/manual-install/Manual_Install_CentOS6.html
+++ b/site/current-book/metron-deployment/other-examples/manual-install/Manual_Install_CentOS6.html
@@ -1,13 +1,13 @@
<!DOCTYPE html>
<!--
- | Generated by Apache Maven Doxia at 2017-09-15
+ | Generated by Apache Maven Doxia at 2018-01-03
| Rendered using Apache Maven Fluido Skin 1.3.0
-->
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
- <meta name="Date-Revision-yyyymmdd" content="20170915" />
+ <meta name="Date-Revision-yyyymmdd" content="20180103" />
<meta http-equiv="Content-Language" content="en" />
<title>Metron – </title>
<link rel="stylesheet" href="../../../css/apache-maven-fluido-1.3.0.min.css" />
@@ -61,8 +61,8 @@
- <li id="publishDate" class="pull-right">Last Published: 2017-09-15</li> <li class="divider pull-right">|</li>
- <li id="projectVersion" class="pull-right">Version: 0.4.1</li>
+ <li id="publishDate" class="pull-right">Last Published: 2018-01-03</li> <li class="divider pull-right">|</li>
+ <li id="projectVersion" class="pull-right">Version: 0.4.2</li>
</ul>
</div>
@@ -103,7 +103,7 @@
<i class="none"></i>
Docker</a>
</li>
-
+
<li>
<a href="../../../metron-deployment/index.html" title="Deployment">
@@ -180,7 +180,7 @@
<i class="icon-chevron-right"></i>
Roles</a>
</li>
-
+
<li>
<a href="../../../metron-deployment/vagrant/index.html" title="Vagrant">
@@ -210,20 +210,27 @@
<i class="none"></i>
Rest</a>
</li>
-
+
<li>
<a href="../../../metron-platform/index.html" title="Platform">
<i class="icon-chevron-right"></i>
Platform</a>
</li>
-
+
<li>
<a href="../../../metron-sensors/index.html" title="Sensors">
<i class="icon-chevron-right"></i>
Sensors</a>
</li>
+
+ <li>
+
+ <a href="../../../metron-stellar/stellar-3rd-party-example/index.html" title="Stellar-3rd-party-example">
+ <i class="none"></i>
+ Stellar-3rd-party-example</a>
+ </li>
<li>
@@ -231,7 +238,7 @@
<i class="icon-chevron-right"></i>
Stellar-common</a>
</li>
-
+
<li>
<a href="../../../use-cases/index.html" title="Use-cases">
@@ -310,14 +317,20 @@
<div class="source">
<div class="source">
<pre># cat /dev/zero | ssh-keygen -q -N "" 2>/dev/null
+# cd ~/.ssh
+# cat id_rsa.pub >> authorized_keys
</pre></div></div>
-<p>If you’re not installing on a single node, add this newly generated key to all the slave nodes:</p>
+
+<ul>
+
+<li>If you’re not installing on a single node, add this newly generated key to all the slave nodes:</li>
+</ul>
<div class="source">
<div class="source">
<pre>ssh-copy-id -i ~/.ssh/id_rsa.pub <replace_with_node_ip>
</pre></div></div>
-<p><i>Side note:</i> You might have to adapt your sshd_config file and add “PermitRootLogin yes” amongst other parameters if you want passwordless root access, but that’s outside the scope of this document.</p>
+<p><b>Side note:</b> You might have to adapt your sshd_config file and add “PermitRootLogin yes” amongst other parameters if you want passwordless root access, but that’s outside the scope of this document.</p>
<ul>
@@ -376,6 +389,24 @@ Afterwards, run:
# grub-install /dev/sda
</pre></div></div>
+
+<ul>
+
+<li>If you do not want to mess with grub/kernel parameters, add the following to /etc/rc.local:</li>
+</ul>
+
+<div class="source">
+<div class="source">
+<pre>vim /etc/rc.local:
+# Disable THP at boot time
+if test -f /sys/kernel/mm/redhat_transparent_hugepage/enabled; then
+ echo never > /sys/kernel/mm/redhat_transparent_hugepage/enabled
+fi
+
+if test -f /sys/kernel/mm/redhat_transparent_hugepage/defrag; then
+ echo never > /sys/kernel/mm/redhat_transparent_hugepage/defrag
+fi
+</pre></div></div>
<p>After reboot check that changes were applied (make sure that word “never” is selected in square-brackets):</p>
<div class="source">
@@ -393,7 +424,7 @@ always madvise [never]
<div class="source">
<div class="source">
-<pre># yum install git wget curl rpm tar unzip bzip2 wget createrepo yum-utils ntp python-pip psutils python-psutil ntp libffi-devel gcc openssl-devel -y
+<pre># yum install git wget curl rpm tar unzip bzip2 wget createrepo yum-utils ntp python-pip psutils python-psutil ntp libffi-devel gcc openssl-devel npm -y
# pip install --upgrade pip
# pip install requests urllib
# pip install --upgrade setuptools
@@ -401,7 +432,7 @@ always madvise [never]
<ul>
-<li>Install Maven on main node and on Metron node install java 1.8 (if you don’t know which it is, install it everywhere):</li>
+<li>Install Maven 3.3.9 on main node and on Metron node install java 1.8 (if you don’t know which it is, install it everywhere):</li>
</ul>
<div class="source">
@@ -433,7 +464,7 @@ always madvise [never]
<ul>
-<li>Download and install Maven:</li>
+<li>Download and install Maven 3.3.9:</li>
</ul>
<div class="source">
@@ -504,11 +535,14 @@ OS name: "linux", version: "3.10.0-514.16.1.el7.x86_64", arc
<ul>
-<li>Remove ipv4 ‘localhost.localdomain’ from /etc/hosts</li>
+<li>
+<p>Remove ipv4 ‘localhost.localdomain’ from /etc/hosts</p></li>
-<li>Remove ipv6 ‘localhost.localdomain’ from /etc/hosts</li>
+<li>
+<p>Remove ipv6 ‘localhost.localdomain’ from /etc/hosts</p></li>
-<li>Add “127.0.0.1 localhost” to /etc/hosts</li>
+<li>
+<p>Add “127.0.0.1 localhost” to /etc/hosts</p></li>
<li>
<p>Install the database we will use for Metron REST:</p></li>
@@ -608,6 +642,14 @@ Cleaning up...
<div class="source">
<pre># git clone https://github.com/apache/metron
</pre></div></div>
+<p>If you want to make sure you’re on the 0.4.0 release branch, do:</p>
+
+<div class="source">
+<div class="source">
+<pre># git clone https://github.com/apache/metron
+# cd metron
+# git checkout Metron_0.4.0
+</pre></div></div>
<ul>
@@ -634,17 +676,19 @@ Cleaning up...
# cp -rp /root/metron/metron-deployment/packaging/docker/rpm-docker/RPMS/noarch/* /localrepo/
# createrepo /localrepo
</pre></div></div>
-<p>If you’re doing a multi node install, also copy the packages to the other nodes:</p>
+<p>If you’re doing a multi node install, also create localrepo on the nodes and copy the packages to the other nodes:</p>
<div class="source">
<div class="source">
-<pre># scp /localrepo/* <replace_with_node_ip>:/localrepo/
-# createrepo /localrepo
+<pre># ssh root@node2 mkdir /localrepo
+# scp /localrepo/*\.rpm root@node2:/localrepo/.
+# ssh root@node2 yum install createrepo -y
+# ssh root@node2 createrepo /localrepo
</pre></div></div>
<ul>
-<li>Make sure to run <tt>createrepo /localrepo</tt> on every node!</li>
+<li>Make sure to do the above on each node.</li>
</ul>
<p>Fetch & create logrotate script for Hadoop Services:</p>
@@ -673,7 +717,7 @@ Cleaning up...
<ul>
-<li>Enable time sync, disable firewall and SElinux:</li>
+<li>Enable time sync, disable firewall and SElinux on every node:</li>
</ul>
<div class="source">
@@ -688,7 +732,7 @@ Cleaning up...
<ul>
-<li>Disable firewall:</li>
+<li>Disable firewall on every node:</li>
</ul>
<div class="source">
@@ -700,7 +744,7 @@ Cleaning up...
<ul>
-<li>Disable IPv6 firewall:</li>
+<li>Disable IPv6 firewall on every node:</li>
</ul>
<div class="source">
@@ -712,7 +756,7 @@ Cleaning up...
<ul>
-<li>Disable SElinux</li>
+<li>Disable SElinux on every node:</li>
</ul>
<div class="source">
@@ -1115,7 +1159,7 @@ Metron JDBC username: <DB USERNAME>
<ul>
<li>
-<p>Install everything. Metron REST will probably not work as we still need to add a user and the database to MySQL.</p></li>
+<p>Install everything. Metron REST will probably not work as we still need to add a user and the database to MySQL. At this point, make sure that all the services are up. You might have to manually start a few.</p></li>
<li>
<p>Configure a user for Metron REST in MySQL. On the node where you installed the Metron REST UI, do:</p></li>
@@ -1216,23 +1260,11 @@ Bye
# yum update -y
# yum install python27 -y
# scl enable python27 bash
-
# cd /opt/rh/python27/root/usr/bin/
# LD_LIBRARY_PATH=$LD_LIBRARY_PATH ./pip2.7 install --upgrade pip
# LD_LIBRARY_PATH=$LD_LIBRARY_PATH ./pip2.7 install requests
-
-(# /opt/rh/python27/root/usr/bin/virtualenv py27venv
-# source py27venv/bin/activate
-# pip install --upgrade pip
-# pip install ansible==2.0.0.2
-# ansible --version
-# deactivate)
-</pre></div></div>
-
-<div class="source">
-<div class="source">
-<pre># yum install @Development python-virtualenv libpcap-devel libselinux-python -y
+# yum install @Development python-virtualenv libpcap-devel libselinux-python -y
# mkdir /usr/local/pycapa
# cd /usr/local/pycapa
# virtualenv pycapa-venv
@@ -1240,7 +1272,6 @@ Bye
# cp -r /root/metron/metron-sensors/pycapa/. /usr/local/pycapa/.
# pip install --upgrade pip
# /usr/local/pycapa/pycapa-venv/bin/pip install -r requirements.txt
-(# pip install -r requirements.txt)
# /usr/local/pycapa/pycapa-venv/bin/python setup.py install
# ln -s /usr/local/lib/librdkafka.so.1 /opt/rh/python27/root/usr/lib64
@@ -1258,7 +1289,6 @@ Bye
# sed -i 's/--kafka {{ kafka_broker_url }}/--kafka-broker <IP:6667>/' /etc/init.d/pycapa
# sed -i 's/--topic {{ pycapa_topic }}/--kafka-topic pcap/' /etc/init.d/pycapa
# sed -i 's/{{ pycapa_sniff_interface }}/tap0/' /etc/init.d/pycapa
-(# sed -i 's/export LD_LIBRARY_PATH=\/opt\/rh\/python27\/root\/usr\/lib64/export LD_LIBRARY_PATH=\/usr\/local\/lib/' /etc/init.d/pycapa)
# chmod 755 /etc/init.d/pycapa
# yum install @Development libdnet-devel rpm-build libpcap libpcap-devel pcre pcre-devel zlib zlib-devel glib2-devel -y
# yum install kafka -y
@@ -1267,9 +1297,9 @@ Bye
<div class="source">
<div class="source">
-<pre># wget -O /tmp/bro-2.4.1.tar.gz https://www.bro.org/downloads/release/bro-2.4.1.tar.gz
-# /bin/gtar --extract -C /tmp -z -f /tmp/bro-2.4.1.tar.gz
-# cd /tmp/bro-2.4.1
+<pre># wget -O /tmp/bro-2.4.2.tar.gz https://www.bro.org/downloads/bro-2.4.2.tar.gz
+# /bin/gtar --extract -C /tmp -z -f /tmp/bro-2.4.2.tar.gz
+# cd /tmp/bro-2.4.2
# ./configure --prefix=/usr/local/bro
# make -j4
# make install
@@ -1292,14 +1322,14 @@ Bye
<div class="source">
<div class="source">
-<pre># cp -r /root/metron/metron-sensors/bro-plugin-kafka /tmp
-# cd /tmp/bro-plugin-kafka
+<pre># git clone https://github.com/apache/metron-bro-plugin-kafka /tmp/metron-bro-plugin-kafka
+# cd /tmp/metron-bro-plugin-kafka
# rm -rf build/
-# ./configure --bro-dist=/tmp/bro-2.4.1 --install-root=/usr/local/bro/lib/bro/plugins/ --with-librdkafka=/usr/local
+# ./configure --bro-dist=/tmp/bro-2.4.2 --install-root=/usr/local/bro/lib/bro/plugins/ --with-librdkafka=/usr/local
# make -j4
# make install
</pre></div></div>
-<p>Configure bro-kafka plugin:</p>
+<p>Configure metron-bro-plugin-kafka:</p>
<div class="source">
<div class="source">
@@ -1476,7 +1506,11 @@ EOF
<ul>
-<li>I had a problem with Zeppelin after rebooting this machine and had to manually create the Zeppelin run directory:</li>
+<li>
+<p>There’s currently a bug in Metron 0.4.0 where Metron REST doesn’t start when restarting the Metron services. This bug was fixed in METRON-990 (<a class="externalLink" href="https://github.com/apache/metron/pull/613/commits/1a9b19a0101ada58cb671ab224934f304df6fff8">https://github.com/apache/metron/pull/613/commits/1a9b19a0101ada58cb671ab224934f304df6fff8</a>) but unfortunately, this fix didn’t make the 0.4.0 release. In order to fix this, edit the file “/etc/rc.d/init.d/metron-rest” and on line 148, change “$0 start” to “$0 start $2”.</p></li>
+
+<li>
+<p>I had a problem with Zeppelin after rebooting this machine and had to manually create the Zeppelin run directory:</p></li>
</ul>
<div class="source">
@@ -1583,12 +1617,13 @@ EOF
<ul>
-<li>Install Apache NiFi. Download nifi-1.2.0-bin.tar.gz from <a class="externalLink" href="https://nifi.apache.org/download.html">https://nifi.apache.org/download.html</a></li>
+<li>Install Apache NiFi in /root (You can pretty much use any directory you want). Download nifi-1.2.0-bin.tar.gz from <a class="externalLink" href="https://nifi.apache.org/download.html">https://nifi.apache.org/download.html</a></li>
</ul>
<div class="source">
<div class="source">
-<pre># wget http://apache.mirror.iweb.ca/nifi/1.2.0/nifi-1.2.0-bin.tar.gz
+<pre># cd /root
+# wget http://apache.mirror.iweb.ca/nifi/1.2.0/nifi-1.2.0-bin.tar.gz
# tar xf nifi-1.2.0-bin.tar.gz
</pre></div></div>
<p>Before we run NiFi, we need to change the port as the default port collides with the Ambari port. To do this, we need to change the value “nifi.web.http.port=8080” to “nifi.web.http.port=8089” in the file “nifi-1.1.2/conf/nifi.properties”. Install and start NiFi afterwards:</p>
@@ -1634,7 +1669,7 @@ EOF
<footer>
<div class="container-fluid">
- <div class="row span12">Copyright © 2017
+ <div class="row span12">Copyright © 2018
<a href="https://www.apache.org">The Apache Software Foundation</a>.
All Rights Reserved.
http://git-wip-us.apache.org/repos/asf/metron/blob/3381b853/site/current-book/metron-deployment/packaging/ambari/index.html
----------------------------------------------------------------------
diff --git a/site/current-book/metron-deployment/packaging/ambari/index.html b/site/current-book/metron-deployment/packaging/ambari/index.html
index 25df81e..8013381 100644
--- a/site/current-book/metron-deployment/packaging/ambari/index.html
+++ b/site/current-book/metron-deployment/packaging/ambari/index.html
@@ -1,13 +1,13 @@
<!DOCTYPE html>
<!--
- | Generated by Apache Maven Doxia at 2017-09-15
+ | Generated by Apache Maven Doxia at 2018-01-03
| Rendered using Apache Maven Fluido Skin 1.3.0
-->
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
- <meta name="Date-Revision-yyyymmdd" content="20170915" />
+ <meta name="Date-Revision-yyyymmdd" content="20180103" />
<meta http-equiv="Content-Language" content="en" />
<title>Metron – Ambari Management Pack Development</title>
<link rel="stylesheet" href="../../../css/apache-maven-fluido-1.3.0.min.css" />
@@ -61,8 +61,8 @@
- <li id="publishDate" class="pull-right">Last Published: 2017-09-15</li> <li class="divider pull-right">|</li>
- <li id="projectVersion" class="pull-right">Version: 0.4.1</li>
+ <li id="publishDate" class="pull-right">Last Published: 2018-01-03</li> <li class="divider pull-right">|</li>
+ <li id="projectVersion" class="pull-right">Version: 0.4.2</li>
</ul>
</div>
@@ -103,7 +103,7 @@
<i class="none"></i>
Docker</a>
</li>
-
+
<li>
<a href="../../../metron-deployment/index.html" title="Deployment">
@@ -171,7 +171,7 @@
<i class="icon-chevron-right"></i>
Roles</a>
</li>
-
+
<li>
<a href="../../../metron-deployment/vagrant/index.html" title="Vagrant">
@@ -201,20 +201,27 @@
<i class="none"></i>
Rest</a>
</li>
-
+
<li>
<a href="../../../metron-platform/index.html" title="Platform">
<i class="icon-chevron-right"></i>
Platform</a>
</li>
-
+
<li>
<a href="../../../metron-sensors/index.html" title="Sensors">
<i class="icon-chevron-right"></i>
Sensors</a>
</li>
+
+ <li>
+
+ <a href="../../../metron-stellar/stellar-3rd-party-example/index.html" title="Stellar-3rd-party-example">
+ <i class="none"></i>
+ Stellar-3rd-party-example</a>
+ </li>
<li>
@@ -222,7 +229,7 @@
<i class="icon-chevron-right"></i>
Stellar-common</a>
</li>
-
+
<li>
<a href="../../../use-cases/index.html" title="Use-cases">
@@ -253,6 +260,29 @@
<h1>Ambari Management Pack Development</h1>
<p><a name="Ambari_Management_Pack_Development"></a></p>
+<div class="section">
+<h2><a name="Table_of_Contents"></a>Table of Contents</h2>
+
+<ul>
+
+<li><a href="#Overview">Overview</a></li>
+
+<li><a href="#Ambari_Zookeeper_Config_Lifecycle">Ambari Zookeeper Config Lifecycle</a></li>
+
+<li><a href="#Adding_a_new_property">Adding a new property</a></li>
+
+<li><a href="#How_to_identify_errors_in_MPack_changes">How to identify errors in MPack changes</a></li>
+
+<li><a href="#Testing_changes_without_cycling_Vagrant_build">Testing changes without cycling Vagrant build</a></li>
+
+<li><a href="#Configuration_involving_dependency_services">Configuration involving dependency services</a></li>
+
+<li><a href="#Kerberos">Kerberos</a></li>
+
+<li><a href="#Best_practices">Best practices</a></li>
+</ul></div>
+<div class="section">
+<h2><a name="Overview"></a>Overview</h2>
<p>Typically, Ambari Management Pack development will be done in the Vagrant environments. These instructions are specific to Vagrant, but can be adapted for other environemnts (e.g. make sure to be on the correct nodes for server vs agent files)</p>
<p>There is an <tt>mpack.json</tt> file which describes what services the mpack will contains, versions, etc.</p>
<p>Alongside this are two directories, <tt>addon-services</tt> and <tt>common-services</tt>.</p>
@@ -359,7 +389,61 @@
<li>Defines the order of service startup and other actions relative to each other.</li>
</ul></li>
+</ul></div>
+<div class="section">
+<h2><a name="Ambari_Zookeeper_Config_Lifecycle"></a>Ambari Zookeeper Config Lifecycle</h2>
+<p>The source of truth for Metron configuration resides in Zookeeper as a series of JSON documents. There are actually multiple locations that you can populate the Zookeeper configs from:</p>
+
+<ul>
+
+<li>$METRON_HOME/config/zookeeper</li>
+
+<li>Stellar REPL</li>
+
+<li>Management UI</li>
+
+<li>Ambari</li>
</ul>
+<p>Any change that works with Metron configuration stored in Zookeeper should always pull the latest config from Zookeeper first before making any changes and publishing them back up to avoid potentially overwriting the latest configs with stale data. The lifecycle for configuration management with Ambari is as follows:</p>
+
+<ul>
+
+<li>First start/install for parsers, enrichment, indexing, or the profiler
+
+<ul>
+
+<li>Perform a PUSH from local disk to Zookeeper</li>
+ </ul></li>
+
+<li>All component start/restart operations for parsers, enrichment, indexing, or the profiler
+
+<ul>
+
+<li>Perform a JSON patch (RFC 6902) on the configs in Zookeeper and push the patched, pretty formatted config up to Zookeeper. Patching is currently applicable to to global.json only.</li>
+
+<li>Pull all configs to the local file system.</li>
+ </ul></li>
+</ul>
+<p>The patching mechanism we introduced to Ambari for managing the global config is by default additive in nature, though it will also overwrite any existing values with the latest values from Ambari. This means that you can add any properties to the global config outside of Ambari you like, provided you are not modifying values explicitly managed by Ambari. This is important because you will not get any errors when modifying the configuration outside of Ambari. Instead, if you modified es.ip and changed global.json outsided Ambari, you would not see this change in Ambari. Meanwhile, the indexing topology would be using the new value stored in Zookeeper. Remember, Zookeeper is the source of truth. If you were to restart a Metron topology component via Ambari, that es.ip property would now be set back to the value stored in Ambari. So in summary, if it’s managed by Ambari, only change that value via Ambari. If it’s not, then feel free to use whatever mechanism you like,
just be sure to pull the latest config from Zookeeper before making any changes. We make no effort to handle race conditions. It’s last commit wins. Below are the global config properties managed by Ambari:</p>
+
+<ul>
+
+<li>es.clustername</li>
+
+<li>es.ip</li>
+
+<li>es.date.format</li>
+
+<li>parser.error.topic</li>
+
+<li>update.hbase.table</li>
+
+<li>update.hbase.cf</li>
+
+<li>profiler.client.period.duration</li>
+
+<li>profiler.client.period.duration.units</li>
+</ul></div>
<div class="section">
<h2><a name="Adding_a_new_property"></a>Adding a new property</h2>
@@ -396,7 +480,7 @@
<div class="source">
<pre> new_property = status_params.new_property
</pre></div></div>
-<p>This behavior is because Ambari doesn’t send all parameters to the status, so it needs to be explicitly provided.</p>
+<p>This behavior is because Ambari doesn’t send all parameters to the status, so it needs to be explicitly provided. Also note that status_params.py parameters are not automatically pulled into the params_linux.py namespace, so we explicitly choose the variables to include. See <a class="externalLink" href="https://docs.python.org/2/howto/doanddont.html#at-module-level">https://docs.python.org/2/howto/doanddont.html#at-module-level</a> for more info.</p>
<ol style="list-style-type: decimal">
@@ -468,7 +552,36 @@
# ...
hdfs_grok_patterns_dir = format("{metron_apps_hdfs_dir}/patterns")
</pre></div></div>
-<p>The <tt>format</tt> function is a special Ambari function that will let you create a string with properties in curly braces replaced by their values. In this case, <tt>hdfs_grok_patterns_dir</tt> will be <tt>/apps/metron/patterns</tt> and will be what we now follow in this example.</p></div>
+<p>The <tt>format</tt> function is a special Ambari function that will let you create a string with properties in curly braces replaced by their values. In this case, <tt>hdfs_grok_patterns_dir</tt> will be <tt>/apps/metron/patterns</tt> and will be what we now follow in this example.</p>
+<p>Note: There are instances where we deviate a bit from this pattern and use format(format(“{{some_prop}}”)). This is not a mistake. Ambari does not natively support nested properties, and this was a workaround that accomplished being able to initialize values from env.xml files and have them also available to users as Ambari properties later. Here’s an example from params_linux.py:</p>
+
+<div class="source">
+<div class="source">
+<pre>metron_apps_indexed_hdfs_dir = format(format(config['configurations']['metron-indexing-env']['metron_apps_indexed_hdfs_dir']))
+</pre></div></div>
+<p>and the relavant properties in metron-env.xml and metron-indexing-env.xml</p>
+<p>metron-env.xml</p>
+
+<div class="source">
+<div class="source">
+<pre><property>
+ <name>metron_apps_hdfs_dir</name>
+ <value>/apps/metron</value>
+ <description>Metron apps HDFS dir</description>
+ <display-name>Metron apps HDFS dir</display-name>
+</property>
+</pre></div></div>
+<p>metron-indexing-env.xml</p>
+
+<div class="source">
+<div class="source">
+<pre><property>
+ <name>metron_apps_indexed_hdfs_dir</name>
+ <value>{{metron_apps_hdfs_dir}}/indexing/indexed</value>
+ <description>Indexing bolts will write to this HDFS directory</description>
+ <display-name>Metron apps indexed HDFS dir</display-name>
+</property>
+</pre></div></div></div>
<div class="section">
<h4><a name="Importing_the_parameters_into_scripts"></a>Importing the parameters into scripts</h4>
<p><tt>hdfs_grok_patterns_dir</tt> is used in <tt>METRON.CURRENT/package/scripts/parser_commands.py</tt>, but before we can reference it, we’ll need the params available to <tt>parser_commands.py</tt></p>
@@ -772,7 +885,7 @@ METRON_LOG_DIR="/var/log/metron"
<footer>
<div class="container-fluid">
- <div class="row span12">Copyright © 2017
+ <div class="row span12">Copyright © 2018
<a href="https://www.apache.org">The Apache Software Foundation</a>.
All Rights Reserved.
http://git-wip-us.apache.org/repos/asf/metron/blob/3381b853/site/current-book/metron-deployment/packaging/docker/ansible-docker/index.html
----------------------------------------------------------------------
diff --git a/site/current-book/metron-deployment/packaging/docker/ansible-docker/index.html b/site/current-book/metron-deployment/packaging/docker/ansible-docker/index.html
index f2045ae..83d98f6 100644
--- a/site/current-book/metron-deployment/packaging/docker/ansible-docker/index.html
+++ b/site/current-book/metron-deployment/packaging/docker/ansible-docker/index.html
@@ -1,13 +1,13 @@
<!DOCTYPE html>
<!--
- | Generated by Apache Maven Doxia at 2017-09-15
+ | Generated by Apache Maven Doxia at 2018-01-03
| Rendered using Apache Maven Fluido Skin 1.3.0
-->
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
- <meta name="Date-Revision-yyyymmdd" content="20170915" />
+ <meta name="Date-Revision-yyyymmdd" content="20180103" />
<meta http-equiv="Content-Language" content="en" />
<title>Metron – Overview</title>
<link rel="stylesheet" href="../../../../css/apache-maven-fluido-1.3.0.min.css" />
@@ -61,8 +61,8 @@
- <li id="publishDate" class="pull-right">Last Published: 2017-09-15</li> <li class="divider pull-right">|</li>
- <li id="projectVersion" class="pull-right">Version: 0.4.1</li>
+ <li id="publishDate" class="pull-right">Last Published: 2018-01-03</li> <li class="divider pull-right">|</li>
+ <li id="projectVersion" class="pull-right">Version: 0.4.2</li>
</ul>
</div>
@@ -103,7 +103,7 @@
<i class="none"></i>
Docker</a>
</li>
-
+
<li>
<a href="../../../../metron-deployment/index.html" title="Deployment">
@@ -171,7 +171,7 @@
<i class="icon-chevron-right"></i>
Roles</a>
</li>
-
+
<li>
<a href="../../../../metron-deployment/vagrant/index.html" title="Vagrant">
@@ -201,20 +201,27 @@
<i class="none"></i>
Rest</a>
</li>
-
+
<li>
<a href="../../../../metron-platform/index.html" title="Platform">
<i class="icon-chevron-right"></i>
Platform</a>
</li>
-
+
<li>
<a href="../../../../metron-sensors/index.html" title="Sensors">
<i class="icon-chevron-right"></i>
Sensors</a>
</li>
+
+ <li>
+
+ <a href="../../../../metron-stellar/stellar-3rd-party-example/index.html" title="Stellar-3rd-party-example">
+ <i class="none"></i>
+ Stellar-3rd-party-example</a>
+ </li>
<li>
@@ -222,7 +229,7 @@
<i class="icon-chevron-right"></i>
Stellar-common</a>
</li>
-
+
<li>
<a href="../../../../use-cases/index.html" title="Use-cases">
@@ -253,7 +260,7 @@
<h1>Overview</h1>
<p><a name="Overview"></a></p>
-<p>The Metron ansible-docker container is provided in an effort reduce the installation burden of deploying Metron in a live envirionment. It is provisioned with software required to sucessfully run the deployment scripts.</p>
+<p>The Metron ansible-docker container is provided in an effort reduce the installation burden of building Metron. It may also be used to deploy Metron in a live environment. It is provisioned with software required to sucessfully build metron and run the deployment scripts.</p>
<div class="section">
<h2><a name="Building_the_Container"></a>Building the Container</h2>
@@ -266,18 +273,35 @@
<li>Build the container <tt>docker build -t ansible-docker:2.0.0.2 .</tt></li>
</ol></div>
<div class="section">
-<h2><a name="Using_the_Container"></a>Using the Container</h2>
-<p>Full instructions are found on the wiki at <a class="externalLink" href="https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=65144361">https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=65144361</a></p>
-<p>tl;dr:</p>
+<h2><a name="Using_the_Container_to_build_metron"></a>Using the Container to build metron</h2>
+<p>anytime after building the container you can run it with the following command</p>
+<p><tt>docker run -it -v \<project-directory\>:/root/metron ansible-docker:2.0.0.2 bash</tt></p>
+<p>If you are going to build metron multiple times, you may want to map the /root/.m2 maven repo from outside of the container so that you don’t start with an empty repo every build and have to download the world.</p>
+<p><tt>docker run -it -v \<project-directory\>:/root/metron -v \<your .m2 directory\>:/root/.m2 ansible-docker:2.0.0.2 bash</tt></p>
+<p>After running the container:</p>
<ol style="list-style-type: decimal">
-<li>docker run -it -v <project-directory>:/root/metron ansible-docker:2.0.0.2 bash</li>
-
<li>cd /root/metron</li>
-<li>mvn clean package -DskipTests</li>
-</ol></div>
+<li>run build commands, for example:
+
+<ul>
+
+<li>build metron without tests : <tt>mvn clean package -DskipTests</tt></li>
+
+<li>build metron and build the rpms as well : <tt>mvn clean install && cd metron-deployment && mvn package -P build-rpms</tt></li>
+ </ul></li>
+</ol>
+<p>If you wish to use this build with a vagrant instance, then after building with rpms as above, modify your usual vagrant up command to skip the build role, as so:</p>
+<p><tt>vagrant --ansible-skip-tags="build" up</tt></p></div>
+<div class="section">
+<h2><a name="Using_the_Container_for_deployment"></a>Using the Container for deployment</h2>
+
+<blockquote>
+<p>Note these instructions are outdated</p>
+</blockquote>
+<p>Full instructions are found on the wiki at <a class="externalLink" href="https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=65144361">https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=65144361</a></p></div>
</div>
</div>
</div>
@@ -286,7 +310,7 @@
<footer>
<div class="container-fluid">
- <div class="row span12">Copyright © 2017
+ <div class="row span12">Copyright © 2018
<a href="https://www.apache.org">The Apache Software Foundation</a>.
All Rights Reserved.
http://git-wip-us.apache.org/repos/asf/metron/blob/3381b853/site/current-book/metron-deployment/packaging/docker/rpm-docker/index.html
----------------------------------------------------------------------
diff --git a/site/current-book/metron-deployment/packaging/docker/rpm-docker/index.html b/site/current-book/metron-deployment/packaging/docker/rpm-docker/index.html
index c216970..53b2885 100644
--- a/site/current-book/metron-deployment/packaging/docker/rpm-docker/index.html
+++ b/site/current-book/metron-deployment/packaging/docker/rpm-docker/index.html
@@ -1,13 +1,13 @@
<!DOCTYPE html>
<!--
- | Generated by Apache Maven Doxia at 2017-09-15
+ | Generated by Apache Maven Doxia at 2018-01-03
| Rendered using Apache Maven Fluido Skin 1.3.0
-->
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
- <meta name="Date-Revision-yyyymmdd" content="20170915" />
+ <meta name="Date-Revision-yyyymmdd" content="20180103" />
<meta http-equiv="Content-Language" content="en" />
<title>Metron – Overview</title>
<link rel="stylesheet" href="../../../../css/apache-maven-fluido-1.3.0.min.css" />
@@ -61,8 +61,8 @@
- <li id="publishDate" class="pull-right">Last Published: 2017-09-15</li> <li class="divider pull-right">|</li>
- <li id="projectVersion" class="pull-right">Version: 0.4.1</li>
+ <li id="publishDate" class="pull-right">Last Published: 2018-01-03</li> <li class="divider pull-right">|</li>
+ <li id="projectVersion" class="pull-right">Version: 0.4.2</li>
</ul>
</div>
@@ -103,7 +103,7 @@
<i class="none"></i>
Docker</a>
</li>
-
+
<li>
<a href="../../../../metron-deployment/index.html" title="Deployment">
@@ -171,7 +171,7 @@
<i class="icon-chevron-right"></i>
Roles</a>
</li>
-
+
<li>
<a href="../../../../metron-deployment/vagrant/index.html" title="Vagrant">
@@ -201,20 +201,27 @@
<i class="none"></i>
Rest</a>
</li>
-
+
<li>
<a href="../../../../metron-platform/index.html" title="Platform">
<i class="icon-chevron-right"></i>
Platform</a>
</li>
-
+
<li>
<a href="../../../../metron-sensors/index.html" title="Sensors">
<i class="icon-chevron-right"></i>
Sensors</a>
</li>
+
+ <li>
+
+ <a href="../../../../metron-stellar/stellar-3rd-party-example/index.html" title="Stellar-3rd-party-example">
+ <i class="none"></i>
+ Stellar-3rd-party-example</a>
+ </li>
<li>
@@ -222,7 +229,7 @@
<i class="icon-chevron-right"></i>
Stellar-common</a>
</li>
-
+
<li>
<a href="../../../../use-cases/index.html" title="Use-cases">
@@ -273,7 +280,7 @@
<footer>
<div class="container-fluid">
- <div class="row span12">Copyright © 2017
+ <div class="row span12">Copyright © 2018
<a href="https://www.apache.org">The Apache Software Foundation</a>.
All Rights Reserved.
http://git-wip-us.apache.org/repos/asf/metron/blob/3381b853/site/current-book/metron-deployment/packaging/packer-build/index.html
----------------------------------------------------------------------
diff --git a/site/current-book/metron-deployment/packaging/packer-build/index.html b/site/current-book/metron-deployment/packaging/packer-build/index.html
index d85fa3e..066073c 100644
--- a/site/current-book/metron-deployment/packaging/packer-build/index.html
+++ b/site/current-book/metron-deployment/packaging/packer-build/index.html
@@ -1,13 +1,13 @@
<!DOCTYPE html>
<!--
- | Generated by Apache Maven Doxia at 2017-09-15
+ | Generated by Apache Maven Doxia at 2018-01-03
| Rendered using Apache Maven Fluido Skin 1.3.0
-->
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
- <meta name="Date-Revision-yyyymmdd" content="20170915" />
+ <meta name="Date-Revision-yyyymmdd" content="20180103" />
<meta http-equiv="Content-Language" content="en" />
<title>Metron – Build Metron Images</title>
<link rel="stylesheet" href="../../../css/apache-maven-fluido-1.3.0.min.css" />
@@ -61,8 +61,8 @@
- <li id="publishDate" class="pull-right">Last Published: 2017-09-15</li> <li class="divider pull-right">|</li>
- <li id="projectVersion" class="pull-right">Version: 0.4.1</li>
+ <li id="publishDate" class="pull-right">Last Published: 2018-01-03</li> <li class="divider pull-right">|</li>
+ <li id="projectVersion" class="pull-right">Version: 0.4.2</li>
</ul>
</div>
@@ -103,7 +103,7 @@
<i class="none"></i>
Docker</a>
</li>
-
+
<li>
<a href="../../../metron-deployment/index.html" title="Deployment">
@@ -171,7 +171,7 @@
<i class="icon-chevron-right"></i>
Roles</a>
</li>
-
+
<li>
<a href="../../../metron-deployment/vagrant/index.html" title="Vagrant">
@@ -201,20 +201,27 @@
<i class="none"></i>
Rest</a>
</li>
-
+
<li>
<a href="../../../metron-platform/index.html" title="Platform">
<i class="icon-chevron-right"></i>
Platform</a>
</li>
-
+
<li>
<a href="../../../metron-sensors/index.html" title="Sensors">
<i class="icon-chevron-right"></i>
Sensors</a>
</li>
+
+ <li>
+
+ <a href="../../../metron-stellar/stellar-3rd-party-example/index.html" title="Stellar-3rd-party-example">
+ <i class="none"></i>
+ Stellar-3rd-party-example</a>
+ </li>
<li>
@@ -222,7 +229,7 @@
<i class="icon-chevron-right"></i>
Stellar-common</a>
</li>
-
+
<li>
<a href="../../../use-cases/index.html" title="Use-cases">
@@ -259,8 +266,6 @@
<ul>
<li>base-centos-6.7: Centos 6.7 + HDP. Used in the full-dev-platform Vagrant image</li>
-
-<li>quick-dev-centos-6.7: Centos 6.7 + HDP + Metron. Used for the quick-dev-platform Vagrant image.</li>
</ul></div>
<div class="section">
<h2><a name="Prerequisites"></a>Prerequisites</h2>
@@ -270,10 +275,14 @@
<li><a class="externalLink" href="https://www.packer.io/">Packer</a> 0.12.2</li>
<li><a class="externalLink" href="https://www.virtualbox.org/">Virtualbox</a> 5.0.16+ (Tested with 5.0.20)</li>
-</ul>
-<p>Build Both Images ———————- Navigate to <your-project-directory>/metron-deployment/packer-build Execute bin/bento build</p>
-<p>Packer will build both images and export .box files to the ./builds directory.</p>
-<p>Build Single Images ———————- Navigate to <i>your-project-directory</i>/metron-deployment/packer-build</p>
+</ul></div>
+<div class="section">
+<h2><a name="Build_Both_Images"></a>Build Both Images</h2>
+<p>Navigate to <your-project-directory>/metron-deployment/packer-build Execute bin/bento build</p>
+<p>Packer will build both images and export .box files to the ./builds directory.</p></div>
+<div class="section">
+<h2><a name="Build_Single_Images"></a>Build Single Images</h2>
+<p>Navigate to <i>your-project-directory</i>/metron-deployment/packer-build</p>
<ul>
@@ -283,18 +292,10 @@
<div class="source">
<div class="source">
<pre>bin/bento build base-centos-6.7.json
-</pre></div></div>
-
-<ul>
-
-<li>Quick Dev</li>
-</ul>
-
-<div class="source">
-<div class="source">
-<pre>bin/bento build quick-dev-centos-6.7.json
-</pre></div></div>
-<p>Using Your New Box File ———————- Modify the relevant Vagrantfile (full-dev-platform or quick-dev-platform) replacing the lines:</p>
+</pre></div></div></div>
+<div class="section">
+<h2><a name="Using_Your_New_Box_File"></a>Using Your New Box File</h2>
+<p>Modify the relevant Vagrantfile (full-dev-platform) replacing the lines:</p>
<div class="source">
<div class="source">
@@ -319,7 +320,7 @@ config.ssh.insert_key = true</code></pre>
<footer>
<div class="container-fluid">
- <div class="row span12">Copyright © 2017
+ <div class="row span12">Copyright © 2018
<a href="https://www.apache.org">The Apache Software Foundation</a>.
All Rights Reserved.
[3/7] metron git commit: METRON-1388 update public web site to point
at 0.4.2 new release (mattf-horton) closes apache/metron#887
Posted by ma...@apache.org.
http://git-wip-us.apache.org/repos/asf/metron/blob/3381b853/site/current-book/metron-platform/metron-elasticsearch/index.html
----------------------------------------------------------------------
diff --git a/site/current-book/metron-platform/metron-elasticsearch/index.html b/site/current-book/metron-platform/metron-elasticsearch/index.html
new file mode 100644
index 0000000..ff4bfb0
--- /dev/null
+++ b/site/current-book/metron-platform/metron-elasticsearch/index.html
@@ -0,0 +1,380 @@
+<!DOCTYPE html>
+<!--
+ | Generated by Apache Maven Doxia at 2018-01-03
+ | Rendered using Apache Maven Fluido Skin 1.3.0
+-->
+<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
+ <head>
+ <meta charset="UTF-8" />
+ <meta name="viewport" content="width=device-width, initial-scale=1.0" />
+ <meta name="Date-Revision-yyyymmdd" content="20180103" />
+ <meta http-equiv="Content-Language" content="en" />
+ <title>Metron – Elasticsearch in Metron</title>
+ <link rel="stylesheet" href="../../css/apache-maven-fluido-1.3.0.min.css" />
+ <link rel="stylesheet" href="../../css/site.css" />
+ <link rel="stylesheet" href="../../css/print.css" media="print" />
+
+
+ <script type="text/javascript" src="../../js/apache-maven-fluido-1.3.0.min.js"></script>
+
+
+
+<script type="text/javascript">$( document ).ready( function() { $( '.carousel' ).carousel( { interval: 3500 } ) } );</script>
+
+ </head>
+ <body class="topBarDisabled">
+
+
+
+
+ <div class="container-fluid">
+ <div id="banner">
+ <div class="pull-left">
+ <a href="http://metron.apache.org/" id="bannerLeft">
+ <img src="../../images/metron-logo.png" alt="Apache Metron" width="148px" height="48px"/>
+ </a>
+ </div>
+ <div class="pull-right"> </div>
+ <div class="clear"><hr/></div>
+ </div>
+
+ <div id="breadcrumbs">
+ <ul class="breadcrumb">
+
+
+ <li class="">
+ <a href="http://www.apache.org" class="externalLink" title="Apache">
+ Apache</a>
+ </li>
+ <li class="divider ">/</li>
+ <li class="">
+ <a href="http://metron.apache.org/" class="externalLink" title="Metron">
+ Metron</a>
+ </li>
+ <li class="divider ">/</li>
+ <li class="">
+ <a href="../../index.html" title="Documentation">
+ Documentation</a>
+ </li>
+ <li class="divider ">/</li>
+ <li class="">Elasticsearch in Metron</li>
+
+
+
+ <li id="publishDate" class="pull-right">Last Published: 2018-01-03</li> <li class="divider pull-right">|</li>
+ <li id="projectVersion" class="pull-right">Version: 0.4.2</li>
+
+ </ul>
+ </div>
+
+
+ <div class="row-fluid">
+ <div id="leftColumn" class="span3">
+ <div class="well sidebar-nav">
+
+
+ <ul class="nav nav-list">
+ <li class="nav-header">User Documentation</li>
+
+ <li>
+
+ <a href="../../index.html" title="Metron">
+ <i class="icon-chevron-down"></i>
+ Metron</a>
+ <ul class="nav nav-list">
+
+ <li>
+
+ <a href="../../Upgrading.html" title="Upgrading">
+ <i class="none"></i>
+ Upgrading</a>
+ </li>
+
+ <li>
+
+ <a href="../../metron-analytics/index.html" title="Analytics">
+ <i class="icon-chevron-right"></i>
+ Analytics</a>
+ </li>
+
+ <li>
+
+ <a href="../../metron-contrib/metron-docker/index.html" title="Docker">
+ <i class="none"></i>
+ Docker</a>
+ </li>
+
+ <li>
+
+ <a href="../../metron-deployment/index.html" title="Deployment">
+ <i class="icon-chevron-right"></i>
+ Deployment</a>
+ </li>
+
+ <li>
+
+ <a href="../../metron-interface/metron-alerts/index.html" title="Alerts">
+ <i class="none"></i>
+ Alerts</a>
+ </li>
+
+ <li>
+
+ <a href="../../metron-interface/metron-config/index.html" title="Config">
+ <i class="none"></i>
+ Config</a>
+ </li>
+
+ <li>
+
+ <a href="../../metron-interface/metron-rest/index.html" title="Rest">
+ <i class="none"></i>
+ Rest</a>
+ </li>
+
+ <li>
+
+ <a href="../../metron-platform/index.html" title="Platform">
+ <i class="icon-chevron-down"></i>
+ Platform</a>
+ <ul class="nav nav-list">
+
+ <li>
+
+ <a href="../../metron-platform/Performance-tuning-guide.html" title="Performance-tuning-guide">
+ <i class="none"></i>
+ Performance-tuning-guide</a>
+ </li>
+
+ <li>
+
+ <a href="../../metron-platform/metron-api/index.html" title="Api">
+ <i class="none"></i>
+ Api</a>
+ </li>
+
+ <li>
+
+ <a href="../../metron-platform/metron-common/index.html" title="Common">
+ <i class="none"></i>
+ Common</a>
+ </li>
+
+ <li>
+
+ <a href="../../metron-platform/metron-data-management/index.html" title="Data-management">
+ <i class="none"></i>
+ Data-management</a>
+ </li>
+
+ <li class="active">
+
+ <a href="#"><i class="none"></i>Elasticsearch</a>
+ </li>
+
+ <li>
+
+ <a href="../../metron-platform/metron-enrichment/index.html" title="Enrichment">
+ <i class="none"></i>
+ Enrichment</a>
+ </li>
+
+ <li>
+
+ <a href="../../metron-platform/metron-indexing/index.html" title="Indexing">
+ <i class="none"></i>
+ Indexing</a>
+ </li>
+
+ <li>
+
+ <a href="../../metron-platform/metron-management/index.html" title="Management">
+ <i class="none"></i>
+ Management</a>
+ </li>
+
+ <li>
+
+ <a href="../../metron-platform/metron-parsers/index.html" title="Parsers">
+ <i class="icon-chevron-right"></i>
+ Parsers</a>
+ </li>
+
+ <li>
+
+ <a href="../../metron-platform/metron-pcap-backend/index.html" title="Pcap-backend">
+ <i class="none"></i>
+ Pcap-backend</a>
+ </li>
+
+ <li>
+
+ <a href="../../metron-platform/metron-writer/index.html" title="Writer">
+ <i class="none"></i>
+ Writer</a>
+ </li>
+ </ul>
+ </li>
+
+ <li>
+
+ <a href="../../metron-sensors/index.html" title="Sensors">
+ <i class="icon-chevron-right"></i>
+ Sensors</a>
+ </li>
+
+ <li>
+
+ <a href="../../metron-stellar/stellar-3rd-party-example/index.html" title="Stellar-3rd-party-example">
+ <i class="none"></i>
+ Stellar-3rd-party-example</a>
+ </li>
+
+ <li>
+
+ <a href="../../metron-stellar/stellar-common/index.html" title="Stellar-common">
+ <i class="icon-chevron-right"></i>
+ Stellar-common</a>
+ </li>
+
+ <li>
+
+ <a href="../../use-cases/index.html" title="Use-cases">
+ <i class="icon-chevron-right"></i>
+ Use-cases</a>
+ </li>
+ </ul>
+ </li>
+ </ul>
+
+
+
+ <hr class="divider" />
+
+ <div id="poweredBy">
+ <div class="clear"></div>
+ <div class="clear"></div>
+ <div class="clear"></div>
+ <a href="http://maven.apache.org/" title="Built by Maven" class="poweredBy">
+ <img class="builtBy" alt="Built by Maven" src="../../images/logos/maven-feather.png" />
+ </a>
+ </div>
+ </div>
+ </div>
+
+
+ <div id="bodyColumn" class="span9" >
+
+ <h1>Elasticsearch in Metron</h1>
+<p><a name="Elasticsearch_in_Metron"></a></p>
+<div class="section">
+<h2><a name="Introduction"></a>Introduction</h2>
+<p>Elasticsearch can be used as the real-time portion of the datastore resulting from <a href="../metron-indexing/index.html">metron-indexing</a>.</p></div>
+<div class="section">
+<h2><a name="Properties"></a>Properties</h2>
+<div class="section">
+<h3><a name="es.clustername"></a><tt>es.clustername</tt></h3>
+<p>The name of the elasticsearch Cluster. See <a class="externalLink" href="https://www.elastic.co/guide/en/elasticsearch/reference/current/important-settings.html#cluster.name">here</a></p></div>
+<div class="section">
+<h3><a name="es.ip"></a><tt>es.ip</tt></h3>
+<p>Specifies the nodes in the elasticsearch cluster to use for writing. The format is one of the following:</p>
+
+<ul>
+
+<li>A hostname or IP address with a port (e.g. <tt>hostname1:1234</tt>), in which case <tt>es.port</tt> is ignored.</li>
+
+<li>A hostname or IP address without a port (e.g. <tt>hostname1</tt>), in which case <tt>es.port</tt> is used.</li>
+
+<li>A string containing a CSV of hostnames without ports (e.g. <tt>hostname1,hostname2,hostname3</tt>) without spaces between. <tt>es.port</tt> is assumed to be the port for each host.</li>
+
+<li>A string containing a CSV of hostnames with ports (e.g. <tt>hostname1:1234,hostname2:1234,hostname3:1234</tt>) without spaces between. <tt>es.port</tt> is ignored.</li>
+
+<li>A list of hostnames with ports (e.g. <tt>[ "hostname1:1234", "hostname2:1234"]</tt>). Note, <tt>es.port</tt> is NOT used in this construction.</li>
+</ul></div>
+<div class="section">
+<h3><a name="es.port"></a><tt>es.port</tt></h3>
+<p>The port for the elasticsearch hosts. This will be used in accordance with the discussion of <tt>es.ip</tt>.</p></div>
+<div class="section">
+<h3><a name="es.date.format"></a><tt>es.date.format</tt></h3>
+<p>The date format to use when constructing the indices. For every message, the date format will be applied to the current time and that will become the last part of the index name where the message is written to.</p>
+<p>For instance, an <tt>es.date.format</tt> of <tt>yyyy.MM.dd.HH</tt> would have the consequence that the indices would roll hourly, whereas an <tt>es.date.format</tt> of <tt>yyyy.MM.dd</tt> would have the consequence that the indices would roll daily.</p></div></div>
+<div class="section">
+<h2><a name="Using_Metron_with_Elasticsearch_2.x"></a>Using Metron with Elasticsearch 2.x</h2>
+<p>With Elasticsearch 2.x, there is a requirement that all sensors templates have a nested alert field defined. This field is a dummy field, and will be obsolete in Elasticsearch 5.x. See <a class="externalLink" href="https://www.elastic.co/guide/en/elasticsearch/reference/current/search-request-sort.html#_ignoring_unmapped_fields">Ignoring Unmapped Fields</a> for more information</p>
+<p>Without this field, an error will be thrown during ALL searches (including from UIs, resulting in no alerts being found for any sensor). This error will be found in the REST service’s logs.</p>
+<p>Exception seen:</p>
+
+<div class="source">
+<div class="source">
+<pre>QueryParsingException[[nested] failed to find nested object under path [alert]];
+</pre></div></div>
+<p>There are two steps to resolve this issue. First is to update the Elasticsearch template for each sensor, so any new indices have the field. This requires retrieving the template, removing an extraneous JSON field so we can put it back later, and adding our new field.</p>
+<p>Make sure to set the ELASTICSEARCH variable appropriately. $SENSOR can contain wildcards, so if rollover has occurred, it’s not necessary to do each index individually. The example here appends <tt>index*</tt> to get all indexes for a the provided sensor.</p>
+
+<div class="source">
+<div class="source">
+<pre>export ELASTICSEARCH="node1"
+export SENSOR="bro"
+curl -XGET "http://${ELASTICSEARCH}:9200/_template/${SENSOR}_index*?pretty=true" -o "${SENSOR}.template"
+sed -i '' '2d;$d' ./${SENSOR}.template
+sed -i '' '/"properties" : {/ a\
+"alert": { "type": "nested"},' ${SENSOR}.template
+</pre></div></div>
+<p>To manually verify this, you can optionally pretty print it again with:</p>
+
+<div class="source">
+<div class="source">
+<pre>python -m json.tool bro.template
+</pre></div></div>
+<p>We’ll want to put the template back into Elasticsearch:</p>
+
+<div class="source">
+<div class="source">
+<pre>curl -XPUT "http://${ELASTICSEARCH}:9200/_template/${SENSOR}_index" -d @${SENSOR}.template
+</pre></div></div>
+<p>To update existing indexes, update Elasticsearch mappings with the new field for each sensor. </p>
+
+<div class="source">
+<div class="source">
+<pre>curl -XPUT "http://${ELASTICSEARCH}:9200/${SENSOR}_index*/_mapping/${SENSOR}_doc" -d '
+{
+ "properties" : {
+ "alert" : {
+ "type" : "nested"
+ }
+ }
+}
+'
+rm ${SENSOR}.template
+</pre></div></div></div>
+<div class="section">
+<h2><a name="Installing_Elasticsearch_Templates"></a>Installing Elasticsearch Templates</h2>
+<p>The stock set of Elasticsearch templates for bro, snort, yaf, error index and meta index are installed automatically during the first time install and startup of Metron Indexing service.</p>
+<p>It is possible that Elasticsearch service is not available when the Metron Indexing Service startup, in that case the Elasticsearch template will not be installed. </p>
+<p>For such a scenario, an Admin can have the template installed in two ways:</p>
+<p><i>Method 1</i> - Manually from the Ambari UI by following the flow: Ambari UI -> Services -> Metron -> Service Actions -> Elasticsearch Template Install</p>
+<p><i>Method 2</i> - Stop the Metron Indexing service, and start it again from Ambari UI. Note that the Metron Indexing service tracks if it has successfully installed the Elasticsearch templates, and will attempt to do so each time it is Started until successful.</p>
+
+<blockquote>
+<p>Note: If you have made any customization to your index templates, then installing Elasticsearch templates afresh will lead to overwriting your existing changes. Please exercise caution.</p>
+</blockquote></div>
+ </div>
+ </div>
+ </div>
+
+ <hr/>
+
+ <footer>
+ <div class="container-fluid">
+ <div class="row span12">Copyright © 2018
+ <a href="https://www.apache.org">The Apache Software Foundation</a>.
+ All Rights Reserved.
+
+ </div>
+
+
+
+ </div>
+ </footer>
+ </body>
+</html>
http://git-wip-us.apache.org/repos/asf/metron/blob/3381b853/site/current-book/metron-platform/metron-enrichment/index.html
----------------------------------------------------------------------
diff --git a/site/current-book/metron-platform/metron-enrichment/index.html b/site/current-book/metron-platform/metron-enrichment/index.html
index 979baf1..584c36f 100644
--- a/site/current-book/metron-platform/metron-enrichment/index.html
+++ b/site/current-book/metron-platform/metron-enrichment/index.html
@@ -1,13 +1,13 @@
<!DOCTYPE html>
<!--
- | Generated by Apache Maven Doxia at 2017-09-15
+ | Generated by Apache Maven Doxia at 2018-01-03
| Rendered using Apache Maven Fluido Skin 1.3.0
-->
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
- <meta name="Date-Revision-yyyymmdd" content="20170915" />
+ <meta name="Date-Revision-yyyymmdd" content="20180103" />
<meta http-equiv="Content-Language" content="en" />
<title>Metron – Enrichment</title>
<link rel="stylesheet" href="../../css/apache-maven-fluido-1.3.0.min.css" />
@@ -61,8 +61,8 @@
- <li id="publishDate" class="pull-right">Last Published: 2017-09-15</li> <li class="divider pull-right">|</li>
- <li id="projectVersion" class="pull-right">Version: 0.4.1</li>
+ <li id="publishDate" class="pull-right">Last Published: 2018-01-03</li> <li class="divider pull-right">|</li>
+ <li id="projectVersion" class="pull-right">Version: 0.4.2</li>
</ul>
</div>
@@ -103,7 +103,7 @@
<i class="none"></i>
Docker</a>
</li>
-
+
<li>
<a href="../../metron-deployment/index.html" title="Deployment">
@@ -131,7 +131,7 @@
<i class="none"></i>
Rest</a>
</li>
-
+
<li>
<a href="../../metron-platform/index.html" title="Platform">
@@ -167,6 +167,13 @@
Data-management</a>
</li>
+ <li>
+
+ <a href="../../metron-platform/metron-elasticsearch/index.html" title="Elasticsearch">
+ <i class="none"></i>
+ Elasticsearch</a>
+ </li>
+
<li class="active">
<a href="#"><i class="none"></i>Enrichment</a>
@@ -208,13 +215,20 @@
</li>
</ul>
</li>
-
+
<li>
<a href="../../metron-sensors/index.html" title="Sensors">
<i class="icon-chevron-right"></i>
Sensors</a>
</li>
+
+ <li>
+
+ <a href="../../metron-stellar/stellar-3rd-party-example/index.html" title="Stellar-3rd-party-example">
+ <i class="none"></i>
+ Stellar-3rd-party-example</a>
+ </li>
<li>
@@ -222,7 +236,7 @@
<i class="icon-chevron-right"></i>
Stellar-common</a>
</li>
-
+
<li>
<a href="../../use-cases/index.html" title="Use-cases">
@@ -274,7 +288,14 @@
<p>There are two types of configurations at the moment, <tt>global</tt> and <tt>sensor</tt> specific. </p></div>
<div class="section">
<h2><a name="Global_Configuration"></a>Global Configuration</h2>
-<p>See the “<a href="../metron-common/index.html">Global Configuration</a>” section.</p></div>
+<p>There are a few enrichments which have independent configurations, such as from the global config.</p>
+<p>Also, see the “<a href="../metron-common/index.html">Global Configuration</a>” section for more discussion of the global config.</p>
+<div class="section">
+<h3><a name="GeoIP"></a>GeoIP</h3>
+<p>Metron supports enrichment of IP information using <a class="externalLink" href="https://dev.maxmind.com/geoip/geoip2/geolite2/">GeoLite2</a>. The location of the file is managed in the global config.</p>
+<div class="section">
+<h4><a name="geo.hdfs.file"></a><tt>geo.hdfs.file</tt></h4>
+<p>The location on HDFS of the GeoLite2 database file to use for GeoIP lookups. This file will be localized on the storm supervisors running the topology and used from there. This is lazy, so if this property changes in a running topology, the file will be localized from HDFS upon first time the file is used via the geo enrichment. </p></div></div></div>
<div class="section">
<h2><a name="Sensor_Enrichment_Configuration"></a>Sensor Enrichment Configuration</h2>
<p>The sensor specific configuration is intended to configure the individual enrichments and threat intelligence enrichments for a given sensor type (e.g. <tt>snort</tt>).</p>
@@ -730,7 +751,7 @@
<footer>
<div class="container-fluid">
- <div class="row span12">Copyright © 2017
+ <div class="row span12">Copyright © 2018
<a href="https://www.apache.org">The Apache Software Foundation</a>.
All Rights Reserved.
http://git-wip-us.apache.org/repos/asf/metron/blob/3381b853/site/current-book/metron-platform/metron-indexing/index.html
----------------------------------------------------------------------
diff --git a/site/current-book/metron-platform/metron-indexing/index.html b/site/current-book/metron-platform/metron-indexing/index.html
index fc5fef2..3d79323 100644
--- a/site/current-book/metron-platform/metron-indexing/index.html
+++ b/site/current-book/metron-platform/metron-indexing/index.html
@@ -1,13 +1,13 @@
<!DOCTYPE html>
<!--
- | Generated by Apache Maven Doxia at 2017-09-15
+ | Generated by Apache Maven Doxia at 2018-01-03
| Rendered using Apache Maven Fluido Skin 1.3.0
-->
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
- <meta name="Date-Revision-yyyymmdd" content="20170915" />
+ <meta name="Date-Revision-yyyymmdd" content="20180103" />
<meta http-equiv="Content-Language" content="en" />
<title>Metron – Indexing</title>
<link rel="stylesheet" href="../../css/apache-maven-fluido-1.3.0.min.css" />
@@ -61,8 +61,8 @@
- <li id="publishDate" class="pull-right">Last Published: 2017-09-15</li> <li class="divider pull-right">|</li>
- <li id="projectVersion" class="pull-right">Version: 0.4.1</li>
+ <li id="publishDate" class="pull-right">Last Published: 2018-01-03</li> <li class="divider pull-right">|</li>
+ <li id="projectVersion" class="pull-right">Version: 0.4.2</li>
</ul>
</div>
@@ -103,7 +103,7 @@
<i class="none"></i>
Docker</a>
</li>
-
+
<li>
<a href="../../metron-deployment/index.html" title="Deployment">
@@ -131,7 +131,7 @@
<i class="none"></i>
Rest</a>
</li>
-
+
<li>
<a href="../../metron-platform/index.html" title="Platform">
@@ -169,6 +169,13 @@
<li>
+ <a href="../../metron-platform/metron-elasticsearch/index.html" title="Elasticsearch">
+ <i class="none"></i>
+ Elasticsearch</a>
+ </li>
+
+ <li>
+
<a href="../../metron-platform/metron-enrichment/index.html" title="Enrichment">
<i class="none"></i>
Enrichment</a>
@@ -208,13 +215,20 @@
</li>
</ul>
</li>
-
+
<li>
<a href="../../metron-sensors/index.html" title="Sensors">
<i class="icon-chevron-right"></i>
Sensors</a>
</li>
+
+ <li>
+
+ <a href="../../metron-stellar/stellar-3rd-party-example/index.html" title="Stellar-3rd-party-example">
+ <i class="none"></i>
+ Stellar-3rd-party-example</a>
+ </li>
<li>
@@ -222,7 +236,7 @@
<i class="icon-chevron-right"></i>
Stellar-common</a>
</li>
-
+
<li>
<a href="../../use-cases/index.html" title="Use-cases">
@@ -306,6 +320,12 @@
<li><tt>enabled</tt> : Whether the writer is enabled (default <tt>true</tt>).</li>
</ul>
<div class="section">
+<h3><a name="Meta_Alerts"></a>Meta Alerts</h3>
+<p>Alerts can be grouped, after appropriate searching, into a set of alerts called a meta alert. A meta alert is useful for maintaining the context of searching and grouping during further investigations. Standard searches can return meta alerts, but grouping and other aggregation or sorting requests will not, because there’s not a clear way to aggregate in many cases if there are multiple alerts contained in the meta alert. All meta alerts will have the source type of metaalert, regardless of the contained alert’s origins.</p></div>
+<div class="section">
+<h3><a name="Elasticsearch"></a>Elasticsearch</h3>
+<p>Metron comes with built-in templates for the default sensors for Elasticsearch. When adding a new sensor, it will be necessary to add a new template defining the output fields appropriately. In addition, there is a requirement for a field <tt>alert</tt> of type <tt>nested</tt> for Elasticsearch 2.x installs. This is detailed at <a href="../metron-elasticsearch/index.html#Using_Metron_with_Elasticsearch_2.x">Using Metron with Elasticsearch 2.x</a></p></div>
+<div class="section">
<h3><a name="Indexing_Configuration_Examples"></a>Indexing Configuration Examples</h3>
<p>For a given sensor, the following scenarios would be indicated by the following cases:</p>
<div class="section">
@@ -452,9 +472,39 @@
<h2><a name="The_IndexDao_Abstraction"></a>The <tt>IndexDao</tt> Abstraction</h2>
<p>The indices mentioned above as part of Update should be pluggable by the developer so that new write-ahead logs or real-time indices can be supported by providing an implementation supporting the data access patterns.</p>
<p>To support a new index, one would need to implement the <tt>org.apache.metron.indexing.dao.IndexDao</tt> abstraction and provide update and search capabilities. IndexDaos may be composed and updates will be performed in parallel. This enables a flexible strategy for specifying your backing store for updates at runtime. For instance, currently the REST API supports the update functionality and may be configured with a list of IndexDao implementations to use to support the updates.</p>
+<p>Updates with the IndexDao.update method replace the current object with the new object. For partial updates, use IndexDao.patch instead.</p>
+<div class="section">
+<h3><a name="The_HBaseDao"></a>The <tt>HBaseDao</tt></h3>
+<p>Updates will be written to HBase. The key structure includes the GUID and sensor type and for each new version, a new column is created with value as the message.</p>
+<p>The HBase table and column family are configured via fields in the global configuration.</p>
+<div class="section">
+<h4><a name="update.hbase.table"></a><tt>update.hbase.table</tt></h4>
+<p>The HBase table to use for message updates.</p></div>
+<div class="section">
+<h4><a name="update.hbase.cf"></a><tt>update.hbase.cf</tt></h4>
+<p>The HBase column family to use for message updates.</p></div></div>
+<div class="section">
+<h3><a name="The_MetaAlertDao"></a>The <tt>MetaAlertDao</tt></h3>
+<p>The goal of meta alerts is to be able to group together a set of alerts while being able to transparently perform actions like searches, as if meta alerts were normal alerts. <tt>org.apache.metron.indexing.dao.MetaAlertDao</tt> extends <tt>IndexDao</tt> and enables several features: </p>
+
+<ul>
+
+<li>the ability to get all meta alerts associated with an alert</li>
+
+<li>creation of a meta alert</li>
+
+<li>adding alerts to a meta alert</li>
+
+<li>removing alerts from a meta alert</li>
+
+<li>changing a meta alert’s status</li>
+</ul>
+<p>The implementation of this is to denormalize the relationship between alerts and meta alerts, and store alerts as a nested field within a meta alert. The use of nested fields is to avoid the limitations of parent-child relationships (one-to-many) and merely linking by IDs (which causes issues with pagination as a result of being unable to join indices). A list of containing meta alerts is stored on an alert for the purpose of keeping source alerts and alerts contained in meta alerts in sync.</p>
+<p>The search functionality of <tt>IndexDao</tt> is wrapped by the <tt>MetaAlertDao</tt> in order to provide both regular and meta alerts side-by-side with sorting. The updating capabilities are similarly wrapped, in order to ensure updates are carried through both the alerts and associated meta alerts. Both of these functions are handled under the hood.</p>
+<p>In addition, API endpoints have been added to expose the features listed above. The denormalization handles the case of going from meta alert to alert automatically.</p>
<p><a name="Notes_on_Performance_Tuning"></a></p>
<h1>Notes on Performance Tuning</h1>
-<p>Default installed Metron is untuned for production deployment. By far and wide, the most likely piece to require TLC from a performance perspective is the indexing layer. An index that does not keep up will back up and you will see errors in the kafka bolt. There are a few knobs to tune to get the most out of your system.</p></div>
+<p>Default installed Metron is untuned for production deployment. By far and wide, the most likely piece to require TLC from a performance perspective is the indexing layer. An index that does not keep up will back up and you will see errors in the kafka bolt. There are a few knobs to tune to get the most out of your system.</p></div></div>
<div class="section">
<h2><a name="Kafka_Queue"></a>Kafka Queue</h2>
<p>The <tt>indexing</tt> kafka queue is a collection point from the enrichment topology. As such, make sure that the number of partitions in the kafka topic is sufficient to handle the throughput that you expect.</p></div>
@@ -491,7 +541,7 @@
<footer>
<div class="container-fluid">
- <div class="row span12">Copyright © 2017
+ <div class="row span12">Copyright © 2018
<a href="https://www.apache.org">The Apache Software Foundation</a>.
All Rights Reserved.
http://git-wip-us.apache.org/repos/asf/metron/blob/3381b853/site/current-book/metron-platform/metron-management/index.html
----------------------------------------------------------------------
diff --git a/site/current-book/metron-platform/metron-management/index.html b/site/current-book/metron-platform/metron-management/index.html
index a760baf..cca14f1 100644
--- a/site/current-book/metron-platform/metron-management/index.html
+++ b/site/current-book/metron-platform/metron-management/index.html
@@ -1,13 +1,13 @@
<!DOCTYPE html>
<!--
- | Generated by Apache Maven Doxia at 2017-09-15
+ | Generated by Apache Maven Doxia at 2018-01-03
| Rendered using Apache Maven Fluido Skin 1.3.0
-->
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
- <meta name="Date-Revision-yyyymmdd" content="20170915" />
+ <meta name="Date-Revision-yyyymmdd" content="20180103" />
<meta http-equiv="Content-Language" content="en" />
<title>Metron – Stellar REPL Management Utilities</title>
<link rel="stylesheet" href="../../css/apache-maven-fluido-1.3.0.min.css" />
@@ -61,8 +61,8 @@
- <li id="publishDate" class="pull-right">Last Published: 2017-09-15</li> <li class="divider pull-right">|</li>
- <li id="projectVersion" class="pull-right">Version: 0.4.1</li>
+ <li id="publishDate" class="pull-right">Last Published: 2018-01-03</li> <li class="divider pull-right">|</li>
+ <li id="projectVersion" class="pull-right">Version: 0.4.2</li>
</ul>
</div>
@@ -103,7 +103,7 @@
<i class="none"></i>
Docker</a>
</li>
-
+
<li>
<a href="../../metron-deployment/index.html" title="Deployment">
@@ -131,7 +131,7 @@
<i class="none"></i>
Rest</a>
</li>
-
+
<li>
<a href="../../metron-platform/index.html" title="Platform">
@@ -169,6 +169,13 @@
<li>
+ <a href="../../metron-platform/metron-elasticsearch/index.html" title="Elasticsearch">
+ <i class="none"></i>
+ Elasticsearch</a>
+ </li>
+
+ <li>
+
<a href="../../metron-platform/metron-enrichment/index.html" title="Enrichment">
<i class="none"></i>
Enrichment</a>
@@ -208,13 +215,20 @@
</li>
</ul>
</li>
-
+
<li>
<a href="../../metron-sensors/index.html" title="Sensors">
<i class="icon-chevron-right"></i>
Sensors</a>
</li>
+
+ <li>
+
+ <a href="../../metron-stellar/stellar-3rd-party-example/index.html" title="Stellar-3rd-party-example">
+ <i class="none"></i>
+ Stellar-3rd-party-example</a>
+ </li>
<li>
@@ -222,7 +236,7 @@
<i class="icon-chevron-right"></i>
Stellar-common</a>
</li>
-
+
<li>
<a href="../../use-cases/index.html" title="Use-cases">
@@ -272,8 +286,47 @@
<li>print structured data in a way that is easier to view (i.e. tabular)</li>
</ul>
<p>This functionality is exposed as a pack of Stellar functions in this project.</p>
+
+<ul>
+
+<li><a href="#Functions">Functions</a>
+
+<ul>
+
+<li><a href="#Grok_Functions">Grok Functions</a></li>
+
+<li><a href="#File_Functions">File Functions</a></li>
+
+<li><a href="#Shell_Functions">Shell Functions</a></li>
+
+<li><a href="#Configuration_Functions">Configuration Functions</a></li>
+
+<li><a href="#Parser_Functions">Parser Functions</a></li>
+
+<li><a href="#Indexing_Functions">Indexing Functions</a></li>
+
+<li><a href="#Enrichment_Functions">Enrichment Functions</a></li>
+
+<li><a href="#Threat_Triage_Functions">Threat Triage Functions</a></li>
+ </ul></li>
+
+<li><a href="#Examples">Examples</a>
+
+<ul>
+
+<li><a href="#Iterate_to_Find_a_Valid_Grok_Pattern">Iterate to Find a Valid Grok Pattern</a></li>
+
+<li><a href="#Manage_Stellar_Field_Transformations">Manage Stellar Field Transformations</a></li>
+
+<li><a href="#Manage_Stellar_Enrichments">Manage Stellar Enrichments</a></li>
+
+<li><a href="#Manage_Threat_Triage_Rules">Manage Threat Triage Rules</a></li>
+
+<li><a href="#Simulate_Threat_Triage_Rules">Simulate Threat Triage Rules</a></li>
+ </ul></li>
+</ul>
<div class="section">
-<h2><a name="Function_Details"></a>Function Details</h2>
+<h2><a name="Functions"></a>Functions</h2>
<p>The functions are split roughly into a few sections:</p>
<ul>
@@ -667,7 +720,7 @@
<li>Returns: The String representation of the config in zookeeper</li>
</ul></li>
-<li><tt>PARSER_STELLAR_TRANSFORM_PRINT</tt>
+<li><tt>PARSER-STELLAR_TRANSFORM_PRINT</tt>
<ul>
@@ -840,37 +893,71 @@
<ul>
-<li><tt>THREAT_TRIAGE_ADD</tt>
+<li><tt>THREAT_TRIAGE_INIT</tt>
<ul>
-<li>Description: Add a threat triage rule.</li>
+<li>Description: Create a threat triage engine.</li>
<li>Input:
<ul>
-<li>sensorConfig - Sensor config to add transformation to.</li>
+<li>config - the threat triage configuration (optional)</li>
+ </ul></li>
+
+<li>Returns: A threat triage engine.</li>
+ </ul></li>
+
+<li><tt>THREAT_TRIAGE_CONFIG</tt>
+
+<ul>
+
+<li>Description: Export the configuration used by a threat triage engine.</li>
+
+<li>Input:
+
+<ul>
-<li>stellarTransforms - A Map associating stellar rules to scores</li>
+<li>engine - threat triage engine returned by THREAT_TRIAGE_INIT.</li>
+ </ul></li>
+
+<li>Returns: The configuration used by the threat triage engine.</li>
+ </ul></li>
+
+<li><tt>THREAT_TRIAGE_SCORE</tt>
+
+<ul>
+
+<li>Description: Scores a message using a set of triage rules.</li>
+
+<li>Inputs:
+
+<ul>
-<li>triageRules - Map (or list of Maps) representing a triage rule. It must contain ‘rule’ and ‘score’ keys, the stellar expression for the rule and triage score respectively. It may contain ‘name’ and ‘comment’, the name of the rule and comment associated with the rule respectively."</li>
+<li>message - a string containing the message to score.</li>
+
+<li>engine - threat triage engine returned by THREAT_TRIAGE_INIT.</li>
</ul></li>
-<li>Returns: The String representation of the threat triage rules</li>
+<li>Returns: A threat triage engine.</li>
</ul></li>
-<li><tt>THREAT_TRIAGE_PRINT</tt>
+<li><tt>THREAT_TRIAGE_ADD</tt>
<ul>
-<li>Description: Retrieve stellar enrichment transformations.</li>
+<li>Description: Add a threat triage rule.</li>
<li>Input:
<ul>
<li>sensorConfig - Sensor config to add transformation to.</li>
+
+<li>stellarTransforms - A Map associating stellar rules to scores</li>
+
+<li>triageRules - Map (or list of Maps) representing a triage rule. It must contain ‘rule’ and ‘score’ keys, the stellar expression for the rule and triage score respectively. It may contain ‘name’ and ‘comment’, the name of the rule and comment associated with the rule respectively."</li>
</ul></li>
<li>Returns: The String representation of the threat triage rules</li>
@@ -894,6 +981,22 @@
<li>Returns: The String representation of the enrichment config</li>
</ul></li>
+<li><tt>THREAT_TRIAGE_PRINT</tt>
+
+<ul>
+
+<li>Description: Retrieve stellar enrichment transformations.</li>
+
+<li>Input:
+
+<ul>
+
+<li>sensorConfig - Sensor config to add transformation to.</li>
+ </ul></li>
+
+<li>Returns: The String representation of the threat triage rules</li>
+ </ul></li>
+
<li><tt>THREAT_TRIAGE_SET_AGGREGATOR</tt>
<ul>
@@ -937,7 +1040,7 @@
<h2><a name="Examples"></a>Examples</h2>
<p>Included for description and education purposes are a couple example Stellar REPL transcripts with helpful comments to illustrate some common operations.</p>
<div class="section">
-<h3><a name="Iterate_in_finding_a_valid_Grok_pattern"></a>Iterate in finding a valid Grok pattern</h3>
+<h3><a name="Iterate_to_Find_a_Valid_Grok_pattern"></a>Iterate to Find a Valid Grok pattern</h3>
<div class="source">
<div class="source">
@@ -1644,7 +1747,133 @@ SION('is_both') ] )
"configuration" : { }
}
[Stellar]>>>
-</pre></div></div></div></div>
+</pre></div></div></div>
+<div class="section">
+<h3><a name="Simulate_Threat_Triage_Rules"></a>Simulate Threat Triage Rules</h3>
+
+<ol style="list-style-type: decimal">
+
+<li>
+<p>Create a threat triage engine.</p>
+
+<div class="source">
+<div class="source">
+<pre>[Stellar]>>> t := THREAT_TRIAGE_INIT()
+[Stellar]>>> t
+ThreatTriage{0 rule(s)}
+</pre></div></div></li>
+
+<li>
+<p>Add a few triage rules.</p>
+
+<div class="source">
+<div class="source">
+<pre>[Stellar]>>> THREAT_TRIAGE_ADD(t, {"name":"rule1", "rule":"value>10",
+</pre></div></div>
+
+<div class="source">
+<div class="source">
+<pre>[Stellar]>>> THREAT_TRIAGE_ADD(t, {"name":"rule2", "rule":"value>20", "score":20})
+</pre></div></div>
+
+<div class="source">
+<div class="source">
+<pre>[Stellar]>>> THREAT_TRIAGE_ADD(t, {"name":"rule3", "rule":"value>30", "score":30})
+</pre></div></div></li>
+
+<li>
+<p>Review the rules that you have created.</p>
+
+<div class="source">
+<div class="source">
+<pre>[Stellar]>>> THREAT_TRIAGE_PRINT(t)
+╔═══════╤═════════╤═════════════╤═══════╤════════╗
+║ Name │ Comment │ Triage Rule │ Score │ Reason ║
+╠═══════╪═════════╪═════════════╪═══════╪════════╣
+║ rule1 │ │ value>10 │ 10 │ ║
+╟───────┼─────────┼─────────────┼───────┼────────╢
+║ rule2 │ │ value>20 │ 20 │ ║
+╟───────┼─────────┼─────────────┼───────┼────────╢
+║ rule3 │ │ value>30 │ 30 │ ║
+╚═══════╧═════════╧═════════════╧═══════╧════════╝
+</pre></div></div></li>
+
+<li>
+<p>Create a few test messages to simulate your telemetry.</p>
+
+<div class="source">
+<div class="source">
+<pre>[Stellar]>>> msg1 := "{ \"value\":22 }"
+[Stellar]>>> msg1
+{ "value":22 }
+</pre></div></div>
+
+<div class="source">
+<div class="source">
+<pre>[Stellar]>>> msg2 := "{ \"value\":44 }"
+[Stellar]>>> msg2
+{ "value":44 }
+</pre></div></div></li>
+
+<li>
+<p>Score a message based on the rules that have been defined. The result allows you to see the total score, the aggregator, along with details about each rule that fired.</p>
+
+<div class="source">
+<div class="source">
+<pre>[Stellar]>>> THREAT_TRIAGE_SCORE( msg1, t)
+{score=20.0, aggregator=MAX, rules=[{score=10.0, name=rule1, rule=value>10}, {score=20.0, name=rule2, rule=value>20}]}
+</pre></div></div>
+
+<div class="source">
+<div class="source">
+<pre>[Stellar]>>> THREAT_TRIAGE_SCORE( msg2, t)
+{score=30.0, aggregator=MAX, rules=[{score=10.0, name=rule1, rule=value>10}, {score=20.0, name=rule2, rule=value>20}, {score=30.0, name=rule3, rule=value>30}]}
+</pre></div></div></li>
+
+<li>
+<p>From here you can iterate on your rule set until it does exactly what you need it to do. Once you have a working rule set, extract the configuration and push it into your live, Metron cluster.</p>
+
+<div class="source">
+<div class="source">
+<pre>[Stellar]>>> conf := THREAT_TRIAGE_CONFIG( t)
+[Stellar]>>> conf
+{
+ "enrichment" : {
+ "fieldMap" : { },
+ "fieldToTypeMap" : { },
+ "config" : { }
+ },
+ "threatIntel" : {
+ "fieldMap" : { },
+ "fieldToTypeMap" : { },
+ "config" : { },
+ "triageConfig" : {
+ "riskLevelRules" : [ {
+ "name" : "rule1",
+ "rule" : "value>10",
+ "score" : 10.0
+ }, {
+ "name" : "rule2",
+ "rule" : "value>20",
+ "score" : 20.0
+ }, {
+ "name" : "rule3",
+ "rule" : "value>30",
+ "score" : 30.0
+ }],
+ "aggregator" : "MAX",
+ "aggregationConfig" : { }
+ }
+ },
+ "configuration" : { }
+}
+</pre></div></div>
+
+<div class="source">
+<div class="source">
+<pre>[Stellar]>>> CONFIG_PUT("ENRICHMENT", conf, "bro")
+</pre></div></div></li>
+</ol></div></div>
</div>
</div>
</div>
@@ -1653,7 +1882,7 @@ SION('is_both') ] )
<footer>
<div class="container-fluid">
- <div class="row span12">Copyright © 2017
+ <div class="row span12">Copyright © 2018
<a href="https://www.apache.org">The Apache Software Foundation</a>.
All Rights Reserved.
http://git-wip-us.apache.org/repos/asf/metron/blob/3381b853/site/current-book/metron-platform/metron-parsers/index.html
----------------------------------------------------------------------
diff --git a/site/current-book/metron-platform/metron-parsers/index.html b/site/current-book/metron-platform/metron-parsers/index.html
index 529badc..23955ac 100644
--- a/site/current-book/metron-platform/metron-parsers/index.html
+++ b/site/current-book/metron-platform/metron-parsers/index.html
@@ -1,13 +1,13 @@
<!DOCTYPE html>
<!--
- | Generated by Apache Maven Doxia at 2017-09-15
+ | Generated by Apache Maven Doxia at 2018-01-03
| Rendered using Apache Maven Fluido Skin 1.3.0
-->
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
- <meta name="Date-Revision-yyyymmdd" content="20170915" />
+ <meta name="Date-Revision-yyyymmdd" content="20180103" />
<meta http-equiv="Content-Language" content="en" />
<title>Metron – Parsers</title>
<link rel="stylesheet" href="../../css/apache-maven-fluido-1.3.0.min.css" />
@@ -61,8 +61,8 @@
- <li id="publishDate" class="pull-right">Last Published: 2017-09-15</li> <li class="divider pull-right">|</li>
- <li id="projectVersion" class="pull-right">Version: 0.4.1</li>
+ <li id="publishDate" class="pull-right">Last Published: 2018-01-03</li> <li class="divider pull-right">|</li>
+ <li id="projectVersion" class="pull-right">Version: 0.4.2</li>
</ul>
</div>
@@ -103,7 +103,7 @@
<i class="none"></i>
Docker</a>
</li>
-
+
<li>
<a href="../../metron-deployment/index.html" title="Deployment">
@@ -131,7 +131,7 @@
<i class="none"></i>
Rest</a>
</li>
-
+
<li>
<a href="../../metron-platform/index.html" title="Platform">
@@ -169,6 +169,13 @@
<li>
+ <a href="../../metron-platform/metron-elasticsearch/index.html" title="Elasticsearch">
+ <i class="none"></i>
+ Elasticsearch</a>
+ </li>
+
+ <li>
+
<a href="../../metron-platform/metron-enrichment/index.html" title="Enrichment">
<i class="none"></i>
Enrichment</a>
@@ -217,13 +224,20 @@
</li>
</ul>
</li>
-
+
<li>
<a href="../../metron-sensors/index.html" title="Sensors">
<i class="icon-chevron-right"></i>
Sensors</a>
</li>
+
+ <li>
+
+ <a href="../../metron-stellar/stellar-3rd-party-example/index.html" title="Stellar-3rd-party-example">
+ <i class="none"></i>
+ Stellar-3rd-party-example</a>
+ </li>
<li>
@@ -231,7 +245,7 @@
<i class="icon-chevron-right"></i>
Stellar-common</a>
</li>
-
+
<li>
<a href="../../use-cases/index.html" title="Use-cases">
@@ -373,7 +387,29 @@
</pre></div></div></div>
<div class="section">
<h2><a name="Global_Configuration"></a>Global Configuration</h2>
-<p>See the “<a href="../metron-common/index.html">Global Configuration</a>” section.</p></div>
+<p>There are a few properties which can be managed in the global configuration that have pertinence to parsers and parsing in general.</p>
+<div class="section">
+<h3><a name="parser.error.topic"></a><tt>parser.error.topic</tt></h3>
+<p>The topic where messages which were unable to be parsed due to error are sent. Error messages will be indexed under a sensor type of <tt>error</tt> and the messages will have the following fields:</p>
+
+<ul>
+
+<li><tt>sensor.type</tt>: <tt>error</tt></li>
+
+<li><tt>failed_sensor_type</tt> : The sensor type of the message which wasn’t able to be parsed</li>
+
+<li><tt>error_type</tt> : The error type, in this case <tt>parser</tt>.</li>
+
+<li><tt>stack</tt> : The stack trace of the error</li>
+
+<li><tt>hostname</tt> : The hostname of the node where the error happened</li>
+
+<li><tt>raw_message</tt> : The raw message in string form</li>
+
+<li><tt>raw_message_bytes</tt> : The raw message bytes</li>
+
+<li><tt>error_hash</tt> : A hash of the error message</li>
+</ul></div></div>
<div class="section">
<h2><a name="Parser_Configuration"></a>Parser Configuration</h2>
<p>The configuration for the various parser topologies is defined by JSON documents stored in zookeeper.</p>
@@ -709,37 +745,41 @@ HH:mm:ss', MAP_GET(dc, dc2tz, 'UTC') )"
<div class="source">
<div class="source">
<pre>usage: start_parser_topology.sh
- -e,--extra_topology_options <JSON_FILE> Extra options in the form
- of a JSON file with a map
- for content.
- -esc,--extra_kafka_spout_config <JSON_FILE> Extra spout config options
- in the form of a JSON file
- with a map for content.
- Possible keys are:
- retryDelayMaxMs,retryDelay
- Multiplier,retryInitialDel
- ayMs,stateUpdateIntervalMs
- ,bufferSizeBytes,fetchMaxW
- ait,fetchSizeBytes,maxOffs
- etBehind,metricsTimeBucket
- SizeInSecs,socketTimeoutMs
- -ewnt,--error_writer_num_tasks <NUM_TASKS> Error Writer Num Tasks
- -ewp,--error_writer_p <PARALLELISM_HINT> Error Writer Parallelism
- Hint
- -h,--help This screen
- -k,--kafka <BROKER_URL> Kafka Broker URL
- -mt,--message_timeout <TIMEOUT_IN_SECS> Message Timeout in Seconds
- -mtp,--max_task_parallelism <MAX_TASK> Max task parallelism
- -na,--num_ackers <NUM_ACKERS> Number of Ackers
- -nw,--num_workers <NUM_WORKERS> Number of Workers
- -pnt,--parser_num_tasks <NUM_TASKS> Parser Num Tasks
- -pp,--parser_p <PARALLELISM_HINT> Parser Parallelism Hint
- -s,--sensor <SENSOR_TYPE> Sensor Type
- -snt,--spout_num_tasks <NUM_TASKS> Spout Num Tasks
- -sp,--spout_p <SPOUT_PARALLELISM_HINT> Spout Parallelism Hint
- -t,--test <TEST> Run in Test Mode
- -z,--zk <ZK_QUORUM> Zookeeper Quroum URL
- (zk1:2181,zk2:2181,...
+ -e,--extra_topology_options <JSON_FILE> Extra options in the form
+ of a JSON file with a map
+ for content.
+ -esc,--extra_kafka_spout_config <JSON_FILE> Extra spout config options
+ in the form of a JSON file
+ with a map for content.
+ Possible keys are:
+ retryDelayMaxMs,retryDelay
+ Multiplier,retryInitialDel
+ ayMs,stateUpdateIntervalMs
+ ,bufferSizeBytes,fetchMaxW
+ ait,fetchSizeBytes,maxOffs
+ etBehind,metricsTimeBucket
+ SizeInSecs,socketTimeoutMs
+ -ewnt,--error_writer_num_tasks <NUM_TASKS> Error Writer Num Tasks
+ -ewp,--error_writer_p <PARALLELISM_HINT> Error Writer Parallelism
+ Hint
+ -h,--help This screen
+ -iwnt,--invalid_writer_num_tasks <NUM_TASKS> Invalid Writer Num Tasks
+ -iwp,--invalid_writer_p <PARALLELISM_HINT> Invalid Message Writer Parallelism Hint
+ -k,--kafka <BROKER_URL> Kafka Broker URL
+ -ksp,--kafka_security_protocol <SECURITY_PROTOCOL> Kafka Security Protocol
+ -mt,--message_timeout <TIMEOUT_IN_SECS> Message Timeout in Seconds
+ -mtp,--max_task_parallelism <MAX_TASK> Max task parallelism
+ -na,--num_ackers <NUM_ACKERS> Number of Ackers
+ -nw,--num_workers <NUM_WORKERS> Number of Workers
+ -ot,--output_topic <KAFKA_TOPIC> Output Kafka Topic
+ -pnt,--parser_num_tasks <NUM_TASKS> Parser Num Tasks
+ -pp,--parser_p <PARALLELISM_HINT> Parser Parallelism Hint
+ -s,--sensor <SENSOR_TYPE> Sensor Type
+ -snt,--spout_num_tasks <NUM_TASKS> Spout Num Tasks
+ -sp,--spout_p <SPOUT_PARALLELISM_HINT> Spout Parallelism Hint
+ -t,--test <TEST> Run in Test Mode
+ -z,--zk <ZK_QUORUM> Zookeeper Quroum URL
+ (zk1:2181,zk2:2181,...
</pre></div></div></div></div>
<div class="section">
<h2><a name="The_--extra_kafka_spout_config_Option"></a>The <tt>--extra_kafka_spout_config</tt> Option</h2>
@@ -793,7 +833,11 @@ HH:mm:ss', MAP_GET(dc, dc2tz, 'UTC') )"
<p>and pass <tt>--extra_topology_options custom_config.json</tt> to <tt>start_parser_topology.sh</tt>.</p>
<p><a name="Notes_on_Performance_Tuning"></a></p>
<h1>Notes on Performance Tuning</h1>
-<p>Default installed Metron is untuned for production deployment. There are a few knobs to tune to get the most out of your system.</p></div>
+<p>Default installed Metron is untuned for production deployment. There are a few knobs to tune to get the most out of your system.</p>
+<p><a name="Notes_on_Adding_a_New_Sensor"></a></p>
+<h1>Notes on Adding a New Sensor</h1>
+<p>In order to allow for meta alerts to be queries alongside regular alerts in Elasticsearch 2.x, it is necessary to add an additional field to the templates and mapping for existing sensors.</p>
+<p>Please see a description of the steps necessary to make this change in the metron-elasticsearch <a href="../../metron-platform/metron-elasticsearch/index.html#Using_Metron_with_Elasticsearch_2.x">Using Metron with Elasticsearch 2.x</a></p></div>
<div class="section">
<h2><a name="Kafka_Queue"></a>Kafka Queue</h2>
<p>The kafka queue associated with your parser is a collection point for all of the data sent to your parser. As such, make sure that the number of partitions in the kafka topic is sufficient to handle the throughput that you expect from your parser topology.</p></div>
@@ -857,7 +901,7 @@ HH:mm:ss', MAP_GET(dc, dc2tz, 'UTC') )"
<footer>
<div class="container-fluid">
- <div class="row span12">Copyright © 2017
+ <div class="row span12">Copyright © 2018
<a href="https://www.apache.org">The Apache Software Foundation</a>.
All Rights Reserved.
http://git-wip-us.apache.org/repos/asf/metron/blob/3381b853/site/current-book/metron-platform/metron-parsers/parser-testing.html
----------------------------------------------------------------------
diff --git a/site/current-book/metron-platform/metron-parsers/parser-testing.html b/site/current-book/metron-platform/metron-parsers/parser-testing.html
index 25288a3..0edfd53 100644
--- a/site/current-book/metron-platform/metron-parsers/parser-testing.html
+++ b/site/current-book/metron-platform/metron-parsers/parser-testing.html
@@ -1,13 +1,13 @@
<!DOCTYPE html>
<!--
- | Generated by Apache Maven Doxia at 2017-09-15
+ | Generated by Apache Maven Doxia at 2018-01-03
| Rendered using Apache Maven Fluido Skin 1.3.0
-->
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
- <meta name="Date-Revision-yyyymmdd" content="20170915" />
+ <meta name="Date-Revision-yyyymmdd" content="20180103" />
<meta http-equiv="Content-Language" content="en" />
<title>Metron – Parser Contribution and Testing</title>
<link rel="stylesheet" href="../../css/apache-maven-fluido-1.3.0.min.css" />
@@ -61,8 +61,8 @@
- <li id="publishDate" class="pull-right">Last Published: 2017-09-15</li> <li class="divider pull-right">|</li>
- <li id="projectVersion" class="pull-right">Version: 0.4.1</li>
+ <li id="publishDate" class="pull-right">Last Published: 2018-01-03</li> <li class="divider pull-right">|</li>
+ <li id="projectVersion" class="pull-right">Version: 0.4.2</li>
</ul>
</div>
@@ -103,7 +103,7 @@
<i class="none"></i>
Docker</a>
</li>
-
+
<li>
<a href="../../metron-deployment/index.html" title="Deployment">
@@ -131,7 +131,7 @@
<i class="none"></i>
Rest</a>
</li>
-
+
<li>
<a href="../../metron-platform/index.html" title="Platform">
@@ -169,6 +169,13 @@
<li>
+ <a href="../../metron-platform/metron-elasticsearch/index.html" title="Elasticsearch">
+ <i class="none"></i>
+ Elasticsearch</a>
+ </li>
+
+ <li>
+
<a href="../../metron-platform/metron-enrichment/index.html" title="Enrichment">
<i class="none"></i>
Enrichment</a>
@@ -217,13 +224,20 @@
</li>
</ul>
</li>
-
+
<li>
<a href="../../metron-sensors/index.html" title="Sensors">
<i class="icon-chevron-right"></i>
Sensors</a>
</li>
+
+ <li>
+
+ <a href="../../metron-stellar/stellar-3rd-party-example/index.html" title="Stellar-3rd-party-example">
+ <i class="none"></i>
+ Stellar-3rd-party-example</a>
+ </li>
<li>
@@ -231,7 +245,7 @@
<i class="icon-chevron-right"></i>
Stellar-common</a>
</li>
-
+
<li>
<a href="../../use-cases/index.html" title="Use-cases">
@@ -358,7 +372,7 @@
<footer>
<div class="container-fluid">
- <div class="row span12">Copyright © 2017
+ <div class="row span12">Copyright © 2018
<a href="https://www.apache.org">The Apache Software Foundation</a>.
All Rights Reserved.
http://git-wip-us.apache.org/repos/asf/metron/blob/3381b853/site/current-book/metron-platform/metron-pcap-backend/index.html
----------------------------------------------------------------------
diff --git a/site/current-book/metron-platform/metron-pcap-backend/index.html b/site/current-book/metron-platform/metron-pcap-backend/index.html
index 601fac8..01961eb 100644
--- a/site/current-book/metron-platform/metron-pcap-backend/index.html
+++ b/site/current-book/metron-platform/metron-pcap-backend/index.html
@@ -1,13 +1,13 @@
<!DOCTYPE html>
<!--
- | Generated by Apache Maven Doxia at 2017-09-15
+ | Generated by Apache Maven Doxia at 2018-01-03
| Rendered using Apache Maven Fluido Skin 1.3.0
-->
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
- <meta name="Date-Revision-yyyymmdd" content="20170915" />
+ <meta name="Date-Revision-yyyymmdd" content="20180103" />
<meta http-equiv="Content-Language" content="en" />
<title>Metron – Metron PCAP Backend</title>
<link rel="stylesheet" href="../../css/apache-maven-fluido-1.3.0.min.css" />
@@ -61,8 +61,8 @@
- <li id="publishDate" class="pull-right">Last Published: 2017-09-15</li> <li class="divider pull-right">|</li>
- <li id="projectVersion" class="pull-right">Version: 0.4.1</li>
+ <li id="publishDate" class="pull-right">Last Published: 2018-01-03</li> <li class="divider pull-right">|</li>
+ <li id="projectVersion" class="pull-right">Version: 0.4.2</li>
</ul>
</div>
@@ -103,7 +103,7 @@
<i class="none"></i>
Docker</a>
</li>
-
+
<li>
<a href="../../metron-deployment/index.html" title="Deployment">
@@ -131,7 +131,7 @@
<i class="none"></i>
Rest</a>
</li>
-
+
<li>
<a href="../../metron-platform/index.html" title="Platform">
@@ -169,6 +169,13 @@
<li>
+ <a href="../../metron-platform/metron-elasticsearch/index.html" title="Elasticsearch">
+ <i class="none"></i>
+ Elasticsearch</a>
+ </li>
+
+ <li>
+
<a href="../../metron-platform/metron-enrichment/index.html" title="Enrichment">
<i class="none"></i>
Enrichment</a>
@@ -208,13 +215,20 @@
</li>
</ul>
</li>
-
+
<li>
<a href="../../metron-sensors/index.html" title="Sensors">
<i class="icon-chevron-right"></i>
Sensors</a>
</li>
+
+ <li>
+
+ <a href="../../metron-stellar/stellar-3rd-party-example/index.html" title="Stellar-3rd-party-example">
+ <i class="none"></i>
+ Stellar-3rd-party-example</a>
+ </li>
<li>
@@ -222,7 +236,7 @@
<i class="icon-chevron-right"></i>
Stellar-common</a>
</li>
-
+
<li>
<a href="../../use-cases/index.html" title="Use-cases">
@@ -648,7 +662,7 @@ dfs.blocksize
<footer>
<div class="container-fluid">
- <div class="row span12">Copyright © 2017
+ <div class="row span12">Copyright © 2018
<a href="https://www.apache.org">The Apache Software Foundation</a>.
All Rights Reserved.
http://git-wip-us.apache.org/repos/asf/metron/blob/3381b853/site/current-book/metron-platform/metron-writer/index.html
----------------------------------------------------------------------
diff --git a/site/current-book/metron-platform/metron-writer/index.html b/site/current-book/metron-platform/metron-writer/index.html
index 9464a4d..b9c2125 100644
--- a/site/current-book/metron-platform/metron-writer/index.html
+++ b/site/current-book/metron-platform/metron-writer/index.html
@@ -1,13 +1,13 @@
<!DOCTYPE html>
<!--
- | Generated by Apache Maven Doxia at 2017-09-15
+ | Generated by Apache Maven Doxia at 2018-01-03
| Rendered using Apache Maven Fluido Skin 1.3.0
-->
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
- <meta name="Date-Revision-yyyymmdd" content="20170915" />
+ <meta name="Date-Revision-yyyymmdd" content="20180103" />
<meta http-equiv="Content-Language" content="en" />
<title>Metron – Writer</title>
<link rel="stylesheet" href="../../css/apache-maven-fluido-1.3.0.min.css" />
@@ -61,8 +61,8 @@
- <li id="publishDate" class="pull-right">Last Published: 2017-09-15</li> <li class="divider pull-right">|</li>
- <li id="projectVersion" class="pull-right">Version: 0.4.1</li>
+ <li id="publishDate" class="pull-right">Last Published: 2018-01-03</li> <li class="divider pull-right">|</li>
+ <li id="projectVersion" class="pull-right">Version: 0.4.2</li>
</ul>
</div>
@@ -103,7 +103,7 @@
<i class="none"></i>
Docker</a>
</li>
-
+
<li>
<a href="../../metron-deployment/index.html" title="Deployment">
@@ -131,7 +131,7 @@
<i class="none"></i>
Rest</a>
</li>
-
+
<li>
<a href="../../metron-platform/index.html" title="Platform">
@@ -169,6 +169,13 @@
<li>
+ <a href="../../metron-platform/metron-elasticsearch/index.html" title="Elasticsearch">
+ <i class="none"></i>
+ Elasticsearch</a>
+ </li>
+
+ <li>
+
<a href="../../metron-platform/metron-enrichment/index.html" title="Enrichment">
<i class="none"></i>
Enrichment</a>
@@ -208,13 +215,20 @@
</li>
</ul>
</li>
-
+
<li>
<a href="../../metron-sensors/index.html" title="Sensors">
<i class="icon-chevron-right"></i>
Sensors</a>
</li>
+
+ <li>
+
+ <a href="../../metron-stellar/stellar-3rd-party-example/index.html" title="Stellar-3rd-party-example">
+ <i class="none"></i>
+ Stellar-3rd-party-example</a>
+ </li>
<li>
@@ -222,7 +236,7 @@
<i class="icon-chevron-right"></i>
Stellar-common</a>
</li>
-
+
<li>
<a href="../../use-cases/index.html" title="Use-cases">
@@ -335,7 +349,7 @@ limitations under the License. --><h1>Writer</h1>
<footer>
<div class="container-fluid">
- <div class="row span12">Copyright © 2017
+ <div class="row span12">Copyright © 2018
<a href="https://www.apache.org">The Apache Software Foundation</a>.
All Rights Reserved.
[4/7] metron git commit: METRON-1388 update public web site to point
at 0.4.2 new release (mattf-horton) closes apache/metron#887
Posted by ma...@apache.org.
http://git-wip-us.apache.org/repos/asf/metron/blob/3381b853/site/current-book/metron-interface/metron-config/index.html
----------------------------------------------------------------------
diff --git a/site/current-book/metron-interface/metron-config/index.html b/site/current-book/metron-interface/metron-config/index.html
index e3bf4b6..e231b51 100644
--- a/site/current-book/metron-interface/metron-config/index.html
+++ b/site/current-book/metron-interface/metron-config/index.html
@@ -1,13 +1,13 @@
<!DOCTYPE html>
<!--
- | Generated by Apache Maven Doxia at 2017-09-15
+ | Generated by Apache Maven Doxia at 2018-01-03
| Rendered using Apache Maven Fluido Skin 1.3.0
-->
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
- <meta name="Date-Revision-yyyymmdd" content="20170915" />
+ <meta name="Date-Revision-yyyymmdd" content="20180103" />
<meta http-equiv="Content-Language" content="en" />
<title>Metron – Metron Management UI</title>
<link rel="stylesheet" href="../../css/apache-maven-fluido-1.3.0.min.css" />
@@ -61,8 +61,8 @@
- <li id="publishDate" class="pull-right">Last Published: 2017-09-15</li> <li class="divider pull-right">|</li>
- <li id="projectVersion" class="pull-right">Version: 0.4.1</li>
+ <li id="publishDate" class="pull-right">Last Published: 2018-01-03</li> <li class="divider pull-right">|</li>
+ <li id="projectVersion" class="pull-right">Version: 0.4.2</li>
</ul>
</div>
@@ -103,7 +103,7 @@
<i class="none"></i>
Docker</a>
</li>
-
+
<li>
<a href="../../metron-deployment/index.html" title="Deployment">
@@ -129,20 +129,27 @@
<i class="none"></i>
Rest</a>
</li>
-
+
<li>
<a href="../../metron-platform/index.html" title="Platform">
<i class="icon-chevron-right"></i>
Platform</a>
</li>
-
+
<li>
<a href="../../metron-sensors/index.html" title="Sensors">
<i class="icon-chevron-right"></i>
Sensors</a>
</li>
+
+ <li>
+
+ <a href="../../metron-stellar/stellar-3rd-party-example/index.html" title="Stellar-3rd-party-example">
+ <i class="none"></i>
+ Stellar-3rd-party-example</a>
+ </li>
<li>
@@ -150,7 +157,7 @@
<i class="icon-chevron-right"></i>
Stellar-common</a>
</li>
-
+
<li>
<a href="../../use-cases/index.html" title="Use-cases">
@@ -189,7 +196,7 @@
<li>A network accessible Metron REST application</li>
-<li>nodejs v6.9+ (nodejs can be installed on quick dev with <tt>curl --silent --location https://rpm.nodesource.com/setup_6.x | bash - && yum install -y nodejs</tt>)</li>
+<li>nodejs v6.9+ (nodejs can be installed on Full Dev with <tt>curl --silent --location https://rpm.nodesource.com/setup_6.x | bash - && yum install -y nodejs</tt>)</li>
</ul></div>
<div class="section">
<h2><a name="Installation"></a>Installation</h2>
@@ -327,7 +334,7 @@ npm install
<footer>
<div class="container-fluid">
- <div class="row span12">Copyright © 2017
+ <div class="row span12">Copyright © 2018
<a href="https://www.apache.org">The Apache Software Foundation</a>.
All Rights Reserved.
http://git-wip-us.apache.org/repos/asf/metron/blob/3381b853/site/current-book/metron-interface/metron-rest/index.html
----------------------------------------------------------------------
diff --git a/site/current-book/metron-interface/metron-rest/index.html b/site/current-book/metron-interface/metron-rest/index.html
index 8f549d5..f673ca6 100644
--- a/site/current-book/metron-interface/metron-rest/index.html
+++ b/site/current-book/metron-interface/metron-rest/index.html
@@ -1,13 +1,13 @@
<!DOCTYPE html>
<!--
- | Generated by Apache Maven Doxia at 2017-09-15
+ | Generated by Apache Maven Doxia at 2018-01-03
| Rendered using Apache Maven Fluido Skin 1.3.0
-->
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
- <meta name="Date-Revision-yyyymmdd" content="20170915" />
+ <meta name="Date-Revision-yyyymmdd" content="20180103" />
<meta http-equiv="Content-Language" content="en" />
<title>Metron – Metron REST</title>
<link rel="stylesheet" href="../../css/apache-maven-fluido-1.3.0.min.css" />
@@ -61,8 +61,8 @@
- <li id="publishDate" class="pull-right">Last Published: 2017-09-15</li> <li class="divider pull-right">|</li>
- <li id="projectVersion" class="pull-right">Version: 0.4.1</li>
+ <li id="publishDate" class="pull-right">Last Published: 2018-01-03</li> <li class="divider pull-right">|</li>
+ <li id="projectVersion" class="pull-right">Version: 0.4.2</li>
</ul>
</div>
@@ -103,7 +103,7 @@
<i class="none"></i>
Docker</a>
</li>
-
+
<li>
<a href="../../metron-deployment/index.html" title="Deployment">
@@ -129,20 +129,27 @@
<a href="#"><i class="none"></i>Rest</a>
</li>
-
+
<li>
<a href="../../metron-platform/index.html" title="Platform">
<i class="icon-chevron-right"></i>
Platform</a>
</li>
-
+
<li>
<a href="../../metron-sensors/index.html" title="Sensors">
<i class="icon-chevron-right"></i>
Sensors</a>
</li>
+
+ <li>
+
+ <a href="../../metron-stellar/stellar-3rd-party-example/index.html" title="Stellar-3rd-party-example">
+ <i class="none"></i>
+ Stellar-3rd-party-example</a>
+ </li>
<li>
@@ -150,7 +157,7 @@
<i class="icon-chevron-right"></i>
Stellar-common</a>
</li>
-
+
<li>
<a href="../../use-cases/index.html" title="Use-cases">
@@ -332,17 +339,6 @@ lib
<tr class="b">
-<td>METRON_USER </td>
-
-<td>Run the application as this user </td>
-
-<td>Optional </td>
-
-<td>metron</td>
- </tr>
-
-<tr class="a">
-
<td>METRON_LOG_DIR </td>
<td>Directory where the log file is written </td>
@@ -352,18 +348,18 @@ lib
<td>/var/log/metron/</td>
</tr>
-<tr class="b">
+<tr class="a">
-<td>METRON_PID_DIR </td>
+<td>METRON_PID_FILE </td>
-<td>Directory where the pid file is written </td>
+<td>File where the pid is written </td>
<td>Optional </td>
<td>/var/run/metron/</td>
</tr>
-<tr class="a">
+<tr class="b">
<td>METRON_REST_PORT </td>
@@ -374,7 +370,7 @@ lib
<td>8082</td>
</tr>
-<tr class="b">
+<tr class="a">
<td>METRON_JDBC_CLIENT_PATH </td>
@@ -385,7 +381,7 @@ lib
<td>H2 is bundled</td>
</tr>
-<tr class="a">
+<tr class="b">
<td>METRON_TEMP_GROK_PATH </td>
@@ -396,7 +392,7 @@ lib
<td>./patterns/temp</td>
</tr>
-<tr class="b">
+<tr class="a">
<td>METRON_DEFAULT_GROK_PATH </td>
@@ -407,7 +403,7 @@ lib
<td>/apps/metron/patterns</td>
</tr>
-<tr class="a">
+<tr class="b">
<td>SECURITY_ENABLED </td>
@@ -482,27 +478,31 @@ lib
</tr>
</tbody>
</table>
-<p>These are set in the <tt>/etc/sysconfig/metron</tt> file.</p></div></div>
+<p>These are set in the <tt>/etc/default/metron</tt> file.</p></div></div>
<div class="section">
<h2><a name="Database_setup"></a>Database setup</h2>
-<p>The REST application persists data in a relational database and requires a dedicated database user and database (see <a class="externalLink" href="https://docs.spring.io/spring-boot/docs/current/reference/html/boot-features-sql.html">https://docs.spring.io/spring-boot/docs/current/reference/html/boot-features-sql.html</a> for more detail).</p>
+<p>The REST application persists data in a relational database and requires a dedicated database user and database (see <a class="externalLink" href="https://docs.spring.io/spring-boot/docs/current/reference/html/boot-features-sql.html">https://docs.spring.io/spring-boot/docs/current/reference/html/boot-features-sql.html</a> for more detail).<br />Spring uses Hibernate as the default ORM framework but another framework is needed becaused Hibernate is not compatible with the Apache 2 license. For this reason Metron uses <a class="externalLink" href="https://docs.spring.io/spring-boot/docs/current/reference/html/boot-features-sql.html#boot-features-embedded-database-support">EclipseLink</a>. See the <a class="externalLink" href="https://github.com/spring-projects/spring-data-examples/tree/master/jpa/eclipselink">Spring Data JPA - EclipseLink</a> project for an example on how to configure EclipseLink in Spring.</p>
<div class="section">
<h3><a name="Development"></a>Development</h3>
-<p>The REST application comes with embedded database support for development purposes (<a class="externalLink" href="https://docs.spring.io/spring-boot/docs/current/reference/html/boot-features-sql.html#boot-features-embedded-database-support)">https://docs.spring.io/spring-boot/docs/current/reference/html/boot-features-sql.html#boot-features-embedded-database-support)</a>.</p>
-<p>For example, edit these variables in <tt>/etc/sysconfig/metron</tt> before starting the application to configure H2:</p>
+<p>The REST application comes with <a class="externalLink" href="https://docs.spring.io/spring-boot/docs/current/reference/html/boot-features-sql.html#boot-features-embedded-database-support">embedded database support</a> for development purposes.</p>
+<p>For example, edit these variables in <tt>/etc/default/metron</tt> before starting the application to configure H2:</p>
<div class="source">
<div class="source">
<pre>METRON_JDBC_DRIVER="org.h2.Driver"
METRON_JDBC_URL="jdbc:h2:file:~/metrondb"
METRON_JDBC_USERNAME="root"
-METRON_JDBC_PASSWORD='root"
METRON_JDBC_PLATFORM="h2"
</pre></div></div></div>
<div class="section">
<h3><a name="Production"></a>Production</h3>
<p>The REST application should be configured with a production-grade database outside of development.</p>
-<p>For example, the following configures the application for MySQL:</p>
+<div class="section">
+<h4><a name="Ambari_Install"></a>Ambari Install</h4>
+<p>Installing with Ambari is recommended for production deployments. Ambari handles setup, configuration, and management of the REST component. This includes managing the PID file, directing logging, etc.</p></div>
+<div class="section">
+<h4><a name="Manual_Install"></a>Manual Install</h4>
+<p>The following configures the application for MySQL:</p>
<ol style="list-style-type: decimal">
@@ -530,7 +530,7 @@ tar xf mysql-connector-java-5.1.41.tar.gz
</pre></div></div></li>
<li>
-<p>Edit these variables in <tt>/etc/sysconfig/metron</tt> to configure the REST application for MySQL:</p>
+<p>Edit these variables in <tt>/etc/default/metron</tt> to configure the REST application for MySQL:</p>
<div class="source">
<div class="source">
@@ -540,21 +540,30 @@ METRON_JDBC_USERNAME="metron"
METRON_JDBC_PLATFORM="mysql"
METRON_JDBC_CLIENT_PATH=$METRON_HOME/lib/mysql-connector-java-5.1.41/mysql-connector-java-5.1.41-bin.jar
</pre></div></div></li>
-</ol></div></div>
-<div class="section">
-<h2><a name="Usage"></a>Usage</h2>
-<p>After configuration is complete, the REST application can be managed as a service:</p>
-
+
+<li>
+<p>Switch to the metron user</p>
+
<div class="source">
<div class="source">
-<pre>service metron-rest start
-</pre></div></div>
-<p>If a production database is configured, the JDBC password should be passed in as the first argument on startup:</p>
-
+<pre>sudo su - metron
+</pre></div></div></li>
+
+<li>
+<p>Start the REST API. Adjust the password as necessary.</p>
+
<div class="source">
<div class="source">
-<pre>service metron-rest start Myp@ssw0rd
-</pre></div></div>
+<pre>set -o allexport;
+source /etc/default/metron;
+set +o allexport;
+export METRON_JDBC_PASSWORD='Myp@ssw0rd';
+$METRON_HOME/bin/metron-rest.sh
+unset METRON_JDBC_PASSWORD;
+</pre></div></div></li>
+</ol></div></div></div>
+<div class="section">
+<h2><a name="Usage"></a>Usage</h2>
<p>The REST application can be accessed with the Swagger UI at <a class="externalLink" href="http://host:port/swagger-ui.html#/">http://host:port/swagger-ui.html#/</a>. The default port is 8082.</p></div>
<div class="section">
<h2><a name="Security"></a>Security</h2>
@@ -570,7 +579,7 @@ insert into authorities (username, authority) values ('your_username', 'ROLE_USE
</pre></div></div></div>
<div class="section">
<h3><a name="Kerberos"></a>Kerberos</h3>
-<p>Metron REST can be configured for a cluster with Kerberos enabled. A client JAAS file is required for Kafka and Zookeeper and a Kerberos keytab for the metron user principal is required for all other services. Configure these settings in the <tt>/etc/sysconfig/metron</tt> file:</p>
+<p>Metron REST can be configured for a cluster with Kerberos enabled. A client JAAS file is required for Kafka and Zookeeper and a Kerberos keytab for the metron user principal is required for all other services. Configure these settings in the <tt>/etc/default/metron</tt> file:</p>
<div class="source">
<div class="source">
@@ -624,7 +633,7 @@ METRON_SERVICE_KEYTAB="/etc/security/keytabs/metron.keytab"
</tr>
</tbody>
</table>
-<p>Setting active profiles is done with the METRON_SPRING_PROFILES_ACTIVE variable. For example, set this variable in <tt>/etc/sysconfig/metron</tt> to configure the REST application for the Vagrant environment and add a test user:</p>
+<p>Setting active profiles is done with the METRON_SPRING_PROFILES_ACTIVE variable. For example, set this variable in <tt>/etc/default/metron</tt> to configure the REST application for the Vagrant environment and add a test user:</p>
<div class="source">
<div class="source">
@@ -651,6 +660,26 @@ METRON_SERVICE_KEYTAB="/etc/security/keytabs/metron.keytab"
<tr class="a">
+<td><a href="#GET_apiv1alertprofile"> <tt>GET /api/v1/alert/profile</tt></a></td>
+ </tr>
+
+<tr class="b">
+
+<td><a href="#GET_apiv1alertprofileall"> <tt>GET /api/v1/alert/profile/all</tt></a></td>
+ </tr>
+
+<tr class="a">
+
+<td><a href="#DELETE_apiv1alertprofile"> <tt>DELETE /api/v1/alert/profile</tt></a></td>
+ </tr>
+
+<tr class="b">
+
+<td><a href="#POST_apiv1alertprofile"> <tt>POST /api/v1/alert/profile</tt></a></td>
+ </tr>
+
+<tr class="a">
+
<td><a href="#GET_apiv1globalconfig"> <tt>GET /api/v1/global/config</tt></a></td>
</tr>
@@ -726,22 +755,52 @@ METRON_SERVICE_KEYTAB="/etc/security/keytabs/metron.keytab"
<tr class="b">
+<td><a href="#GET_apiv1metaalertsearchByAlert"> <tt>GET /api/v1/metaalert/searchByAlert</tt></a></td>
+ </tr>
+
+<tr class="a">
+
+<td><a href="#GET_apiv1metaalertcreate"> <tt>GET /api/v1/metaalert/create</tt></a></td>
+ </tr>
+
+<tr class="b">
+
+<td><a href="#GET_apiv1metaalertaddalert"> <tt>GET /api/v1/metaalert/add/alert</tt></a></td>
+ </tr>
+
+<tr class="a">
+
+<td><a href="#GET_apiv1metaalertremovealert"> <tt>GET /api/v1/metaalert/remove/alert</tt></a></td>
+ </tr>
+
+<tr class="b">
+
+<td><a href="#GET_apiv1metaalertupdatestatusguidstatus"> <tt>GET /api/v1/metaalert/update/status/{guid}/{status}</tt></a></td>
+ </tr>
+
+<tr class="a">
+
<td><a href="#GET_apiv1searchsearch"> <tt>GET /api/v1/search/search</tt></a></td>
</tr>
+<tr class="b">
+
+<td><a href="#get-apiv1searchsearch"> <tt>POST /api/v1/search/search</tt></a></td>
+ </tr>
+
<tr class="a">
-<td><a href="#GET_apiv1searchfindOne"> <tt>GET /api/v1/search/findOne</tt></a></td>
+<td><a href="#get-apiv1searchgroup"> <tt>POST /api/v1/search/group</tt></a></td>
</tr>
<tr class="b">
-<td><a href="#get-apiv1searchcolumnmetadata"> <tt>GET /api/v1/search/search</tt></a></td>
+<td><a href="#GET_apiv1searchfindOne"> <tt>GET /api/v1/search/findOne</tt></a></td>
</tr>
<tr class="a">
-<td><a href="#get-apiv1searchcolumnmetadatacommon"> <tt>GET /api/v1/search/search</tt></a></td>
+<td><a href="#GET_apiv1searchcolumnmetadata"> <tt>GET /api/v1/search/column/metadata</tt></a></td>
</tr>
<tr class="b">
@@ -982,6 +1041,86 @@ METRON_SERVICE_KEYTAB="/etc/security/keytabs/metron.keytab"
</ul></li>
</ul></div>
<div class="section">
+<h3><a name="GET_apiv1alertprofile"></a><tt>GET /api/v1/alert/profile</tt></h3>
+
+<ul>
+
+<li>Description: Retrieves the current user’s alerts profile</li>
+
+<li>Returns:
+
+<ul>
+
+<li>200 - Alerts profile</li>
+
+<li>404 - The current user does not have an alerts profile</li>
+ </ul></li>
+</ul></div>
+<div class="section">
+<h3><a name="GET_apiv1alertprofileall"></a><tt>GET /api/v1/alert/profile/all</tt></h3>
+
+<ul>
+
+<li>Description: Retrieves all users’ alerts profiles. Only users that are part of the “ROLE_ADMIN” role are allowed to get all alerts profiles.</li>
+
+<li>Returns:
+
+<ul>
+
+<li>200 - List of all alerts profiles</li>
+
+<li>403 - The current user does not have permission to get all alerts profiles</li>
+ </ul></li>
+</ul></div>
+<div class="section">
+<h3><a name="DELETE_apiv1alertprofile"></a><tt>DELETE /api/v1/alert/profile</tt></h3>
+
+<ul>
+
+<li>Description: Deletes a user’s alerts profile. Only users that are part of the “ROLE_ADMIN” role are allowed to delete user alerts profiles.</li>
+
+<li>Input:
+
+<ul>
+
+<li>user - The user whose prolife will be deleted</li>
+ </ul></li>
+
+<li>Returns:
+
+<ul>
+
+<li>200 - Alerts profile was deleted</li>
+
+<li>403 - The current user does not have permission to delete alerts profiles</li>
+
+<li>404 - Alerts profile could not be found</li>
+ </ul></li>
+</ul></div>
+<div class="section">
+<h3><a name="POST_apiv1alertprofile"></a><tt>POST /api/v1/alert/profile</tt></h3>
+
+<ul>
+
+<li>Description: Creates or updates the current user’s alerts profile</li>
+
+<li>Input:
+
+<ul>
+
+<li>alertsProfile - The alerts profile to be saved</li>
+ </ul></li>
+
+<li>Returns:
+
+<ul>
+
+<li>200 - Alerts profile updated. Returns saved alerts profile.</li>
+
+<li>201 - Alerts profile created. Returns saved alerts profile.</li>
+ </ul></li>
+</ul></div>
+<div class="section">
<h3><a name="GET_apiv1globalconfig"></a><tt>GET /api/v1/global/config</tt></h3>
<ul>
@@ -1097,7 +1236,7 @@ METRON_SERVICE_KEYTAB="/etc/security/keytabs/metron.keytab"
<ul>
-<li>Description: Writes contents to an HDFS file. Warning: this will overwrite the contents of a file if it already exists.</li>
+<li>Description: Writes contents to an HDFS file. Warning: this will overwrite the contents of a file if it already exists. Permissions must be set for all three groups if they are to be set. If any are missing, the default permissions will be used, and if any are invalid an exception will be thrown.</li>
<li>Input:
@@ -1106,6 +1245,12 @@ METRON_SERVICE_KEYTAB="/etc/security/keytabs/metron.keytab"
<li>path - Path to HDFS file</li>
<li>contents - File contents</li>
+
+<li>userMode - [optional] symbolic permission string for user portion of the permissions to be set on the file written. For example ‘rwx’ or read, write, execute. The symbol ‘-’ is used to exclude that permission such as ‘rw-’ for read, write, no execute</li>
+
+<li>groupMode - [optional] symbolic permission string for group portion of the permissions to be set on the file written. For example ‘rwx’ or read, write, execute. The symbol ‘-’ is used to exclude that permission such as ‘rw-’ for read, write, no execute</li>
+
+<li>otherMode - [optional] symbolic permission string for other portion of the permissions to be set on the file written. For example ‘rwx’ or read, write, execute. The symbol ‘-’ is used to exclude that permission such as ‘rw-’ for read, write, no execute</li>
</ul></li>
<li>Returns:
@@ -1287,54 +1432,118 @@ METRON_SERVICE_KEYTAB="/etc/security/keytabs/metron.keytab"
</ul></li>
</ul></div>
<div class="section">
-<h3><a name="GET_apiv1searchfindOne"></a><tt>GET /api/v1/search/findOne</tt></h3>
+<h3><a name="POST_apiv1metaalertsearchByAlert"></a><tt>POST /api/v1/metaalert/searchByAlert</tt></h3>
<ul>
-<li>Description: Returns latest document for a guid and sensor</li>
+<li>Description: Get all meta alerts that contain an alert.</li>
<li>Input:
<ul>
-<li>getRequest - Get request
+<li>guid - GUID of the alert</li>
+ </ul></li>
+
+<li>Returns:
+
+<ul>
+<li>200 - Search results</li>
+ </ul></li>
+</ul></div>
+<div class="section">
+<h3><a name="POST_apiv1metaalertcreate"></a><tt>POST /api/v1/metaalert/create</tt></h3>
+
+<ul>
+
+<li>Description: Creates a new meta alert from a list of existing alerts. The meta alert status will initially be set to ‘ACTIVE’ and summary statistics will be computed from the list of alerts. A list of groups included in the request are also added to the meta alert.</li>
+
+<li>Input:
+
<ul>
-
-<li>guid - message UUID</li>
-
-<li>sensorType - Sensor Type</li>
- </ul></li>
-<li>Example: Return <tt>bro</tt> document with UUID of <tt>000-000-0000</tt></li>
+<li>request - Meta alert create request which includes a list of alert get requests and a list of custom groups used to annotate a meta alert.</li>
</ul></li>
-</ul>
+
+<li>Returns:
+
+<ul>
+
+<li>200 - The GUID of the new meta alert</li>
+ </ul></li>
+</ul></div>
+<div class="section">
+<h3><a name="POST_apiv1metaalertaddalert"></a><tt>POST /api/v1/metaalert/add/alert</tt></h3>
-<div class="source">
-<div class="source">
-<pre>{
- "guid" : "000-000-0000",
- "sensorType" : "bro"
-}
-</pre></div></div>
+<ul>
+
+<li>Description: Adds an alert to an existing meta alert. An alert will not be added if it is already contained in a meta alert.</li>
+
+<li>Input:
+
+<ul>
+
+<li>request - Meta alert add request which includes a meta alert GUID and list of alert get requests</li>
+ </ul></li>
+
+<li>Returns:
+
+<ul>
+
+<li>200 - Returns ‘true’ if the alert was added and ‘false’ if the meta alert did not change.</li>
+ </ul></li>
+</ul></div>
+<div class="section">
+<h3><a name="POST_apiv1metaalertremovealert"></a><tt>POST /api/v1/metaalert/remove/alert</tt></h3>
<ul>
+<li>Description: Removes an alert from an existing meta alert. If the alert to be removed is not in a meta alert, ‘false’ will be returned.</li>
+
+<li>Input:
+
+<ul>
+
+<li>request - Meta alert remove request which includes a meta alert GUID and list of alert get requests</li>
+ </ul></li>
+
<li>Returns:
<ul>
-<li>200 - Document representing the output</li>
+<li>200 - Returns ‘true’ if the alert was removed and ‘false’ if the meta alert did not change.</li>
+ </ul></li>
+</ul></div>
+<div class="section">
+<h3><a name="POST_apiv1metaalertupdatestatusguidstatus"></a><tt>POST /api/v1/metaalert/update/status/{guid}/{status}</tt></h3>
+
+<ul>
+
+<li>Description: Updates the status of a meta alert to either ‘ACTIVE’ or ‘INACTIVE’.</li>
+
+<li>Input:
+
+<ul>
-<li>404 - Document with UUID and sensor type not found</li>
+<li>guid - Meta alert GUID</li>
+
+<li>status - Meta alert status with a value of either ‘ACTIVE’ or ‘INACTIVE’</li>
+ </ul></li>
+
+<li>Returns:
+
+<ul>
+
+<li>200 - Returns ‘true’ if the status changed and ‘false’ if it did not.</li>
</ul></li>
</ul></div>
<div class="section">
-<h3><a name="GET_apiv1searchsearch"></a><tt>GET /api/v1/search/search</tt></h3>
+<h3><a name="POST_apiv1searchsearch"></a><tt>POST /api/v1/search/search</tt></h3>
<ul>
-<li>Description: Searches the indexing store</li>
+<li>Description: Searches the indexing store. GUIDs must be quoted to ensure correct results.</li>
<li>Input:
@@ -1347,49 +1556,103 @@ METRON_SERVICE_KEYTAB="/etc/security/keytabs/metron.keytab"
<ul>
-<li>200 - Search results</li>
+<li>200 - Search response</li>
</ul></li>
</ul></div>
<div class="section">
-<h3><a name="GET_apiv1searchcolumnmetadata"></a><tt>GET /api/v1/search/column/metadata</tt></h3>
+<h3><a name="POST_apiv1searchgroup"></a><tt>POST /api/v1/search/group</tt></h3>
<ul>
-<li>Description: Get column metadata for each index in the list of indicies</li>
+<li>Description: Searches the indexing store and returns field groups. GUIDs must be quoted to ensure correct results. Groups are hierarchical and nested in the order the fields appear in the ‘groups’ request parameter. The default sorting within groups is by count descending. A groupOrder type of count will sort based on then number of documents in a group while a groupType of term will sort by the groupBy term.</li>
<li>Input:
<ul>
-<li>indices - Indices</li>
+<li>groupRequest - Group request
+
+<ul>
+
+<li>indices - list of indices to search</li>
+
+<li>query - lucene query</li>
+
+<li>scoreField - field used to compute a total score for each group</li>
+
+<li>groups - List of groups (field name and sort order)</li>
+ </ul></li>
</ul></li>
<li>Returns:
<ul>
-<li>200 - Column Metadata</li>
+<li>200 - Group response</li>
+ </ul></li>
+</ul></div>
+<div class="section">
+<h3><a name="GET_apiv1searchfindOne"></a><tt>GET /api/v1/search/findOne</tt></h3>
+
+<ul>
+
+<li>Description: Returns latest document for a guid and sensor</li>
+
+<li>Input:
+
+<ul>
+
+<li>getRequest - Get request
+
+<ul>
+
+<li>guid - message UUID</li>
+
+<li>sensorType - Sensor Type</li>
+ </ul></li>
+
+<li>Example: Return <tt>bro</tt> document with UUID of <tt>000-000-0000</tt></li>
+ </ul></li>
+</ul>
+
+<div class="source">
+<div class="source">
+<pre>{
+ "guid" : "000-000-0000",
+ "sensorType" : "bro"
+}
+</pre></div></div>
+
+<ul>
+
+<li>Returns:
+
+<ul>
+
+<li>200 - Document representing the output</li>
+
+<li>404 - Document with UUID and sensor type not found</li>
</ul></li>
</ul></div>
<div class="section">
-<h3><a name="GET_apiv1searchcolumnmetadatacommon"></a><tt>GET /api/v1/search/column/metadata/common</tt></h3>
+<h3><a name="GET_apiv1searchcolumnmetadata"></a><tt>GET /api/v1/search/column/metadata</tt></h3>
<ul>
-<li>Description: Get metadata for columns shared by the list of indices</li>
+<li>Description: Get index column metadata for a list of sensor types with duplicates removed. Column names and types for each sensor are retrieved from the most recent index. Columns that exist in multiple indices with different types will default to type ‘other’.</li>
<li>Input:
<ul>
-<li>indices - Indices</li>
+<li>sensorTypes - Sensor Types</li>
</ul></li>
<li>Returns:
<ul>
-<li>200 - Common Column Metadata</li>
+<li>200 - Column Metadata</li>
</ul></li>
</ul></div>
<div class="section">
@@ -2234,7 +2497,7 @@ METRON_SERVICE_KEYTAB="/etc/security/keytabs/metron.keytab"
</ul></div></div>
<div class="section">
<h2><a name="Testing"></a>Testing</h2>
-<p>Profiles are includes for both the metron-docker and Quick Dev environments.</p>
+<p>Profiles are includes for both the metron-docker and Full Dev environments.</p>
<div class="section">
<h3><a name="metron-docker"></a>metron-docker</h3>
<p>Start the <a href="../../metron-docker/index.html">metron-docker</a> environment. Build the metron-rest module and start it with the Spring Boot Maven plugin:</p>
@@ -2246,8 +2509,8 @@ mvn spring-boot:run -Drun.profiles=docker,dev
</pre></div></div>
<p>The metron-rest application will be available at <a class="externalLink" href="http://localhost:8080/swagger-ui.html#/">http://localhost:8080/swagger-ui.html#/</a>.</p></div>
<div class="section">
-<h3><a name="Quick_Dev"></a>Quick Dev</h3>
-<p>Start the <a href="../../metron-deployment/vagrant/quick-dev-platform/index.html">Quick Dev</a> environment. Build the metron-rest module and start it with the Spring Boot Maven plugin:</p>
+<h3><a name="Full_Dev"></a>Full Dev</h3>
+<p>Start the <a href="../../metron-deployment/vagrant/full-dev-platform/index.html">Full Dev</a> environment. Build the metron-rest module and start it with the Spring Boot Maven plugin:</p>
<div class="source">
<div class="source">
@@ -2255,7 +2518,7 @@ mvn spring-boot:run -Drun.profiles=docker,dev
mvn spring-boot:run -Drun.profiles=vagrant,dev
</pre></div></div>
<p>The metron-rest application will be available at <a class="externalLink" href="http://localhost:8080/swagger-ui.html#/">http://localhost:8080/swagger-ui.html#/</a>.</p>
-<p>To run the application locally on the Quick Dev host (node1), follow the <a href="#Installation">Installation</a> instructions above. Then set the METRON_SPRING_PROFILES_ACTIVE variable in <tt>/etc/sysconfig/metron</tt>:</p>
+<p>To run the application locally on the Full Dev host (node1), follow the <a href="#Installation">Installation</a> instructions above. Then set the METRON_SPRING_PROFILES_ACTIVE variable in <tt>/etc/default/metron</tt>:</p>
<div class="source">
<div class="source">
@@ -2267,7 +2530,7 @@ mvn spring-boot:run -Drun.profiles=vagrant,dev
<div class="source">
<pre>service metron-rest start
</pre></div></div>
-<p>In a cluster with Kerberos enabled, update the security settings in <tt>/etc/sysconfig/metron</tt>. Security is disabled by default in the <tt>vagrant</tt> Spring profile so that setting must be overriden with the METRON_SPRING_OPTIONS variable:</p>
+<p>In a cluster with Kerberos enabled, update the security settings in <tt>/etc/default/metron</tt>. Security is disabled by default in the <tt>vagrant</tt> Spring profile so that setting must be overriden with the METRON_SPRING_OPTIONS variable:</p>
<div class="source">
<div class="source">
@@ -2287,7 +2550,7 @@ METRON_SPRING_OPTIONS="--kerberos.enabled=true"
<footer>
<div class="container-fluid">
- <div class="row span12">Copyright © 2017
+ <div class="row span12">Copyright © 2018
<a href="https://www.apache.org">The Apache Software Foundation</a>.
All Rights Reserved.
http://git-wip-us.apache.org/repos/asf/metron/blob/3381b853/site/current-book/metron-platform/Performance-tuning-guide.html
----------------------------------------------------------------------
diff --git a/site/current-book/metron-platform/Performance-tuning-guide.html b/site/current-book/metron-platform/Performance-tuning-guide.html
index e985bdd..941e2d7 100644
--- a/site/current-book/metron-platform/Performance-tuning-guide.html
+++ b/site/current-book/metron-platform/Performance-tuning-guide.html
@@ -1,13 +1,13 @@
<!DOCTYPE html>
<!--
- | Generated by Apache Maven Doxia at 2017-09-15
+ | Generated by Apache Maven Doxia at 2018-01-03
| Rendered using Apache Maven Fluido Skin 1.3.0
-->
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
- <meta name="Date-Revision-yyyymmdd" content="20170915" />
+ <meta name="Date-Revision-yyyymmdd" content="20180103" />
<meta http-equiv="Content-Language" content="en" />
<title>Metron – Metron Performance Tuning Guide</title>
<link rel="stylesheet" href="../css/apache-maven-fluido-1.3.0.min.css" />
@@ -61,8 +61,8 @@
- <li id="publishDate" class="pull-right">Last Published: 2017-09-15</li> <li class="divider pull-right">|</li>
- <li id="projectVersion" class="pull-right">Version: 0.4.1</li>
+ <li id="publishDate" class="pull-right">Last Published: 2018-01-03</li> <li class="divider pull-right">|</li>
+ <li id="projectVersion" class="pull-right">Version: 0.4.2</li>
</ul>
</div>
@@ -103,7 +103,7 @@
<i class="none"></i>
Docker</a>
</li>
-
+
<li>
<a href="../metron-deployment/index.html" title="Deployment">
@@ -131,7 +131,7 @@
<i class="none"></i>
Rest</a>
</li>
-
+
<li>
<a href="../metron-platform/index.html" title="Platform">
@@ -167,6 +167,13 @@
<li>
+ <a href="../metron-platform/metron-elasticsearch/index.html" title="Elasticsearch">
+ <i class="none"></i>
+ Elasticsearch</a>
+ </li>
+
+ <li>
+
<a href="../metron-platform/metron-enrichment/index.html" title="Enrichment">
<i class="none"></i>
Enrichment</a>
@@ -208,13 +215,20 @@
</li>
</ul>
</li>
-
+
<li>
<a href="../metron-sensors/index.html" title="Sensors">
<i class="icon-chevron-right"></i>
Sensors</a>
</li>
+
+ <li>
+
+ <a href="../metron-stellar/stellar-3rd-party-example/index.html" title="Stellar-3rd-party-example">
+ <i class="none"></i>
+ Stellar-3rd-party-example</a>
+ </li>
<li>
@@ -222,7 +236,7 @@
<i class="icon-chevron-right"></i>
Stellar-common</a>
</li>
-
+
<li>
<a href="../use-cases/index.html" title="Use-cases">
@@ -471,39 +485,50 @@ enrichments enrichments 43 29754331 297
<div class="source">
<div class="source">
-<pre>/usr/metron/0.4.0/bin/start_parser_topology.sh -k $BROKERLIST -z $ZOOKEEPER -s bro -ksp SASL_PLAINTEXT
- -ot enrichments
+<pre>/usr/metron/0.4.2/bin/start_parser_topology.sh \
-e ~metron/.storm/storm-bro.config \
-esc ~/.storm/spout-bro.config \
- -sp 24 \
- -snt 24 \
+ -k $BROKERLIST \
+ -ksp SASL_PLAINTEXT \
-nw 1 \
+ -ot enrichments \
-pnt 24 \
-pp 24 \
+ -s bro \
+ -snt 24 \
+ -sp 24 \
+ -z $ZOOKEEPER \
</pre></div></div>
-<p>From the usage docs, here are the options we’ve used. The full reference can be found here - <a class="externalLink" href="https://github.com/apache/metron/blob/master/metron-platform/metron-parsers/README.md">https://github.com/apache/metron/blob/master/metron-platform/metron-parsers/README.md</a></p>
-
-<div class="source">
-<div class="source">
-<pre>-e,--extra_topology_options <JSON_FILE> Extra options in the form
- of a JSON file with a map
- for content.
--esc,--extra_kafka_spout_config <JSON_FILE> Extra spout config options
- in the form of a JSON file
- with a map for content.
- Possible keys are:
- retryDelayMaxMs,retryDelay
- Multiplier,retryInitialDel
- ayMs,stateUpdateIntervalMs
- ,bufferSizeBytes,fetchMaxW
- ait,fetchSizeBytes,maxOffs
- etBehind,metricsTimeBucket
- SizeInSecs,socketTimeoutMs
--sp,--spout_p <SPOUT_PARALLELISM_HINT> Spout Parallelism Hint
--snt,--spout_num_tasks <NUM_TASKS> Spout Num Tasks
--nw,--num_workers <NUM_WORKERS> Number of Workers
--pnt,--parser_num_tasks <NUM_TASKS> Parser Num Tasks
--pp,--parser_p <PARALLELISM_HINT> Parser Parallelism Hint
+<p>From the usage docs, here are the options we’ve used. The full reference can be found <a href="../metron-platform/metron-parsers/index.html#Starting_the_Parser_Topology">here</a>.</p>
+
+<div class="source">
+<div class="source">
+<pre>usage: start_parser_topology.sh
+ -e,--extra_topology_options <JSON_FILE> Extra options in the form
+ of a JSON file with a map
+ for content.
+ -esc,--extra_kafka_spout_config <JSON_FILE> Extra spout config options
+ in the form of a JSON file
+ with a map for content.
+ Possible keys are:
+ retryDelayMaxMs,retryDelay
+ Multiplier,retryInitialDel
+ ayMs,stateUpdateIntervalMs
+ ,bufferSizeBytes,fetchMaxW
+ ait,fetchSizeBytes,maxOffs
+ etBehind,metricsTimeBucket
+ SizeInSecs,socketTimeoutMs
+ -k,--kafka <BROKER_URL> Kafka Broker URL
+ -ksp,--kafka_security_protocol <SECURITY_PROTOCOL> Kafka Security Protocol
+ -nw,--num_workers <NUM_WORKERS> Number of Workers
+ -ot,--output_topic <KAFKA_TOPIC> Output Kafka Topic
+ -pnt,--parser_num_tasks <NUM_TASKS> Parser Num Tasks
+ -pp,--parser_p <PARALLELISM_HINT> Parser Parallelism Hint
+ -s,--sensor <SENSOR_TYPE> Sensor Type
+ -snt,--spout_num_tasks <NUM_TASKS> Spout Num Tasks
+ -sp,--spout_p <SPOUT_PARALLELISM_HINT> Spout Parallelism Hint
+ -z,--zk <ZK_QUORUM> Zookeeper Quroum URL
+ (zk1:2181,zk2:2181,...
</pre></div></div></div>
<div class="section">
<h3><a name="Enrichment_Tuning"></a>Enrichment Tuning</h3>
@@ -663,7 +688,7 @@ session timeout or by reducing the maximum size of batches returned in poll() wi
<footer>
<div class="container-fluid">
- <div class="row span12">Copyright © 2017
+ <div class="row span12">Copyright © 2018
<a href="https://www.apache.org">The Apache Software Foundation</a>.
All Rights Reserved.
http://git-wip-us.apache.org/repos/asf/metron/blob/3381b853/site/current-book/metron-platform/index.html
----------------------------------------------------------------------
diff --git a/site/current-book/metron-platform/index.html b/site/current-book/metron-platform/index.html
index b58ddcc..ee219ed 100644
--- a/site/current-book/metron-platform/index.html
+++ b/site/current-book/metron-platform/index.html
@@ -1,13 +1,13 @@
<!DOCTYPE html>
<!--
- | Generated by Apache Maven Doxia at 2017-09-15
+ | Generated by Apache Maven Doxia at 2018-01-03
| Rendered using Apache Maven Fluido Skin 1.3.0
-->
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
- <meta name="Date-Revision-yyyymmdd" content="20170915" />
+ <meta name="Date-Revision-yyyymmdd" content="20180103" />
<meta http-equiv="Content-Language" content="en" />
<title>Metron – Current Build</title>
<link rel="stylesheet" href="../css/apache-maven-fluido-1.3.0.min.css" />
@@ -61,8 +61,8 @@
- <li id="publishDate" class="pull-right">Last Published: 2017-09-15</li> <li class="divider pull-right">|</li>
- <li id="projectVersion" class="pull-right">Version: 0.4.1</li>
+ <li id="publishDate" class="pull-right">Last Published: 2018-01-03</li> <li class="divider pull-right">|</li>
+ <li id="projectVersion" class="pull-right">Version: 0.4.2</li>
</ul>
</div>
@@ -103,7 +103,7 @@
<i class="none"></i>
Docker</a>
</li>
-
+
<li>
<a href="../metron-deployment/index.html" title="Deployment">
@@ -131,7 +131,7 @@
<i class="none"></i>
Rest</a>
</li>
-
+
<li class="active">
<a href="#"><i class="icon-chevron-down"></i>Platform</a>
@@ -167,6 +167,13 @@
<li>
+ <a href="../metron-platform/metron-elasticsearch/index.html" title="Elasticsearch">
+ <i class="none"></i>
+ Elasticsearch</a>
+ </li>
+
+ <li>
+
<a href="../metron-platform/metron-enrichment/index.html" title="Enrichment">
<i class="none"></i>
Enrichment</a>
@@ -208,13 +215,20 @@
</li>
</ul>
</li>
-
+
<li>
<a href="../metron-sensors/index.html" title="Sensors">
<i class="icon-chevron-right"></i>
Sensors</a>
</li>
+
+ <li>
+
+ <a href="../metron-stellar/stellar-3rd-party-example/index.html" title="Stellar-3rd-party-example">
+ <i class="none"></i>
+ Stellar-3rd-party-example</a>
+ </li>
<li>
@@ -222,7 +236,7 @@
<i class="icon-chevron-right"></i>
Stellar-common</a>
</li>
-
+
<li>
<a href="../use-cases/index.html" title="Use-cases">
@@ -266,7 +280,7 @@ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License. --><h1>Current Build</h1>
<p><a name="Current_Build"></a></p>
-<p>The latest build of metron-platform is 0.4.1.</p>
+<p>The latest build of metron-platform is 0.4.2.</p>
<p>We are still in the process of merging/porting additional features from our production code base into this open source release. This release will be followed by a number of additional beta releases until the port is complete. We will also work on getting additional documentation and user/developer guides to the community as soon as we can. At this time we offer no support for the beta software, but will try to respond to requests as promptly as we can.</p>
<p><a name="metron-platform"></a></p>
<h1>metron-platform</h1>
@@ -282,7 +296,7 @@ limitations under the License. --><h1>Current Build</h1>
<footer>
<div class="container-fluid">
- <div class="row span12">Copyright © 2017
+ <div class="row span12">Copyright © 2018
<a href="https://www.apache.org">The Apache Software Foundation</a>.
All Rights Reserved.
http://git-wip-us.apache.org/repos/asf/metron/blob/3381b853/site/current-book/metron-platform/metron-api/index.html
----------------------------------------------------------------------
diff --git a/site/current-book/metron-platform/metron-api/index.html b/site/current-book/metron-platform/metron-api/index.html
index ddc7e78..690ab94 100644
--- a/site/current-book/metron-platform/metron-api/index.html
+++ b/site/current-book/metron-platform/metron-api/index.html
@@ -1,13 +1,13 @@
<!DOCTYPE html>
<!--
- | Generated by Apache Maven Doxia at 2017-09-15
+ | Generated by Apache Maven Doxia at 2018-01-03
| Rendered using Apache Maven Fluido Skin 1.3.0
-->
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
- <meta name="Date-Revision-yyyymmdd" content="20170915" />
+ <meta name="Date-Revision-yyyymmdd" content="20180103" />
<meta http-equiv="Content-Language" content="en" />
<title>Metron – Metron PCAP Service</title>
<link rel="stylesheet" href="../../css/apache-maven-fluido-1.3.0.min.css" />
@@ -61,8 +61,8 @@
- <li id="publishDate" class="pull-right">Last Published: 2017-09-15</li> <li class="divider pull-right">|</li>
- <li id="projectVersion" class="pull-right">Version: 0.4.1</li>
+ <li id="publishDate" class="pull-right">Last Published: 2018-01-03</li> <li class="divider pull-right">|</li>
+ <li id="projectVersion" class="pull-right">Version: 0.4.2</li>
</ul>
</div>
@@ -103,7 +103,7 @@
<i class="none"></i>
Docker</a>
</li>
-
+
<li>
<a href="../../metron-deployment/index.html" title="Deployment">
@@ -131,7 +131,7 @@
<i class="none"></i>
Rest</a>
</li>
-
+
<li>
<a href="../../metron-platform/index.html" title="Platform">
@@ -167,6 +167,13 @@
<li>
+ <a href="../../metron-platform/metron-elasticsearch/index.html" title="Elasticsearch">
+ <i class="none"></i>
+ Elasticsearch</a>
+ </li>
+
+ <li>
+
<a href="../../metron-platform/metron-enrichment/index.html" title="Enrichment">
<i class="none"></i>
Enrichment</a>
@@ -208,13 +215,20 @@
</li>
</ul>
</li>
-
+
<li>
<a href="../../metron-sensors/index.html" title="Sensors">
<i class="icon-chevron-right"></i>
Sensors</a>
</li>
+
+ <li>
+
+ <a href="../../metron-stellar/stellar-3rd-party-example/index.html" title="Stellar-3rd-party-example">
+ <i class="none"></i>
+ Stellar-3rd-party-example</a>
+ </li>
<li>
@@ -222,7 +236,7 @@
<i class="icon-chevron-right"></i>
Stellar-common</a>
</li>
-
+
<li>
<a href="../../use-cases/index.html" title="Use-cases">
@@ -318,7 +332,7 @@
<footer>
<div class="container-fluid">
- <div class="row span12">Copyright © 2017
+ <div class="row span12">Copyright © 2018
<a href="https://www.apache.org">The Apache Software Foundation</a>.
All Rights Reserved.
http://git-wip-us.apache.org/repos/asf/metron/blob/3381b853/site/current-book/metron-platform/metron-common/index.html
----------------------------------------------------------------------
diff --git a/site/current-book/metron-platform/metron-common/index.html b/site/current-book/metron-platform/metron-common/index.html
index 92bf3a2..7111111 100644
--- a/site/current-book/metron-platform/metron-common/index.html
+++ b/site/current-book/metron-platform/metron-common/index.html
@@ -1,13 +1,13 @@
<!DOCTYPE html>
<!--
- | Generated by Apache Maven Doxia at 2017-09-15
+ | Generated by Apache Maven Doxia at 2018-01-03
| Rendered using Apache Maven Fluido Skin 1.3.0
-->
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
- <meta name="Date-Revision-yyyymmdd" content="20170915" />
+ <meta name="Date-Revision-yyyymmdd" content="20180103" />
<meta http-equiv="Content-Language" content="en" />
<title>Metron – Contents</title>
<link rel="stylesheet" href="../../css/apache-maven-fluido-1.3.0.min.css" />
@@ -61,8 +61,8 @@
- <li id="publishDate" class="pull-right">Last Published: 2017-09-15</li> <li class="divider pull-right">|</li>
- <li id="projectVersion" class="pull-right">Version: 0.4.1</li>
+ <li id="publishDate" class="pull-right">Last Published: 2018-01-03</li> <li class="divider pull-right">|</li>
+ <li id="projectVersion" class="pull-right">Version: 0.4.2</li>
</ul>
</div>
@@ -103,7 +103,7 @@
<i class="none"></i>
Docker</a>
</li>
-
+
<li>
<a href="../../metron-deployment/index.html" title="Deployment">
@@ -131,7 +131,7 @@
<i class="none"></i>
Rest</a>
</li>
-
+
<li>
<a href="../../metron-platform/index.html" title="Platform">
@@ -167,6 +167,13 @@
<li>
+ <a href="../../metron-platform/metron-elasticsearch/index.html" title="Elasticsearch">
+ <i class="none"></i>
+ Elasticsearch</a>
+ </li>
+
+ <li>
+
<a href="../../metron-platform/metron-enrichment/index.html" title="Enrichment">
<i class="none"></i>
Enrichment</a>
@@ -208,13 +215,20 @@
</li>
</ul>
</li>
-
+
<li>
<a href="../../metron-sensors/index.html" title="Sensors">
<i class="icon-chevron-right"></i>
Sensors</a>
</li>
+
+ <li>
+
+ <a href="../../metron-stellar/stellar-3rd-party-example/index.html" title="Stellar-3rd-party-example">
+ <i class="none"></i>
+ Stellar-3rd-party-example</a>
+ </li>
<li>
@@ -222,7 +236,7 @@
<i class="icon-chevron-right"></i>
Stellar-common</a>
</li>
-
+
<li>
<a href="../../use-cases/index.html" title="Use-cases">
@@ -315,7 +329,7 @@
"es.ip": "node1",
"es.port": "9300",
"es.date.format": "yyyy.MM.dd.HH",
- "parser.error.topic": "indexing"
+ "parser.error.topic": "indexing",
"fieldValidations" : [
{
"input" : [ "ip_src_addr", "ip_dst_addr" ],
@@ -327,6 +341,182 @@
]
}
</pre></div></div>
+<p>Various parts of our stack uses the global config are documented throughout the Metron documentation, but a convenient index is provided here:</p>
+
+<table border="0" class="table table-striped">
+ <thead>
+
+<tr class="a">
+
+<th>Property Name </th>
+
+<th>Subsystem </th>
+
+<th>Type </th>
+
+<th>Ambari Property </th>
+ </tr>
+ </thead>
+ <tbody>
+
+<tr class="b">
+
+<td><a href="../metron-elasticsearch/index.html#es.clustername"><tt>es.clustername</tt></a> </td>
+
+<td>Indexing </td>
+
+<td>String </td>
+
+<td><tt>es_cluster_name</tt> </td>
+ </tr>
+
+<tr class="a">
+
+<td><a href="../metron-elasticsearch/index.html#es.ip"><tt>es.ip</tt></a> </td>
+
+<td>Indexing </td>
+
+<td>String </td>
+
+<td><tt>es_hosts</tt> </td>
+ </tr>
+
+<tr class="b">
+
+<td><a href="../metron-elasticsearch/index.html#es.port"><tt>es.port</tt></a> </td>
+
+<td>Indexing </td>
+
+<td>String </td>
+
+<td><tt>es_port</tt> </td>
+ </tr>
+
+<tr class="a">
+
+<td><a href="../metron-elasticsearch/index.html#es.date.format"><tt>es.date.format</tt></a> </td>
+
+<td>Indexing </td>
+
+<td>String </td>
+
+<td><tt>es_date_format</tt> </td>
+ </tr>
+
+<tr class="b">
+
+<td><a href="#validation-framework"><tt>fieldValidations</tt></a> </td>
+
+<td>Parsing </td>
+
+<td>Object </td>
+
+<td>N/A </td>
+ </tr>
+
+<tr class="a">
+
+<td><a href="../metron-parsers/index.html#parser.error.topic"><tt>parser.error.topic</tt></a> </td>
+
+<td>Parsing </td>
+
+<td>String </td>
+
+<td>N/A </td>
+ </tr>
+
+<tr class="b">
+
+<td><a href="../../metron-stellar/stellar-common/index.html#stellar.function.paths"><tt>stellar.function.paths</tt></a> </td>
+
+<td>Stellar </td>
+
+<td>CSV String </td>
+
+<td>N/A </td>
+ </tr>
+
+<tr class="a">
+
+<td><a href="../../metron-stellar/stellar-common/index.html#stellarfunctionresolverincludesexcludes"><tt>stellar.function.resolver.includes</tt></a> </td>
+
+<td>Stellar </td>
+
+<td>CSV String </td>
+
+<td>N/A </td>
+ </tr>
+
+<tr class="b">
+
+<td><a href="../../metron-stellar/stellar-common/index.html#stellarfunctionresolverincludesexcludes"><tt>stellar.function.resolver.excludes</tt></a> </td>
+
+<td>Stellar </td>
+
+<td>CSV String </td>
+
+<td>N/A </td>
+ </tr>
+
+<tr class="a">
+
+<td><a href="../../metron-analytics/metron-profiler/index.html#profiler.period.duration"><tt>profiler.period.duration</tt></a> </td>
+
+<td>Profiler </td>
+
+<td>Integer </td>
+
+<td><tt>profiler_period_duration</tt> </td>
+ </tr>
+
+<tr class="b">
+
+<td><a href="../../metron-analytics/metron-profiler/index.html#profiler.period.duration.units"><tt>profiler.period.duration.units</tt></a> </td>
+
+<td>Profiler </td>
+
+<td>String </td>
+
+<td><tt>profiler_period_units</tt> </td>
+ </tr>
+
+<tr class="a">
+
+<td><a href="../metron-indexing/index.html#update.hbase.table"><tt>update.hbase.table</tt></a> </td>
+
+<td>REST/Indexing </td>
+
+<td>String </td>
+
+<td><tt>update_hbase_table</tt> </td>
+ </tr>
+
+<tr class="b">
+
+<td><a href="../metron-indexing/index.html#update.hbase.cf"><tt>update.hbase.cf</tt></a> </td>
+
+<td>REST/Indexing </td>
+
+<td>String </td>
+
+<td><tt>update_hbase_cf</tt> </td>
+ </tr>
+
+<tr class="a">
+
+<td><a href="../metron-enrichment/index.html#geo.hdfs.file"><tt>geo.hdfs.file</tt></a> </td>
+
+<td>Enrichment </td>
+
+<td>String </td>
+
+<td><tt>geo_hdfs_file</tt> </td>
+ </tr>
+ </tbody>
+</table>
+<div class="section">
+<h2><a name="Note_Configs_in_Ambari"></a>Note Configs in Ambari</h2>
+<p>If a field is managed via ambari, you should change the field via ambari. Otherwise, upon service restarts, you may find your update overwritten.</p>
<p><a name="Validation_Framework"></a></p>
<h1>Validation Framework</h1>
<p>Inside of the global configuration, there is a validation framework in place that enables the validation that messages coming from all parsers are valid. This is done in the form of validation plugins where assertions about fields or whole messages can be made. </p>
@@ -377,16 +567,28 @@
<div class="source">
<div class="source">
-<pre> -f,--force Force operation
+<pre> -c,--config_type <CONFIG_TYPE> The configuration type: GLOBAL,
+ PARSER, ENRICHMENT, INDEXING,
+ PROFILER
+ -f,--force Force operation
-h,--help Generate Help screen
-i,--input_dir <DIR> The input directory containing
the configuration files named
like "$source.json"
-m,--mode <MODE> The mode of operation: DUMP,
- PULL, PUSH
+ PULL, PUSH, PATCH
+ -n,--config_name <CONFIG_NAME> The configuration name: bro,
+ yaf, snort, squid, etc.
-o,--output_dir <DIR> The output directory which will
store the JSON configuration
from Zookeeper
+ -pk,--patch_key <PATCH_KEY> The key to modify
+ -pm,--patch_mode <PATCH_MODE> One of: ADD, REMOVE - relevant
+ only for key/value patches,
+ i.e. when a patch file is not
+ used.
+ -pf,--patch_file <PATCH_FILE> Path to the patch file.
+ -pv,--patch_value <PATCH_VALUE> Value to use in the patch.
-z,--zk_quorum <host:port,[host:port]*> Zookeeper Quorum URL
(zk1:port,zk2:port,...)
</pre></div></div>
@@ -396,10 +598,163 @@
<li>To dump the existing configs from zookeeper on the singlenode vagrant machine: <tt>$METRON_HOME/bin/zk_load_configs.sh -z node1:2181 -m DUMP</tt></li>
+<li>To dump the existing GLOBAL configs from zookeeper on the singlenode vagrant machine: <tt>$METRON_HOME/bin/zk_load_configs.sh -z node1:2181 -m DUMP -c GLOBAL</tt></li>
+
<li>To push the configs into zookeeper on the singlenode vagrant machine: <tt>$METRON_HOME/bin/zk_load_configs.sh -z node1:2181 -m PUSH -i $METRON_HOME/config/zookeeper</tt></li>
-<li>To pull the configs from zookeeper to the singlenode vagrant machine disk: <tt>$METRON_HOME/bin/zk_load_configs.sh -z node1:2181 -m PULL -o $METRON_HOME/config/zookeeper -f</tt></li>
+<li>To push only the GLOBAL configs into zookeeper on the singlenode vagrant machine: <tt>$METRON_HOME/bin/zk_load_configs.sh -z node1:2181 -m PUSH -i $METRON_HOME/config/zookeeper -c GLOBAL</tt></li>
+
+<li>To push only the PARSER configs into zookeeper on the singlenode vagrant machine: <tt>$METRON_HOME/bin/zk_load_configs.sh -z node1:2181 -m PUSH -i $METRON_HOME/config/zookeeper -c PARSER</tt></li>
+
+<li>To push only the PARSER ‘bro’ configs into zookeeper on the singlenode vagrant machine: <tt>$METRON_HOME/bin/zk_load_configs.sh -z node1:2181 -m PUSH -i $METRON_HOME/config/zookeeper -c PARSER -n bro</tt></li>
+
+<li>To pull all configs from zookeeper to the singlenode vagrant machine disk: <tt>$METRON_HOME/bin/zk_load_configs.sh -z node1:2181 -m PULL -o $METRON_HOME/config/zookeeper -f</tt></li>
+</ul></div>
+<div class="section">
+<h2><a name="Patching_mechanism"></a>Patching mechanism</h2>
+<p>The configuration management utility leverages a JSON patching library that conforms to <a class="externalLink" href="https://tools.ietf.org/html/rfc6902">RFC-6902 spec</a>. We’re using the zjsonpatch library implementation from here - <a class="externalLink" href="https://github.com/flipkart-incubator/zjsonpatch">https://github.com/flipkart-incubator/zjsonpatch</a>. There are a couple options for leveraging patching. You can choose to patch the Zookeeper config via patch file:</p>
+<p><tt>$METRON_HOME/bin/zk_load_configs.sh -z $ZOOKEEPER -m PATCH -c GLOBAL -pf /tmp/mypatch.txt</tt></p>
+<p>or key/value pair:</p>
+<p><tt>$METRON_HOME/bin/zk_load_configs.sh -z $ZOOKEEPER -m PATCH -c GLOBAL -pm ADD -pk foo -pv \"\"bar\"\"</tt></p>
+<p>The options exposed via patch file are the full range of options from RFC-6902:</p>
+
+<ul>
+
+<li>ADD</li>
+
+<li>REMOVE</li>
+
+<li>REPLACE</li>
+
+<li>MOVE</li>
+
+<li>COPY</li>
+
+<li>TEST</li>
</ul>
+<p>whereas with key/value patching, we only current expose ADD and REMOVE. Note that ADD will function as a REPLACE when the key already exists.</p>
+<div class="section">
+<h3><a name="Patch_File"></a>Patch File</h3>
+<p>Let’s say we want to add a complex JSON object to our configuration with a patch file. e.g.</p>
+
+<div class="source">
+<div class="source">
+<pre>"foo" : {
+ "bar" : {
+ "baz" : [ "bazval1", "bazval2" ]
+ }
+ }
+</pre></div></div>
+<p>We would write a patch file “/tmp/mypatch.txt” with contents:</p>
+
+<div class="source">
+<div class="source">
+<pre>[
+ {
+ "op": "add",
+ "path": "/foo",
+ "value": { "bar" : { "baz" : [ "bazval1", "bazval2" ] } }
+ }
+]
+</pre></div></div>
+<p>And submit via zk_load_configs as follows:</p>
+
+<div class="source">
+<div class="source">
+<pre> $METRON_HOME/bin/zk_load_configs.sh -z $ZOOKEEPER -m PATCH -c GLOBAL -pf /tmp/mypatch.txt
+</pre></div></div></div>
+<div class="section">
+<h3><a name="Patch_KeyValue"></a>Patch Key/Value</h3>
+<p>Now let’s try the same without using a patch file, instead using the patch_key and patch_value options right from the command line utility. This would like like the following.</p>
+
+<div class="source">
+<div class="source">
+<pre>$METRON_HOME/bin/zk_load_configs.sh -z $ZOOKEEPER -m PATCH -c GLOBAL -pm ADD -pk "/foo" -pv "{ \"bar\" : { \"baz\" : [ \"bazval1\", \"bazval2\" ] } }"
+</pre></div></div></div>
+<div class="section">
+<h3><a name="Applying_Multiple_Patches"></a>Applying Multiple Patches</h3>
+<p>Applying multiple patches is also pretty straightforward. You can achieve this in a single command using patch files, or simply execute multiple commands in sequence using the patch_key/value approach.</p>
+<p>Let’s say we wanted to add the following to our global config:</p>
+
+<div class="source">
+<div class="source">
+<pre>"apache" : "metron",
+"is" : "the best",
+"streaming" : "analytics platform"
+</pre></div></div>
+<p>and remove the /foo key from the previous example.</p>
+<p>Create a patch file /tmp/mypatch.txt with four separate patch operations.</p>
+
+<div class="source">
+<div class="source">
+<pre>[
+ {
+ "op": "remove",
+ "path": "/foo"
+ },
+ {
+ "op": "add",
+ "path": "/apache",
+ "value": "metron"
+ },
+ {
+ "op": "add",
+ "path": "/is",
+ "value": "the best"
+ },
+ {
+ "op": "add",
+ "path": "/streaming",
+ "value": "analytics platform"
+ }
+]
+</pre></div></div>
+<p>Now submit again and you should see a Global config with the “foo” key removed and three new keys added.</p>
+
+<div class="source">
+<div class="source">
+<pre> $METRON_HOME/bin/zk_load_configs.sh -z $ZOOKEEPER -m PATCH -c GLOBAL -pf /tmp/mypatch.txt
+</pre></div></div></div>
+<div class="section">
+<h3><a name="Notes_On_Patching"></a>Notes On Patching</h3>
+<p>For any given patch key, the last/leaf node in the key’s parent <i>must</i> exist, otherwise an exception will be thrown. For example, if you want to add the following:</p>
+
+<div class="source">
+<div class="source">
+<pre>"foo": {
+ "bar": "baz"
+}
+</pre></div></div>
+<p>It is not sufficient to use /foo/bar as a key if foo does not already exist. You would either need to incrementally build the JSON and make this a two step process</p>
+
+<div class="source">
+<div class="source">
+<pre>[
+ {
+ "op": "add",
+ "path": "/foo",
+ "value": { }
+ },
+ {
+ "op": "add",
+ "path": "/foo/bar",
+ "value": "baz"
+ }
+]
+</pre></div></div>
+<p>Or provide the value as a complete JSON object.</p>
+
+<div class="source">
+<div class="source">
+<pre>[
+ {
+ "op": "add",
+ "path": "/foo",
+ "value": { "bar" : "baz" }
+ }
+]
+</pre></div></div>
+<p>The REMOVE operation is idempotent. Running the remove command on the same key multiple times will not fail once the key has been removed.</p>
<p><a name="Topology_Errors"></a></p>
<h1>Topology Errors</h1>
<p>Errors generated in Metron topologies are transformed into JSON format and follow this structure:</p>
@@ -499,7 +854,7 @@ if (perfLog.isDebugEnabled()) {
}
</pre></div></div>
<p><b>Side Effects</b></p>
-<p>Calling the mark() method multiple times simply resets the start time to the current nano time. Calling log() with a non-existent mark name will log 0 ns elapsed time with a warning indicating that log has been invoked for a mark name that does not exist. The class is not thread-safe and makes no attempt at keeping multiple threads from modifying the same markers.</p>
+<p>Calling the mark() method multiple times simply resets the start time to the current nano time. Calling log() with a non-existent mark name will log 0 ns elapsed time with a warning indicating that log has been invoked for a mark name that does not exist. The class is not thread-safe and makes no attempt at keeping multiple threads from modifying the same markers.</p></div></div>
</div>
</div>
</div>
@@ -508,7 +863,7 @@ if (perfLog.isDebugEnabled()) {
<footer>
<div class="container-fluid">
- <div class="row span12">Copyright © 2017
+ <div class="row span12">Copyright © 2018
<a href="https://www.apache.org">The Apache Software Foundation</a>.
All Rights Reserved.
http://git-wip-us.apache.org/repos/asf/metron/blob/3381b853/site/current-book/metron-platform/metron-data-management/index.html
----------------------------------------------------------------------
diff --git a/site/current-book/metron-platform/metron-data-management/index.html b/site/current-book/metron-platform/metron-data-management/index.html
index b04a6f4..df34389 100644
--- a/site/current-book/metron-platform/metron-data-management/index.html
+++ b/site/current-book/metron-platform/metron-data-management/index.html
@@ -1,13 +1,13 @@
<!DOCTYPE html>
<!--
- | Generated by Apache Maven Doxia at 2017-09-15
+ | Generated by Apache Maven Doxia at 2018-01-03
| Rendered using Apache Maven Fluido Skin 1.3.0
-->
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
- <meta name="Date-Revision-yyyymmdd" content="20170915" />
+ <meta name="Date-Revision-yyyymmdd" content="20180103" />
<meta http-equiv="Content-Language" content="en" />
<title>Metron – Resource Data Management</title>
<link rel="stylesheet" href="../../css/apache-maven-fluido-1.3.0.min.css" />
@@ -61,8 +61,8 @@
- <li id="publishDate" class="pull-right">Last Published: 2017-09-15</li> <li class="divider pull-right">|</li>
- <li id="projectVersion" class="pull-right">Version: 0.4.1</li>
+ <li id="publishDate" class="pull-right">Last Published: 2018-01-03</li> <li class="divider pull-right">|</li>
+ <li id="projectVersion" class="pull-right">Version: 0.4.2</li>
</ul>
</div>
@@ -103,7 +103,7 @@
<i class="none"></i>
Docker</a>
</li>
-
+
<li>
<a href="../../metron-deployment/index.html" title="Deployment">
@@ -131,7 +131,7 @@
<i class="none"></i>
Rest</a>
</li>
-
+
<li>
<a href="../../metron-platform/index.html" title="Platform">
@@ -167,6 +167,13 @@
<li>
+ <a href="../../metron-platform/metron-elasticsearch/index.html" title="Elasticsearch">
+ <i class="none"></i>
+ Elasticsearch</a>
+ </li>
+
+ <li>
+
<a href="../../metron-platform/metron-enrichment/index.html" title="Enrichment">
<i class="none"></i>
Enrichment</a>
@@ -208,13 +215,20 @@
</li>
</ul>
</li>
-
+
<li>
<a href="../../metron-sensors/index.html" title="Sensors">
<i class="icon-chevron-right"></i>
Sensors</a>
</li>
+
+ <li>
+
+ <a href="../../metron-stellar/stellar-3rd-party-example/index.html" title="Stellar-3rd-party-example">
+ <i class="none"></i>
+ Stellar-3rd-party-example</a>
+ </li>
<li>
@@ -222,7 +236,7 @@
<i class="icon-chevron-right"></i>
Stellar-common</a>
</li>
-
+
<li>
<a href="../../use-cases/index.html" title="Use-cases">
@@ -1041,7 +1055,7 @@
<footer>
<div class="container-fluid">
- <div class="row span12">Copyright © 2017
+ <div class="row span12">Copyright © 2018
<a href="https://www.apache.org">The Apache Software Foundation</a>.
All Rights Reserved.
[2/7] metron git commit: METRON-1388 update public web site to point
at 0.4.2 new release (mattf-horton) closes apache/metron#887
Posted by ma...@apache.org.
http://git-wip-us.apache.org/repos/asf/metron/blob/3381b853/site/current-book/metron-sensors/bro-plugin-kafka/index.html
----------------------------------------------------------------------
diff --git a/site/current-book/metron-sensors/bro-plugin-kafka/index.html b/site/current-book/metron-sensors/bro-plugin-kafka/index.html
index 3456bc7..638fdcd 100644
--- a/site/current-book/metron-sensors/bro-plugin-kafka/index.html
+++ b/site/current-book/metron-sensors/bro-plugin-kafka/index.html
@@ -1,13 +1,13 @@
<!DOCTYPE html>
<!--
- | Generated by Apache Maven Doxia at 2017-09-15
+ | Generated by Apache Maven Doxia at 2018-01-03
| Rendered using Apache Maven Fluido Skin 1.3.0
-->
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
- <meta name="Date-Revision-yyyymmdd" content="20170915" />
+ <meta name="Date-Revision-yyyymmdd" content="20180103" />
<meta http-equiv="Content-Language" content="en" />
<title>Metron – Logging Bro Output to Kafka</title>
<link rel="stylesheet" href="../../css/apache-maven-fluido-1.3.0.min.css" />
@@ -61,8 +61,8 @@
- <li id="publishDate" class="pull-right">Last Published: 2017-09-15</li> <li class="divider pull-right">|</li>
- <li id="projectVersion" class="pull-right">Version: 0.4.1</li>
+ <li id="publishDate" class="pull-right">Last Published: 2018-01-03</li> <li class="divider pull-right">|</li>
+ <li id="projectVersion" class="pull-right">Version: 0.4.2</li>
</ul>
</div>
@@ -75,7 +75,7 @@
<ul class="nav nav-list">
<li class="nav-header">User Documentation</li>
-
+
<li>
<a href="../../index.html" title="Metron">
@@ -131,7 +131,7 @@
<i class="none"></i>
Rest</a>
</li>
-
+
<li>
<a href="../../metron-platform/index.html" title="Platform">
@@ -166,6 +166,13 @@
</li>
</ul>
</li>
+
+ <li>
+
+ <a href="../../metron-stellar/stellar-3rd-party-example/index.html" title="Stellar-3rd-party-example">
+ <i class="none"></i>
+ Stellar-3rd-party-example</a>
+ </li>
<li>
@@ -173,7 +180,7 @@
<i class="icon-chevron-right"></i>
Stellar-common</a>
</li>
-
+
<li>
<a href="../../use-cases/index.html" title="Use-cases">
@@ -371,7 +378,7 @@ event bro_init() &priority=-5
<li><tt>logs_to_send</tt> is mutually exclusive with <tt>$pred</tt>, thus for each log you want to set <tt>$pred</tt> on, you must individually setup a <tt>Log::add_filter</tt> and refrain from including that log in <tt>logs_to_send</tt>.</li>
-<li>You can also filter IPv6 logs from within your Metron cluster <a href="../../metron-platform/metron-common/index.html#IS_IP">using Stellar</a>. In that case, you wouldn’t apply a predicate in your bro configuration, and instead Stellar would filter the logs out before they were processed by the enrichment layer of Metron.</li>
+<li>You can also filter IPv6 logs from within your Metron cluster <a href="../../metron-stellar/stellar-common/index.html#IS_IP">using Stellar</a>. In that case, you wouldn’t apply a predicate in your bro configuration, and instead Stellar would filter the logs out before they were processed by the enrichment layer of Metron.</li>
<li>It is also possible to use the <tt>is_v6_subnet()</tt> bro function in your predicate, as of their <a class="externalLink" href="https://www.bro.org/sphinx-git/install/release-notes.html#bro-2-5">2.5 release</a>, however the above example should work on <a class="externalLink" href="https://www.bro.org/sphinx-git/install/release-notes.html#bro-2-4">bro 2.4</a> and newer, which has been the focus of the kafka plugin.</li>
</ul></div></div></div>
@@ -498,7 +505,7 @@ redef Kafka::kafka_conf = table( ["metadata.broker.list"] = "node
<footer>
<div class="container-fluid">
- <div class="row span12">Copyright © 2017
+ <div class="row span12">Copyright © 2018
<a href="https://www.apache.org">The Apache Software Foundation</a>.
All Rights Reserved.
http://git-wip-us.apache.org/repos/asf/metron/blob/3381b853/site/current-book/metron-sensors/fastcapa/index.html
----------------------------------------------------------------------
diff --git a/site/current-book/metron-sensors/fastcapa/index.html b/site/current-book/metron-sensors/fastcapa/index.html
index 38bae8a..59beb04 100644
--- a/site/current-book/metron-sensors/fastcapa/index.html
+++ b/site/current-book/metron-sensors/fastcapa/index.html
@@ -1,13 +1,13 @@
<!DOCTYPE html>
<!--
- | Generated by Apache Maven Doxia at 2017-09-15
+ | Generated by Apache Maven Doxia at 2018-01-03
| Rendered using Apache Maven Fluido Skin 1.3.0
-->
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
- <meta name="Date-Revision-yyyymmdd" content="20170915" />
+ <meta name="Date-Revision-yyyymmdd" content="20180103" />
<meta http-equiv="Content-Language" content="en" />
<title>Metron – Fastcapa</title>
<link rel="stylesheet" href="../../css/apache-maven-fluido-1.3.0.min.css" />
@@ -61,8 +61,8 @@
- <li id="publishDate" class="pull-right">Last Published: 2017-09-15</li> <li class="divider pull-right">|</li>
- <li id="projectVersion" class="pull-right">Version: 0.4.1</li>
+ <li id="publishDate" class="pull-right">Last Published: 2018-01-03</li> <li class="divider pull-right">|</li>
+ <li id="projectVersion" class="pull-right">Version: 0.4.2</li>
</ul>
</div>
@@ -103,7 +103,7 @@
<i class="none"></i>
Docker</a>
</li>
-
+
<li>
<a href="../../metron-deployment/index.html" title="Deployment">
@@ -131,14 +131,14 @@
<i class="none"></i>
Rest</a>
</li>
-
+
<li>
<a href="../../metron-platform/index.html" title="Platform">
<i class="icon-chevron-right"></i>
Platform</a>
</li>
-
+
<li>
<a href="../../metron-sensors/index.html" title="Sensors">
@@ -146,13 +146,6 @@
Sensors</a>
<ul class="nav nav-list">
- <li>
-
- <a href="../../metron-sensors/bro-plugin-kafka/index.html" title="Bro-plugin-kafka">
- <i class="none"></i>
- Bro-plugin-kafka</a>
- </li>
-
<li class="active">
<a href="#"><i class="none"></i>Fastcapa</a>
@@ -166,6 +159,13 @@
</li>
</ul>
</li>
+
+ <li>
+
+ <a href="../../metron-stellar/stellar-3rd-party-example/index.html" title="Stellar-3rd-party-example">
+ <i class="none"></i>
+ Stellar-3rd-party-example</a>
+ </li>
<li>
@@ -173,7 +173,7 @@
<i class="icon-chevron-right"></i>
Stellar-common</a>
</li>
-
+
<li>
<a href="../../use-cases/index.html" title="Use-cases">
@@ -1087,7 +1087,7 @@ mount -a
<footer>
<div class="container-fluid">
- <div class="row span12">Copyright © 2017
+ <div class="row span12">Copyright © 2018
<a href="https://www.apache.org">The Apache Software Foundation</a>.
All Rights Reserved.
http://git-wip-us.apache.org/repos/asf/metron/blob/3381b853/site/current-book/metron-sensors/index.html
----------------------------------------------------------------------
diff --git a/site/current-book/metron-sensors/index.html b/site/current-book/metron-sensors/index.html
index f3ab985..8aff298 100644
--- a/site/current-book/metron-sensors/index.html
+++ b/site/current-book/metron-sensors/index.html
@@ -1,13 +1,13 @@
<!DOCTYPE html>
<!--
- | Generated by Apache Maven Doxia at 2017-09-15
+ | Generated by Apache Maven Doxia at 2018-01-03
| Rendered using Apache Maven Fluido Skin 1.3.0
-->
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
- <meta name="Date-Revision-yyyymmdd" content="20170915" />
+ <meta name="Date-Revision-yyyymmdd" content="20180103" />
<meta http-equiv="Content-Language" content="en" />
<title>Metron – Metron Sensors</title>
<link rel="stylesheet" href="../css/apache-maven-fluido-1.3.0.min.css" />
@@ -61,8 +61,8 @@
- <li id="publishDate" class="pull-right">Last Published: 2017-09-15</li> <li class="divider pull-right">|</li>
- <li id="projectVersion" class="pull-right">Version: 0.4.1</li>
+ <li id="publishDate" class="pull-right">Last Published: 2018-01-03</li> <li class="divider pull-right">|</li>
+ <li id="projectVersion" class="pull-right">Version: 0.4.2</li>
</ul>
</div>
@@ -103,7 +103,7 @@
<i class="none"></i>
Docker</a>
</li>
-
+
<li>
<a href="../metron-deployment/index.html" title="Deployment">
@@ -131,14 +131,14 @@
<i class="none"></i>
Rest</a>
</li>
-
+
<li>
<a href="../metron-platform/index.html" title="Platform">
<i class="icon-chevron-right"></i>
Platform</a>
</li>
-
+
<li class="active">
<a href="#"><i class="icon-chevron-down"></i>Sensors</a>
@@ -146,13 +146,6 @@
<li>
- <a href="../metron-sensors/bro-plugin-kafka/index.html" title="Bro-plugin-kafka">
- <i class="none"></i>
- Bro-plugin-kafka</a>
- </li>
-
- <li>
-
<a href="../metron-sensors/fastcapa/index.html" title="Fastcapa">
<i class="none"></i>
Fastcapa</a>
@@ -166,6 +159,13 @@
</li>
</ul>
</li>
+
+ <li>
+
+ <a href="../metron-stellar/stellar-3rd-party-example/index.html" title="Stellar-3rd-party-example">
+ <i class="none"></i>
+ Stellar-3rd-party-example</a>
+ </li>
<li>
@@ -173,7 +173,7 @@
<i class="icon-chevron-right"></i>
Stellar-common</a>
</li>
-
+
<li>
<a href="../use-cases/index.html" title="Use-cases">
@@ -208,7 +208,7 @@
<ul>
<li>
-<p><a href="bro-plugin-kafka/index.html"><tt>bro-plugin-kafka</tt></a>: Provides integration between <a class="externalLink" href="https://www.bro.org/">Bro</a> and Kafka. A Bro plugin that sends logging output to Kafka. This provides a convenient means for tools in the Hadoop ecosystem, such as Storm, Spark, and others to process the data generated by Bro.</p></li>
+<p><a class="externalLink" href="https://github.com/apache/metron-bro-plugin-kafka"><tt>metron-bro-plugin-kafka</tt></a>: Provides integration between <a class="externalLink" href="https://www.bro.org/">Bro</a> and Kafka. A Bro plugin that sends logging output to Kafka. This provides a convenient means for tools in the Hadoop ecosystem, such as Storm, Spark, and others to process the data generated by Bro.</p></li>
<li>
<p><a href="fastcapa/index.html"><tt>fastcapa</tt></a>: Performs fast network packet capture by leveraging Linux kernel-bypass and user space networking technology. The probe will bind to a network interface, capture network packets, and send the raw packet data to Kafka. This provides a scalable mechanism for ingesting high-volumes of network packet data.</p></li>
@@ -224,7 +224,7 @@
<footer>
<div class="container-fluid">
- <div class="row span12">Copyright © 2017
+ <div class="row span12">Copyright © 2018
<a href="https://www.apache.org">The Apache Software Foundation</a>.
All Rights Reserved.
http://git-wip-us.apache.org/repos/asf/metron/blob/3381b853/site/current-book/metron-sensors/pycapa/index.html
----------------------------------------------------------------------
diff --git a/site/current-book/metron-sensors/pycapa/index.html b/site/current-book/metron-sensors/pycapa/index.html
index 52d8f5a..8c46464 100644
--- a/site/current-book/metron-sensors/pycapa/index.html
+++ b/site/current-book/metron-sensors/pycapa/index.html
@@ -1,13 +1,13 @@
<!DOCTYPE html>
<!--
- | Generated by Apache Maven Doxia at 2017-09-15
+ | Generated by Apache Maven Doxia at 2018-01-03
| Rendered using Apache Maven Fluido Skin 1.3.0
-->
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
- <meta name="Date-Revision-yyyymmdd" content="20170915" />
+ <meta name="Date-Revision-yyyymmdd" content="20180103" />
<meta http-equiv="Content-Language" content="en" />
<title>Metron – Pycapa</title>
<link rel="stylesheet" href="../../css/apache-maven-fluido-1.3.0.min.css" />
@@ -61,8 +61,8 @@
- <li id="publishDate" class="pull-right">Last Published: 2017-09-15</li> <li class="divider pull-right">|</li>
- <li id="projectVersion" class="pull-right">Version: 0.4.1</li>
+ <li id="publishDate" class="pull-right">Last Published: 2018-01-03</li> <li class="divider pull-right">|</li>
+ <li id="projectVersion" class="pull-right">Version: 0.4.2</li>
</ul>
</div>
@@ -103,7 +103,7 @@
<i class="none"></i>
Docker</a>
</li>
-
+
<li>
<a href="../../metron-deployment/index.html" title="Deployment">
@@ -131,14 +131,14 @@
<i class="none"></i>
Rest</a>
</li>
-
+
<li>
<a href="../../metron-platform/index.html" title="Platform">
<i class="icon-chevron-right"></i>
Platform</a>
</li>
-
+
<li>
<a href="../../metron-sensors/index.html" title="Sensors">
@@ -148,13 +148,6 @@
<li>
- <a href="../../metron-sensors/bro-plugin-kafka/index.html" title="Bro-plugin-kafka">
- <i class="none"></i>
- Bro-plugin-kafka</a>
- </li>
-
- <li>
-
<a href="../../metron-sensors/fastcapa/index.html" title="Fastcapa">
<i class="none"></i>
Fastcapa</a>
@@ -166,6 +159,13 @@
</li>
</ul>
</li>
+
+ <li>
+
+ <a href="../../metron-stellar/stellar-3rd-party-example/index.html" title="Stellar-3rd-party-example">
+ <i class="none"></i>
+ Stellar-3rd-party-example</a>
+ </li>
<li>
@@ -173,7 +173,7 @@
<i class="icon-chevron-right"></i>
Stellar-common</a>
</li>
-
+
<li>
<a href="../../use-cases/index.html" title="Use-cases">
@@ -539,14 +539,19 @@ ${KAFKA_HOME}/bin/kafka-acls.sh \
INFO:root:'10' packet(s) in, '10' packet(s) out
</pre></div></div></li>
</ol>
-<h1>FAQs</h1>
-<p><b>Question</b>: How do I get more logs?</p>
+<h1>FAQs</h1></div>
+<div class="section">
+<h3><a name="How_do_I_get_more_logs"></a>How do I get more logs?</h3>
<p>Use the following two command-line arguments to get detailed logging.</p>
<div class="source">
<div class="source">
<pre>-X debug=all --log-level DEBUG
-</pre></div></div></div></div>
+</pre></div></div></div>
+<div class="section">
+<h3><a name="When_I_run_Pycapa_against_a_Kafka_broker_with_Kerberos_enabled_why_do_I_get_an_error_like_No_such_configuration_property:_sasl.kerberos.principal"></a>When I run Pycapa against a Kafka broker with Kerberos enabled, why do I get an error like “No such configuration property: ‘sasl.kerberos.principal’”?</h3>
+<p>This can be a confusing error message because <tt>sasl.kerberos.principal</tt> is indeed a valid property for librdkafka as defined <a class="externalLink" href="https://github.com/edenhill/librdkafka/blob/master/CONFIGURATION.md">here</a>. This is most likely because Pycapa is running against a version of Librdkafka without SASL support enabled. This might happen if you have accidentally installed multiple versions of Librdkafka and Pycapa is unexpectedly using the version without SASL support enabled.</p>
+<p>Bottom Line: Make sure that Pycapa is running against a version of Librdkafka with SASL support enabled.</p></div></div>
</div>
</div>
</div>
@@ -555,7 +560,7 @@ ${KAFKA_HOME}/bin/kafka-acls.sh \
<footer>
<div class="container-fluid">
- <div class="row span12">Copyright © 2017
+ <div class="row span12">Copyright © 2018
<a href="https://www.apache.org">The Apache Software Foundation</a>.
All Rights Reserved.
http://git-wip-us.apache.org/repos/asf/metron/blob/3381b853/site/current-book/metron-stellar/stellar-3rd-party-example/index.html
----------------------------------------------------------------------
diff --git a/site/current-book/metron-stellar/stellar-3rd-party-example/index.html b/site/current-book/metron-stellar/stellar-3rd-party-example/index.html
new file mode 100644
index 0000000..d60a128
--- /dev/null
+++ b/site/current-book/metron-stellar/stellar-3rd-party-example/index.html
@@ -0,0 +1,211 @@
+<!DOCTYPE html>
+<!--
+ | Generated by Apache Maven Doxia at 2018-01-03
+ | Rendered using Apache Maven Fluido Skin 1.3.0
+-->
+<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
+ <head>
+ <meta charset="UTF-8" />
+ <meta name="viewport" content="width=device-width, initial-scale=1.0" />
+ <meta name="Date-Revision-yyyymmdd" content="20180103" />
+ <meta http-equiv="Content-Language" content="en" />
+ <title>Metron – Introduction</title>
+ <link rel="stylesheet" href="../../css/apache-maven-fluido-1.3.0.min.css" />
+ <link rel="stylesheet" href="../../css/site.css" />
+ <link rel="stylesheet" href="../../css/print.css" media="print" />
+
+
+ <script type="text/javascript" src="../../js/apache-maven-fluido-1.3.0.min.js"></script>
+
+
+
+<script type="text/javascript">$( document ).ready( function() { $( '.carousel' ).carousel( { interval: 3500 } ) } );</script>
+
+ </head>
+ <body class="topBarDisabled">
+
+
+
+
+ <div class="container-fluid">
+ <div id="banner">
+ <div class="pull-left">
+ <a href="http://metron.apache.org/" id="bannerLeft">
+ <img src="../../images/metron-logo.png" alt="Apache Metron" width="148px" height="48px"/>
+ </a>
+ </div>
+ <div class="pull-right"> </div>
+ <div class="clear"><hr/></div>
+ </div>
+
+ <div id="breadcrumbs">
+ <ul class="breadcrumb">
+
+
+ <li class="">
+ <a href="http://www.apache.org" class="externalLink" title="Apache">
+ Apache</a>
+ </li>
+ <li class="divider ">/</li>
+ <li class="">
+ <a href="http://metron.apache.org/" class="externalLink" title="Metron">
+ Metron</a>
+ </li>
+ <li class="divider ">/</li>
+ <li class="">
+ <a href="../../index.html" title="Documentation">
+ Documentation</a>
+ </li>
+ <li class="divider ">/</li>
+ <li class="">Introduction</li>
+
+
+
+ <li id="publishDate" class="pull-right">Last Published: 2018-01-03</li> <li class="divider pull-right">|</li>
+ <li id="projectVersion" class="pull-right">Version: 0.4.2</li>
+
+ </ul>
+ </div>
+
+
+ <div class="row-fluid">
+ <div id="leftColumn" class="span3">
+ <div class="well sidebar-nav">
+
+
+ <ul class="nav nav-list">
+ <li class="nav-header">User Documentation</li>
+
+ <li>
+
+ <a href="../../index.html" title="Metron">
+ <i class="icon-chevron-down"></i>
+ Metron</a>
+ <ul class="nav nav-list">
+
+ <li>
+
+ <a href="../../Upgrading.html" title="Upgrading">
+ <i class="none"></i>
+ Upgrading</a>
+ </li>
+
+ <li>
+
+ <a href="../../metron-analytics/index.html" title="Analytics">
+ <i class="icon-chevron-right"></i>
+ Analytics</a>
+ </li>
+
+ <li>
+
+ <a href="../../metron-contrib/metron-docker/index.html" title="Docker">
+ <i class="none"></i>
+ Docker</a>
+ </li>
+
+ <li>
+
+ <a href="../../metron-deployment/index.html" title="Deployment">
+ <i class="icon-chevron-right"></i>
+ Deployment</a>
+ </li>
+
+ <li>
+
+ <a href="../../metron-interface/metron-alerts/index.html" title="Alerts">
+ <i class="none"></i>
+ Alerts</a>
+ </li>
+
+ <li>
+
+ <a href="../../metron-interface/metron-config/index.html" title="Config">
+ <i class="none"></i>
+ Config</a>
+ </li>
+
+ <li>
+
+ <a href="../../metron-interface/metron-rest/index.html" title="Rest">
+ <i class="none"></i>
+ Rest</a>
+ </li>
+
+ <li>
+
+ <a href="../../metron-platform/index.html" title="Platform">
+ <i class="icon-chevron-right"></i>
+ Platform</a>
+ </li>
+
+ <li>
+
+ <a href="../../metron-sensors/index.html" title="Sensors">
+ <i class="icon-chevron-right"></i>
+ Sensors</a>
+ </li>
+
+ <li class="active">
+
+ <a href="#"><i class="none"></i>Stellar-3rd-party-example</a>
+ </li>
+
+ <li>
+
+ <a href="../../metron-stellar/stellar-common/index.html" title="Stellar-common">
+ <i class="icon-chevron-right"></i>
+ Stellar-common</a>
+ </li>
+
+ <li>
+
+ <a href="../../use-cases/index.html" title="Use-cases">
+ <i class="icon-chevron-right"></i>
+ Use-cases</a>
+ </li>
+ </ul>
+ </li>
+ </ul>
+
+
+
+ <hr class="divider" />
+
+ <div id="poweredBy">
+ <div class="clear"></div>
+ <div class="clear"></div>
+ <div class="clear"></div>
+ <a href="http://maven.apache.org/" title="Built by Maven" class="poweredBy">
+ <img class="builtBy" alt="Built by Maven" src="../../images/logos/maven-feather.png" />
+ </a>
+ </div>
+ </div>
+ </div>
+
+
+ <div id="bodyColumn" class="span9" >
+
+ <h1>Introduction</h1>
+<p><a name="Introduction"></a></p>
+<p>This is just a sample 3rd party stellar function (which is used in our unit testing). If Stellar changes its API (e.g. the annotation changes classes), then this will need to be built and the output jar of this (in <tt>target</tt>) will need to be placed in <tt>../stellar-common/src/test/classpath-resources/</tt>.</p>
+ </div>
+ </div>
+ </div>
+
+ <hr/>
+
+ <footer>
+ <div class="container-fluid">
+ <div class="row span12">Copyright © 2018
+ <a href="https://www.apache.org">The Apache Software Foundation</a>.
+ All Rights Reserved.
+
+ </div>
+
+
+
+ </div>
+ </footer>
+ </body>
+</html>
http://git-wip-us.apache.org/repos/asf/metron/blob/3381b853/site/current-book/metron-stellar/stellar-common/3rdPartyStellar.html
----------------------------------------------------------------------
diff --git a/site/current-book/metron-stellar/stellar-common/3rdPartyStellar.html b/site/current-book/metron-stellar/stellar-common/3rdPartyStellar.html
index d5633cc..eb9e054 100644
--- a/site/current-book/metron-stellar/stellar-common/3rdPartyStellar.html
+++ b/site/current-book/metron-stellar/stellar-common/3rdPartyStellar.html
@@ -1,13 +1,13 @@
<!DOCTYPE html>
<!--
- | Generated by Apache Maven Doxia at 2017-09-15
+ | Generated by Apache Maven Doxia at 2018-01-03
| Rendered using Apache Maven Fluido Skin 1.3.0
-->
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
- <meta name="Date-Revision-yyyymmdd" content="20170915" />
+ <meta name="Date-Revision-yyyymmdd" content="20180103" />
<meta http-equiv="Content-Language" content="en" />
<title>Metron – Custom Stellar Functions</title>
<link rel="stylesheet" href="../../css/apache-maven-fluido-1.3.0.min.css" />
@@ -61,8 +61,8 @@
- <li id="publishDate" class="pull-right">Last Published: 2017-09-15</li> <li class="divider pull-right">|</li>
- <li id="projectVersion" class="pull-right">Version: 0.4.1</li>
+ <li id="publishDate" class="pull-right">Last Published: 2018-01-03</li> <li class="divider pull-right">|</li>
+ <li id="projectVersion" class="pull-right">Version: 0.4.2</li>
</ul>
</div>
@@ -103,7 +103,7 @@
<i class="none"></i>
Docker</a>
</li>
-
+
<li>
<a href="../../metron-deployment/index.html" title="Deployment">
@@ -131,20 +131,27 @@
<i class="none"></i>
Rest</a>
</li>
-
+
<li>
<a href="../../metron-platform/index.html" title="Platform">
<i class="icon-chevron-right"></i>
Platform</a>
</li>
-
+
<li>
<a href="../../metron-sensors/index.html" title="Sensors">
<i class="icon-chevron-right"></i>
Sensors</a>
</li>
+
+ <li>
+
+ <a href="../../metron-stellar/stellar-3rd-party-example/index.html" title="Stellar-3rd-party-example">
+ <i class="none"></i>
+ Stellar-3rd-party-example</a>
+ </li>
<li>
@@ -159,7 +166,7 @@
</li>
</ul>
</li>
-
+
<li>
<a href="../../use-cases/index.html" title="Use-cases">
@@ -195,7 +202,7 @@
<div class="section">
<h2><a name="Building_Your_Own_Function"></a>Building Your Own Function</h2>
<p>Let’s say that I need a function that returns the current time in milliseconds since the epoch. I notice that there’s nothing like that currently in Metron, so I embark on the adventure of adding it for my cluster.</p>
-<p>I will presume that you have an installed Metron into your local maven repo via <tt>mvn install</tt> . In the future, when we publish to a maven repo, you will not need this. I will depend on 0.4.1 for the purpose of this demonstration</p>
+<p>I will presume that you have an installed Metron into your local maven repo via <tt>mvn install</tt> . In the future, when we publish to a maven repo, you will not need this. I will depend on 0.4.2 for the purpose of this demonstration</p>
<div class="section">
<h3><a name="Hack_Hack_Hack"></a>Hack, Hack, Hack</h3>
<p>I like to use Maven, so we’ll use that for this demonstration, but you can use whatever build system that you like. Here’s my favorite way to build a project with groupId <tt>com.mycompany.stellar</tt> and artifactId of <tt>tempus</tt> <tt>mvn archetype:create -DgroupId=com.mycompany.stellar -DartifactId=tempus -DarchetypeArtifactId=maven-archetype-quickstart</tt></p>
@@ -223,7 +230,7 @@
<dependency>
<groupId>org.apache.metron</groupId>
<artifactId>metron-common</artifactId>
- <version>0.4.1</version>
+ <version>0.4.2</version>
<!-- NOTE: We will want to depend on the deployed common on the classpath. -->
<scope>provided</scope>
</dependency>
@@ -340,7 +347,7 @@ Returns: Timestamp
<footer>
<div class="container-fluid">
- <div class="row span12">Copyright © 2017
+ <div class="row span12">Copyright © 2018
<a href="https://www.apache.org">The Apache Software Foundation</a>.
All Rights Reserved.
http://git-wip-us.apache.org/repos/asf/metron/blob/3381b853/site/current-book/metron-stellar/stellar-common/index.html
----------------------------------------------------------------------
diff --git a/site/current-book/metron-stellar/stellar-common/index.html b/site/current-book/metron-stellar/stellar-common/index.html
index 933d2a9..0465dbb 100644
--- a/site/current-book/metron-stellar/stellar-common/index.html
+++ b/site/current-book/metron-stellar/stellar-common/index.html
@@ -1,15 +1,15 @@
<!DOCTYPE html>
<!--
- | Generated by Apache Maven Doxia at 2017-09-15
+ | Generated by Apache Maven Doxia at 2018-01-03
| Rendered using Apache Maven Fluido Skin 1.3.0
-->
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
- <meta name="Date-Revision-yyyymmdd" content="20170915" />
+ <meta name="Date-Revision-yyyymmdd" content="20180103" />
<meta http-equiv="Content-Language" content="en" />
- <title>Metron – Contents</title>
+ <title>Metron – Stellar Language</title>
<link rel="stylesheet" href="../../css/apache-maven-fluido-1.3.0.min.css" />
<link rel="stylesheet" href="../../css/site.css" />
<link rel="stylesheet" href="../../css/print.css" media="print" />
@@ -57,12 +57,12 @@
Documentation</a>
</li>
<li class="divider ">/</li>
- <li class="">Contents</li>
+ <li class="">Stellar Language</li>
- <li id="publishDate" class="pull-right">Last Published: 2017-09-15</li> <li class="divider pull-right">|</li>
- <li id="projectVersion" class="pull-right">Version: 0.4.1</li>
+ <li id="publishDate" class="pull-right">Last Published: 2018-01-03</li> <li class="divider pull-right">|</li>
+ <li id="projectVersion" class="pull-right">Version: 0.4.2</li>
</ul>
</div>
@@ -103,7 +103,7 @@
<i class="none"></i>
Docker</a>
</li>
-
+
<li>
<a href="../../metron-deployment/index.html" title="Deployment">
@@ -131,20 +131,27 @@
<i class="none"></i>
Rest</a>
</li>
-
+
<li>
<a href="../../metron-platform/index.html" title="Platform">
<i class="icon-chevron-right"></i>
Platform</a>
</li>
-
+
<li>
<a href="../../metron-sensors/index.html" title="Sensors">
<i class="icon-chevron-right"></i>
Sensors</a>
</li>
+
+ <li>
+
+ <a href="../../metron-stellar/stellar-3rd-party-example/index.html" title="Stellar-3rd-party-example">
+ <i class="none"></i>
+ Stellar-3rd-party-example</a>
+ </li>
<li class="active">
@@ -159,7 +166,7 @@
</li>
</ul>
</li>
-
+
<li>
<a href="../../use-cases/index.html" title="Use-cases">
@@ -188,30 +195,38 @@
<div id="bodyColumn" class="span9" >
- <h1>Contents</h1>
-<p><a name="Contents"></a></p>
+ <h1>Stellar Language</h1>
+<p><a name="Stellar_Language"></a></p>
+<p>For a variety of components (threat intelligence triage and field transformations) we have the need to do simple computation and transformation using the data from messages as variables. For those purposes, there exists a simple, scaled down DSL created to do simple computation and transformation.</p>
<ul>
-<li><a href="#Stellar_Language">Stellar Language</a>
+<li><a href="#Introduction">Introduction</a></li>
+
+<li><a href="#Stellar_Core_Functions">Stellar Core Functions</a></li>
+
+<li><a href="#Stellar_Benchmarks">Stellar Benchmarks</a></li>
+
+<li><a href="#Stellar_Shell">Stellar Shell</a>
<ul>
-<li><a href="#Stellar_Language_Keywords">Stellar Language Keywords</a></li>
+<li><a href="#Getting_Started">Getting Started</a></li>
-<li><a href="#Stellar_Core_Functions">Stellar Core Functions</a></li>
+<li><a href="#Command_Line_Options">Command Line Options</a></li>
-<li><a href="#Stellar_Benchmarks">Stellar Benchmarks</a></li>
+<li><a href="#Variable_Assignment">Variable Assignment</a></li>
-<li><a href="#Stellar_Shell">Stellar Shell</a></li>
+<li><a href="#Magic_Commands">Magic Commands</a></li>
+
+<li><a href="#Advanced_Usage">Advanced Usage</a></li>
</ul></li>
<li><a href="#Stellar_Configuration">Stellar Configuration</a></li>
</ul>
-<p><a name="Stellar_Language"></a></p>
-<h1>Stellar Language</h1>
-<p>For a variety of components (threat intelligence triage and field transformations) we have the need to do simple computation and transformation using the data from messages as variables.<br />For those purposes, there exists a simple, scaled down DSL created to do simple computation and transformation.</p>
-<p>The query language supports the following:</p>
+<div class="section">
+<h2><a name="Introduction"></a>Introduction</h2>
+<p>The Stellar language supports the following:</p>
<ul>
@@ -240,6 +255,8 @@
<li>if/then/else comparisons (i.e. <tt>if var1 < 10 then 'less than 10' else '10 or more'</tt>)</li>
+<li>Simple match evaluations (i.e. <tt>match{ var1 < 10 => 'warn', var1 >= 10 => 'critical', default => 'info'}</tt></li>
+
<li>Determining whether a field exists (via <tt>exists</tt>)</li>
<li>An <tt>in</tt> operator that works like the <tt>in</tt> in Python</li>
@@ -249,7 +266,7 @@
<li>User defined functions, including Lambda expressions</li>
</ul>
<div class="section">
-<h2><a name="Stellar_Language_Keywords"></a>Stellar Language Keywords</h2>
+<h3><a name="Stellar_Language_Keywords"></a>Stellar Language Keywords</h3>
<p>The following keywords need to be single quote escaped in order to be used in Stellar expressions:</p>
<table border="0" class="table table-striped">
@@ -291,41 +308,67 @@
<td align="center">in </td>
-<td align="center">== </td>
+<td align="center">NaN </td>
-<td align="center">!= </td>
+<td align="center">match </td>
</tr>
<tr class="b">
+<td align="center">default </td>
+
+<td align="center">== </td>
+
+<td align="center">!= </td>
+
<td align="center"><= </td>
<td align="center">> </td>
+ </tr>
+
+<tr class="a">
<td align="center">>= </td>
<td align="center">+ </td>
<td align="center">- </td>
- </tr>
-
-<tr class="a">
<td align="center">< </td>
<td align="center">? </td>
+ </tr>
+
+<tr class="b">
<td align="center">* </td>
<td align="center">/ </td>
<td align="center">, </td>
+
+<td align="center">{ </td>
+
+<td align="center">} </td>
+ </tr>
+
+<tr class="a">
+
+<td align="center">\=> </td>
+
+<td align="center"> </td>
+
+<td align="center"> </td>
+
+<td align="center"> </td>
+
+<td align="center"> </td>
</tr>
</tbody>
</table>
<p>Using parens such as: “foo” : “<ok>” requires escaping; “foo”: “'<ok>'”</p></div>
<div class="section">
-<h2><a name="Stellar_Language_Inclusion_Checks_in_and_not_in"></a>Stellar Language Inclusion Checks (<tt>in</tt> and <tt>not in</tt>)</h2>
+<h3><a name="Stellar_Language_Inclusion_Checks_in_and_not_in"></a>Stellar Language Inclusion Checks (<tt>in</tt> and <tt>not in</tt>)</h3>
<ol style="list-style-type: decimal">
@@ -338,7 +381,7 @@
<li><tt>not in</tt> is the negation of the in expression. e.g. <tt>'grok' not in 'foobar' == true</tt></li>
</ol></div>
<div class="section">
-<h2><a name="Stellar_Language_Comparisons____"></a>Stellar Language Comparisons (<tt><</tt>, <tt><=</tt>, <tt>></tt>, <tt>>=</tt>)</h2>
+<h3><a name="Stellar_Language_Comparisons____"></a>Stellar Language Comparisons (<tt><</tt>, <tt><=</tt>, <tt>></tt>, <tt>>=</tt>)</h3>
<ol style="list-style-type: decimal">
@@ -362,7 +405,7 @@
<li>If none of the above are met then an exception is thrown.</li>
</ol></div>
<div class="section">
-<h2><a name="Stellar_Language_Equality_Check__"></a>Stellar Language Equality Check (<tt>==</tt>, <tt>!=</tt>)</h2>
+<h3><a name="Stellar_Language_Equality_Check__"></a>Stellar Language Equality Check (<tt>==</tt>, <tt>!=</tt>)</h3>
<p>Below is how the <tt>==</tt> operator is expected to work:</p>
<ol style="list-style-type: decimal">
@@ -386,7 +429,7 @@
</ol>
<p>The <tt>!=</tt> operator is the negation of the above.</p></div>
<div class="section">
-<h2><a name="Stellar_Language_Lambda_Expressions"></a>Stellar Language Lambda Expressions</h2>
+<h3><a name="Stellar_Language_Lambda_Expressions"></a>Stellar Language Lambda Expressions</h3>
<p>Stellar provides the capability to pass lambda expressions to functions which wish to support that layer of indirection. The syntax is:</p>
<ul>
@@ -431,6 +474,35 @@
<li><tt>REDUCE</tt> - Applies a function over a list of input. For instance <tt>REDUCE([ 1, 2, 3], (sum, x) -> sum + x, 0 )</tt> returns <tt>6</tt></li>
</ul></div>
<div class="section">
+<h3><a name="Stellar_Language_Match_Expression"></a>Stellar Language Match Expression</h3>
+<p>Stellar provides the capability to write match expressions, which are similar to switch statements commonly found in c like languages.</p>
+<p>The syntax is:</p>
+
+<ul>
+
+<li><tt>match{ logical_expression1 => evaluation expression1, logical_expression2 => evaluation_expression2, default => default_expression}</tt></li>
+</ul>
+<p>Where:</p>
+
+<ul>
+
+<li><tt>logical_expression</tt> is a Stellar expression that evaluates to true or false. For instance <tt>var > 0</tt> or <tt>var > 0 AND var2 == 'foo'</tt> or <tt>IF ... THEN ... ELSE</tt></li>
+
+<li><tt>evaluation_expression</tt> is a Stellar Expression</li>
+
+<li><tt>default</tt> is a required default return value, should no logical expression match</li>
+</ul>
+
+<blockquote>
+<p>default is required </p>
+<p>Lambda expressions are supported, but they must be no argument lambdas such as <tt>() -> STATEMENT</tt></p>
+</blockquote>
+
+<ul>
+
+<li>Only the first clause that evaluates to true will be executed.</li>
+</ul></div></div>
+<div class="section">
<h2><a name="Stellar_Core_Functions"></a>Stellar Core Functions</h2>
<table border="0" class="table table-striped">
@@ -585,17 +657,17 @@
<tr class="b">
-<td><a href="#FUZZY_LANGS"> <tt>FUZZY_LANGS</tt></a> </td>
+<td><a href="#FORMAT"> <tt>FORMAT</tt></a> </td>
</tr>
<tr class="a">
-<td><a href="#FUZZY_SCORE"> <tt>FUZZY_SCORE</tt></a> </td>
+<td><a href="#FUZZY_LANGS"> <tt>FUZZY_LANGS</tt></a> </td>
</tr>
<tr class="b">
-<td><a href="#FORMAT"> <tt>FORMAT</tt></a> </td>
+<td><a href="#FUZZY_SCORE"> <tt>FUZZY_SCORE</tt></a> </td>
</tr>
<tr class="a">
@@ -660,22 +732,22 @@
<tr class="a">
-<td><a href="../../metron-analytics/metron-statistics/index.html#HLLP_CARDINALITY"> <tt>HLLP_CARDINALITY</tt></a> </td>
+<td><a href="../../metron-analytics/metron-statistics/index.html#HLLP_ADD"> <tt>HLLP_ADD</tt></a> </td>
</tr>
<tr class="b">
-<td><a href="../../metron-analytics/metron-statistics/index.html#HLLP_INIT"> <tt>HLLP_INIT</tt></a> </td>
+<td><a href="../../metron-analytics/metron-statistics/index.html#HLLP_CARDINALITY"> <tt>HLLP_CARDINALITY</tt></a> </td>
</tr>
<tr class="a">
-<td><a href="../../metron-analytics/metron-statistics/index.html#HLLP_MERGE"> <tt>HLLP_MERGE</tt></a> </td>
+<td><a href="../../metron-analytics/metron-statistics/index.html#HLLP_INIT"> <tt>HLLP_INIT</tt></a> </td>
</tr>
<tr class="b">
-<td><a href="../../metron-analytics/metron-statistics/index.html#HLLP_OFFER"> <tt>HLLP_OFFER</tt></a> </td>
+<td><a href="../../metron-analytics/metron-statistics/index.html#HLLP_MERGE"> <tt>HLLP_MERGE</tt></a> </td>
</tr>
<tr class="a">
@@ -720,79 +792,99 @@
<tr class="a">
-<td><a href="#IS_URL"> <tt>IS_URL</tt></a> </td>
+<td><a href="#IS_NAN"> <tt>IS_NAN</tt></a> </td>
</tr>
<tr class="b">
-<td><a href="#JOIN"> <tt>JOIN</tt></a> </td>
+<td><a href="#IS_URL"> <tt>IS_URL</tt></a> </td>
</tr>
<tr class="a">
-<td><a href="#KAFKA_GET"> <tt>KAFKA_GET</tt></a> </td>
+<td><a href="#JOIN"> <tt>JOIN</tt></a> </td>
</tr>
<tr class="b">
-<td><a href="#KAFKA_PROPS"> <tt>KAFKA_PROPS</tt></a> </td>
+<td><a href="#KAFKA_GET"> <tt>KAFKA_GET</tt></a> </td>
</tr>
<tr class="a">
-<td><a href="#KAFKA_PUT"> <tt>KAFKA_PUT</tt></a> </td>
+<td><a href="#KAFKA_PROPS"> <tt>KAFKA_PROPS</tt></a> </td>
</tr>
<tr class="b">
-<td><a href="#KAFKA_TAIL"> <tt>KAFKA_TAIL</tt></a> </td>
+<td><a href="#KAFKA_PUT"> <tt>KAFKA_PUT</tt></a> </td>
</tr>
<tr class="a">
-<td><a href="#LENGTH"> <tt>LENGTH</tt></a> </td>
+<td><a href="#KAFKA_TAIL"> <tt>KAFKA_TAIL</tt></a> </td>
</tr>
<tr class="b">
-<td><a href="#LIST_ADD"> <tt>LIST_ADD</tt></a> </td>
+<td><a href="#LENGTH"> <tt>LENGTH</tt></a> </td>
</tr>
<tr class="a">
-<td><a href="#LOG2"> <tt>LOG2</tt></a> </td>
+<td><a href="#LIST_ADD"> <tt>LIST_ADD</tt></a> </td>
</tr>
<tr class="b">
-<td><a href="#LOG10"> <tt>LOG10</tt></a> </td>
+<td><a href="#LOG2"> <tt>LOG2</tt></a> </td>
</tr>
<tr class="a">
-<td><a href="#LN"> <tt>LN</tt></a> </td>
+<td><a href="#LOG10"> <tt>LOG10</tt></a> </td>
</tr>
<tr class="b">
-<td><a href="#MAAS_GET_ENDPOINT"> <tt>MAAS_GET_ENDPOINT</tt></a> </td>
+<td><a href="#LN"> <tt>LN</tt></a> </td>
</tr>
<tr class="a">
-<td><a href="#MAAS_MODEL_APPLY"> <tt>MAAS_MODEL_APPLY</tt></a> </td>
+<td><a href="#MAAS_GET_ENDPOINT"> <tt>MAAS_GET_ENDPOINT</tt></a> </td>
</tr>
<tr class="b">
-<td><a href="#MAP"> <tt>MAP</tt></a> </td>
+<td><a href="#MAAS_MODEL_APPLY"> <tt>MAAS_MODEL_APPLY</tt></a> </td>
</tr>
<tr class="a">
+<td><a href="#MAP"> <tt>MAP</tt></a> </td>
+ </tr>
+
+<tr class="b">
+
<td><a href="#MAP_EXISTS"> <tt>MAP_EXISTS</tt></a> </td>
</tr>
+<tr class="a">
+
+<td><a href="#MAP_GET"> <tt>MAP_GET</tt></a> </td>
+ </tr>
+
+<tr class="b">
+
+<td><a href="#MAX"> <tt>MAX</tt></a> </td>
+ </tr>
+
+<tr class="a">
+
+<td><a href="#MIN"> <tt>MIN</tt></a> </td>
+ </tr>
+
<tr class="b">
<td><a href="#MONTH"> <tt>MONTH</tt></a> </td>
@@ -890,12 +982,12 @@
<tr class="a">
-<td><a href="#SPLIT"> <tt>SPLIT</tt></a> </td>
+<td><a href="#SIN"> <tt>SIN</tt></a> </td>
</tr>
<tr class="b">
-<td><a href="#SIN"> <tt>SIN</tt></a> </td>
+<td><a href="#SPLIT"> <tt>SPLIT</tt></a> </td>
</tr>
<tr class="a">
@@ -1030,24 +1122,44 @@
<tr class="a">
-<td><a href="#TO_DOUBLE"> <tt>TO_DOUBLE</tt></a> </td>
+<td><a href="#TLSH_DIST"> <tt>TLSH_DIST</tt></a> </td>
</tr>
<tr class="b">
-<td><a href="#TO_EPOCH_TIMESTAMP"> <tt>TO_EPOCH_TIMESTAMP</tt></a> </td>
+<td><a href="#TO_DOUBLE"> <tt>TO_DOUBLE</tt></a> </td>
</tr>
<tr class="a">
-<td><a href="#TO_FLOAT"> <tt>TO_FLOAT</tt></a> </td>
+<td><a href="#TO_EPOCH_TIMESTAMP"> <tt>TO_EPOCH_TIMESTAMP</tt></a> </td>
</tr>
<tr class="b">
+<td><a href="#TO_FLOAT"> <tt>TO_FLOAT</tt></a> </td>
+ </tr>
+
+<tr class="a">
+
<td><a href="#TO_INTEGER"> <tt>TO_INTEGER</tt></a> </td>
</tr>
+<tr class="b">
+
+<td><a href="#TO_JSON_LIST"> <tt>TO_JSON_LIST</tt></a> </td>
+ </tr>
+
+<tr class="a">
+
+<td><a href="#TO_JSON_MAP"> <tt>TO_JSON_MAP</tt></a> </td>
+ </tr>
+
+<tr class="b">
+
+<td><a href="#TO_JSON_OBJECT"> <tt>TO_JSON_OBJECT</tt></a> </td>
+ </tr>
+
<tr class="a">
<td><a href="#TO_LONG"> <tt>TO_LONG</tt></a> </td>
@@ -1115,7 +1227,7 @@
<tr class="b">
-<td><a href="#ZIP_JAGGED"> <tt>ZIP_JAGGED</tt></a> </td>
+<td><a href="#ZIP_LONGEST"> <tt>ZIP_LONGEST</tt></a> </td>
</tr>
</tbody>
</table>
@@ -1152,7 +1264,7 @@
<li>bloom - The bloom filter</li>
-<li>value* - The values to add</li>
+<li>value(s) - The value(s) to add</li>
</ul></li>
<li>Returns: Bloom Filter</li>
@@ -1830,6 +1942,15 @@
<li>Returns: Last element of the list</li>
</ul></div>
<div class="section">
+<h3><a name="GET_SUPPORTED_ENCODINGS"></a><tt>GET_SUPPORTED_ENCODINGS</tt></h3>
+
+<ul>
+
+<li>Description: Returns a list of the encodings that are currently supported.</li>
+
+<li>Returns: A List of String</li>
+</ul></div>
+<div class="section">
<h3><a name="HASH"></a><tt>HASH</tt></h3>
<ul>
@@ -1843,9 +1964,34 @@
<li>toHash - value to hash.</li>
<li>hashType - A valid string representation of a hashing algorithm. See ‘GET_HASHES_AVAILABLE’.</li>
+
+<li>config? - Configuration for the hash function in the form of a String to object map.
+
+<ul>
+
+<li>For forensic hash TLSH (see <a class="externalLink" href="https://github.com/trendmicro/tlsh">https://github.com/trendmicro/tlsh</a> and Jonathan Oliver, Chun Cheng, and Yanggui Chen, TLSH - A Locality Sensitive Hash. 4th Cybercrime and Trustworthy Computing Workshop, Sydney, November 2013):
+
+<ul>
+
+<li>bucketSize : This defines the size of the hash created. Valid values are 128 (default) or 256 (the former results in a 70 character hash and latter results in 134 characters)</li>
+
+<li>checksumBytes : This defines how many bytes are used to capture the checksum. Valid values are 1 (default) and 3</li>
+
+<li>force : If true (the default) then a hash can be generated from as few as 50 bytes. If false, then at least 256 bytes are required. Insufficient variation or size in the bytes result in a null being returned.</li>
+
+<li>hashes : You can compute a second hash for use in fuzzy clustering TLSH signatures. The number of hashes is the lever to adjust the size of those clusters and "fuzzy" the clusters are. If this is specified, then one or more bins are created based on the specified size and the function will return a Map containing the bins.</li>
+ </ul></li>
+
+<li>For all other hashes:
+
+<ul>
+
+<li>charset : The character set to use (UTF8 is default).</li>
+ </ul></li>
+ </ul></li>
</ul></li>
-<li>Returns: A hex encoded string of a hashed value using the given algorithm. If ‘hashType’ is null then ‘00’, padded to the necessary length, will be returned. If ‘toHash’ is not able to be hashed or ‘hashType’ is null then null is returned.</li>
+<li>Returns = A hex encoded string of a hashed value using the given algorithm. If ‘hashType’ is null then ‘00’, padded to the necessary length, will be returned. If ‘toHash’ is not able to be hashed or ‘hashType’ is null then null is returned.</li>
</ul></div>
<div class="section">
<h3><a name="IN_SUBNET"></a><tt>IN_SUBNET</tt></h3>
@@ -1984,6 +2130,22 @@
<li>Returns: True if the string is an IP and false otherwise.</li>
</ul></div>
<div class="section">
+<h3><a name="IS_NAN"></a><tt>IS_NAN</tt></h3>
+
+<ul>
+
+<li>Description: Evaluates if the passed number is NaN. The number is evaluated as a double.</li>
+
+<li>Input:
+
+<ul>
+
+<li>number - number to evaluate"</li>
+ </ul></li>
+
+<li>Returns: True if the number is NaN, false if it is</li>
+</ul></div>
+<div class="section">
<h3><a name="IS_URL"></a><tt>IS_URL</tt></h3>
<ul>
@@ -2128,15 +2290,6 @@
<li>Returns: Resulting list with the item added at the end.</li>
</ul></div>
<div class="section">
-<h3><a name="GET_SUPPORTED_ENCODINGS"></a><tt>GET_SUPPORTED_ENCODINGS</tt></h3>
-
-<ul>
-
-<li>Description: Returns a list of the encodings that are currently supported.</li>
-
-<li>Returns: A List of String</li>
-</ul></div>
-<div class="section">
<h3><a name="LOG2"></a><tt>LOG2</tt></h3>
<ul>
@@ -2279,6 +2432,38 @@
<li>Returns: The object associated with the key in the map. If no value is associated with the key and default is specified, then default is returned. If no value is associated with the key or default, then null is returned.</li>
</ul></div>
<div class="section">
+<h3><a name="MAX"></a><tt>MAX</tt></h3>
+
+<ul>
+
+<li>Description: Returns the maximum value of a list of input values.
+
+<ul>
+
+<li>Input:</li>
+
+<li>"list - List of arguments. The list may only contain objects that are mutually comparable / ordinal (implement java.lang.Comparable interface). Multi type numeric comparisons are supported: MAX([10,15L,15.3]) would return 15.3, but MAX([‘23’,25]) will fail and return null as strings and numbers can’t be compared.</li>
+ </ul></li>
+
+<li>Returns: The maximum value of the list, or null if the list is empty or the input values were not comparable.</li>
+</ul></div>
+<div class="section">
+<h3><a name="MIN"></a><tt>MIN</tt></h3>
+
+<ul>
+
+<li>Description: Returns the minimum value of a list of input values.
+
+<ul>
+
+<li>Input:</li>
+
+<li>"list - List of arguments. The list may only contain objects that are mutually comparable / ordinal (implement java.lang.Comparable interface). Multi type numeric comparisons are supported: MIN([10,15L,15.3]) would return 10, but MIN([‘23’,25]) will fail and return null as strings and numbers can’t be compared.</li>
+ </ul></li>
+
+<li>Returns: The minimum value of the list, or null if the list is empty or the input values were not comparable.</li>
+</ul></div>
+<div class="section">
<h3><a name="MONTH"></a><tt>MONTH</tt></h3>
<ul>
@@ -2292,7 +2477,7 @@
<li>dateTime - The datetime as a long representing the milliseconds since unix epoch</li>
</ul></li>
-<li>Returns: The current month (0-based).</li>
+<li>Returns: The current month (0-based)</li>
</ul></div>
<div class="section">
<h3><a name="MULTISET_ADD"></a><tt>MULTISET_ADD</tt></h3>
@@ -2323,7 +2508,7 @@
<ul>
-<li>input? - An initialization of the multiset</li>
+<li>input (optional) - An initialization of the multiset</li>
</ul></li>
<li>Returns: A multiset</li>
@@ -2581,7 +2766,7 @@
<ul>
-<li>input? - An initialization of the set</li>
+<li>input (optional) - An initialization of the set</li>
</ul></li>
<li>Returns: A Set</li>
@@ -2637,54 +2822,54 @@
<li>Returns: The sine of the number passed in.</li>
</ul></div>
<div class="section">
-<h3><a name="SQRT"></a><tt>SQRT</tt></h3>
+<h3><a name="SPLIT"></a><tt>SPLIT</tt></h3>
<ul>
-<li>Description: Returns the square root of a number.</li>
+<li>Description: Splits the string by the delimiter.</li>
<li>Input:
<ul>
-<li>number - The number to take the square root of</li>
+<li>input - String to split</li>
+
+<li>delim - String delimiter</li>
</ul></li>
-<li>Returns: The square root of the number passed in.</li>
+<li>Returns: List of strings</li>
</ul></div>
<div class="section">
-<h3><a name="STRING_ENTROPY"></a><tt>STRING_ENTROPY</tt></h3>
+<h3><a name="SQRT"></a><tt>SQRT</tt></h3>
<ul>
-<li>Description: Computes the base-2 shannon entropy of a string.</li>
+<li>Description: Returns the square root of a number.</li>
<li>Input:
<ul>
-<li>input - String</li>
+<li>number - The number to take the square root of</li>
</ul></li>
-<li>Returns: The base-2 shannon entropy of the string (<a class="externalLink" href="https://en.wikipedia.org/wiki/Entropy_(information_theory)#Definition)">https://en.wikipedia.org/wiki/Entropy_(information_theory)#Definition)</a>. The unit of this is bits.</li>
+<li>Returns: The square root of the number passed in.</li>
</ul></div>
<div class="section">
-<h3><a name="SPLIT"></a><tt>SPLIT</tt></h3>
+<h3><a name="STRING_ENTROPY"></a><tt>STRING_ENTROPY</tt></h3>
<ul>
-<li>Description: Splits the string by the delimiter.</li>
+<li>Description: Computes the base-2 shannon entropy of a string.</li>
<li>Input:
<ul>
-<li>input - String to split</li>
-
-<li>delim - String delimiter</li>
+<li>input - String</li>
</ul></li>
-<li>Returns: List of strings</li>
+<li>Returns: The base-2 shannon entropy of the string (<a class="externalLink" href="https://en.wikipedia.org/wiki/Entropy_(information_theory)#Definition)">https://en.wikipedia.org/wiki/Entropy_(information_theory)#Definition)</a>. The unit of this is bits.</li>
</ul></div>
<div class="section">
<h3><a name="STARTS_WITH"></a><tt>STARTS_WITH</tt></h3>
@@ -2773,6 +2958,24 @@
<li>Returns: The tangent of the number passed in.</li>
</ul></div>
<div class="section">
+<h3><a name="TLSH_DIST"></a><tt>TLSH_DIST</tt></h3>
+
+<ul>
+
+<li>Description: Will return the hamming distance between two TLSH hashes (note: must be computed with the same params). For more information, see <a class="externalLink" href="https://github.com/trendmicro/tlsh">https://github.com/trendmicro/tlsh</a> and Jonathan Oliver, Chun Cheng, and Yanggui Chen, TLSH - A Locality Sensitive Hash. 4th Cybercrime and Trustworthy Computing Workshop, Sydney, November 2013. For a discussion of tradeoffs, see Table II on page 5 of <a class="externalLink" href="https://github.com/trendmicro/tlsh/blob/master/TLSH_CTC_final.pdf">https://github.com/trendmicro/tlsh/blob/master/TLSH_CTC_final.pdf</a></li>
+
+<li>Input:
+
+<ul>
+
+<li>hash1 - The first TLSH hash</li>
+
+<li>hash2 - The first TLSH hash</li>
+
+<li>includeLength? - Include the length in the distance calculation or not? Returns: An integer representing the distance between hash1 and hash2. The distance is roughly hamming distance, so 0 is very similar.</li>
+ </ul></li>
+</ul></div>
+<div class="section">
<h3><a name="TO_DOUBLE"></a><tt>TO_DOUBLE</tt></h3>
<ul>
@@ -2809,7 +3012,7 @@
<li>Returns: Epoch timestamp</li>
</ul></div>
<div class="section">
-<h3><a name="TO_FOAT"></a><tt>TO_FOAT</tt></h3>
+<h3><a name="TO_FLOAT"></a><tt>TO_FLOAT</tt></h3>
<ul>
@@ -2841,6 +3044,54 @@
<li>Returns: Integer version of the first argument</li>
</ul></div>
<div class="section">
+<h3><a name="TO_JSON_LIST"></a><tt>TO_JSON_LIST</tt></h3>
+
+<ul>
+
+<li>Description: Accepts JSON string as an input and returns a List object parsed by Jackson. You need to be aware of content of JSON string that is to be parsed. For e.g. <tt>GET_FIRST( TO_JSON_LIST( '[ "foo", 2]')</tt> would yield <tt>foo</tt></li>
+
+<li>Input:
+
+<ul>
+
+<li>string - The JSON string to be parsed</li>
+ </ul></li>
+
+<li>Returns: A parsed List object</li>
+</ul></div>
+<div class="section">
+<h3><a name="TO_JSON_MAP"></a><tt>TO_JSON_MAP</tt></h3>
+
+<ul>
+
+<li>Description: Accepts JSON string as an input and returns a Map object parsed by Jackson. You need to be aware of content of JSON string that is to be parsed. For e.g. <tt>MAP_GET( 'bar', TO_JSON_MAP( '{ "foo" : 1, "bar" : 2}' )</tt> would yield <tt>2</tt></li>
+
+<li>Input:
+
+<ul>
+
+<li>string - The JSON string to be parsed</li>
+ </ul></li>
+
+<li>Returns: A parsed Map object</li>
+</ul></div>
+<div class="section">
+<h3><a name="TO_JSON_OBJECT"></a><tt>TO_JSON_OBJECT</tt></h3>
+
+<ul>
+
+<li>Description: Accepts JSON string as an input and returns a JSON Object parsed by Jackson. You need to be aware of content of JSON string that is to be parsed. For e.g. <tt>MAP_GET( 'bar', TO_JSON_OBJECT( '{ "foo" : 1, "bar" : 2}' )</tt> would yield <tt>2</tt></li>
+
+<li>Input:
+
+<ul>
+
+<li>string - The JSON string to be parsed</li>
+ </ul></li>
+
+<li>Returns: A parsed JSON object</li>
+</ul></div>
+<div class="section">
<h3><a name="TO_LONG"></a><tt>TO_LONG</tt></h3>
<ul>
@@ -3043,7 +3294,7 @@
<ul>
-<li>list* - Lists to zip.</li>
+<li>list(s) - List(s) to zip.</li>
</ul></li>
<li>Returns: The zip of the lists. The returned list is the min size of all the lists. e.g. <tt>ZIP( [ 1, 2 ], [ 3, 4, 5] ) == [ [1, 3], [2, 4] ]</tt></li>
@@ -3059,7 +3310,7 @@
<ul>
-<li>list* - Lists to zip.</li>
+<li>list(s) - List(s) to zip.</li>
</ul></li>
<li>Returns: The zip of the lists. The returned list is the max size of all the lists. Empty elements are null e.g. <tt>ZIP_LONGEST( [ 1, 2 ], [ 3, 4, 5] ) == [ [1, 3], [2, 4], [null, 5] ]</tt></li>
@@ -3171,6 +3422,16 @@ mvn exec:java -Dexec.mainClass="org.apache.metron.stellar.common.benchmark.
</ul>
<p>Note: Stellar classpath configuration from the global config is honored here if the REPL knows about zookeeper.</p>
<div class="section">
+<h3><a name="Environment_Variables"></a>Environment Variables</h3>
+<p>When starting the REPL via <tt>$METRON_HOME/bin/stellar</tt> you can specify certain environment variables to customize the experience:</p>
+
+<ul>
+
+<li><tt>JVMFLAGS</tt> - Arbitrary JVM flags to pass to the <tt>java</tt> command when starting the REPL.</li>
+
+<li><tt>CONTRIB</tt> - Directory where jars with Stellar functions can be placed. The default is <tt>$METRON_HOME/contrib</tt>.</li>
+</ul></div>
+<div class="section">
<h3><a name="Getting_Started"></a>Getting Started</h3>
<p>To run the Stellar Shell from within a deployed Metron cluster, run the following command on the host where Metron is installed.</p>
@@ -3270,15 +3531,44 @@ IN_SUBNET
<div class="source">
<div class="source">
-<pre>Stellar, Go!
-{es.clustername=metron, es.ip=node1, es.port=9300, es.date.format=yyyy.MM.dd.HH}
-[Stellar]>>> %vars
+<pre>[Stellar]>>> %vars
[Stellar]>>> foo := 2 + 2
4.0
[Stellar]>>> %vars
foo = 4.0
</pre></div></div></div>
<div class="section">
+<h4><a name="aglobals"></a><tt>%globals</tt></h4>
+<p>Lists all values that are defined in the global configuration.</p>
+<p>Most of Metron’s functional components have access to what is called the global configuration. This is a key/value configuration store that can be used to customize Metron. Many Stellar functions accept configuration values from the global configuration. The Stellar Shell also leverages the global configuration for customizing the behavior of many Stellar functions. </p>
+
+<div class="source">
+<div class="source">
+<pre>[Stellar]>>> %globals
+{es.clustername=metron, es.ip=node1:9300, es.date.format=yyyy.MM.dd.HH, parser.error.topic=indexing, update.hbase.table=metron_update, update.hbase.cf=t}
+</pre></div></div></div>
+<div class="section">
+<h4><a name="adefine"></a><tt>%define</tt></h4>
+<p>Defines a global configuration value in the current shell session. This value will be forgotten once the session is ended.</p>
+
+<div class="source">
+<div class="source">
+<pre>[Stellar]>>> %define bootstrap.servers := "node1:6667"
+node1:6667
+[Stellar]>>> %globals
+{bootstrap.servers=node1:6667}
+</pre></div></div></div>
+<div class="section">
+<h4><a name="aundefine"></a><tt>%undefine</tt></h4>
+<p>Undefine a global configuration value in the current shell session. This will not modify the persisted global configuration.</p>
+
+<div class="source">
+<div class="source">
+<pre>[Stellar]>>> %undefine bootstrap.servers
+[Stellar]>>> %globals
+{}
+</pre></div></div></div>
+<div class="section">
<h4><a name="afunction"></a><tt>?<function></tt></h4>
<p>Returns formatted documentation of the Stellar function. Provides the description of the function along with the expected arguments.</p>
@@ -3326,9 +3616,9 @@ Please note that functions are loading lazily in the background and will be unav
[Stellar]>>> Functions loaded, you may refer to functions now...
%functions
ABS, APPEND_IF_MISSING, BIN, BLOOM_ADD, BLOOM_EXISTS, BLOOM_INIT, BLOOM_MERGE, CHOMP, CHOP, COUNT_MATCHES, DAY_OF_MONTH, DAY_OF_WEEK, DAY_OF_YEAR, DOMAIN_REMOVE_SUBDOMAINS, DOMAIN_REMOVE_TLD, DOMAIN_TO_TLD, ENDS_WITH, FILL_LEFT, FILL_RIGHT, FILTER, FORMAT, GET, GET_FIRST, GET_LAST, HLLP_ADD, HLLP_CARDINALITY, HLLP_INIT, HLLP_MERGE, IN_SUBNET, IS_DATE, IS_DOMAIN, IS_EMAIL, IS_EMPTY, IS_INTEGER, IS_IP, IS_URL, JOIN, LENGTH, LIST_ADD, MAAS_GET_ENDPOINT, MAAS_MODEL_APPLY, MAP, MAP_EXISTS, MAP_GET, MONTH, OUTLIER_MAD_ADD, OUTLIER_MAD_SCORE, OUTLIER_MAD_STATE_MERGE, PREPEND_IF_MISSING, PROFILE_FIXED, PROFILE_GET, PROFILE_WINDOW, PROTOCOL_TO_NAME, REDUCE, REGEXP_MATCH, SPLIT, STARTS_WITH, STATS_ADD, STATS_BIN, STATS_COUNT, STATS_GEOMETRIC_MEAN, STATS_INIT, STATS_KURTOSIS, STATS_MAX, STATS_MEAN, STATS_MERGE, STATS_MIN, STATS_PERCENTILE, STATS_POPULATION_VARIANCE, STATS_QUADRATIC_MEAN, STATS_SD, STATS_SKEWNESS, STATS_SUM, STATS_SUM_LOGS, STATS_SUM_SQUARES, STATS_VARIANCE, STRING_ENTROPY, SYS
TEM_ENV_GET, SYSTEM_PROPERTY_GET, TO_DOUBLE, TO_EPOCH_TIMESTAMP, TO_FLOAT, TO_INTEGER, TO_LONG, TO_LOWER, TO_STRING, TO_UPPER, TRIM, URL_TO_HOST, URL_TO_PATH, URL_TO_PORT, URL_TO_PROTOCOL, WEEK_OF_MONTH, WEEK_OF_YEAR, YEAR
-</pre></div></div>
-<p><a name="Stellar_Configuration"></a></p>
-<h1>Stellar Configuration</h1>
+</pre></div></div></div></div>
+<div class="section">
+<h2><a name="Stellar_Configuration"></a>Stellar Configuration</h2>
<p>Stellar can be configured in a variety of ways from the <a href="../../metron-platform/metron-common/index.html#Global_Configuration">Global Configuration</a>. In particular, there are three main configuration parameters around configuring Stellar:</p>
<ul>
@@ -3338,7 +3628,7 @@ ABS, APPEND_IF_MISSING, BIN, BLOOM_ADD, BLOOM_EXISTS, BLOOM_INIT, BLOOM_MERGE, C
<li><tt>stellar.function.resolver.includes</tt></li>
<li><tt>stellar.function.resolver.excludes</tt></li>
-</ul></div></div>
+</ul></div>
<div class="section">
<h2><a name="stellar.function.paths"></a><tt>stellar.function.paths</tt></h2>
<p>If specified, Stellar will use a custom classloader which will wrap the context classloader and allow for the resolution of classes stored in jars not shipped with Metron and stored in a variety of mediums:</p>
@@ -3366,7 +3656,7 @@ ABS, APPEND_IF_MISSING, BIN, BLOOM_ADD, BLOOM_EXISTS, BLOOM_INIT, BLOOM_MERGE, C
<div class="source">
<pre>{
...
- "stellar.function.paths" : "hdfs://node1:8020/apps/metron/stellar/metron-management-0.4.1.jar, hdfs://node1:8020/apps/metron/3rdparty/.*.jar"
+ "stellar.function.paths" : "hdfs://node1:8020/apps/metron/stellar/metron-management-0.4.2.jar, hdfs://node1:8020/apps/metron/3rdparty/.*.jar"
}
</pre></div></div>
<p>Please be aware that this classloader does not reload functions dynamically and the classpath specified here in the global config is read on topology start. A change in classpath, to be picked up, would necessitate a topology restart at the moment</p></div>
@@ -3396,7 +3686,7 @@ ABS, APPEND_IF_MISSING, BIN, BLOOM_ADD, BLOOM_EXISTS, BLOOM_INIT, BLOOM_MERGE, C
<footer>
<div class="container-fluid">
- <div class="row span12">Copyright © 2017
+ <div class="row span12">Copyright © 2018
<a href="https://www.apache.org">The Apache Software Foundation</a>.
All Rights Reserved.
[5/7] metron git commit: METRON-1388 update public web site to point
at 0.4.2 new release (mattf-horton) closes apache/metron#887
Posted by ma...@apache.org.
http://git-wip-us.apache.org/repos/asf/metron/blob/3381b853/site/current-book/metron-deployment/roles/index.html
----------------------------------------------------------------------
diff --git a/site/current-book/metron-deployment/roles/index.html b/site/current-book/metron-deployment/roles/index.html
index a7cf149..53372ea 100644
--- a/site/current-book/metron-deployment/roles/index.html
+++ b/site/current-book/metron-deployment/roles/index.html
@@ -1,13 +1,13 @@
<!DOCTYPE html>
<!--
- | Generated by Apache Maven Doxia at 2017-09-15
+ | Generated by Apache Maven Doxia at 2018-01-03
| Rendered using Apache Maven Fluido Skin 1.3.0
-->
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
- <meta name="Date-Revision-yyyymmdd" content="20170915" />
+ <meta name="Date-Revision-yyyymmdd" content="20180103" />
<meta http-equiv="Content-Language" content="en" />
<title>Metron – Ansible Roles</title>
<link rel="stylesheet" href="../../css/apache-maven-fluido-1.3.0.min.css" />
@@ -61,8 +61,8 @@
- <li id="publishDate" class="pull-right">Last Published: 2017-09-15</li> <li class="divider pull-right">|</li>
- <li id="projectVersion" class="pull-right">Version: 0.4.1</li>
+ <li id="publishDate" class="pull-right">Last Published: 2018-01-03</li> <li class="divider pull-right">|</li>
+ <li id="projectVersion" class="pull-right">Version: 0.4.2</li>
</ul>
</div>
@@ -103,7 +103,7 @@
<i class="none"></i>
Docker</a>
</li>
-
+
<li>
<a href="../../metron-deployment/index.html" title="Deployment">
@@ -208,7 +208,7 @@
</li>
</ul>
</li>
-
+
<li>
<a href="../../metron-deployment/vagrant/index.html" title="Vagrant">
@@ -238,20 +238,27 @@
<i class="none"></i>
Rest</a>
</li>
-
+
<li>
<a href="../../metron-platform/index.html" title="Platform">
<i class="icon-chevron-right"></i>
Platform</a>
</li>
-
+
<li>
<a href="../../metron-sensors/index.html" title="Sensors">
<i class="icon-chevron-right"></i>
Sensors</a>
</li>
+
+ <li>
+
+ <a href="../../metron-stellar/stellar-3rd-party-example/index.html" title="Stellar-3rd-party-example">
+ <i class="none"></i>
+ Stellar-3rd-party-example</a>
+ </li>
<li>
@@ -259,7 +266,7 @@
<i class="icon-chevron-right"></i>
Stellar-common</a>
</li>
-
+
<li>
<a href="../../use-cases/index.html" title="Use-cases">
@@ -311,7 +318,7 @@
<footer>
<div class="container-fluid">
- <div class="row span12">Copyright © 2017
+ <div class="row span12">Copyright © 2018
<a href="https://www.apache.org">The Apache Software Foundation</a>.
All Rights Reserved.
http://git-wip-us.apache.org/repos/asf/metron/blob/3381b853/site/current-book/metron-deployment/roles/monit/index.html
----------------------------------------------------------------------
diff --git a/site/current-book/metron-deployment/roles/monit/index.html b/site/current-book/metron-deployment/roles/monit/index.html
index 54c659a..09684dc 100644
--- a/site/current-book/metron-deployment/roles/monit/index.html
+++ b/site/current-book/metron-deployment/roles/monit/index.html
@@ -1,13 +1,13 @@
<!DOCTYPE html>
<!--
- | Generated by Apache Maven Doxia at 2017-09-15
+ | Generated by Apache Maven Doxia at 2018-01-03
| Rendered using Apache Maven Fluido Skin 1.3.0
-->
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
- <meta name="Date-Revision-yyyymmdd" content="20170915" />
+ <meta name="Date-Revision-yyyymmdd" content="20180103" />
<meta http-equiv="Content-Language" content="en" />
<title>Metron – Monit Integration</title>
<link rel="stylesheet" href="../../../css/apache-maven-fluido-1.3.0.min.css" />
@@ -61,8 +61,8 @@
- <li id="publishDate" class="pull-right">Last Published: 2017-09-15</li> <li class="divider pull-right">|</li>
- <li id="projectVersion" class="pull-right">Version: 0.4.1</li>
+ <li id="publishDate" class="pull-right">Last Published: 2018-01-03</li> <li class="divider pull-right">|</li>
+ <li id="projectVersion" class="pull-right">Version: 0.4.2</li>
</ul>
</div>
@@ -103,7 +103,7 @@
<i class="none"></i>
Docker</a>
</li>
-
+
<li>
<a href="../../../metron-deployment/index.html" title="Deployment">
@@ -208,7 +208,7 @@
</li>
</ul>
</li>
-
+
<li>
<a href="../../../metron-deployment/vagrant/index.html" title="Vagrant">
@@ -238,20 +238,27 @@
<i class="none"></i>
Rest</a>
</li>
-
+
<li>
<a href="../../../metron-platform/index.html" title="Platform">
<i class="icon-chevron-right"></i>
Platform</a>
</li>
-
+
<li>
<a href="../../../metron-sensors/index.html" title="Sensors">
<i class="icon-chevron-right"></i>
Sensors</a>
</li>
+
+ <li>
+
+ <a href="../../../metron-stellar/stellar-3rd-party-example/index.html" title="Stellar-3rd-party-example">
+ <i class="none"></i>
+ Stellar-3rd-party-example</a>
+ </li>
<li>
@@ -259,7 +266,7 @@
<i class="icon-chevron-right"></i>
Stellar-common</a>
</li>
-
+
<li>
<a href="../../../use-cases/index.html" title="Use-cases">
@@ -367,7 +374,7 @@
<footer>
<div class="container-fluid">
- <div class="row span12">Copyright © 2017
+ <div class="row span12">Copyright © 2018
<a href="https://www.apache.org">The Apache Software Foundation</a>.
All Rights Reserved.
http://git-wip-us.apache.org/repos/asf/metron/blob/3381b853/site/current-book/metron-deployment/roles/opentaxii/index.html
----------------------------------------------------------------------
diff --git a/site/current-book/metron-deployment/roles/opentaxii/index.html b/site/current-book/metron-deployment/roles/opentaxii/index.html
index 89abff1..29d8700 100644
--- a/site/current-book/metron-deployment/roles/opentaxii/index.html
+++ b/site/current-book/metron-deployment/roles/opentaxii/index.html
@@ -1,13 +1,13 @@
<!DOCTYPE html>
<!--
- | Generated by Apache Maven Doxia at 2017-09-15
+ | Generated by Apache Maven Doxia at 2018-01-03
| Rendered using Apache Maven Fluido Skin 1.3.0
-->
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
- <meta name="Date-Revision-yyyymmdd" content="20170915" />
+ <meta name="Date-Revision-yyyymmdd" content="20180103" />
<meta http-equiv="Content-Language" content="en" />
<title>Metron – OpenTAXII</title>
<link rel="stylesheet" href="../../../css/apache-maven-fluido-1.3.0.min.css" />
@@ -61,8 +61,8 @@
- <li id="publishDate" class="pull-right">Last Published: 2017-09-15</li> <li class="divider pull-right">|</li>
- <li id="projectVersion" class="pull-right">Version: 0.4.1</li>
+ <li id="publishDate" class="pull-right">Last Published: 2018-01-03</li> <li class="divider pull-right">|</li>
+ <li id="projectVersion" class="pull-right">Version: 0.4.2</li>
</ul>
</div>
@@ -103,7 +103,7 @@
<i class="none"></i>
Docker</a>
</li>
-
+
<li>
<a href="../../../metron-deployment/index.html" title="Deployment">
@@ -208,7 +208,7 @@
</li>
</ul>
</li>
-
+
<li>
<a href="../../../metron-deployment/vagrant/index.html" title="Vagrant">
@@ -238,20 +238,27 @@
<i class="none"></i>
Rest</a>
</li>
-
+
<li>
<a href="../../../metron-platform/index.html" title="Platform">
<i class="icon-chevron-right"></i>
Platform</a>
</li>
-
+
<li>
<a href="../../../metron-sensors/index.html" title="Sensors">
<i class="icon-chevron-right"></i>
Sensors</a>
</li>
+
+ <li>
+
+ <a href="../../../metron-stellar/stellar-3rd-party-example/index.html" title="Stellar-3rd-party-example">
+ <i class="none"></i>
+ Stellar-3rd-party-example</a>
+ </li>
<li>
@@ -259,7 +266,7 @@
<i class="icon-chevron-right"></i>
Stellar-common</a>
</li>
-
+
<li>
<a href="../../../use-cases/index.html" title="Use-cases">
@@ -459,7 +466,7 @@ taxii-poll --discovery http://hailataxii.com/taxii-data -c guest.phishtank_com -
<footer>
<div class="container-fluid">
- <div class="row span12">Copyright © 2017
+ <div class="row span12">Copyright © 2018
<a href="https://www.apache.org">The Apache Software Foundation</a>.
All Rights Reserved.
http://git-wip-us.apache.org/repos/asf/metron/blob/3381b853/site/current-book/metron-deployment/roles/pcap_replay/index.html
----------------------------------------------------------------------
diff --git a/site/current-book/metron-deployment/roles/pcap_replay/index.html b/site/current-book/metron-deployment/roles/pcap_replay/index.html
index e350bea..22bf7c0 100644
--- a/site/current-book/metron-deployment/roles/pcap_replay/index.html
+++ b/site/current-book/metron-deployment/roles/pcap_replay/index.html
@@ -1,13 +1,13 @@
<!DOCTYPE html>
<!--
- | Generated by Apache Maven Doxia at 2017-09-15
+ | Generated by Apache Maven Doxia at 2018-01-03
| Rendered using Apache Maven Fluido Skin 1.3.0
-->
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
- <meta name="Date-Revision-yyyymmdd" content="20170915" />
+ <meta name="Date-Revision-yyyymmdd" content="20180103" />
<meta http-equiv="Content-Language" content="en" />
<title>Metron – Pcap Replay</title>
<link rel="stylesheet" href="../../../css/apache-maven-fluido-1.3.0.min.css" />
@@ -61,8 +61,8 @@
- <li id="publishDate" class="pull-right">Last Published: 2017-09-15</li> <li class="divider pull-right">|</li>
- <li id="projectVersion" class="pull-right">Version: 0.4.1</li>
+ <li id="publishDate" class="pull-right">Last Published: 2018-01-03</li> <li class="divider pull-right">|</li>
+ <li id="projectVersion" class="pull-right">Version: 0.4.2</li>
</ul>
</div>
@@ -103,7 +103,7 @@
<i class="none"></i>
Docker</a>
</li>
-
+
<li>
<a href="../../../metron-deployment/index.html" title="Deployment">
@@ -208,7 +208,7 @@
</li>
</ul>
</li>
-
+
<li>
<a href="../../../metron-deployment/vagrant/index.html" title="Vagrant">
@@ -238,20 +238,27 @@
<i class="none"></i>
Rest</a>
</li>
-
+
<li>
<a href="../../../metron-platform/index.html" title="Platform">
<i class="icon-chevron-right"></i>
Platform</a>
</li>
-
+
<li>
<a href="../../../metron-sensors/index.html" title="Sensors">
<i class="icon-chevron-right"></i>
Sensors</a>
</li>
+
+ <li>
+
+ <a href="../../../metron-stellar/stellar-3rd-party-example/index.html" title="Stellar-3rd-party-example">
+ <i class="none"></i>
+ Stellar-3rd-party-example</a>
+ </li>
<li>
@@ -259,7 +266,7 @@
<i class="icon-chevron-right"></i>
Stellar-common</a>
</li>
-
+
<li>
<a href="../../../use-cases/index.html" title="Use-cases">
@@ -334,7 +341,7 @@
<footer>
<div class="container-fluid">
- <div class="row span12">Copyright © 2017
+ <div class="row span12">Copyright © 2018
<a href="https://www.apache.org">The Apache Software Foundation</a>.
All Rights Reserved.
http://git-wip-us.apache.org/repos/asf/metron/blob/3381b853/site/current-book/metron-deployment/roles/sensor-stubs/index.html
----------------------------------------------------------------------
diff --git a/site/current-book/metron-deployment/roles/sensor-stubs/index.html b/site/current-book/metron-deployment/roles/sensor-stubs/index.html
index 3756024..436a523 100644
--- a/site/current-book/metron-deployment/roles/sensor-stubs/index.html
+++ b/site/current-book/metron-deployment/roles/sensor-stubs/index.html
@@ -1,13 +1,13 @@
<!DOCTYPE html>
<!--
- | Generated by Apache Maven Doxia at 2017-09-15
+ | Generated by Apache Maven Doxia at 2018-01-03
| Rendered using Apache Maven Fluido Skin 1.3.0
-->
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
- <meta name="Date-Revision-yyyymmdd" content="20170915" />
+ <meta name="Date-Revision-yyyymmdd" content="20180103" />
<meta http-equiv="Content-Language" content="en" />
<title>Metron – Sensor Stubs</title>
<link rel="stylesheet" href="../../../css/apache-maven-fluido-1.3.0.min.css" />
@@ -61,8 +61,8 @@
- <li id="publishDate" class="pull-right">Last Published: 2017-09-15</li> <li class="divider pull-right">|</li>
- <li id="projectVersion" class="pull-right">Version: 0.4.1</li>
+ <li id="publishDate" class="pull-right">Last Published: 2018-01-03</li> <li class="divider pull-right">|</li>
+ <li id="projectVersion" class="pull-right">Version: 0.4.2</li>
</ul>
</div>
@@ -103,7 +103,7 @@
<i class="none"></i>
Docker</a>
</li>
-
+
<li>
<a href="../../../metron-deployment/index.html" title="Deployment">
@@ -208,7 +208,7 @@
</li>
</ul>
</li>
-
+
<li>
<a href="../../../metron-deployment/vagrant/index.html" title="Vagrant">
@@ -238,20 +238,27 @@
<i class="none"></i>
Rest</a>
</li>
-
+
<li>
<a href="../../../metron-platform/index.html" title="Platform">
<i class="icon-chevron-right"></i>
Platform</a>
</li>
-
+
<li>
<a href="../../../metron-sensors/index.html" title="Sensors">
<i class="icon-chevron-right"></i>
Sensors</a>
</li>
+
+ <li>
+
+ <a href="../../../metron-stellar/stellar-3rd-party-example/index.html" title="Stellar-3rd-party-example">
+ <i class="none"></i>
+ Stellar-3rd-party-example</a>
+ </li>
<li>
@@ -259,7 +266,7 @@
<i class="icon-chevron-right"></i>
Stellar-common</a>
</li>
-
+
<li>
<a href="../../../use-cases/index.html" title="Use-cases">
@@ -301,7 +308,7 @@
<div class="section">
<h3><a name="aQ_How_do_I_install_the_sensor_stubs"></a>(Q) How do I install the sensor stubs?</h3>
<p>Using the default playbooks, this role can be installed by using the Ansible tag <tt>sensor-stubs</tt>. This service is installed on the same hosts where the sensors would be; defined by the <tt>sensors</tt> host group.</p>
-<p>The defaults for the “Quick Dev” and “Full Dev” environments have been changed so that the Sensor Stubs are installed by default, rather than the sensors themselves. The Amazon EC2 environment continues to install the original sensors by default.</p></div>
+<p>The defaults for the “Full Dev” environment have been changed so that the Sensor Stubs are installed by default, rather than the sensors themselves. The Amazon EC2 environment continues to install the original sensors by default.</p></div>
<div class="section">
<h3><a name="aQ_How_do_I_use_the_sensor_stubs"></a>(Q) How do I use the sensor stubs?</h3>
<p>Start all sensor stubs. The output includes the PID for each running sensor stub.</p>
@@ -362,11 +369,11 @@ Stopping sensor-stubs...
</pre></div></div></div>
<div class="section">
<h3><a name="aQ_How_do_I_install_the_original_sensors"></a>(Q) How do I install the original sensors?</h3>
-<p>The default behavior can be changed by skipping the <tt>sensor-stubs</tt> flag and including the <tt>sensors</tt> flag. For example, to deploy “Quick Dev” with the original sensors run the following command.</p>
+<p>The default behavior can be changed by skipping the <tt>sensor-stubs</tt> flag and including the <tt>sensors</tt> flag. For example, to deploy “Full Dev” with the original sensors run the following command.</p>
<div class="source">
<div class="source">
-<pre>cd metron-deployment/vagrant/quick-dev-platform
+<pre>cd metron-deployment/vagrant/full-dev-platform
vagrant --ansible-skip-tags="sensor-stubs,solr" up
</pre></div></div></div>
<div class="section">
@@ -380,7 +387,7 @@ vagrant --ansible-skip-tags="sensor-stubs,solr" up
<footer>
<div class="container-fluid">
- <div class="row span12">Copyright © 2017
+ <div class="row span12">Copyright © 2018
<a href="https://www.apache.org">The Apache Software Foundation</a>.
All Rights Reserved.
http://git-wip-us.apache.org/repos/asf/metron/blob/3381b853/site/current-book/metron-deployment/roles/sensor-test-mode/index.html
----------------------------------------------------------------------
diff --git a/site/current-book/metron-deployment/roles/sensor-test-mode/index.html b/site/current-book/metron-deployment/roles/sensor-test-mode/index.html
index d6c62c6..bf88d45 100644
--- a/site/current-book/metron-deployment/roles/sensor-test-mode/index.html
+++ b/site/current-book/metron-deployment/roles/sensor-test-mode/index.html
@@ -1,13 +1,13 @@
<!DOCTYPE html>
<!--
- | Generated by Apache Maven Doxia at 2017-09-15
+ | Generated by Apache Maven Doxia at 2018-01-03
| Rendered using Apache Maven Fluido Skin 1.3.0
-->
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
- <meta name="Date-Revision-yyyymmdd" content="20170915" />
+ <meta name="Date-Revision-yyyymmdd" content="20180103" />
<meta http-equiv="Content-Language" content="en" />
<title>Metron – Sensor Test Mode</title>
<link rel="stylesheet" href="../../../css/apache-maven-fluido-1.3.0.min.css" />
@@ -61,8 +61,8 @@
- <li id="publishDate" class="pull-right">Last Published: 2017-09-15</li> <li class="divider pull-right">|</li>
- <li id="projectVersion" class="pull-right">Version: 0.4.1</li>
+ <li id="publishDate" class="pull-right">Last Published: 2018-01-03</li> <li class="divider pull-right">|</li>
+ <li id="projectVersion" class="pull-right">Version: 0.4.2</li>
</ul>
</div>
@@ -103,7 +103,7 @@
<i class="none"></i>
Docker</a>
</li>
-
+
<li>
<a href="../../../metron-deployment/index.html" title="Deployment">
@@ -208,7 +208,7 @@
</li>
</ul>
</li>
-
+
<li>
<a href="../../../metron-deployment/vagrant/index.html" title="Vagrant">
@@ -238,20 +238,27 @@
<i class="none"></i>
Rest</a>
</li>
-
+
<li>
<a href="../../../metron-platform/index.html" title="Platform">
<i class="icon-chevron-right"></i>
Platform</a>
</li>
-
+
<li>
<a href="../../../metron-sensors/index.html" title="Sensors">
<i class="icon-chevron-right"></i>
Sensors</a>
</li>
+
+ <li>
+
+ <a href="../../../metron-stellar/stellar-3rd-party-example/index.html" title="Stellar-3rd-party-example">
+ <i class="none"></i>
+ Stellar-3rd-party-example</a>
+ </li>
<li>
@@ -259,7 +266,7 @@
<i class="icon-chevron-right"></i>
Stellar-common</a>
</li>
-
+
<li>
<a href="../../../use-cases/index.html" title="Use-cases">
@@ -324,7 +331,7 @@
<footer>
<div class="container-fluid">
- <div class="row span12">Copyright © 2017
+ <div class="row span12">Copyright © 2018
<a href="https://www.apache.org">The Apache Software Foundation</a>.
All Rights Reserved.
http://git-wip-us.apache.org/repos/asf/metron/blob/3381b853/site/current-book/metron-deployment/vagrant/codelab-platform/index.html
----------------------------------------------------------------------
diff --git a/site/current-book/metron-deployment/vagrant/codelab-platform/index.html b/site/current-book/metron-deployment/vagrant/codelab-platform/index.html
index 8edf5ac..e4b8eca 100644
--- a/site/current-book/metron-deployment/vagrant/codelab-platform/index.html
+++ b/site/current-book/metron-deployment/vagrant/codelab-platform/index.html
@@ -1,13 +1,13 @@
<!DOCTYPE html>
<!--
- | Generated by Apache Maven Doxia at 2017-09-15
+ | Generated by Apache Maven Doxia at 2018-01-03
| Rendered using Apache Maven Fluido Skin 1.3.0
-->
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
- <meta name="Date-Revision-yyyymmdd" content="20170915" />
+ <meta name="Date-Revision-yyyymmdd" content="20180103" />
<meta http-equiv="Content-Language" content="en" />
<title>Metron – Developer Image for Apache Metron on Virtualbox</title>
<link rel="stylesheet" href="../../../css/apache-maven-fluido-1.3.0.min.css" />
@@ -61,8 +61,8 @@
- <li id="publishDate" class="pull-right">Last Published: 2017-09-15</li> <li class="divider pull-right">|</li>
- <li id="projectVersion" class="pull-right">Version: 0.4.1</li>
+ <li id="publishDate" class="pull-right">Last Published: 2018-01-03</li> <li class="divider pull-right">|</li>
+ <li id="projectVersion" class="pull-right">Version: 0.4.2</li>
</ul>
</div>
@@ -75,7 +75,7 @@
<ul class="nav nav-list">
<li class="nav-header">User Documentation</li>
-
+
<li>
<a href="../../../index.html" title="Metron">
@@ -231,7 +231,7 @@
<i class="none"></i>
Rest</a>
</li>
-
+
<li>
<a href="../../../metron-platform/index.html" title="Platform">
@@ -245,6 +245,13 @@
<i class="icon-chevron-right"></i>
Sensors</a>
</li>
+
+ <li>
+
+ <a href="../../../metron-stellar/stellar-3rd-party-example/index.html" title="Stellar-3rd-party-example">
+ <i class="none"></i>
+ Stellar-3rd-party-example</a>
+ </li>
<li>
@@ -252,7 +259,7 @@
<i class="icon-chevron-right"></i>
Stellar-common</a>
</li>
-
+
<li>
<a href="../../../use-cases/index.html" title="Use-cases">
@@ -362,7 +369,7 @@
<footer>
<div class="container-fluid">
- <div class="row span12">Copyright © 2017
+ <div class="row span12">Copyright © 2018
<a href="https://www.apache.org">The Apache Software Foundation</a>.
All Rights Reserved.
http://git-wip-us.apache.org/repos/asf/metron/blob/3381b853/site/current-book/metron-deployment/vagrant/fastcapa-test-platform/index.html
----------------------------------------------------------------------
diff --git a/site/current-book/metron-deployment/vagrant/fastcapa-test-platform/index.html b/site/current-book/metron-deployment/vagrant/fastcapa-test-platform/index.html
index 7f0c87c..2fcb617 100644
--- a/site/current-book/metron-deployment/vagrant/fastcapa-test-platform/index.html
+++ b/site/current-book/metron-deployment/vagrant/fastcapa-test-platform/index.html
@@ -1,13 +1,13 @@
<!DOCTYPE html>
<!--
- | Generated by Apache Maven Doxia at 2017-09-15
+ | Generated by Apache Maven Doxia at 2018-01-03
| Rendered using Apache Maven Fluido Skin 1.3.0
-->
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
- <meta name="Date-Revision-yyyymmdd" content="20170915" />
+ <meta name="Date-Revision-yyyymmdd" content="20180103" />
<meta http-equiv="Content-Language" content="en" />
<title>Metron – Fastcapa Test Environment</title>
<link rel="stylesheet" href="../../../css/apache-maven-fluido-1.3.0.min.css" />
@@ -61,8 +61,8 @@
- <li id="publishDate" class="pull-right">Last Published: 2017-09-15</li> <li class="divider pull-right">|</li>
- <li id="projectVersion" class="pull-right">Version: 0.4.1</li>
+ <li id="publishDate" class="pull-right">Last Published: 2018-01-03</li> <li class="divider pull-right">|</li>
+ <li id="projectVersion" class="pull-right">Version: 0.4.2</li>
</ul>
</div>
@@ -103,7 +103,7 @@
<i class="none"></i>
Docker</a>
</li>
-
+
<li>
<a href="../../../metron-deployment/index.html" title="Deployment">
@@ -173,7 +173,7 @@
<i class="icon-chevron-right"></i>
Roles</a>
</li>
-
+
<li>
<a href="../../../metron-deployment/vagrant/index.html" title="Vagrant">
@@ -181,13 +181,6 @@
Vagrant</a>
<ul class="nav nav-list">
- <li>
-
- <a href="../../../metron-deployment/vagrant/codelab-platform/index.html" title="Codelab-platform">
- <i class="none"></i>
- Codelab-platform</a>
- </li>
-
<li class="active">
<a href="#"><i class="none"></i>Fastcapa-test-platform</a>
@@ -199,13 +192,6 @@
<i class="none"></i>
Full-dev-platform</a>
</li>
-
- <li>
-
- <a href="../../../metron-deployment/vagrant/quick-dev-platform/index.html" title="Quick-dev-platform">
- <i class="none"></i>
- Quick-dev-platform</a>
- </li>
</ul>
</li>
</ul>
@@ -231,20 +217,27 @@
<i class="none"></i>
Rest</a>
</li>
-
+
<li>
<a href="../../../metron-platform/index.html" title="Platform">
<i class="icon-chevron-right"></i>
Platform</a>
</li>
-
+
<li>
<a href="../../../metron-sensors/index.html" title="Sensors">
<i class="icon-chevron-right"></i>
Sensors</a>
</li>
+
+ <li>
+
+ <a href="../../../metron-stellar/stellar-3rd-party-example/index.html" title="Stellar-3rd-party-example">
+ <i class="none"></i>
+ Stellar-3rd-party-example</a>
+ </li>
<li>
@@ -252,7 +245,7 @@
<i class="icon-chevron-right"></i>
Stellar-common</a>
</li>
-
+
<li>
<a href="../../../use-cases/index.html" title="Use-cases">
@@ -297,7 +290,15 @@
</ul>
<div class="section">
<h2><a name="Getting_Started"></a>Getting Started</h2>
-<p>Simply run <tt>vagrant up</tt> to launch the environment. Automated tests are executed after provisioning completes to ensure that Fastcapa and the rest of the environment is functioning properly.</p>
+<p>The Fastcapa test environment can be executed on different operating systems. There is a sub-directory for each operating system that Fastcapa can be tested on. </p>
+<p>To run, simply execute <tt>vagrant up</tt> within the appropriate directory. For example, to run the tests on CentOS 7.1 then execute the following commands.</p>
+
+<div class="source">
+<div class="source">
+<pre>cd centos-7.1
+vagrant up
+</pre></div></div>
+<p>Automated tests are executed after provisioning completes to ensure that Fastcapa and the rest of the environment is functioning properly. If you see something like the following, then the tests have passed.</p>
<div class="source">
<div class="source">
@@ -314,6 +315,21 @@ TASK [debug] *******************************************************************
ok: [source] => {
"msg": "Successfully received a Kafka message from fastcapa!"
}
+</pre></div></div>
+<p>If the deployment process fails mid-course, running <tt>vagrant provision</tt> will continue the process from where it left off. This can sometimes occur when the VM reboots as part of the deployment process. The error might look like the following.</p>
+
+<div class="source">
+<div class="source">
+<pre>TASK [fastcapa : Restart for modified kernel params] ***************************
+fatal: [sink]: UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via ssh: Shared connection to 127.0.0.1 closed.\r\n", "unreachable": true}
+ to retry, use: --limit @/Users/nallen/Development/metron/metron-deployment/vagrant/fastcapa-test-platform/playbook.retry
+
+PLAY RECAP *********************************************************************
+sink : ok=11 changed=9 unreachable=1 failed=0
+source : ok=29 changed=25 unreachable=0 failed=0
+
+Ansible failed to complete successfully. Any error output should be
+visible above. Please fix these errors and try again.
</pre></div></div></div>
<div class="section">
<h2><a name="Going_Deeper"></a>Going Deeper</h2>
@@ -352,6 +368,32 @@ service fastcapa status
<pre>/usr/hdp/current/kafka-broker/bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic pcap
</pre></div></div>
<p>Enter ‘CTRL-C’ to kill the <tt>kafka-console-consumer</tt> process once you are able to see that packets are being sent. These packets will appear to be gibberish in the console. This is the raw binary network packet data after all.</p></div></div>
+<div class="section">
+<h2><a name="FAQ"></a>FAQ</h2>
+<div class="section">
+<h3><a name="Error_Message:_Timed_out_while_waiting_for_the_machine_to_boot"></a>Error Message: Timed out while waiting for the machine to boot</h3>
+
+<div class="source">
+<div class="source">
+<pre>Timed out while waiting for the machine to boot. This means that
+Vagrant was unable to communicate with the guest machine within
+the configured ("config.vm.boot_timeout" value) time period.
+If you look above, you should be able to see the error(s) that
+Vagrant had when attempting to connect to the machine. These errors
+are usually good hints as to what may be wrong.
+If you're using a custom box, make sure that networking is properly
+working and you're able to connect to the machine. It is a common
+problem that networking isn't setup properly in these boxes.
+Verify that authentication configurations are also setup properly,
+as well.
+If the box appears to be booting properly, you may want to increase
+the timeout ("config.vm.boot_timeout") value.
+➜ centos-7.4 git:(master) ✗ vagrant status
+Current machine states:
+source running (virtualbox)
+sink not created (virtualbox)
+</pre></div></div>
+<p>If you are unable to launch any of the Fastcapa test environments, which results in a message like the one above, then you may need to upgrade your version of Virtualbox. Success has been reported with versions of VirtualBox 5.1.22+.</p></div></div>
</div>
</div>
</div>
@@ -360,7 +402,7 @@ service fastcapa status
<footer>
<div class="container-fluid">
- <div class="row span12">Copyright © 2017
+ <div class="row span12">Copyright © 2018
<a href="https://www.apache.org">The Apache Software Foundation</a>.
All Rights Reserved.
http://git-wip-us.apache.org/repos/asf/metron/blob/3381b853/site/current-book/metron-deployment/vagrant/full-dev-platform/index.html
----------------------------------------------------------------------
diff --git a/site/current-book/metron-deployment/vagrant/full-dev-platform/index.html b/site/current-book/metron-deployment/vagrant/full-dev-platform/index.html
index 401a3ac..bee92c5 100644
--- a/site/current-book/metron-deployment/vagrant/full-dev-platform/index.html
+++ b/site/current-book/metron-deployment/vagrant/full-dev-platform/index.html
@@ -1,13 +1,13 @@
<!DOCTYPE html>
<!--
- | Generated by Apache Maven Doxia at 2017-09-15
+ | Generated by Apache Maven Doxia at 2018-01-03
| Rendered using Apache Maven Fluido Skin 1.3.0
-->
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
- <meta name="Date-Revision-yyyymmdd" content="20170915" />
+ <meta name="Date-Revision-yyyymmdd" content="20180103" />
<meta http-equiv="Content-Language" content="en" />
<title>Metron – Full Development Platform</title>
<link rel="stylesheet" href="../../../css/apache-maven-fluido-1.3.0.min.css" />
@@ -61,8 +61,8 @@
- <li id="publishDate" class="pull-right">Last Published: 2017-09-15</li> <li class="divider pull-right">|</li>
- <li id="projectVersion" class="pull-right">Version: 0.4.1</li>
+ <li id="publishDate" class="pull-right">Last Published: 2018-01-03</li> <li class="divider pull-right">|</li>
+ <li id="projectVersion" class="pull-right">Version: 0.4.2</li>
</ul>
</div>
@@ -103,7 +103,7 @@
<i class="none"></i>
Docker</a>
</li>
-
+
<li>
<a href="../../../metron-deployment/index.html" title="Deployment">
@@ -173,7 +173,7 @@
<i class="icon-chevron-right"></i>
Roles</a>
</li>
-
+
<li>
<a href="../../../metron-deployment/vagrant/index.html" title="Vagrant">
@@ -183,13 +183,6 @@
<li>
- <a href="../../../metron-deployment/vagrant/codelab-platform/index.html" title="Codelab-platform">
- <i class="none"></i>
- Codelab-platform</a>
- </li>
-
- <li>
-
<a href="../../../metron-deployment/vagrant/fastcapa-test-platform/index.html" title="Fastcapa-test-platform">
<i class="none"></i>
Fastcapa-test-platform</a>
@@ -199,13 +192,6 @@
<a href="#"><i class="none"></i>Full-dev-platform</a>
</li>
-
- <li>
-
- <a href="../../../metron-deployment/vagrant/quick-dev-platform/index.html" title="Quick-dev-platform">
- <i class="none"></i>
- Quick-dev-platform</a>
- </li>
</ul>
</li>
</ul>
@@ -231,20 +217,27 @@
<i class="none"></i>
Rest</a>
</li>
-
+
<li>
<a href="../../../metron-platform/index.html" title="Platform">
<i class="icon-chevron-right"></i>
Platform</a>
</li>
-
+
<li>
<a href="../../../metron-sensors/index.html" title="Sensors">
<i class="icon-chevron-right"></i>
Sensors</a>
</li>
+
+ <li>
+
+ <a href="../../../metron-stellar/stellar-3rd-party-example/index.html" title="Stellar-3rd-party-example">
+ <i class="none"></i>
+ Stellar-3rd-party-example</a>
+ </li>
<li>
@@ -252,7 +245,7 @@
<i class="icon-chevron-right"></i>
Stellar-common</a>
</li>
-
+
<li>
<a href="../../../use-cases/index.html" title="Use-cases">
@@ -320,7 +313,8 @@
<div class="source">
<div class="source">
-<pre>brew cask install vagrant virtualbox java docker
+<pre>brew cask install vagrant virtualbox docker
+brew cask install caskroom/versions/java8
brew install maven@3.3 git
pip install ansible==2.2.2.0
vagrant plugin install vagrant-hostmanager
@@ -412,7 +406,7 @@ vagrant up
<footer>
<div class="container-fluid">
- <div class="row span12">Copyright © 2017
+ <div class="row span12">Copyright © 2018
<a href="https://www.apache.org">The Apache Software Foundation</a>.
All Rights Reserved.
http://git-wip-us.apache.org/repos/asf/metron/blob/3381b853/site/current-book/metron-deployment/vagrant/index.html
----------------------------------------------------------------------
diff --git a/site/current-book/metron-deployment/vagrant/index.html b/site/current-book/metron-deployment/vagrant/index.html
index 8a29917..744ea5b 100644
--- a/site/current-book/metron-deployment/vagrant/index.html
+++ b/site/current-book/metron-deployment/vagrant/index.html
@@ -1,13 +1,13 @@
<!DOCTYPE html>
<!--
- | Generated by Apache Maven Doxia at 2017-09-15
+ | Generated by Apache Maven Doxia at 2018-01-03
| Rendered using Apache Maven Fluido Skin 1.3.0
-->
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
- <meta name="Date-Revision-yyyymmdd" content="20170915" />
+ <meta name="Date-Revision-yyyymmdd" content="20180103" />
<meta http-equiv="Content-Language" content="en" />
<title>Metron – Vagrant Deployment</title>
<link rel="stylesheet" href="../../css/apache-maven-fluido-1.3.0.min.css" />
@@ -61,8 +61,8 @@
- <li id="publishDate" class="pull-right">Last Published: 2017-09-15</li> <li class="divider pull-right">|</li>
- <li id="projectVersion" class="pull-right">Version: 0.4.1</li>
+ <li id="publishDate" class="pull-right">Last Published: 2018-01-03</li> <li class="divider pull-right">|</li>
+ <li id="projectVersion" class="pull-right">Version: 0.4.2</li>
</ul>
</div>
@@ -103,7 +103,7 @@
<i class="none"></i>
Docker</a>
</li>
-
+
<li>
<a href="../../metron-deployment/index.html" title="Deployment">
@@ -173,7 +173,7 @@
<i class="icon-chevron-right"></i>
Roles</a>
</li>
-
+
<li class="active">
<a href="#"><i class="icon-chevron-down"></i>Vagrant</a>
@@ -181,13 +181,6 @@
<li>
- <a href="../../metron-deployment/vagrant/codelab-platform/index.html" title="Codelab-platform">
- <i class="none"></i>
- Codelab-platform</a>
- </li>
-
- <li>
-
<a href="../../metron-deployment/vagrant/fastcapa-test-platform/index.html" title="Fastcapa-test-platform">
<i class="none"></i>
Fastcapa-test-platform</a>
@@ -199,13 +192,6 @@
<i class="none"></i>
Full-dev-platform</a>
</li>
-
- <li>
-
- <a href="../../metron-deployment/vagrant/quick-dev-platform/index.html" title="Quick-dev-platform">
- <i class="none"></i>
- Quick-dev-platform</a>
- </li>
</ul>
</li>
</ul>
@@ -231,20 +217,27 @@
<i class="none"></i>
Rest</a>
</li>
-
+
<li>
<a href="../../metron-platform/index.html" title="Platform">
<i class="icon-chevron-right"></i>
Platform</a>
</li>
-
+
<li>
<a href="../../metron-sensors/index.html" title="Sensors">
<i class="icon-chevron-right"></i>
Sensors</a>
</li>
+
+ <li>
+
+ <a href="../../metron-stellar/stellar-3rd-party-example/index.html" title="Stellar-3rd-party-example">
+ <i class="none"></i>
+ Stellar-3rd-party-example</a>
+ </li>
<li>
@@ -252,7 +245,7 @@
<i class="icon-chevron-right"></i>
Stellar-common</a>
</li>
-
+
<li>
<a href="../../use-cases/index.html" title="Use-cases">
@@ -286,13 +279,9 @@
<ul>
-<li>Codelab Platform</li>
-
<li>Fast CAPA Test Platform</li>
<li>Full Dev Platform</li>
-
-<li>Quick Dev Platform</li>
</ul>
</div>
</div>
@@ -302,7 +291,7 @@
<footer>
<div class="container-fluid">
- <div class="row span12">Copyright © 2017
+ <div class="row span12">Copyright © 2018
<a href="https://www.apache.org">The Apache Software Foundation</a>.
All Rights Reserved.
http://git-wip-us.apache.org/repos/asf/metron/blob/3381b853/site/current-book/metron-deployment/vagrant/quick-dev-platform/index.html
----------------------------------------------------------------------
diff --git a/site/current-book/metron-deployment/vagrant/quick-dev-platform/index.html b/site/current-book/metron-deployment/vagrant/quick-dev-platform/index.html
index 136b790..f72df78 100644
--- a/site/current-book/metron-deployment/vagrant/quick-dev-platform/index.html
+++ b/site/current-book/metron-deployment/vagrant/quick-dev-platform/index.html
@@ -1,13 +1,13 @@
<!DOCTYPE html>
<!--
- | Generated by Apache Maven Doxia at 2017-09-15
+ | Generated by Apache Maven Doxia at 2018-01-03
| Rendered using Apache Maven Fluido Skin 1.3.0
-->
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
- <meta name="Date-Revision-yyyymmdd" content="20170915" />
+ <meta name="Date-Revision-yyyymmdd" content="20180103" />
<meta http-equiv="Content-Language" content="en" />
<title>Metron – Quick Development Platform</title>
<link rel="stylesheet" href="../../../css/apache-maven-fluido-1.3.0.min.css" />
@@ -61,8 +61,8 @@
- <li id="publishDate" class="pull-right">Last Published: 2017-09-15</li> <li class="divider pull-right">|</li>
- <li id="projectVersion" class="pull-right">Version: 0.4.1</li>
+ <li id="publishDate" class="pull-right">Last Published: 2018-01-03</li> <li class="divider pull-right">|</li>
+ <li id="projectVersion" class="pull-right">Version: 0.4.2</li>
</ul>
</div>
@@ -75,7 +75,7 @@
<ul class="nav nav-list">
<li class="nav-header">User Documentation</li>
-
+
<li>
<a href="../../../index.html" title="Metron">
@@ -231,7 +231,7 @@
<i class="none"></i>
Rest</a>
</li>
-
+
<li>
<a href="../../../metron-platform/index.html" title="Platform">
@@ -245,6 +245,13 @@
<i class="icon-chevron-right"></i>
Sensors</a>
</li>
+
+ <li>
+
+ <a href="../../../metron-stellar/stellar-3rd-party-example/index.html" title="Stellar-3rd-party-example">
+ <i class="none"></i>
+ Stellar-3rd-party-example</a>
+ </li>
<li>
@@ -252,7 +259,7 @@
<i class="icon-chevron-right"></i>
Stellar-common</a>
</li>
-
+
<li>
<a href="../../../use-cases/index.html" title="Use-cases">
@@ -385,7 +392,7 @@ vagrant up
<footer>
<div class="container-fluid">
- <div class="row span12">Copyright © 2017
+ <div class="row span12">Copyright © 2018
<a href="https://www.apache.org">The Apache Software Foundation</a>.
All Rights Reserved.
http://git-wip-us.apache.org/repos/asf/metron/blob/3381b853/site/current-book/metron-interface/metron-alerts/index.html
----------------------------------------------------------------------
diff --git a/site/current-book/metron-interface/metron-alerts/index.html b/site/current-book/metron-interface/metron-alerts/index.html
index cbfbe36..cf30176 100644
--- a/site/current-book/metron-interface/metron-alerts/index.html
+++ b/site/current-book/metron-interface/metron-alerts/index.html
@@ -1,13 +1,13 @@
<!DOCTYPE html>
<!--
- | Generated by Apache Maven Doxia at 2017-09-15
+ | Generated by Apache Maven Doxia at 2018-01-03
| Rendered using Apache Maven Fluido Skin 1.3.0
-->
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
- <meta name="Date-Revision-yyyymmdd" content="20170915" />
+ <meta name="Date-Revision-yyyymmdd" content="20180103" />
<meta http-equiv="Content-Language" content="en" />
<title>Metron – </title>
<link rel="stylesheet" href="../../css/apache-maven-fluido-1.3.0.min.css" />
@@ -61,8 +61,8 @@
- <li id="publishDate" class="pull-right">Last Published: 2017-09-15</li> <li class="divider pull-right">|</li>
- <li id="projectVersion" class="pull-right">Version: 0.4.1</li>
+ <li id="publishDate" class="pull-right">Last Published: 2018-01-03</li> <li class="divider pull-right">|</li>
+ <li id="projectVersion" class="pull-right">Version: 0.4.2</li>
</ul>
</div>
@@ -103,7 +103,7 @@
<i class="none"></i>
Docker</a>
</li>
-
+
<li>
<a href="../../metron-deployment/index.html" title="Deployment">
@@ -129,20 +129,27 @@
<i class="none"></i>
Rest</a>
</li>
-
+
<li>
<a href="../../metron-platform/index.html" title="Platform">
<i class="icon-chevron-right"></i>
Platform</a>
</li>
-
+
<li>
<a href="../../metron-sensors/index.html" title="Sensors">
<i class="icon-chevron-right"></i>
Sensors</a>
</li>
+
+ <li>
+
+ <a href="../../metron-stellar/stellar-3rd-party-example/index.html" title="Stellar-3rd-party-example">
+ <i class="none"></i>
+ Stellar-3rd-party-example</a>
+ </li>
<li>
@@ -150,7 +157,7 @@
<i class="icon-chevron-right"></i>
Stellar-common</a>
</li>
-
+
<li>
<a href="../../use-cases/index.html" title="Use-cases">
@@ -195,127 +202,164 @@
</ul>
<div class="section">
<h2><a name="Caveats"></a>Caveats</h2>
-
-<ul>
-
-<li>UI doesn’t have an authentication module yet</li>
-
-<li>UI uses local storage to save all the data. A middleware needs to be designed and developed for persisting the data</li>
-</ul></div>
+<div class="section">
+<h3><a name="Local_Storage"></a>Local Storage</h3>
+<p>UI uses local storage to save all the data. A middleware needs to be designed and developed for persisting the data</p></div>
+<div class="section">
+<h3><a name="Search_for_Alert_GUIDs"></a>Search for Alert GUIDs</h3>
+<p>Alert GUIDs must be double-quoted when being searched on to ensure correctness of results, e.g. guid:“id1”.</p></div>
+<div class="section">
+<h3><a name="Search_for_Comments"></a>Search for Comments</h3>
+<p>Users cannot search for the contents of the comment’s in the Alerts-UI</p></div>
+<div class="section">
+<h3><a name="Meta_alerts"></a>Meta alerts</h3>
+<p>Grouping/faceting requests and other aggregations do not return meta alerts. This is because it’s not clear what the intended results should be when there are multiple matching items.</p>
+<p>Sorting has a similar caveat, in that if we are matching on multiple alerts, there is no well defined sort.</p>
+<p>Alerts that are contained in a a meta alert are generally excluded from search results, because a user has already grouped them in a meaningful way.</p></div></div>
<div class="section">
<h2><a name="Prerequisites"></a>Prerequisites</h2>
<ul>
-<li>Elastic search should be up and running and should have some alerts populated by metron topologies</li>
+<li>The Metron REST application should be up and running and Elasticsearch should have some alerts populated by Metron topologies</li>
-<li>The alerts can be populated using Quick Dev, Full Dev or any other setup</li>
+<li>The Management UI should be installed (which includes <a class="externalLink" href="https://expressjs.com/">Express</a>)</li>
+
+<li>The alerts can be populated using Full Dev or any other setup</li>
<li>UI is developed using angular4 and uses angular-cli</li>
<li>node.JS >= 7.8.0</li>
</ul></div>
<div class="section">
-<h2><a name="Development_Setup"></a>Development Setup</h2>
+<h2><a name="Installation"></a>Installation</h2>
+<div class="section">
+<h3><a name="From_Source"></a>From Source</h3>
<ol style="list-style-type: decimal">
<li>
-<p>Install all the dependent node_modules using the following command</p>
+<p>Package the application with Maven:</p>
<div class="source">
<div class="source">
-<pre>cd metron/metron-interface/metron-alerts
-npm install
+<pre>cd metron-interface/metron-alerts
+mvn clean package
</pre></div></div></li>
<li>
-<p>UI can be run by using the following command</p>
+<p>Untar the archive in the $METRON_HOME directory. The directory structure will look like:</p>
<div class="source">
<div class="source">
-<pre>./scripts/start-dev.sh
+<pre>bin
+ metron-alerts-ui
+web
+ expressjs
+ alerts-server.js
+ alerts-ui
+ web assets (html, css, js, ...)
</pre></div></div></li>
-<li>You can view the GUI @http://localhost:4200 . The default credentials for login are admin/password</li>
-</ol>
-<p><b>NOTE</b>: <i>In the development mode ui by default connects to ES at <a class="externalLink" href="http://node1:9200">http://node1:9200</a> for fetching data. If you wish to change it you can change the ES url at metron/metron-interface/metron-alerts/proxy.conf.json</i></p></div>
+<li>
+<p>Copy the <tt>$METRON_HOME/bin/metron-alerts-ui</tt> script to <tt>/etc/init.d/metron-alerts-ui</tt></p></li>
+
+<li>
+<p><a class="externalLink" href="https://expressjs.com/">Express</a> is installed at <tt>$METRON_HOME/web/expressjs/</tt> as part of the Management UI installation process. The Management UI should be installed first on the same host as the Alerts UI.</p></li>
+</ol></div>
<div class="section">
-<h2><a name="E2E_Tests"></a>E2E Tests</h2>
-<p>An expressjs server is available for mocking the elastic search api.</p>
+<h3><a name="From_Package_Manager"></a>From Package Manager</h3>
<ol style="list-style-type: decimal">
<li>
-<p>Run e2e webserver :</p>
+<p>Deploy the RPM at <tt>/metron/metron-deployment/packaging/docker/rpm-docker/target/RPMS/noarch/metron-alerts-$METRON_VERSION-*.noarch.rpm</tt></p></li>
+
+<li>
+<p>Install the RPM with:</p>
<div class="source">
<div class="source">
-<pre>cd metron/metron-interface/metron-alerts
-sh ./scripts/start-server-for-e2e.sh
+<pre>rpm -ih metron-alerts-$METRON_VERSION-*.noarch.rpm
</pre></div></div></li>
+</ol></div>
+<div class="section">
+<h3><a name="From_Ambari_MPack"></a>From Ambari MPack</h3>
+<p>The Alerts UI is included in the Metron Ambari MPack. It can be accessed through the Quick Links in the Metron service.</p></div></div>
+<div class="section">
+<h2><a name="Configuration"></a>Configuration</h2>
+<p>The Alerts UI is configured in the <tt>$METRON_HOME/config/alerts_ui.yml</tt> file. Create this file and set the values to match your environment:</p>
+
+<div class="source">
+<div class="source">
+<pre>port: port the alerts UI will run on
+
+rest:
+ host: REST application host
+ port: REST applciation port
+</pre></div></div></div>
+<div class="section">
+<h2><a name="Usage"></a>Usage</h2>
+<p>After configuration is complete, the Management UI can be managed as a service:</p>
+
+<div class="source">
+<div class="source">
+<pre>service metron-alerts-ui start
+</pre></div></div>
+<p>The application will be available at <a class="externalLink" href="http://host:4201">http://host:4201</a> assuming the port is set to <tt>4201</tt>. Logs can be found at <tt>/var/log/metron/metron-alerts-ui.log</tt>.</p></div>
+<div class="section">
+<h2><a name="Development_Setup"></a>Development Setup</h2>
+
+<ol style="list-style-type: decimal">
<li>
-<p>run e2e test using the following command</p>
+<p>Install all the dependent node_modules using the following command</p>
<div class="source">
<div class="source">
<pre>cd metron/metron-interface/metron-alerts
-npm run e2e
+npm install
</pre></div></div></li>
<li>
-<p>E2E tests uses data from full-dev wherever applicable. The tests assume rest-api’s are available @http://node1:8082</p></li>
+<p>UI can be run by using the following command</p>
+
+<div class="source">
+<div class="source">
+<pre>./scripts/start-dev.sh
+</pre></div></div></li>
+
+<li>You can view the GUI @http://localhost:4201. The default credentials for login are admin/password</li>
</ol>
-<p><b>NOTE</b>: <i>e2e tests covers all the general workflows and we will extend them as we need</i></p></div>
+<p><b>NOTE</b>: <i>In the development mode ui by default connects to REST at <a class="externalLink" href="http://node1:8082">http://node1:8082</a> for fetching data. If you wish to change it you can change the REST url at metron/metron-interface/metron-alerts/proxy.conf.json</i></p></div>
<div class="section">
-<h2><a name="Mpack_Integration"></a>Mpack Integration</h2>
-<p>Yet to come</p></div>
-<div class="section">
-<h2><a name="Installing_on_an_existing_Cluster"></a>Installing on an existing Cluster</h2>
+<h2><a name="E2E_Tests"></a>E2E Tests</h2>
+<p>An expressjs server is available for mocking the elastic search api.</p>
<ol style="list-style-type: decimal">
<li>
-<p>Build Metron:</p>
+<p>Run e2e webserver :</p>
<div class="source">
<div class="source">
-<pre>mvn clean package -DskipTests
+<pre>cd metron/metron-interface/metron-alerts
+sh ./scripts/start-server-for-e2e.sh
</pre></div></div></li>
<li>
-<p>Copy <tt>metron/metron-interface/metron-alerts/target/metron-alerts-METRON_VERSION-archive.tar.gz</tt> to the desired host.</p></li>
-
-<li>
-<p>Untar the archive in the target directory. The directory structure will look like:</p>
+<p>run e2e test using the following command</p>
<div class="source">
<div class="source">
-<pre>bin
- start_alerts_ui.sh
-web
- alerts-ui
- package.json
- server.js
- web assets (html, css, js, ...)
+<pre>cd metron/metron-interface/metron-alerts
+npm run e2e
</pre></div></div></li>
<li>
-<p><a class="externalLink" href="https://github.com/expressjs/express">Expressjs</a> webserver script is included in the build that will serve the application. (The script has few rewrite rules and we can replace expressjs with any other webserver)</p></li>
-
-<li>
-<p>Then start the application with the script:</p>
-
-<div class="source">
-<div class="source">
-<pre>./bin/start_alerts_ui.sh
-Usage: server.js -p [port] -r [restUrl]
-Options:
- -p Port to run metron alerts ui [required]
- -r, --resturl Url where elastic search rest api is available [required]
-</pre></div></div></li>
-</ol></div>
+<p>E2E tests uses data from full-dev wherever applicable. The tests assume rest-api’s are available @http://node1:8082</p></li>
+</ol>
+<p><b>NOTE</b>: <i>e2e tests covers all the general workflows and we will extend them as we need</i></p></div>
</div>
</div>
</div>
@@ -324,7 +368,7 @@ Options:
<footer>
<div class="container-fluid">
- <div class="row span12">Copyright © 2017
+ <div class="row span12">Copyright © 2018
<a href="https://www.apache.org">The Apache Software Foundation</a>.
All Rights Reserved.
[7/7] metron git commit: METRON-1388 update public web site to point
at 0.4.2 new release (mattf-horton) closes apache/metron#887
Posted by ma...@apache.org.
METRON-1388 update public web site to point at 0.4.2 new release (mattf-horton) closes apache/metron#887
Project: http://git-wip-us.apache.org/repos/asf/metron/repo
Commit: http://git-wip-us.apache.org/repos/asf/metron/commit/3381b853
Tree: http://git-wip-us.apache.org/repos/asf/metron/tree/3381b853
Diff: http://git-wip-us.apache.org/repos/asf/metron/diff/3381b853
Branch: refs/heads/master
Commit: 3381b853dca1c08a7a083593045dec2c7d4d92db
Parents: 01c26a7
Author: mattf-horton <mf...@hortonworks.com>
Authored: Thu Jan 4 12:30:30 2018 -0800
Committer: mattf <ma...@apache.org>
Committed: Thu Jan 4 12:30:30 2018 -0800
----------------------------------------------------------------------
site/current-book/Upgrading.html | 68 +-
site/current-book/index.html | 39 +-
site/current-book/metron-analytics/index.html | 25 +-
.../metron-maas-service/index.html | 25 +-
.../metron-profiler-client/index.html | 29 +-
.../metron-analytics/metron-profiler/index.html | 331 +++++++--
.../metron-statistics/HLLP.html | 25 +-
.../metron-statistics/index.html | 27 +-
.../metron-contrib/metron-docker/index.html | 25 +-
.../Kerberos-ambari-setup.html | 27 +-
.../Kerberos-manual-setup.html | 110 ++-
.../metron-deployment/amazon-ec2/index.html | 27 +-
site/current-book/metron-deployment/index.html | 38 +-
.../metron-deployment/other-examples/index.html | 27 +-
.../manual-install/Manual_Install_CentOS6.html | 139 ++--
.../packaging/ambari/index.html | 137 +++-
.../packaging/docker/ansible-docker/index.html | 60 +-
.../packaging/docker/rpm-docker/index.html | 27 +-
.../packaging/packer-build/index.html | 57 +-
.../metron-deployment/roles/index.html | 27 +-
.../metron-deployment/roles/monit/index.html | 27 +-
.../roles/opentaxii/index.html | 27 +-
.../roles/pcap_replay/index.html | 27 +-
.../roles/sensor-stubs/index.html | 33 +-
.../roles/sensor-test-mode/index.html | 27 +-
.../vagrant/codelab-platform/index.html | 23 +-
.../vagrant/fastcapa-test-platform/index.html | 92 ++-
.../vagrant/full-dev-platform/index.html | 44 +-
.../metron-deployment/vagrant/index.html | 45 +-
.../vagrant/quick-dev-platform/index.html | 23 +-
.../metron-interface/metron-alerts/index.html | 178 +++--
.../metron-interface/metron-config/index.html | 27 +-
.../metron-interface/metron-rest/index.html | 437 ++++++++---
.../Performance-tuning-guide.html | 97 ++-
site/current-book/metron-platform/index.html | 34 +-
.../metron-platform/metron-api/index.html | 32 +-
.../metron-platform/metron-common/index.html | 383 +++++++++-
.../metron-data-management/index.html | 32 +-
.../metron-elasticsearch/index.html | 380 ++++++++++
.../metron-enrichment/index.html | 41 +-
.../metron-platform/metron-indexing/index.html | 70 +-
.../metron-management/index.html | 271 ++++++-
.../metron-platform/metron-parsers/index.html | 128 ++--
.../metron-parsers/parser-testing.html | 32 +-
.../metron-pcap-backend/index.html | 32 +-
.../metron-platform/metron-writer/index.html | 32 +-
.../metron-sensors/bro-plugin-kafka/index.html | 25 +-
.../metron-sensors/fastcapa/index.html | 32 +-
site/current-book/metron-sensors/index.html | 34 +-
.../metron-sensors/pycapa/index.html | 43 +-
.../stellar-3rd-party-example/index.html | 211 ++++++
.../stellar-common/3rdPartyStellar.html | 29 +-
.../metron-stellar/stellar-common/index.html | 486 ++++++++++---
.../use-cases/forensic_clustering/index.html | 726 +++++++++++++++++++
.../geographic_login_outliers/index.html | 37 +-
site/current-book/use-cases/index.html | 32 +-
site/documentation/index.md | 14 +-
57 files changed, 4472 insertions(+), 1041 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/metron/blob/3381b853/site/current-book/Upgrading.html
----------------------------------------------------------------------
diff --git a/site/current-book/Upgrading.html b/site/current-book/Upgrading.html
index 00a5595..78fe2ac 100644
--- a/site/current-book/Upgrading.html
+++ b/site/current-book/Upgrading.html
@@ -1,13 +1,13 @@
<!DOCTYPE html>
<!--
- | Generated by Apache Maven Doxia at 2017-09-15
+ | Generated by Apache Maven Doxia at 2018-01-03
| Rendered using Apache Maven Fluido Skin 1.3.0
-->
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
- <meta name="Date-Revision-yyyymmdd" content="20170915" />
+ <meta name="Date-Revision-yyyymmdd" content="20180103" />
<meta http-equiv="Content-Language" content="en" />
<title>Metron – Upgrading</title>
<link rel="stylesheet" href="./css/apache-maven-fluido-1.3.0.min.css" />
@@ -61,8 +61,8 @@
- <li id="publishDate" class="pull-right">Last Published: 2017-09-15</li> <li class="divider pull-right">|</li>
- <li id="projectVersion" class="pull-right">Version: 0.4.1</li>
+ <li id="publishDate" class="pull-right">Last Published: 2018-01-03</li> <li class="divider pull-right">|</li>
+ <li id="projectVersion" class="pull-right">Version: 0.4.2</li>
</ul>
</div>
@@ -101,7 +101,7 @@
<i class="none"></i>
Docker</a>
</li>
-
+
<li>
<a href="metron-deployment/index.html" title="Deployment">
@@ -129,20 +129,27 @@
<i class="none"></i>
Rest</a>
</li>
-
+
<li>
<a href="metron-platform/index.html" title="Platform">
<i class="icon-chevron-right"></i>
Platform</a>
</li>
-
+
<li>
<a href="metron-sensors/index.html" title="Sensors">
<i class="icon-chevron-right"></i>
Sensors</a>
</li>
+
+ <li>
+
+ <a href="metron-stellar/stellar-3rd-party-example/index.html" title="Stellar-3rd-party-example">
+ <i class="none"></i>
+ Stellar-3rd-party-example</a>
+ </li>
<li>
@@ -150,7 +157,7 @@
<i class="icon-chevron-right"></i>
Stellar-common</a>
</li>
-
+
<li>
<a href="use-cases/index.html" title="Use-cases">
@@ -183,6 +190,49 @@
<p><a name="Upgrading"></a></p>
<p>This document constitutes a per-version listing of changes of configuration which are non-backwards compatible.</p>
<div class="section">
+<h2><a name="a0.4.1_to_0.4.2"></a>0.4.1 to 0.4.2</h2>
+<div class="section">
+<h3><a name="METRON-1277:_STELLAR_Add_Match_functionality_to_language"></a><a class="externalLink" href="https://issues.apache.org/jira/browse/METRON-1277">METRON-1277: STELLAR Add Match functionality to language</a></h3>
+<p>As we continue to evolve the Stellar language, it is possible that new keywords will be added to the language. This may cause compatablity issues where these reserved words and symbols are used in existing scripts.</p>
+<p>Adding <tt>match</tt> to the Stellar lanaguage has introduced the following new reserved keywords and symbols:</p>
+<p><tt>match</tt>, <tt>default</tt>, <tt>{</tt>, <tt>}</tt>, ‘=>’</p>
+<p>Any stellar expressions which use these keywords not in quotes will need to be modified.</p></div>
+<div class="section">
+<h3><a name="METRON-1158:_Build_backend_for_grouping_alerts_into_meta_alerts"></a><a class="externalLink" href="https://issues.apache.org/jira/browse/METRON-1158">METRON-1158: Build backend for grouping alerts into meta alerts</a></h3>
+<p>In order to allow for meta alerts to be queries alongside regular alerts in Elasticsearch 2.x, it is necessary to add an additional field to the templates and mapping for existing sensors.</p>
+<p>Two steps must be done for each sensor, but not on each index for each sensor.</p>
+<p>First is to update the Elasticsearch template for each sensor, so any new indices have the field:</p>
+
+<div class="source">
+<div class="source">
+<pre>export ELASTICSEARCH="node1"
+export SENSOR="bro"
+curl -XGET "http://${ELASTICSEARCH}:9200/_template/${SENSOR}_index*?pretty=true" -o "${SENSOR}.template"
+sed -i '' '2d;$d' ./${SENSOR}.template
+sed -i '' '/"properties" : {/ a\
+"alert": { "type": "nested"},' ${SENSOR}.template
+curl -XPUT "http://${ELASTICSEARCH}:9200/_template/${SENSOR}_index" -d @${SENSOR}.template
+</pre></div></div>
+<p>To update existing indexes, update Elasticsearch mappings with the new field for each sensor. Make sure to set the ELASTICSEARCH variable appropriately.</p>
+
+<div class="source">
+<div class="source">
+<pre>curl -XPUT "http://${ELASTICSEARCH}:9200/${SENSOR}_index*/_mapping/${SENSOR}_doc" -d '
+{
+ "properties" : {
+ "alert" : {
+ "type" : "nested"
+ }
+ }
+}
+'
+rm ${SENSOR}.template
+</pre></div></div>
+<p>For a more detailed description, please see metron-platform/metron-elasticsearch/README.md</p></div>
+<div class="section">
+<h3><a name="Description"></a>Description</h3>
+<p>In the 0.4.2 release, </p></div></div>
+<div class="section">
<h2><a name="a0.3.1_to_0.4.0"></a>0.3.1 to 0.4.0</h2>
<div class="section">
<h3><a name="METRON-671:_Refactor_existing_Ansible_deployment_to_use_Ambari_MPack"></a><a class="externalLink" href="https://issues.apache.org/jira/browse/METRON-671">METRON-671: Refactor existing Ansible deployment to use Ambari MPack</a></h3>
@@ -319,7 +369,7 @@ name conflict, enrichment already defined
<footer>
<div class="container-fluid">
- <div class="row span12">Copyright © 2017
+ <div class="row span12">Copyright © 2018
<a href="https://www.apache.org">The Apache Software Foundation</a>.
All Rights Reserved.
http://git-wip-us.apache.org/repos/asf/metron/blob/3381b853/site/current-book/index.html
----------------------------------------------------------------------
diff --git a/site/current-book/index.html b/site/current-book/index.html
index a213776..729fff7 100644
--- a/site/current-book/index.html
+++ b/site/current-book/index.html
@@ -1,15 +1,15 @@
<!DOCTYPE html>
<!--
- | Generated by Apache Maven Doxia at 2017-09-15
+ | Generated by Apache Maven Doxia at 2018-01-03
| Rendered using Apache Maven Fluido Skin 1.3.0
-->
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
- <meta name="Date-Revision-yyyymmdd" content="20170915" />
+ <meta name="Date-Revision-yyyymmdd" content="20180103" />
<meta http-equiv="Content-Language" content="en" />
- <title>Metron – </title>
+ <title>Metron – Apache Metron</title>
<link rel="stylesheet" href="./css/apache-maven-fluido-1.3.0.min.css" />
<link rel="stylesheet" href="./css/site.css" />
<link rel="stylesheet" href="./css/print.css" media="print" />
@@ -57,12 +57,12 @@
Documentation</a>
</li>
<li class="divider ">/</li>
- <li class=""></li>
+ <li class="">Apache Metron</li>
- <li id="publishDate" class="pull-right">Last Published: 2017-09-15</li> <li class="divider pull-right">|</li>
- <li id="projectVersion" class="pull-right">Version: 0.4.1</li>
+ <li id="publishDate" class="pull-right">Last Published: 2018-01-03</li> <li class="divider pull-right">|</li>
+ <li id="projectVersion" class="pull-right">Version: 0.4.2</li>
</ul>
</div>
@@ -101,7 +101,7 @@
<i class="none"></i>
Docker</a>
</li>
-
+
<li>
<a href="metron-deployment/index.html" title="Deployment">
@@ -129,20 +129,27 @@
<i class="none"></i>
Rest</a>
</li>
-
+
<li>
<a href="metron-platform/index.html" title="Platform">
<i class="icon-chevron-right"></i>
Platform</a>
</li>
-
+
<li>
<a href="metron-sensors/index.html" title="Sensors">
<i class="icon-chevron-right"></i>
Sensors</a>
</li>
+
+ <li>
+
+ <a href="metron-stellar/stellar-3rd-party-example/index.html" title="Stellar-3rd-party-example">
+ <i class="none"></i>
+ Stellar-3rd-party-example</a>
+ </li>
<li>
@@ -150,7 +157,7 @@
<i class="icon-chevron-right"></i>
Stellar-common</a>
</li>
-
+
<li>
<a href="use-cases/index.html" title="Use-cases">
@@ -179,8 +186,7 @@
<div id="bodyColumn" class="span9" >
- <p><a class="externalLink" href="https://travis-ci.org/apache/metron"><img src="https://travis-ci.org/apache/metron.svg?branch=master" alt="Build Status" /></a></p>
-<h1>Apache Metron</h1>
+ <h1>Apache Metron</h1>
<p><a name="Apache_Metron"></a></p>
<p>Metron integrates a variety of open source big data technologies in order to offer a centralized tool for security monitoring and analysis. Metron provides capabilities for log aggregation, full packet capture indexing, storage, advanced behavioral analytics and data enrichment, while applying the most current threat intelligence information to security telemetry within a single platform.</p>
<p>For the latest information, please visit our website at <a class="externalLink" href="http://metron.apache.org/">http://metron.apache.org/</a></p>
@@ -264,6 +270,9 @@ $ mvn site site:stage-deploy site:deploy
<pre>$ mvn clean install -DskipTests site site:stage-deploy site:deploy
</pre></div></div>
<p>The staged site is deployed to /tmp/metron/site/index.html, and can be viewed by opening the file in a browser.</p>
+<div class="section">
+<h2><a name="Building_with_Docker"></a>Building with Docker</h2>
+<p>A Docker container with all the required software, with the proper versions, is available to be used as well. see <a href="metron-deployment/packaging/docker/ansible-docker/index.html">ansible-docker</a></p>
<p><a name="Navigating_the_Architecture"></a></p>
<h1>Navigating the Architecture</h1>
<p>Metron is at its core a Kappa architecture with Apache Storm as the processing component and Apache Kafka as the unified data bus.</p>
@@ -289,6 +298,10 @@ $ mvn site site:stage-deploy site:deploy
<li><a href="metron-analytics/metron-profiler/index.html">Profiler</a> : A feature extraction mechanism that can generate a profile describing the behavior of an entity. An entity might be a server, user, subnet or application. Once a profile has been generated defining what normal behavior looks-like, models can be built that identify anomalous behavior.</li>
</ul>
+<p><a name="Notes_on_Adding_a_New_Sensor"></a></p>
+<h1>Notes on Adding a New Sensor</h1>
+<p>In order to allow for meta alerts to be queries alongside regular alerts in Elasticsearch 2.x, it is necessary to add an additional field to the templates and mapping for existing sensors.</p>
+<p>Please see a description of the steps necessary to make this change in the metron-elasticsearch <a href="./metron-platform/metron-elasticsearch/index.html#Using_Metron_with_Elasticsearch_2.x">Using Metron with Elasticsearch 2.x</a></p></div>
</div>
</div>
</div>
@@ -297,7 +310,7 @@ $ mvn site site:stage-deploy site:deploy
<footer>
<div class="container-fluid">
- <div class="row span12">Copyright © 2017
+ <div class="row span12">Copyright © 2018
<a href="https://www.apache.org">The Apache Software Foundation</a>.
All Rights Reserved.
http://git-wip-us.apache.org/repos/asf/metron/blob/3381b853/site/current-book/metron-analytics/index.html
----------------------------------------------------------------------
diff --git a/site/current-book/metron-analytics/index.html b/site/current-book/metron-analytics/index.html
index f1b83b4..fc4137a 100644
--- a/site/current-book/metron-analytics/index.html
+++ b/site/current-book/metron-analytics/index.html
@@ -1,13 +1,13 @@
<!DOCTYPE html>
<!--
- | Generated by Apache Maven Doxia at 2017-09-15
+ | Generated by Apache Maven Doxia at 2018-01-03
| Rendered using Apache Maven Fluido Skin 1.3.0
-->
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
- <meta name="Date-Revision-yyyymmdd" content="20170915" />
+ <meta name="Date-Revision-yyyymmdd" content="20180103" />
<meta http-equiv="Content-Language" content="en" />
<title>Metron – Metron Analytics</title>
<link rel="stylesheet" href="../css/apache-maven-fluido-1.3.0.min.css" />
@@ -61,8 +61,8 @@
- <li id="publishDate" class="pull-right">Last Published: 2017-09-15</li> <li class="divider pull-right">|</li>
- <li id="projectVersion" class="pull-right">Version: 0.4.1</li>
+ <li id="publishDate" class="pull-right">Last Published: 2018-01-03</li> <li class="divider pull-right">|</li>
+ <li id="projectVersion" class="pull-right">Version: 0.4.2</li>
</ul>
</div>
@@ -131,7 +131,7 @@
<i class="none"></i>
Docker</a>
</li>
-
+
<li>
<a href="../metron-deployment/index.html" title="Deployment">
@@ -159,20 +159,27 @@
<i class="none"></i>
Rest</a>
</li>
-
+
<li>
<a href="../metron-platform/index.html" title="Platform">
<i class="icon-chevron-right"></i>
Platform</a>
</li>
-
+
<li>
<a href="../metron-sensors/index.html" title="Sensors">
<i class="icon-chevron-right"></i>
Sensors</a>
</li>
+
+ <li>
+
+ <a href="../metron-stellar/stellar-3rd-party-example/index.html" title="Stellar-3rd-party-example">
+ <i class="none"></i>
+ Stellar-3rd-party-example</a>
+ </li>
<li>
@@ -180,7 +187,7 @@
<i class="icon-chevron-right"></i>
Stellar-common</a>
</li>
-
+
<li>
<a href="../use-cases/index.html" title="Use-cases">
@@ -229,7 +236,7 @@
<footer>
<div class="container-fluid">
- <div class="row span12">Copyright © 2017
+ <div class="row span12">Copyright © 2018
<a href="https://www.apache.org">The Apache Software Foundation</a>.
All Rights Reserved.
http://git-wip-us.apache.org/repos/asf/metron/blob/3381b853/site/current-book/metron-analytics/metron-maas-service/index.html
----------------------------------------------------------------------
diff --git a/site/current-book/metron-analytics/metron-maas-service/index.html b/site/current-book/metron-analytics/metron-maas-service/index.html
index 820b149..0298a29 100644
--- a/site/current-book/metron-analytics/metron-maas-service/index.html
+++ b/site/current-book/metron-analytics/metron-maas-service/index.html
@@ -1,13 +1,13 @@
<!DOCTYPE html>
<!--
- | Generated by Apache Maven Doxia at 2017-09-15
+ | Generated by Apache Maven Doxia at 2018-01-03
| Rendered using Apache Maven Fluido Skin 1.3.0
-->
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
- <meta name="Date-Revision-yyyymmdd" content="20170915" />
+ <meta name="Date-Revision-yyyymmdd" content="20180103" />
<meta http-equiv="Content-Language" content="en" />
<title>Metron – Model Management Infrastructure</title>
<link rel="stylesheet" href="../../css/apache-maven-fluido-1.3.0.min.css" />
@@ -61,8 +61,8 @@
- <li id="publishDate" class="pull-right">Last Published: 2017-09-15</li> <li class="divider pull-right">|</li>
- <li id="projectVersion" class="pull-right">Version: 0.4.1</li>
+ <li id="publishDate" class="pull-right">Last Published: 2018-01-03</li> <li class="divider pull-right">|</li>
+ <li id="projectVersion" class="pull-right">Version: 0.4.2</li>
</ul>
</div>
@@ -131,7 +131,7 @@
<i class="none"></i>
Docker</a>
</li>
-
+
<li>
<a href="../../metron-deployment/index.html" title="Deployment">
@@ -159,20 +159,27 @@
<i class="none"></i>
Rest</a>
</li>
-
+
<li>
<a href="../../metron-platform/index.html" title="Platform">
<i class="icon-chevron-right"></i>
Platform</a>
</li>
-
+
<li>
<a href="../../metron-sensors/index.html" title="Sensors">
<i class="icon-chevron-right"></i>
Sensors</a>
</li>
+
+ <li>
+
+ <a href="../../metron-stellar/stellar-3rd-party-example/index.html" title="Stellar-3rd-party-example">
+ <i class="none"></i>
+ Stellar-3rd-party-example</a>
+ </li>
<li>
@@ -180,7 +187,7 @@
<i class="icon-chevron-right"></i>
Stellar-common</a>
</li>
-
+
<li>
<a href="../../use-cases/index.html" title="Use-cases">
@@ -509,7 +516,7 @@ su - hdfs -c "hadoop fs -chown metron:metron /user/metron"
<footer>
<div class="container-fluid">
- <div class="row span12">Copyright © 2017
+ <div class="row span12">Copyright © 2018
<a href="https://www.apache.org">The Apache Software Foundation</a>.
All Rights Reserved.
http://git-wip-us.apache.org/repos/asf/metron/blob/3381b853/site/current-book/metron-analytics/metron-profiler-client/index.html
----------------------------------------------------------------------
diff --git a/site/current-book/metron-analytics/metron-profiler-client/index.html b/site/current-book/metron-analytics/metron-profiler-client/index.html
index 658dfe9..4fc31d6 100644
--- a/site/current-book/metron-analytics/metron-profiler-client/index.html
+++ b/site/current-book/metron-analytics/metron-profiler-client/index.html
@@ -1,13 +1,13 @@
<!DOCTYPE html>
<!--
- | Generated by Apache Maven Doxia at 2017-09-15
+ | Generated by Apache Maven Doxia at 2018-01-03
| Rendered using Apache Maven Fluido Skin 1.3.0
-->
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
- <meta name="Date-Revision-yyyymmdd" content="20170915" />
+ <meta name="Date-Revision-yyyymmdd" content="20180103" />
<meta http-equiv="Content-Language" content="en" />
<title>Metron – Metron Profiler Client</title>
<link rel="stylesheet" href="../../css/apache-maven-fluido-1.3.0.min.css" />
@@ -61,8 +61,8 @@
- <li id="publishDate" class="pull-right">Last Published: 2017-09-15</li> <li class="divider pull-right">|</li>
- <li id="projectVersion" class="pull-right">Version: 0.4.1</li>
+ <li id="publishDate" class="pull-right">Last Published: 2018-01-03</li> <li class="divider pull-right">|</li>
+ <li id="projectVersion" class="pull-right">Version: 0.4.2</li>
</ul>
</div>
@@ -131,7 +131,7 @@
<i class="none"></i>
Docker</a>
</li>
-
+
<li>
<a href="../../metron-deployment/index.html" title="Deployment">
@@ -159,20 +159,27 @@
<i class="none"></i>
Rest</a>
</li>
-
+
<li>
<a href="../../metron-platform/index.html" title="Platform">
<i class="icon-chevron-right"></i>
Platform</a>
</li>
-
+
<li>
<a href="../../metron-sensors/index.html" title="Sensors">
<i class="icon-chevron-right"></i>
Sensors</a>
</li>
+
+ <li>
+
+ <a href="../../metron-stellar/stellar-3rd-party-example/index.html" title="Stellar-3rd-party-example">
+ <i class="none"></i>
+ Stellar-3rd-party-example</a>
+ </li>
<li>
@@ -180,7 +187,7 @@
<i class="icon-chevron-right"></i>
Stellar-common</a>
</li>
-
+
<li>
<a href="../../use-cases/index.html" title="Use-cases">
@@ -716,7 +723,7 @@ Tuesday 11:15pm-11:45pm
<p>The most common result of incorrect <tt>PROFILE_GET</tt> arguments or Client configuration parameters is an empty result set, rather than an error. The Client cannot effectively validate the arguments, because the Profiler configuration parameters may be changed and the profile itself does not store them. The person doing the querying must carry forward the knowledge of the Profiler configuration parameters from the time of profile creation, and use corresponding <tt>PROFILE_GET</tt> arguments and Client configuration parameters when querying the data.</p></div>
<div class="section">
<h3><a name="Examples"></a>Examples</h3>
-<p>The following are usage examples that show how the Stellar API can be used to read profiles generated by the <a href="../metron-profiler/index.html">Metron Profiler</a>. This API would be used in conjunction with other Stellar functions like <a href="../../metron-platform/metron-common/index.html#MAAS_MODEL_APPLY"><tt>MAAS_MODEL_APPLY</tt></a> to perform model scoring on streaming data.</p>
+<p>The following are usage examples that show how the Stellar API can be used to read profiles generated by the <a href="../metron-profiler/index.html">Metron Profiler</a>. This API would be used in conjunction with other Stellar functions like <a href="../../metron-stellar/stellar-common/index.html#MAAS_MODEL_APPLY"><tt>MAAS_MODEL_APPLY</tt></a> to perform model scoring on streaming data.</p>
<p>These examples assume a profile has been defined called ‘snort-alerts’ that tracks the number of Snort alerts associated with an IP address over time. The profile definition might look similar to the following.</p>
<div class="source">
@@ -772,7 +779,7 @@ Tuesday 11:15pm-11:45pm
<div class="source">
<div class="source">
-<pre>[root@node1 0.4.1]# bin/stellar -z node1:2181
+<pre>[root@node1 0.4.2]# bin/stellar -z node1:2181
Stellar, Go!
Please note that functions are loading lazily in the background and will be unavailable until loaded fully.
{es.clustername=metron, es.ip=node1, es.port=9300, es.date.format=yyyy.MM.dd.HH}
@@ -899,7 +906,7 @@ Profiler{1 profile(s), 3 messages(s), 3 route(s)}
<footer>
<div class="container-fluid">
- <div class="row span12">Copyright © 2017
+ <div class="row span12">Copyright © 2018
<a href="https://www.apache.org">The Apache Software Foundation</a>.
All Rights Reserved.
http://git-wip-us.apache.org/repos/asf/metron/blob/3381b853/site/current-book/metron-analytics/metron-profiler/index.html
----------------------------------------------------------------------
diff --git a/site/current-book/metron-analytics/metron-profiler/index.html b/site/current-book/metron-analytics/metron-profiler/index.html
index c35bdb6..c3bf7ad 100644
--- a/site/current-book/metron-analytics/metron-profiler/index.html
+++ b/site/current-book/metron-analytics/metron-profiler/index.html
@@ -1,13 +1,13 @@
<!DOCTYPE html>
<!--
- | Generated by Apache Maven Doxia at 2017-09-15
+ | Generated by Apache Maven Doxia at 2018-01-03
| Rendered using Apache Maven Fluido Skin 1.3.0
-->
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
- <meta name="Date-Revision-yyyymmdd" content="20170915" />
+ <meta name="Date-Revision-yyyymmdd" content="20180103" />
<meta http-equiv="Content-Language" content="en" />
<title>Metron – Metron Profiler</title>
<link rel="stylesheet" href="../../css/apache-maven-fluido-1.3.0.min.css" />
@@ -61,8 +61,8 @@
- <li id="publishDate" class="pull-right">Last Published: 2017-09-15</li> <li class="divider pull-right">|</li>
- <li id="projectVersion" class="pull-right">Version: 0.4.1</li>
+ <li id="publishDate" class="pull-right">Last Published: 2018-01-03</li> <li class="divider pull-right">|</li>
+ <li id="projectVersion" class="pull-right">Version: 0.4.2</li>
</ul>
</div>
@@ -131,7 +131,7 @@
<i class="none"></i>
Docker</a>
</li>
-
+
<li>
<a href="../../metron-deployment/index.html" title="Deployment">
@@ -159,20 +159,27 @@
<i class="none"></i>
Rest</a>
</li>
-
+
<li>
<a href="../../metron-platform/index.html" title="Platform">
<i class="icon-chevron-right"></i>
Platform</a>
</li>
-
+
<li>
<a href="../../metron-sensors/index.html" title="Sensors">
<i class="icon-chevron-right"></i>
Sensors</a>
</li>
+
+ <li>
+
+ <a href="../../metron-stellar/stellar-3rd-party-example/index.html" title="Stellar-3rd-party-example">
+ <i class="none"></i>
+ Stellar-3rd-party-example</a>
+ </li>
<li>
@@ -180,7 +187,7 @@
<i class="icon-chevron-right"></i>
Stellar-common</a>
</li>
-
+
<li>
<a href="../../use-cases/index.html" title="Use-cases">
@@ -219,10 +226,12 @@
<li><a href="#Installation">Installation</a></li>
-<li><a href="#Getting_Started">Getting Started</a></li>
-
<li><a href="#Creating_Profiles">Creating Profiles</a></li>
+<li><a href="#Deploying_Profiles">Deploying Profiles</a></li>
+
+<li><a href="#Anatomy_of_a_Profile">Anatomy of a Profile</a></li>
+
<li><a href="#Configuring_the_Profiler">Configuring the Profiler</a></li>
<li><a href="#Examples">Examples</a></li>
@@ -231,7 +240,20 @@
</ul>
<div class="section">
<h2><a name="Installation"></a>Installation</h2>
-<p>Follow these instructions to install the Profiler. This assumes that core Metron has already been installed and validated. </p>
+<p>The Profiler can be installed with either of these two methods.</p>
+
+<ul>
+
+<li><a href="#Ambari_Installation">Ambari Installation</a></li>
+
+<li><a href="#Manual_Installation">Manual Installation</a></li>
+</ul>
+<div class="section">
+<h3><a name="Ambari_Installation"></a>Ambari Installation</h3>
+<p>The Metron Profiler is installed automatically when installing Metron using the Ambari MPack. You can skip the <a href="#Installation">Installation</a> section and move ahead to <a href="#Creating_Profiles">Creating Profiles</a> should this be the case.</p></div>
+<div class="section">
+<h3><a name="Manual_Installation"></a>Manual Installation</h3>
+<p>This section will describe the steps necessary to manually install the Profiler on an RPM-based Linux distribution. This assumes that core Metron has already been installed and validated. If you installed Metron using the <a href="#Ambari_MPack">Ambari MPack</a>, then the Profiler has already been installed and you can skip this section.</p>
<ol style="list-style-type: decimal">
@@ -252,7 +274,7 @@ metron-deployment//packaging/docker/rpm-docker/RPMS/noarch/metron-profiler-0.4.1
<div class="source">
<div class="source">
<pre>[root@node1 ~]# find /localrepo/ -name "metron-profiler*.rpm"
-/localrepo/metron-profiler-0.4.0-201707112313.noarch.rpm
+/localrepo/metron-profiler-0.4.1-201707112313.noarch.rpm
</pre></div></div></li>
<li>
@@ -269,19 +291,35 @@ Preparing... ########################################### [100%]
<div class="source">
<pre>[root@node1 ~]# rpm -ql metron-profiler
/usr/metron
-/usr/metron/0.4.1
-/usr/metron/0.4.1/bin
-/usr/metron/0.4.1/bin/start_profiler_topology.sh
-/usr/metron/0.4.1/config
-/usr/metron/0.4.1/config/profiler.properties
-/usr/metron/0.4.1/flux
-/usr/metron/0.4.1/flux/profiler
-/usr/metron/0.4.1/flux/profiler/remote.yaml
-/usr/metron/0.4.1/lib
-/usr/metron/0.4.1/lib/metron-profiler-0.4.0-uber.jar
+/usr/metron/0.4.2
+/usr/metron/0.4.2/bin
+/usr/metron/0.4.2/bin/start_profiler_topology.sh
+/usr/metron/0.4.2/config
+/usr/metron/0.4.2/config/profiler.properties
+/usr/metron/0.4.2/flux
+/usr/metron/0.4.2/flux/profiler
+/usr/metron/0.4.2/flux/profiler/remote.yaml
+/usr/metron/0.4.2/lib
+/usr/metron/0.4.2/lib/metron-profiler-0.4.2-uber.jar
</pre></div></div></li>
<li>
+<p>Edit the configuration file located at <tt>$METRON_HOME/config/profiler.properties</tt>. </p>
+
+<div class="source">
+<div class="source">
+<pre>kafka.zk=node1:2181
+kafka.broker=node1:6667
+</pre></div></div>
+
+<ul>
+
+<li></li>
+
+<li></li>
+ </ul></li>
+
+<li>
<p>Create a table within HBase that will store the profile data. By default, the table is named <tt>profiler</tt> with a column family <tt>P</tt>. The table name and column family must match the Profiler’s configuration (see <a href="#Configuring_the_Profiler">Configuring the Profiler</a>). </p>
<div class="source">
@@ -291,28 +329,203 @@ hbase(main):001:0> create 'profiler', 'P'
</pre></div></div></li>
<li>
-<p>Edit the configuration file located at <tt>$METRON_HOME/config/profiler.properties</tt>. </p>
+<p>Start the Profiler topology.</p>
<div class="source">
<div class="source">
-<pre>kafka.zk=node1:2181
-kafka.broker=node1:6667
+<pre>$ cd $METRON_HOME
+$ bin/start_profiler_topology.sh
+</pre></div></div></li>
+</ol>
+<p>At this point the Profiler is running and consuming telemetry messages. We have not defined any profiles yet, so it is not doing anything very useful. The next section walks you through the steps to create your very first “Hello, World!” profile.</p></div></div>
+<div class="section">
+<h2><a name="Creating_Profiles"></a>Creating Profiles</h2>
+<p>This section will describe how to create your very first “Hello, World” profile. It will also outline a useful workflow for creating, testing, and deploying profiles.</p>
+<p>Creating and refining profiles is an iterative process. Iterating against a live stream of data is slow, difficult and error prone. The Profile Debugger was created to provide a controlled and isolated execution environment to create, refine and troubleshoot profiles.</p>
+
+<ol style="list-style-type: decimal">
+
+<li>
+<p>Launch the Stellar Shell. We will leverage the Profiler Debugger from within the Stellar Shell.</p>
+
+<div class="source">
+<div class="source">
+<pre>[root@node1 ~]# $METRON_HOME/bin/stellar
+Stellar, Go!
+[Stellar]>>> %functions PROFILER
+PROFILER_APPLY, PROFILER_FLUSH, PROFILER_INIT
+</pre></div></div></li>
+
+<li>
+<p>Create a simple <tt>hello-world</tt> profile that will count the number of messages for each <tt>ip_src_addr</tt>. The <tt>SHELL_EDIT</tt> function will open an editor in which you can copy/paste the following Profiler configuration.</p>
+
+<div class="source">
+<div class="source">
+<pre>[Stellar]>>> conf := SHELL_EDIT()
+[Stellar]>>> conf
+{
+ "profiles": [
+ {
+ "profile": "hello-world",
+ "onlyif": "exists(ip_src_addr)",
+ "foreach": "ip_src_addr",
+ "init": { "count": "0" },
+ "update": { "count": "count + 1" },
+ "result": "count"
+ }
+ ]
+}
+</pre></div></div></li>
+
+<li>
+<p>Create a Profile execution environment; the Profile Debugger. </p>
+<p>The Profiler will output the number of profiles that have been defined, the number of messages that have been applied and the number of routes that have been followed. </p>
+<p>A route is defined when a message is applied to a specific profile.</p>
+
+<ul>
+
+<li>If a message is not needed by any profile, then there are no routes.</li>
+
+<li>If a message is needed by one profile, then one route has been followed.</li>
+
+<li>If a message is needed by two profiles, then two routes have been followed.</li>
+ </ul>
+
+<div class="source">
+<div class="source">
+<pre>[Stellar]>>> p := PROFILER_INIT(conf)
+[Stellar]>>> p
+Profiler{1 profile(s), 0 messages(s), 0 route(s)}
+</pre></div></div></li>
+
+<li>
+<p>Create a message that mimics the telemetry that your profile will consume. </p>
+<p>This message can be as simple or complex as you like. For the <tt>hello-world</tt> profile, all you need is a message containing an <tt>ip_src_addr</tt> field.</p>
+
+<div class="source">
+<div class="source">
+<pre>[Stellar]>>> msg := SHELL_EDIT()
+[Stellar]>>> msg
+{
+ "ip_src_addr": "10.0.0.1"
+}
+</pre></div></div></li>
+
+<li>
+<p>Apply the message to your Profiler, as many times as you like.</p>
+
+<div class="source">
+<div class="source">
+<pre>[Stellar]>>> PROFILER_APPLY(msg, p)
+Profiler{1 profile(s), 1 messages(s), 1 route(s)}
</pre></div></div>
-<p>Change <tt>kafka.zk</tt> to refer to Zookeeper in your environment.<br />Change <tt>kafka.broker</tt> to refer to a Kafka Broker in your environment.</p></li>
+
+<div class="source">
+<div class="source">
+<pre>[Stellar]>>> PROFILER_APPLY(msg, p)
+Profiler{1 profile(s), 2 messages(s), 2 route(s)}
+</pre></div></div></li>
<li>
-<p>Start the Profiler topology.</p>
+<p>Flush the Profiler. </p>
+<p>A flush is what occurs at the end of each 15 minute period in the Profiler. The result is a list of Profile Measurements. Each measurement is a map containing detailed information about the profile data that has been generated. The <tt>value</tt> field is what is written to HBase when running this profile in the Profiler topology. </p>
+<p>There will always be one measurement for each [profile, entity] pair. This profile simply counts the number of messages by IP source address. Notice that the value is ‘3’ for the entity ‘10.0.0.1’ as we applied 3 messages with an ‘ip_src_addr’ of ’10.0.0.1’.</p>
<div class="source">
<div class="source">
-<pre>$ cd $METRON_HOME
-$ bin/start_profiler_topology.sh
+<pre>[Stellar]>>> values := PROFILER_FLUSH(profiler)
+[Stellar]>>> values
+[{period={duration=900000, period=1669628, start=1502665200000, end=1502666100000},
+profile=hello-world, groups=[], value=3, entity=10.0.0.1}]
</pre></div></div></li>
-</ol>
-<p>At this point the Profiler is running and consuming telemetry messages. We have not defined any profiles yet, so it is not doing anything very useful. The next section walks you through the steps to create your very first “Hello, World!” profile.</p></div>
+
+<li>
+<p>Apply real, live telemetry to your profile.</p>
+<p>Once you are happy with your profile against a controlled data set, it can be useful to introduce more complex, live data. This example extracts 10 messages of live, enriched telemetry to test your profile(s).</p>
+
+<div class="source">
+<div class="source">
+<pre>[Stellar]>>> %define bootstrap.servers := "node1:6667"
+node1:6667
+[Stellar]>>> msgs := KAFKA_GET("indexing", 10)
+[Stellar]>>> LENGTH(msgs)
+10
+</pre></div></div>
+<p>Apply those 10 messages to your profile(s).</p>
+
+<div class="source">
+<div class="source">
+<pre>[Stellar]>>> PROFILER_APPLY(msgs, p)
+ Profiler{1 profile(s), 10 messages(s), 10 route(s)}
+</pre></div></div></li>
+</ol></div>
+<div class="section">
+<h2><a name="Deploying_Profiles"></a>Deploying Profiles</h2>
+<p>This section will describe the steps required to get your first “Hello, World!”" profile running. This assumes that you have a successful Profiler <a href="#Installation">Installation</a> and have it running. You can deploy profiles in two different ways.</p>
+
+<ul>
+
+<li><a href="#Deploying_Profiles_with_the_Stellar_Shell">Deploying Profiles with the Stellar Shell</a></li>
+
+<li><a href="#Deploying_Profiles_from_the_Command_Line">Deploying Profiles from the Command Line</a></li>
+</ul>
+<div class="section">
+<h3><a name="Deploying_Profiles_with_the_Stellar_Shell"></a>Deploying Profiles with the Stellar Shell</h3>
+<p>Continuing the previous running example, at this point, you have seen how your profile behaves against real, live telemetry in a controlled execution environment. The next step is to deploy your profile to the live, actively running Profiler topology.</p>
+
+<ol style="list-style-type: decimal">
+
+<li>
+<p>Start the Stellar Shell with the <tt>-z ZK:2181</tt> command line argument. This is required when deploying a new profile to the active Profiler topology. Replace <tt>ZK:2181</tt> with a URL that is appropriate to your environment.</p>
+
+<div class="source">
+<div class="source">
+<pre>[root@node1 ~]# $METRON_HOME/bin/stellar -z ZK:2181
+Stellar, Go!
+[Stellar]>>>
+[Stellar]>>> %functions CONFIG CONFIG_GET, CONFIG_PUT
+</pre></div></div></li>
+
+<li>
+<p>If you haven’t already, define your profile.</p>
+
+<div class="source">
+<div class="source">
+<pre>[Stellar]>>> conf := SHELL_EDIT()
+[Stellar]>>> conf
+{
+ "profiles": [
+ {
+ "profile": "hello-world",
+ "onlyif": "exists(ip_src_addr)",
+ "foreach": "ip_src_addr",
+ "init": { "count": "0" },
+ "update": { "count": "count + 1" },
+ "result": "count"
+ }
+ ]
+}
+</pre></div></div></li>
+
+<li>
+<p>Check what is already deployed. </p>
+<p>Pushing a new profile configuration is destructive. It will overwrite any existing configuration. Check what you have out there. Manually merge the existing configuration with your new profile definition.</p>
+
+<div class="source">
+<div class="source">
+<pre>[Stellar]>>> existing := CONFIG_GET("PROFILER")
+</pre></div></div></li>
+
+<li>
+<p>Deploy your profile. This will push the configuration to to the live, actively running Profiler topology. This will overwrite any existing profile definitions.</p>
+
+<div class="source">
+<div class="source">
+<pre>[Stellar]>>> CONFIG_PUT("PROFILER", conf)
+</pre></div></div></li>
+</ol></div>
<div class="section">
-<h2><a name="Getting_Started"></a>Getting Started</h2>
-<p>This section will describe the steps required to get your first “Hello, World!”" profile running. This assumes that you have a successful Profiler <a href="#Installation">Installation</a> and have it running.</p>
+<h3><a name="Deploying_Profiles_from_the_Command_Line"></a>Deploying Profiles from the Command Line</h3>
<ol style="list-style-type: decimal">
@@ -388,10 +601,10 @@ hbase(main):001:0> count 'profiler'
</pre></div></div>
<p>This result indicates that over the past 30 minutes, the Profiler stored two values related to the source IP address “10.0.0.1”. In the first 15 minute period, the IP <tt>10.0.0.1</tt> was seen in 451 telemetry messages. In the second 15 minute period, the same IP was seen in 448 telemetry messages.</p>
<p>It is assumed that the <tt>PROFILE_GET</tt> client is correctly configured to match the Profile configuration before using it to read that Profile. More information on configuring and using the Profiler client can be found <a href="../metron-profiler-client/index.html">here</a>. </p></li>
-</ol></div>
+</ol></div></div>
<div class="section">
-<h2><a name="Creating_Profiles"></a>Creating Profiles</h2>
-<p>The Profiler specification requires a JSON-formatted set of elements, many of which can contain Stellar code. The specification contains the following elements. (For the impatient, skip ahead to the <a href="#Examples">Examples</a>.) The specification for the Profiler topology is stored in Zookeeper at <tt>/metron/topology/profiler</tt>. These properties also exist in the local filesystem at <tt>$METRON_HOME/config/zookeeper/profiler.json</tt>. The values can be changed on disk and then uploaded to Zookeeper using <tt>$METRON_HOME/bin/zk_load_configs.sh</tt>.</p>
+<h2><a name="Anatomy_of_a_Profile"></a>Anatomy of a Profile</h2>
+<p>A profile definition requires a JSON-formatted set of elements, many of which can contain Stellar code. The specification contains the following elements. (For the impatient, skip ahead to the <a href="#Examples">Examples</a>.)</p>
<table border="0" class="table table-striped">
<thead>
@@ -761,29 +974,33 @@ hbase(main):001:0> count 'profiler'
<p>The maximum number of seconds between batch writes to HBase.</p></div></div>
<div class="section">
<h2><a name="Examples"></a>Examples</h2>
-<p>The following examples are intended to highlight the functionality provided by the Profiler. Each shows the configuration that would be required to generate the profile. </p>
-<p>These examples assume a fictitious input message stream that looks something like the following.</p>
+<p>The following examples are intended to highlight the functionality provided by the Profiler. Try out these examples easily in the Stellar Shell as described in the <a href="#Creating_Profiles">Creating Profiles</a> section.</p>
+<p>These examples assume a fictitious input message stream that looks like the following. </p>
<div class="source">
<div class="source">
-<pre>{
- "ip_src_addr": "10.0.0.1",
- "protocol": "HTTPS",
- "length": "10",
- "bytes_in": "234"
-},
-{
- "ip_src_addr": "10.0.0.2",
- "protocol": "HTTP",
- "length": "20",
- "bytes_in": "390"
-},
-{
- "ip_src_addr": "10.0.0.3",
- "protocol": "DNS",
- "length": "30",
- "bytes_in": "560"
-}
+<pre>[Stellar]>>> msgs := SHELL_EDIT()
+[Stellar]>>> msgs
+[
+ {
+ "ip_src_addr": "10.0.0.1",
+ "protocol": "HTTPS",
+ "length": "10",
+ "bytes_in": "234"
+ },
+ {
+ "ip_src_addr": "10.0.0.2",
+ "protocol": "HTTP",
+ "length": "20",
+ "bytes_in": "390"
+ },
+ {
+ "ip_src_addr": "10.0.0.3",
+ "protocol": "DNS",
+ "length": "30",
+ "bytes_in": "560"
+ }
+]
</pre></div></div>
<div class="section">
<h3><a name="Example_1"></a>Example 1</h3>
@@ -998,7 +1215,7 @@ hbase(main):001:0> count 'profiler'
<footer>
<div class="container-fluid">
- <div class="row span12">Copyright © 2017
+ <div class="row span12">Copyright © 2018
<a href="https://www.apache.org">The Apache Software Foundation</a>.
All Rights Reserved.
http://git-wip-us.apache.org/repos/asf/metron/blob/3381b853/site/current-book/metron-analytics/metron-statistics/HLLP.html
----------------------------------------------------------------------
diff --git a/site/current-book/metron-analytics/metron-statistics/HLLP.html b/site/current-book/metron-analytics/metron-statistics/HLLP.html
index d1175e9..f56beba 100644
--- a/site/current-book/metron-analytics/metron-statistics/HLLP.html
+++ b/site/current-book/metron-analytics/metron-statistics/HLLP.html
@@ -1,13 +1,13 @@
<!DOCTYPE html>
<!--
- | Generated by Apache Maven Doxia at 2017-09-15
+ | Generated by Apache Maven Doxia at 2018-01-03
| Rendered using Apache Maven Fluido Skin 1.3.0
-->
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
- <meta name="Date-Revision-yyyymmdd" content="20170915" />
+ <meta name="Date-Revision-yyyymmdd" content="20180103" />
<meta http-equiv="Content-Language" content="en" />
<title>Metron – HyperLogLogPlus</title>
<link rel="stylesheet" href="../../css/apache-maven-fluido-1.3.0.min.css" />
@@ -61,8 +61,8 @@
- <li id="publishDate" class="pull-right">Last Published: 2017-09-15</li> <li class="divider pull-right">|</li>
- <li id="projectVersion" class="pull-right">Version: 0.4.1</li>
+ <li id="publishDate" class="pull-right">Last Published: 2018-01-03</li> <li class="divider pull-right">|</li>
+ <li id="projectVersion" class="pull-right">Version: 0.4.2</li>
</ul>
</div>
@@ -140,7 +140,7 @@
<i class="none"></i>
Docker</a>
</li>
-
+
<li>
<a href="../../metron-deployment/index.html" title="Deployment">
@@ -168,20 +168,27 @@
<i class="none"></i>
Rest</a>
</li>
-
+
<li>
<a href="../../metron-platform/index.html" title="Platform">
<i class="icon-chevron-right"></i>
Platform</a>
</li>
-
+
<li>
<a href="../../metron-sensors/index.html" title="Sensors">
<i class="icon-chevron-right"></i>
Sensors</a>
</li>
+
+ <li>
+
+ <a href="../../metron-stellar/stellar-3rd-party-example/index.html" title="Stellar-3rd-party-example">
+ <i class="none"></i>
+ Stellar-3rd-party-example</a>
+ </li>
<li>
@@ -189,7 +196,7 @@
<i class="icon-chevron-right"></i>
Stellar-common</a>
</li>
-
+
<li>
<a href="../../use-cases/index.html" title="Use-cases">
@@ -7606,7 +7613,7 @@ format err as %: true
<footer>
<div class="container-fluid">
- <div class="row span12">Copyright © 2017
+ <div class="row span12">Copyright © 2018
<a href="https://www.apache.org">The Apache Software Foundation</a>.
All Rights Reserved.
http://git-wip-us.apache.org/repos/asf/metron/blob/3381b853/site/current-book/metron-analytics/metron-statistics/index.html
----------------------------------------------------------------------
diff --git a/site/current-book/metron-analytics/metron-statistics/index.html b/site/current-book/metron-analytics/metron-statistics/index.html
index 4cdd493..29084e8 100644
--- a/site/current-book/metron-analytics/metron-statistics/index.html
+++ b/site/current-book/metron-analytics/metron-statistics/index.html
@@ -1,13 +1,13 @@
<!DOCTYPE html>
<!--
- | Generated by Apache Maven Doxia at 2017-09-15
+ | Generated by Apache Maven Doxia at 2018-01-03
| Rendered using Apache Maven Fluido Skin 1.3.0
-->
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
- <meta name="Date-Revision-yyyymmdd" content="20170915" />
+ <meta name="Date-Revision-yyyymmdd" content="20180103" />
<meta http-equiv="Content-Language" content="en" />
<title>Metron – Statistics and Mathematical Functions</title>
<link rel="stylesheet" href="../../css/apache-maven-fluido-1.3.0.min.css" />
@@ -61,8 +61,8 @@
- <li id="publishDate" class="pull-right">Last Published: 2017-09-15</li> <li class="divider pull-right">|</li>
- <li id="projectVersion" class="pull-right">Version: 0.4.1</li>
+ <li id="publishDate" class="pull-right">Last Published: 2018-01-03</li> <li class="divider pull-right">|</li>
+ <li id="projectVersion" class="pull-right">Version: 0.4.2</li>
</ul>
</div>
@@ -140,7 +140,7 @@
<i class="none"></i>
Docker</a>
</li>
-
+
<li>
<a href="../../metron-deployment/index.html" title="Deployment">
@@ -168,20 +168,27 @@
<i class="none"></i>
Rest</a>
</li>
-
+
<li>
<a href="../../metron-platform/index.html" title="Platform">
<i class="icon-chevron-right"></i>
Platform</a>
</li>
-
+
<li>
<a href="../../metron-sensors/index.html" title="Sensors">
<i class="icon-chevron-right"></i>
Sensors</a>
</li>
+
+ <li>
+
+ <a href="../../metron-stellar/stellar-3rd-party-example/index.html" title="Stellar-3rd-party-example">
+ <i class="none"></i>
+ Stellar-3rd-party-example</a>
+ </li>
<li>
@@ -189,7 +196,7 @@
<i class="icon-chevron-right"></i>
Stellar-common</a>
</li>
-
+
<li>
<a href="../../use-cases/index.html" title="Use-cases">
@@ -344,7 +351,7 @@
<li>stats - The Stellar statistics object. If null, then a new one is initialized.</li>
-<li>value+ - One or more numbers to add</li>
+<li>value+ - One or more items to add. Each item may be a number or a list of numbers. If an item is a list, each number in the list will be added.</li>
</ul></li>
<li>Returns: A Stellar statistics object</li>
@@ -950,7 +957,7 @@ PROFILE_GET( 'sketchy_mad', 'global', PROFILE_FIXED(10, 'MINUTES')) ), value)&qu
<footer>
<div class="container-fluid">
- <div class="row span12">Copyright © 2017
+ <div class="row span12">Copyright © 2018
<a href="https://www.apache.org">The Apache Software Foundation</a>.
All Rights Reserved.
http://git-wip-us.apache.org/repos/asf/metron/blob/3381b853/site/current-book/metron-contrib/metron-docker/index.html
----------------------------------------------------------------------
diff --git a/site/current-book/metron-contrib/metron-docker/index.html b/site/current-book/metron-contrib/metron-docker/index.html
index bdbfa9d..761c3b0 100644
--- a/site/current-book/metron-contrib/metron-docker/index.html
+++ b/site/current-book/metron-contrib/metron-docker/index.html
@@ -1,13 +1,13 @@
<!DOCTYPE html>
<!--
- | Generated by Apache Maven Doxia at 2017-09-15
+ | Generated by Apache Maven Doxia at 2018-01-03
| Rendered using Apache Maven Fluido Skin 1.3.0
-->
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
- <meta name="Date-Revision-yyyymmdd" content="20170915" />
+ <meta name="Date-Revision-yyyymmdd" content="20180103" />
<meta http-equiv="Content-Language" content="en" />
<title>Metron – Metron Docker</title>
<link rel="stylesheet" href="../../css/apache-maven-fluido-1.3.0.min.css" />
@@ -61,8 +61,8 @@
- <li id="publishDate" class="pull-right">Last Published: 2017-09-15</li> <li class="divider pull-right">|</li>
- <li id="projectVersion" class="pull-right">Version: 0.4.1</li>
+ <li id="publishDate" class="pull-right">Last Published: 2018-01-03</li> <li class="divider pull-right">|</li>
+ <li id="projectVersion" class="pull-right">Version: 0.4.2</li>
</ul>
</div>
@@ -101,7 +101,7 @@
<a href="#"><i class="none"></i>Docker</a>
</li>
-
+
<li>
<a href="../../metron-deployment/index.html" title="Deployment">
@@ -129,20 +129,27 @@
<i class="none"></i>
Rest</a>
</li>
-
+
<li>
<a href="../../metron-platform/index.html" title="Platform">
<i class="icon-chevron-right"></i>
Platform</a>
</li>
-
+
<li>
<a href="../../metron-sensors/index.html" title="Sensors">
<i class="icon-chevron-right"></i>
Sensors</a>
</li>
+
+ <li>
+
+ <a href="../../metron-stellar/stellar-3rd-party-example/index.html" title="Stellar-3rd-party-example">
+ <i class="none"></i>
+ Stellar-3rd-party-example</a>
+ </li>
<li>
@@ -150,7 +157,7 @@
<i class="icon-chevron-right"></i>
Stellar-common</a>
</li>
-
+
<li>
<a href="../../use-cases/index.html" title="Use-cases">
@@ -495,7 +502,7 @@ yellow open bro_index_2016.12.19.18 5 1 180 0 475kb
<footer>
<div class="container-fluid">
- <div class="row span12">Copyright © 2017
+ <div class="row span12">Copyright © 2018
<a href="https://www.apache.org">The Apache Software Foundation</a>.
All Rights Reserved.
http://git-wip-us.apache.org/repos/asf/metron/blob/3381b853/site/current-book/metron-deployment/Kerberos-ambari-setup.html
----------------------------------------------------------------------
diff --git a/site/current-book/metron-deployment/Kerberos-ambari-setup.html b/site/current-book/metron-deployment/Kerberos-ambari-setup.html
index 662dca9..1ed07c3 100644
--- a/site/current-book/metron-deployment/Kerberos-ambari-setup.html
+++ b/site/current-book/metron-deployment/Kerberos-ambari-setup.html
@@ -1,13 +1,13 @@
<!DOCTYPE html>
<!--
- | Generated by Apache Maven Doxia at 2017-09-15
+ | Generated by Apache Maven Doxia at 2018-01-03
| Rendered using Apache Maven Fluido Skin 1.3.0
-->
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
- <meta name="Date-Revision-yyyymmdd" content="20170915" />
+ <meta name="Date-Revision-yyyymmdd" content="20180103" />
<meta http-equiv="Content-Language" content="en" />
<title>Metron – Setting Up Kerberos in Vagrant Full Dev</title>
<link rel="stylesheet" href="../css/apache-maven-fluido-1.3.0.min.css" />
@@ -61,8 +61,8 @@
- <li id="publishDate" class="pull-right">Last Published: 2017-09-15</li> <li class="divider pull-right">|</li>
- <li id="projectVersion" class="pull-right">Version: 0.4.1</li>
+ <li id="publishDate" class="pull-right">Last Published: 2018-01-03</li> <li class="divider pull-right">|</li>
+ <li id="projectVersion" class="pull-right">Version: 0.4.2</li>
</ul>
</div>
@@ -103,7 +103,7 @@
<i class="none"></i>
Docker</a>
</li>
-
+
<li>
<a href="../metron-deployment/index.html" title="Deployment">
@@ -171,7 +171,7 @@
<i class="icon-chevron-right"></i>
Roles</a>
</li>
-
+
<li>
<a href="../metron-deployment/vagrant/index.html" title="Vagrant">
@@ -201,20 +201,27 @@
<i class="none"></i>
Rest</a>
</li>
-
+
<li>
<a href="../metron-platform/index.html" title="Platform">
<i class="icon-chevron-right"></i>
Platform</a>
</li>
-
+
<li>
<a href="../metron-sensors/index.html" title="Sensors">
<i class="icon-chevron-right"></i>
Sensors</a>
</li>
+
+ <li>
+
+ <a href="../metron-stellar/stellar-3rd-party-example/index.html" title="Stellar-3rd-party-example">
+ <i class="none"></i>
+ Stellar-3rd-party-example</a>
+ </li>
<li>
@@ -222,7 +229,7 @@
<i class="icon-chevron-right"></i>
Stellar-common</a>
</li>
-
+
<li>
<a href="../use-cases/index.html" title="Use-cases">
@@ -296,7 +303,7 @@
<footer>
<div class="container-fluid">
- <div class="row span12">Copyright © 2017
+ <div class="row span12">Copyright © 2018
<a href="https://www.apache.org">The Apache Software Foundation</a>.
All Rights Reserved.