You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@metron.apache.org by ma...@apache.org on 2017/07/05 20:01:03 UTC
[01/10] metron git commit: METRON-1012 Update Metron public web site
for 0.4.0 release - push to asf-site from the generated code on master
Repository: metron
Updated Branches:
refs/heads/asf-site 7152bce7b -> cf4c2ecde
http://git-wip-us.apache.org/repos/asf/metron/blob/cf4c2ecd/current-book/metron-sensors/pycapa/index.html
----------------------------------------------------------------------
diff --git a/current-book/metron-sensors/pycapa/index.html b/current-book/metron-sensors/pycapa/index.html
index 0841338..4bdd35c 100644
--- a/current-book/metron-sensors/pycapa/index.html
+++ b/current-book/metron-sensors/pycapa/index.html
@@ -1,13 +1,13 @@
<!DOCTYPE html>
<!--
- | Generated by Apache Maven Doxia at 2017-02-23
+ | Generated by Apache Maven Doxia at 2017-06-27
| Rendered using Apache Maven Fluido Skin 1.3.0
-->
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
- <meta name="Date-Revision-yyyymmdd" content="20170223" />
+ <meta name="Date-Revision-yyyymmdd" content="20170627" />
<meta http-equiv="Content-Language" content="en" />
<title>Metron – Pycapa</title>
<link rel="stylesheet" href="../../css/apache-maven-fluido-1.3.0.min.css" />
@@ -30,14 +30,11 @@
<div class="container-fluid">
<div id="banner">
<div class="pull-left">
- <a href="http://metron.incubator.apache.org/" id="bannerLeft">
- <img src="../../images/metron-logo.png" alt="Apache Metron - Incubating" width="148px" height="48px"/>
+ <a href="http://metron.apache.org/" id="bannerLeft">
+ <img src="../../images/metron-logo.png" alt="Apache Metron" width="148px" height="48px"/>
</a>
</div>
- <div class="pull-right"> <a href="http://incubator.apache.org/" id="bannerRight">
- <img src="../../images/ApacheIncubating_Logo.png" alt="Apache Incubating" width="192px" height="48px"/>
- </a>
- </div>
+ <div class="pull-right"> </div>
<div class="clear"><hr/></div>
</div>
@@ -51,8 +48,8 @@
</li>
<li class="divider ">/</li>
<li class="">
- <a href="http://metron.incubator.apache.org/" class="externalLink" title="Metron-Incubating">
- Metron-Incubating</a>
+ <a href="http://metron.apache.org/" class="externalLink" title="Metron">
+ Metron</a>
</li>
<li class="divider ">/</li>
<li class="">
@@ -64,8 +61,8 @@
- <li id="publishDate" class="pull-right">Last Published: 2017-02-23</li> <li class="divider pull-right">|</li>
- <li id="projectVersion" class="pull-right">Version: 0.3.1</li>
+ <li id="publishDate" class="pull-right">Last Published: 2017-06-27</li> <li class="divider pull-right">|</li>
+ <li id="projectVersion" class="pull-right">Version: 0.4.0</li>
</ul>
</div>
@@ -78,7 +75,7 @@
<ul class="nav nav-list">
<li class="nav-header">User Documentation</li>
-
+
<li>
<a href="../../index.html" title="Metron">
@@ -99,7 +96,7 @@
<i class="icon-chevron-right"></i>
Analytics</a>
</li>
-
+
<li>
<a href="../../metron-deployment/index.html" title="Deployment">
@@ -113,14 +110,28 @@
<i class="none"></i>
Docker</a>
</li>
-
+
+ <li>
+
+ <a href="../../metron-interface/metron-config/index.html" title="Config">
+ <i class="none"></i>
+ Config</a>
+ </li>
+
+ <li>
+
+ <a href="../../metron-interface/metron-rest/index.html" title="Rest">
+ <i class="none"></i>
+ Rest</a>
+ </li>
+
<li>
<a href="../../metron-platform/index.html" title="Platform">
<i class="icon-chevron-right"></i>
Platform</a>
</li>
-
+
<li>
<a href="../../metron-sensors/index.html" title="Sensors">
@@ -130,6 +141,13 @@
<li>
+ <a href="../../metron-sensors/bro-plugin-kafka/index.html" title="Bro-plugin-kafka">
+ <i class="none"></i>
+ Bro-plugin-kafka</a>
+ </li>
+
+ <li>
+
<a href="../../metron-sensors/fastcapa/index.html" title="Fastcapa">
<i class="none"></i>
Fastcapa</a>
@@ -163,84 +181,351 @@
<div id="bodyColumn" class="span9" >
- <h1>Pycapa</h1>
-<p><a name="Pycapa"></a></p>
-<div class="section">
-<h2><a name="Overview"></a>Overview</h2>
-<p>Pycapa performs network packet capture, both off-the-wire and from Kafka, which is useful for the testing and development of <a class="externalLink" href="https://github.com/apache/incubator-metron">Apache Metron</a>. It is not intended for production use. The tool will capture packets from a specified interface and push them into a Kafka Topic.</p></div>
-<div class="section">
-<h2><a name="Installation"></a>Installation</h2>
+ <div class="section">
+<h2><a name="Pycapa"></a>Pycapa</h2>
+<ul>
+
+<li><a href="#Overview">Overview</a></li>
+
+<li><a href="#Installation">Installation</a></li>
+
+<li><a href="#Usage">Usage</a>
+
+<ul>
+
+<li><a href="#Parameters">Parameters</a></li>
+
+<li><a href="#Examples">Examples</a></li>
+
+<li><a href="#Kerberos">Kerberos</a></li>
+ </ul></li>
+
+<li><a href="#FAQs">FAQs</a></li>
+</ul>
+<h1>Overview</h1>
+<p>Pycapa performs network packet capture, both off-the-wire and from a Kafka topic, which is useful for the testing and development of <a class="externalLink" href="https://github.com/apache/metron">Apache Metron</a>. It is not intended for production use. The tool will capture packets from a specified interface and push them into a Kafka Topic. The tool can also do the reverse. It can consume packets from Kafka and reconstruct each network packet. This can then be used to create a <a class="externalLink" href="https://wiki.wireshark.org/Development/LibpcapFileFormat">libpcap-compliant file</a> or even to feed directly into a tool like Wireshark to monitor ongoing activity.</p>
+<h1>Installation</h1>
+<p>General notes on the installation of Pycapa.</p>
+
+<ul>
+
+<li>Python 2.7 is required.</li>
+
+<li>The following package dependencies are required and can be installed automatically with <tt>pip</tt>.
+
+<ul>
+
+<li><a class="externalLink" href="https://github.com/confluentinc/confluent-kafka-python">confluent-kafka-python</a></li>
+
+<li><a class="externalLink" href="https://github.com/CoreSecurity/pcapy">pcapy</a></li>
+ </ul></li>
+
+<li>These instructions can be used directly on CentOS 7+.</li>
+
+<li>Other Linux distributions that come with Python 2.7 can use these instructions with some minor modifications.</li>
+
+<li>Older distributions, like CentOS 6, that come with Python 2.6 installed, should install Python 2.7 within a virtual environment and then run Pycapa from within the virtual environment.</li>
+</ul>
+
+<ol style="list-style-type: decimal">
+
+<li>
+<p>Install system dependencies including the core development tools, Python libraries and header files, and Libpcap libraries and header files. On CentOS 7+, you can install these requirements with the following command.</p>
+
+<div class="source">
+<div class="source">
+<pre> yum -y install "@Development tools" python-devel libpcap-devel
+</pre></div></div></li>
+
+<li>
+<p>Install Librdkafka at your chosen $PREFIX.</p>
+
+<div class="source">
+<div class="source">
+<pre> export PREFIX=/usr
+ wget https://github.com/edenhill/librdkafka/archive/v0.9.4.tar.gz -O - | tar -xz
+ cd librdkafka-0.9.4/
+ ./configure --prefix=$PREFIX
+ make
+ make install
+</pre></div></div></li>
+
+<li>
+<p>Add Librdkafka to the dynamic library load path.</p>
+
+<div class="source">
+<div class="source">
+<pre>echo "$PREFIX/lib" >> /etc/ld.so.conf.d/pycapa.conf
+ldconfig -v
+</pre></div></div></li>
+
+<li>
+<p>Install Pycapa. This assumes that you already have the Metron source code on the host.</p>
+
<div class="source">
<div class="source">
-<pre>pip install -r requirements.txt
+<pre>cd metron/metron-sensors/pycapa
+pip install -r requirements.txt
python setup.py install
-</pre></div></div></div>
+</pre></div></div></li>
+</ol>
+<h1>Usage</h1>
+<p>Pycapa has two primary runtime modes.</p>
+
+<ul>
+
+<li>
+<p><b>Producer Mode</b>: Pycapa can capture packets from a network interface and forward those packets to a Kafka topic. Pycapa embeds the raw network packet data in the Kafka message body. The message key contains the timestamp indicating when the packet was captured in microseconds from the epoch, in network byte order.</p></li>
+
+<li>
+<p><b>Consumer Mode</b>: Pycapa can also perform the reverse operation. It can consume packets from Kafka and reconstruct each network packet. This can then be used to create a <a class="externalLink" href="https://wiki.wireshark.org/Development/LibpcapFileFormat">libpcap-compliant file</a> or even to feed directly into a tool like Wireshark to monitor activity.</p></li>
+</ul>
<div class="section">
-<h2><a name="Usage"></a>Usage</h2>
+<h3><a name="Parameters"></a>Parameters</h3>
<div class="source">
<div class="source">
<pre>$ pycapa --help
-usage: pycapa [-h] [-p] [-c] [-k KAFKA_BROKERS] [-t TOPIC] [-n PACKET_COUNT]
- [-d DEBUG] [-i INTERFACE]
+usage: pycapa [-h] [-p] [-c] [-k KAFKA_BROKERS] [-t KAFKA_TOPIC]
+ [-o {begin,end,stored}] [-i NETWORK_IFACE] [-m MAX_PACKETS]
+ [-pp PRETTY_PRINT] [-ll LOG_LEVEL] [-X KAFKA_CONFIGS]
+ [-s SNAPLEN]
optional arguments:
-h, --help show this help message and exit
-p, --producer sniff packets and send to kafka
-c, --consumer read packets from kafka
- -k KAFKA_BROKERS, --kafka KAFKA_BROKERS
- kafka broker(s)
- -t TOPIC, --topic TOPIC
+ -k KAFKA_BROKERS, --kafka-broker KAFKA_BROKERS
+ kafka broker(s) as host:port
+ -t KAFKA_TOPIC, --kafka-topic KAFKA_TOPIC
kafka topic
- -n PACKET_COUNT, --number PACKET_COUNT
- number of packets to consume
- -d DEBUG, --debug DEBUG
- debug every X packets
- -i INTERFACE, --interface INTERFACE
- interface to listen on
-</pre></div></div>
-<p>Pycapa has two primary runtime modes.</p>
+ -o {begin,end,stored}, --kafka-offset {begin,end,stored}
+ kafka offset to consume from; default=end
+ -i NETWORK_IFACE, --interface NETWORK_IFACE
+ network interface to listen on
+ -m MAX_PACKETS, --max-packets MAX_PACKETS
+ stop after this number of packets
+ -pp PRETTY_PRINT, --pretty-print PRETTY_PRINT
+ pretty print every X packets
+ -ll LOG_LEVEL, --log-level LOG_LEVEL
+ set the log level; DEBUG, INFO, WARN
+ -X KAFKA_CONFIGS define a kafka client parameter; key=value
+ -s SNAPLEN, --snaplen SNAPLEN
+ capture only the first X bytes of each packet;
+ default=65535
+</pre></div></div></div>
+<div class="section">
+<h3><a name="Examples"></a>Examples</h3>
<div class="section">
-<h3><a name="Producer_Mode"></a>Producer Mode</h3>
-<p>Pycapa can be configured to capture packets from a network interface and then forward those packets to a Kafka topic. The following example will capture packets from the <tt>eth0</tt> network interface and forward those to a Kafka topic called <tt>pcap</tt> running on <tt>localhost</tt>.</p>
+<h4><a name="Example_1"></a>Example 1</h4>
+<p>Capture 10 packets from the <tt>eth0</tt> network interface and forward those to a Kafka topic called <tt>pcap</tt> running on <tt>localhost:9092</tt>. The process will not terminate until all messages have been delivered to Kafka.</p>
<div class="source">
<div class="source">
-<pre>pycapa --producer --kafka localhost:9092 --topic pcap -i eth0
-</pre></div></div>
-<p>To output debug messages every 100 captured packets, run the following.</p>
+<pre>$ pycapa --producer \
+ --interface eth0 \
+ --kafka-broker localhost:9092 \
+ --kafka-topic pcap \
+ --max-packets 10
+INFO:root:Connecting to Kafka; {'bootstrap.servers': 'localhost:9092', 'group.id': 'AWBHMIAESAHJ'}
+INFO:root:Starting packet capture
+INFO:root:Waiting for '6' message(s) to flush
+INFO:root:'10' packet(s) in, '10' packet(s) out
+</pre></div></div></div>
+<div class="section">
+<h4><a name="Example_2"></a>Example 2</h4>
+<p>Capture packets until SIGINT is received (the interrupt signal sent when entering CTRL-C in the console.) In this example, nothing will be reported as packets are captured and delivered to Kafka. Simply wait a few seconds, then type CTRL-C and the number of packets will be reported.</p>
<div class="source">
<div class="source">
-<pre>pycapa --producer --kafka localhost:9092 --topic pcap -i eth0 --debug 100
+<pre>$ pycapa --producer \
+ --interface en0 \
+ --kafka-broker localhost:9092 \
+ --kafka-topic pcap
+INFO:root:Connecting to Kafka; {'bootstrap.servers': 'localhost:9092', 'group.id': 'EULLGDOMZDCT'}
+INFO:root:Starting packet capture
+^C
+INFO:root:Clean shutdown process started
+INFO:root:Waiting for '2' message(s) to flush
+INFO:root:'21' packet(s) in, '21' packet(s) out
</pre></div></div></div>
<div class="section">
-<h3><a name="Consumer_Mode"></a>Consumer Mode</h3>
-<p>Pycapa can be configured to consume packets from a Kafka topic and then write those packets to a <a class="externalLink" href="https://wiki.wireshark.org/Development/LibpcapFileFormat">libpcap-compliant file</a>. To read 100 packets from a kafka topic and then write those to a <a class="externalLink" href="https://wiki.wireshark.org/Development/LibpcapFileFormat">libpcap-compliant file</a>, run the following command. The file <tt>out.pcap</tt> can then be opened with a tool such as Wireshark for further validation.</p>
+<h4><a name="Example_3"></a>Example 3</h4>
+<p>While capturing packets, output diagnostic information every 5 packets. Diagnostics will report when packets have been received from the network interface and when they have been successfully delivered to Kafka.</p>
<div class="source">
<div class="source">
-<pre>pycapa --consumer --kafka localhost:9092 --topic pcap --n 100 > out.pcap
-</pre></div></div>
-<p>To consume packets from Kafka continuously and print debug messages every 10 packets, run the following command. </p>
+<pre>$ pycapa --producer \
+ --interface eth0 \
+ --kafka-broker localhost:9092 \
+ --kafka-topic pcap \
+ --pretty-print 5
+ INFO:root:Connecting to Kafka; {'bootstrap.servers': 'localhost:9092', 'group.id': 'UAWINMBDNQEH'}
+ INFO:root:Starting packet capture
+ Packet received[5]
+ Packet delivered[5]: date=2017-05-08 14:48:54.474031 topic=pcap partition=0 offset=29086 len=42
+ Packet received[10]
+ Packet received[15]
+ Packet delivered[10]: date=2017-05-08 14:48:58.879710 topic=pcap partition=0 offset=0 len=187
+ Packet delivered[15]: date=2017-05-08 14:48:59.633127 topic=pcap partition=0 offset=0 len=43
+ Packet received[20]
+ Packet delivered[20]: date=2017-05-08 14:49:01.949628 topic=pcap partition=0 offset=29101 len=134
+ Packet received[25]
+ ^C
+ INFO:root:Clean shutdown process started
+ Packet delivered[25]: date=2017-05-08 14:49:03.589940 topic=pcap partition=0 offset=0 len=142
+ INFO:root:Waiting for '1' message(s) to flush
+ INFO:root:'27' packet(s) in, '27' packet(s) out
+
+</pre></div></div></div>
+<div class="section">
+<h4><a name="Example_4"></a>Example 4</h4>
+<p>Consume 10 packets and create a libpcap-compliant pcap file.</p>
+
+<div class="source">
+<div class="source">
+<pre> $ pycapa --consumer \
+ --kafka-broker localhost:9092 \
+ --kafka-topic pcap \
+ --max-packets 10 \
+ > out.pcap
+ $ tshark -r out.pcap
+ 1 0.000000 199.193.204.147 → 192.168.0.3 TLSv1.2 151 Application Data
+ 2 0.000005 199.193.204.147 → 192.168.0.3 TLSv1.2 1191 Application Data
+ 3 0.000088 192.168.0.3 → 199.193.204.147 TCP 66 54788 → 443 [ACK] Seq=1 Ack=86 Win=4093 Len=0 TSval=961284465 TSecr=943744612
+ 4 0.000089 192.168.0.3 → 199.193.204.147 TCP 66 54788 → 443 [ACK] Seq=1 Ack=1211 Win=4058 Len=0 TSval=961284465 TSecr=943744612
+ 5 0.948788 192.168.0.3 → 192.30.253.125 TCP 54 54671 → 443 [ACK] Seq=1 Ack=1 Win=4096 Len=0
+ 6 1.005175 192.30.253.125 → 192.168.0.3 TCP 66 [TCP ACKed unseen segment] 443 → 54671 [ACK] Seq=1 Ack=2 Win=31 Len=0 TSval=2658544467 TSecr=961240339
+ 7 1.636312 fe80::1286:8cff:fe0e:65df → ff02::1 ICMPv6 134 Router Advertisement from 10:86:8c:0e:65:df
+ 8 2.253052 192.175.27.112 → 192.168.0.3 TLSv1.2 928 Application Data
+ 9 2.253140 192.168.0.3 → 192.175.27.112 TCP 66 55078 → 443 [ACK] Seq=1 Ack=863 Win=4069 Len=0 TSval=961286699 TSecr=967172238
+ 10 2.494769 192.168.0.3 → 224.0.0.251 MDNS 82 Standard query 0x0000 PTR _googlecast._tcp.local, "QM" question
+</pre></div></div></div>
+<div class="section">
+<h4><a name="Example_5"></a>Example 5</h4>
+<p>Consume 10 packets from the Kafka topic <tt>pcap</tt> running on <tt>localhost:9092</tt>, then pipe those into Wireshark for DPI.</p>
<div class="source">
<div class="source">
-<pre>pycapa --consumer --kafka localhost:9092 --topic pcap --debug 10
+<pre>$ pycapa --consumer \
+ --kafka-broker localhost:9092 \
+ --kafka-topic pcap \
+ --max-packets 10 \
+ | tshark -i -
+Capturing on 'Standard input'
+ 1 0.000000 ArrisGro_0e:65:df → Apple_bf:0d:43 ARP 56 Who has 192.168.0.3? Tell 192.168.0.1
+ 2 0.000044 Apple_bf:0d:43 → ArrisGro_0e:65:df ARP 42 192.168.0.3 is at ac:bc:32:bf:0d:43
+ 3 0.203495 fe80::1286:8cff:fe0e:65df → ff02::1 ICMPv6 134 Router Advertisement from 10:86:8c:0e:65:df
+ 4 2.031988 192.168.0.3 → 96.27.183.249 TCP 54 55110 → 443 [ACK] Seq=1 Ack=1 Win=4108 Len=0
+ 5 2.035816 192.30.253.125 → 192.168.0.3 TLSv1.2 97 Application Data
+ 6 2.035892 192.168.0.3 → 192.30.253.125 TCP 66 54671 → 443 [ACK] Seq=1 Ack=32 Win=4095 Len=0 TSval=961120495 TSecr=2658503052
+ 7 2.035994 192.168.0.3 → 192.30.253.125 TLSv1.2 101 Application Data
+ 8 2.053866 96.27.183.249 → 192.168.0.3 TCP 66 [TCP ACKed unseen segment] 443 → 55110 [ACK] Seq=1 Ack=2 Win=243 Len=0 TSval=728145145 TSecr=961030381
+ 9 2.083872 192.30.253.125 → 192.168.0.3 TCP 66 443 → 54671 [ACK] Seq=32 Ack=36 Win=31 Len=0 TSval=2658503087 TSecr=961120495
+ 10 3.173189 fe80::1286:8cff:fe0e:65df → ff02::1 ICMPv6 134 Router Advertisement from 10:86:8c:0e:65:df
+10 packets captured
</pre></div></div></div></div>
<div class="section">
-<h2><a name="Dependencies"></a>Dependencies</h2>
+<h3><a name="Kerberos"></a>Kerberos</h3>
+<p>The probe can be used in a Kerberized environment. Follow these additional steps to use Pycapa with Kerberos. The following assumptions have been made. These may need altered to fit your environment.</p>
<ul>
-<li><a class="externalLink" href="https://github.com/dpkp/kafka-python">kafka-python</a></li>
+<li>The Kafka broker is at <tt>kafka1:6667</tt></li>
-<li><a class="externalLink" href="https://github.com/CoreSecurity/pcapy">pcapy</a></li>
-</ul></div>
-<div class="section">
-<h2><a name="Implementation"></a>Implementation</h2>
-<p>When run in Producer Mode, Pycapa embeds the raw network packet data in the Kafka message. The message key contains the timestamp indicating when the packet was captured in microseconds from the epoch. This value is in network byte order.</p></div>
+<li>Zookeeper is at <tt>zookeeper1:2181</tt></li>
+
+<li>The Kafka security protocol is <tt>SASL_PLAINTEXT</tt></li>
+
+<li>The keytab used is located at <tt>/etc/security/keytabs/metron.headless.keytab</tt></li>
+
+<li>The service principal is <tt>metron@EXAMPLE.COM</tt></li>
+</ul>
+
+<ol style="list-style-type: decimal">
+
+<li>
+<p>Build Librdkafka with SASL support (<tt>--enable-sasl</tt>) and install at your chosen $PREFIX.</p>
+
+<div class="source">
+<div class="source">
+<pre>wget https://github.com/edenhill/librdkafka/archive/v0.9.4.tar.gz -O - | tar -xz
+cd librdkafka-0.9.4/
+./configure --prefix=$PREFIX --enable-sasl
+make
+make install
+</pre></div></div></li>
+
+<li>
+<p>Validate Librdkafka does indeed support SASL. Run the following command and ensure that <tt>sasl</tt> is returned as a built-in feature.</p>
+
+<div class="source">
+<div class="source">
+<pre>$ examples/rdkafka_example -X builtin.features
+builtin.features = gzip,snappy,ssl,sasl,regex
+</pre></div></div>
+<p>If it is not, ensure that you have <tt>libsasl</tt> or <tt>libsasl2</tt> installed. On CentOS, this can be installed with the following command.</p>
+
+<div class="source">
+<div class="source">
+<pre>yum install -y cyrus-sasl cyrus-sasl-devel cyrus-sasl-gssapi
+</pre></div></div></li>
+
+<li>
+<p>Grant access to your Kafka topic. In this example the topic is simply named <tt>pcap</tt>.</p>
+
+<div class="source">
+<div class="source">
+<pre>${KAFKA_HOME}/bin/kafka-acls.sh \
+ --authorizer kafka.security.auth.SimpleAclAuthorizer \
+ --authorizer-properties zookeeper.connect=zookeeper1:2181 \
+ --add \
+ --allow-principal User:metron \
+ --topic pcap
+${KAFKA_HOME}/bin/kafka-acls.sh \
+ --authorizer kafka.security.auth.SimpleAclAuthorizer \
+ --authorizer-properties zookeeper.connect=zookeeper1:2181 \
+ --add \
+ --allow-principal User:metron \
+ --group pycapa
+</pre></div></div></li>
+
+<li>
+<p>Use Pycapa as you normally would, but append the following three additional parameters</p>
+
+<ul>
+
+<li><tt>security.protocol</tt></li>
+
+<li><tt>sasl.kerberos.keytab</tt></li>
+
+<li><tt>sasl.kerberos.principal</tt></li>
+ </ul>
+
+<div class="source">
+<div class="source">
+<pre> $ pycapa --producer \
+ --interface eth0 \
+ --kafka-broker kafka1:6667 \
+ --kafka-topic pcap --max-packets 10 \
+ -X security.protocol=SASL_PLAINTEXT \
+ -X sasl.kerberos.keytab=/etc/security/keytabs/metron.headless.keytab \
+ -X sasl.kerberos.principal=metron-metron@METRONEXAMPLE.COM
+ INFO:root:Connecting to Kafka; {'sasl.kerberos.principal': 'metron-metron@METRONEXAMPLE.COM', 'group.id': 'ORNLVWJZZUAA', 'security.protocol': 'SASL_PLAINTEXT', 'sasl.kerberos.keytab': '/etc/security/keytabs/metron.headless.keytab', 'bootstrap.servers': 'kafka1:6667'}
+ INFO:root:Starting packet capture
+ INFO:root:Waiting for '1' message(s) to flush
+ INFO:root:'10' packet(s) in, '10' packet(s) out
+</pre></div></div></li>
+</ol>
+<h1>FAQs</h1>
+<p><b>Question</b>: How do I get more logs?</p>
+<p>Use the following two command-line arguments to get detailed logging.</p>
+
+<div class="source">
+<div class="source">
+<pre>-X debug=all --log-level DEBUG
+</pre></div></div></div></div>
</div>
</div>
</div>
@@ -249,8 +534,9 @@ optional arguments:
<footer>
<div class="container-fluid">
- <div class="row span12">Copyright © 2017.
- All Rights Reserved.
+ <div class="row span12">Copyright © 2017
+ <a href="https://www.apache.org">The Apache Software Foundation</a>.
+ All Rights Reserved.
</div>
http://git-wip-us.apache.org/repos/asf/metron/blob/cf4c2ecd/develop/bylaws/index.html
----------------------------------------------------------------------
diff --git a/develop/bylaws/index.html b/develop/bylaws/index.html
new file mode 100644
index 0000000..8a2be3b
--- /dev/null
+++ b/develop/bylaws/index.html
@@ -0,0 +1,508 @@
+<!DOCTYPE html>
+
+<html lang="en-US" prefix="og: http://ogp.me/ns#" ng-app="hortonWorksApp"><head><meta http-equiv="Content-Type" content="text/html; charset=UTF-8"><style type="text/css">@charset "UTF-8";[ng\:cloak],[ng-cloak],[data-ng-cloak],[x-ng-cloak],.ng-cloak,.x-ng-cloak,.ng-hide:not(.ng-hide-animate){display:none !important;}ng\:form{display:block;}.ng-animate-shim{visibility:hidden;}.ng-anchor{position:absolute;}</style>
+
+ <meta name="viewport" content="width=device-width, height=device-height, initial-scale=1.0, user-scalable=0, minimum-scale=1.0, maximum-scale=1.0">
+ <link rel="profile" href="http://gmpg.org/xfn/11">
+ <title>Apache Metron Bylaws</title>
+
+ <style type="text/css">
+ img.wp-smiley,
+ img.emoji {
+ display: inline !important;
+ border: none !important;
+ box-shadow: none !important;
+ height: 1em !important;
+ width: 1em !important;
+ margin: 0 .07em !important;
+ vertical-align: -0.1em !important;
+ background: none !important;
+ padding: 0 !important;
+ }
+ </style>
+
+ <script>
+ (function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
+ (i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),
+ m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
+ })(window,document,'script','https://www.google-analytics.com/analytics.js','ga');
+
+ ga('create', 'UA-78972211-1', 'auto');
+ ga('send', 'pageview');
+
+ </script>
+
+ <link rel="stylesheet" id="default-css" href="/css/default.css" type="text/css" media="all">
+ <link rel="stylesheet" id="legacy-css" href="/css/legacy.css" type="text/css" media="all">
+ <link rel="stylesheet" id="gravityforms-css" href="/css/gravityforms.css" type="text/css" media="all">
+ <link rel="stylesheet" id="selectboxit-css" href="/css/selectBoxIt.css" type="text/css" media="all">
+ <link href='http://fonts.googleapis.com/css?family=Roboto%20Condensed' rel='stylesheet' type='text/css'>
+
+ <script type="text/javascript" src="/js/jquery.js"></script>
+ <script type="text/javascript" src="/js/jquery-migrate.min.js"></script>
+ <script type="text/javascript" src="/js/jquery.mobile.custom.min.js"></script>
+ <script type="text/javascript" src="/js/isotope.pkgd.min.js"></script>
+ <script type="text/javascript" src="/js/navigation.js"></script>
+ <script type="text/javascript" src="/js/slider.js"></script>
+ <script type="text/javascript" src="/js/jquery.classyloader.js"></script>
+ <script type="text/javascript" src="/js/hero-slider.js"></script>
+ <script type="text/javascript" src="/js/footer.js"></script>
+ <script type="text/javascript" src="/js/top-form.js"></script>
+ <script type="text/javascript" src="/js/local6.js"></script>
+ <script type="text/javascript" src="/js/select-list.js"></script>
+ <script type="text/javascript" src="/js/legacy.js"></script>
+ <script type="text/javascript" src="/js/paralax-scroll.js"></script>
+ <script type="text/javascript" src="/js/jquery-ui.min.js"></script>
+ <script type="text/javascript" src="/js/jquery.selectBoxIt.min.js"></script>
+ <script type="text/javascript" src="/js/scroll-slider.js"></script>
+ <script type="text/javascript" src="/js/charts-animate.js"></script>
+ <script type="text/javascript" src="/js/dotdotdot.js"></script>
+ <script type="text/javascript" src="/js/products_scroll.js"></script>
+ <script type="text/javascript" src="/js/parallax.min.js"></script>
+ <script type="text/javascript" src="/js/parallax.js"></script>
+ <script type="text/javascript" src="/js/shb0wjr.js"></script>
+
+ <script>
+ window.lsfDataLayer = window.lsfDataLayer || [];
+ </script>
+
+ <style>
+ body {
+ font-family: "roboto-condensed", sans-serif;
+ }
+ </style>
+
+ <!-- Angular js files -->
+ <script src="/js/angular.min.js"></script>
+ <script src="/js/angular-animate.min.js"></script>
+ <script src="/js/app.js"></script>
+ <script src="/js/pressReleaseController.js"></script>
+ <script src="/js/eventCtrl.js"></script>
+ <!-- End Angular js files -->
+ <link rel="stylesheet" type="text/css" href="/css/prettify.css">
+ <script type="text/javascript" async="" src="/js/munchkin-beta.js"></script>
+ <link type="text/css" href="/css/jquery-ui-insightera-custom-1.8.css" rel="stylesheet">
+ <style id="style-1-cropbar-clipper">
+ /* Copyright 2014 Evernote Corporation. All rights reserved. */
+
+ .en-markup-crop-options {
+ top: 18px !important;
+ left: 50% !important;
+ margin-left: -100px !important;
+ width: 200px !important;
+ border: 2px rgba(255, 255, 255, .38) solid !important;
+ border-radius: 4px !important;
+ }
+
+ .en-markup-crop-options div div:first-of-type {
+ margin-left: 0px !important;
+ }
+ </style>
+</head>
+
+
+<body class="wrap">
+ <header>
+ <div class="main-navigation" style="height: 85px;">
+ <div class="main-nav fixed">
+ <div class="wrapper">
+ <div class="logo">
+ <a href=""><img src="/img/logo.png" alt="Logo"></a>
+ </div>
+ <nav>
+ <div class="toggle">
+ <span></span>
+ <span></span>
+ <span></span>
+ </div>
+ <div class="menu-wrapper" style="height: auto; display: block;">
+ <ul class="main-menu">
+ <li class="products-menu"><a href="/">Home</a>
+ </li>
+ <li class="solutions-menu"><a href="/about/">About</a>
+ </li>
+ <li class="services-menu"><a href="/documentation/">Documentation</a>
+ </li>
+ <li class="services-menu"><a href="/community/">Community</a>
+ </li>
+ <li class="training-menu"><a href="/news/">News</a>
+ </li>
+ <li>
+ <button class="button-default button-green" style="line-height:43px;"> <a>Download </a></button>
+ </li>
+</ul>
+
+ </div>
+ </nav>
+ </div>
+ </div>
+ </div>
+</header>
+
+
+ <section class="standalone">
+ <div class="grid">
+
+ <div class="unit whole">
+ <article>
+ <h1>Apache Metron Bylaws</h1>
+ <h2 id="introduction">Introduction</h2>
+
+<p>This document defines the bylaws under which the Apache Metron project
+operates. It defines the roles and responsibilities of the project,
+who may vote, how voting works, how conflicts are resolved, etc.</p>
+
+<p>Metron is a project of the Apache Software Foundation (ASF) and the foundation
+holds the trademark on the name “Metron” and copyright on the combined
+code base. The <a href="http://www.apache.org/foundation/faq.html">Apache Foundation
+FAQ</a> and
+<a href="http://www.apache.org/foundation/how-it-works.html">How-It-Works</a>
+explain the operation and background of the foundation.</p>
+
+<p>Apache has a <a href="http://www.apache.org/foundation/policies/conduct.html">code of
+conduct</a> that
+it expects its members to follow. In particular:</p>
+
+<ul>
+ <li>
+ <p>Be <strong>open and welcoming</strong>. It is important that we grow and
+encourage the community of users and developers for our project.</p>
+ </li>
+ <li>
+ <p>Be <strong>collaborative</strong>. Working together on the open mailing lists and
+bug database to make decisions helps the project grow.</p>
+ </li>
+ <li>
+ <p>Be <strong>respectful</strong> of others. Everyone is volunteering their time and
+efforts to work on this project. Please be respectful of everyone
+and their views.</p>
+ </li>
+</ul>
+
+<p>Metron is typical of Apache projects in that it operates under a set of
+principles, known collectively as the “Apache Way”. If you are new to
+Apache development, please refer to
+<a href="http://www.apache.org/foundation/how-it-works.html">this</a> for more
+information on how Apache projects operate.</p>
+
+<h2 id="roles-and-responsibilities">Roles and Responsibilities</h2>
+
+<p>Apache projects define a set of roles with associated rights and
+responsibilities. These roles govern what tasks an individual may
+perform within the project. The roles are defined in the following
+sections.</p>
+
+<h3 id="users">Users</h3>
+
+<p>The most important participants in the project are people who use our
+software. The majority of our developers start out as users and guide
+their development efforts from the user’s perspective. Users
+contribute to the Apache projects by providing feedback to developers
+in the form of bug reports and feature suggestions. As well, users
+participate in the Apache community by helping other users on mailing
+lists and user support forums.</p>
+
+<h3 id="contributors">Contributors</h3>
+
+<p>Contributors include all of the volunteers who donate time, code,
+documentation,
+or resources to the Metron Project. A contributor that makes sustained,
+welcome contributions to the project may be invited to become a
+committer, though the exact timing of such invitations depends on many
+factors.</p>
+
+<h3 id="committers">Committers</h3>
+
+<p>The project’s committers are responsible for the project’s technical
+management. Committers have the right to commit to the project’s git
+repository. Committers may cast binding votes on any technical
+discussion.</p>
+
+<p>Committer access is by invitation only and must be approved by
+consensus approval of the active Project Management Committee (PMC)
+members.</p>
+
+<p>If a committer wishes to leave the project or does not contribute to
+the project in any form for six months, the PMC may make them emeritus.
+Emeritus committers lose their ability to commit code or cast binding
+votes. An emeritus committer may
+request reinstatement of commit access from the PMC. Such
+reinstatement is subject to consensus approval of active PMC members.</p>
+
+<p>All Apache committers are required to have a signed <a href="https://www.apache.org/licenses/icla.txt">Individual
+Contributor License
+Agreement</a> (ICLA) on file
+with the Apache Software Foundation. There is a <a href="http://www.apache.org/dev/committers.html">Committer
+FAQ</a> which provides more
+details on the requirements for Committers.</p>
+
+<p>A committer who makes a
+sustained contribution to the project may be invited to become a
+member of the PMC. The form of contribution
+is not limited to code. It can also include code review, helping out
+users on the mailing lists, documentation, testing, etc.</p>
+
+<h3 id="release-manager">Release Manager</h3>
+
+<p>A Release Manager (RM) is a committer who volunteers to produce a
+Release Candidate. The RM shall publish a Release Plan on the
+dev mailing list stating the branch from which they intend to
+make a Release Candidate.</p>
+
+<h3 id="project-management-committee">Project Management Committee</h3>
+
+<p>The Project Management Committee (PMC) for Apache Metron was created by
+the Apache Board in December 2015 when Metron moved out of Cisco’s OpenSOC<br />
+project and became an incubated project at Apache.
+The PMC is responsible to the board and
+the ASF for the management and oversight of the Apache Metron
+codebase. The responsibilities of the PMC include</p>
+
+<ul>
+ <li>
+ <p>Deciding what is distributed as products of the Apache Metron
+project. In particular all releases must be approved by the PMC.</p>
+ </li>
+ <li>
+ <p>Maintaining the project’s shared resources, including the codebase
+repository, mailing lists, and websites.</p>
+ </li>
+ <li>
+ <p>Speaking on behalf of the project.</p>
+ </li>
+ <li>
+ <p>Resolving license disputes regarding products of the project</p>
+ </li>
+ <li>
+ <p>Nominating new PMC members and committers</p>
+ </li>
+ <li>
+ <p>Maintaining these bylaws and other guidelines of the project</p>
+ </li>
+</ul>
+
+<p>Membership of the PMC is by invitation only and must be approved by a
+consensus approval of active PMC members.</p>
+
+<p>A PMC member is considered
+emeritus by their own declaration or by not contributing in any form
+to the project for over six months. An emeritus member may request
+reinstatement to the PMC. Such reinstatement is subject to consensus
+approval of the active PMC members.</p>
+
+<p>The chair of the PMC is appointed by the ASF board. The chair is an
+office holder of the Apache Software Foundation (Vice President,
+Apache Metron) and has primary responsibility to the board for the
+management of the project within the scope of the Metron PMC. The
+chair reports to the board quarterly on developments within the Metron
+project.</p>
+
+<p>When the project desires a new PMC chair, the PMC votes to recommend a
+new chair using <a href="http://wiki.apache.org/general/BoardVoting">Single Transferable
+Vote</a> voting. The decision
+must be ratified by the Apache board.</p>
+
+<h2 id="decision-making">Decision Making</h2>
+
+<p>Within the Metron project, different types of decisions require
+different forms of approval. For example, the previous section
+describes several decisions which require “consensus approval.”
+This section defines how voting is performed, the types of
+approvals, and which types of decision require which type of approval.</p>
+
+<h3 id="voting">Voting</h3>
+
+<p>Decisions regarding the project are made by votes on the primary
+project development mailing list (dev@Metron.apache.org). Where
+necessary, PMC voting may take place on the private Metron PMC mailing
+list. Votes are clearly indicated by subject line starting with
+[VOTE]. Votes may contain multiple items for approval and these should
+be clearly separated. Voting is carried out by replying to the vote
+mail. Voting may take five flavors:</p>
+
+<ul>
+ <li>
+ <p><strong>+1</strong> – “Yes,” “Agree,” or “the action should be performed.” In general,
+this vote also indicates a willingness on the behalf of the voter in
+“making it happen.”</p>
+ </li>
+ <li>
+ <p><strong>+0</strong> – This vote indicates a willingness for the action under
+consideration to go ahead. The voter, however, will not be able to
+help.</p>
+ </li>
+ <li>
+ <p><strong>0</strong> – The voter is neutral on the topic under discussion.</p>
+ </li>
+ <li>
+ <p><strong>-0</strong> – This vote indicates that the voter does not, in general, agree
+ with the proposed action but is not concerned enough to prevent the
+ action going ahead.</p>
+ </li>
+ <li>
+ <p><strong>-1</strong> – This is a negative vote. On issues where consensus is required,
+ this vote counts as a veto. All vetoes must contain an explanation
+ of why the veto is appropriate. Vetoes with no explanation are
+ void. It may also be appropriate for a -1 vote to include an
+ alternative course of action.</p>
+ </li>
+</ul>
+
+<p>All participants in the Metron project are encouraged to show their
+agreement for or against a particular action by voting, regardless of
+whether their vote is binding. Nonbinding votes are useful for
+encouraging discussion and understanding the scope of opinions within
+the project.</p>
+
+<h3 id="approvals">Approvals</h3>
+
+<p>These are the types of approvals that can be sought. Different actions
+require different types of approvals.</p>
+
+<ul>
+ <li>
+ <p><strong>Consensus Approval</strong> – Consensus approval requires 3 binding +1
+votes and no binding vetoes.</p>
+ </li>
+ <li>
+ <p><strong>Lazy Consensus</strong> – Lazy consensus requires at least one +1 vote and
+no -1 votes (‘silence gives assent’).</p>
+ </li>
+ <li>
+ <p><strong>Lazy Majority</strong> – A lazy majority vote requires 3 binding +1 votes
+ and more binding +1 votes than -1 votes.</p>
+ </li>
+ <li>
+ <p><strong>Lazy 2/3 Majority</strong> – Lazy 2/3 majority votes requires at least 3
+votes and twice as many +1 votes as -1 votes.</p>
+ </li>
+</ul>
+
+<h3 id="vetoes">Vetoes</h3>
+
+<p>A valid, binding veto cannot be overruled. If a veto is cast, it must
+be accompanied by a valid reason explaining the reasons for the
+veto. The validity of a veto, if challenged, can be confirmed by
+anyone who has a binding vote. This does not necessarily signify
+agreement with the veto - merely that the veto is valid. If you
+disagree with a valid veto, you must lobby the person casting the veto
+to withdraw their veto. If a veto is not withdrawn, any action that
+has already been taken must be reversed in a timely manner.</p>
+
+<h3 id="actions">Actions</h3>
+
+<p>This section describes the various actions which are undertaken within
+the project, the corresponding approval required for that action and
+those who have binding votes over the action.</p>
+
+<h4 id="code-change">Code Change</h4>
+
+<p>A change made to a codebase of the project requires <em>lazy consensus</em>
+of active committers other than the author of the patch. The code can
+be committed after the first +1.</p>
+
+<h4 id="product-release">Product Release</h4>
+
+<p>To make a release, the release manager creates a release candidate and
+a vote requiring a <em>lazy majority</em> of the active PMC members is
+required. Once the vote passes, the release candidate becomes an
+official release.</p>
+
+<h4 id="adoption-of-new-codebase">Adoption of New Codebase</h4>
+
+<p>When the codebase for an existing, released product is to be replaced
+with an alternative codebase, it requires a <em>lazy 2/3 majority</em> of PMC
+members. This also covers the creation of new sub-projects and
+submodules within the project.</p>
+
+<h4 id="new-committer">New Committer</h4>
+
+<p>When a new committer is proposed for the project, <em>consensus approval</em>
+of the active PMC members is required.</p>
+
+<h4 id="new-pmc-member">New PMC Member</h4>
+
+<p>To promote a committer to a PMC member requires <em>consensus approval</em>
+of active PMC members.</p>
+
+<p>If the vote passes, the Apache Board must be notified to make the change
+official.</p>
+
+<h4 id="committer-removal">Committer Removal</h4>
+
+<p>Removal of commit privileges requires a <em>lazy 2/3 majority</em> of active
+PMC members.</p>
+
+<h4 id="pmc-member-removal">PMC Member Removal</h4>
+
+<p>Removing a PMC member requires a <em>lazy 2/3 majority</em> of active PMC
+members, excluding the member in question.</p>
+
+<p>If the vote passes, the Apache Board must be notified to make the change
+official.</p>
+
+<h4 id="modifying-bylaws">Modifying Bylaws</h4>
+
+<p>Modifying this document requires a <em>lazy majority</em> of active PMC members.</p>
+
+<h3 id="voting-timeframes">Voting Timeframes</h3>
+
+<p>Votes are open for a minimum period of 72 hours to allow all active
+voters time to consider the vote. For holiday weekends or conferences,
+consider using a longer vote window. Votes relating to code changes are
+not subject to a strict timetable but should be made as timely as
+possible.</p>
+
+ </article>
+ </div>
+
+ <div class="clear"></div>
+
+ </div>
+</section>
+
+
+ </section>
+<section class="intro-block darken" id="docshome">
+ <div class="content-960 hover-btn text-center disclaimer">
+ The contents of this website are © 2016 Apache Software Foundation under the terms of the Apache License v2. Apache Metron and its logo are trademarks of the Apache Software Foundation.
+ </div>
+</section>
+
+ <script>
+ var anchorForId = function (id) {
+ var anchor = document.createElement("a");
+ anchor.className = "header-link";
+ anchor.href = "#" + id;
+ anchor.innerHTML = "<span class=\"sr-only\">Permalink</span><i class=\"fa fa-link\"></i>";
+ anchor.title = "Permalink";
+ return anchor;
+ };
+
+ var linkifyAnchors = function (level, containingElement) {
+ var headers = containingElement.getElementsByTagName("h" + level);
+ for (var h = 0; h < headers.length; h++) {
+ var header = headers[h];
+
+ if (typeof header.id !== "undefined" && header.id !== "") {
+ header.appendChild(anchorForId(header.id));
+ }
+ }
+ };
+
+ document.onreadystatechange = function () {
+ if (this.readyState === "complete") {
+ var contentBlock = document.getElementsByClassName("docs")[0] || document.getElementsByClassName("news")[0];
+ if (!contentBlock) {
+ return;
+ }
+ for (var level = 1; level <= 6; level++) {
+ linkifyAnchors(level, contentBlock);
+ }
+ }
+ };
+</script>
+
+
+</body>
+</html>
http://git-wip-us.apache.org/repos/asf/metron/blob/cf4c2ecd/develop/coding/index.html
----------------------------------------------------------------------
diff --git a/develop/coding/index.html b/develop/coding/index.html
new file mode 100644
index 0000000..59ea441
--- /dev/null
+++ b/develop/coding/index.html
@@ -0,0 +1,243 @@
+<!DOCTYPE html>
+
+<html lang="en-US" prefix="og: http://ogp.me/ns#" ng-app="hortonWorksApp"><head><meta http-equiv="Content-Type" content="text/html; charset=UTF-8"><style type="text/css">@charset "UTF-8";[ng\:cloak],[ng-cloak],[data-ng-cloak],[x-ng-cloak],.ng-cloak,.x-ng-cloak,.ng-hide:not(.ng-hide-animate){display:none !important;}ng\:form{display:block;}.ng-animate-shim{visibility:hidden;}.ng-anchor{position:absolute;}</style>
+
+ <meta name="viewport" content="width=device-width, height=device-height, initial-scale=1.0, user-scalable=0, minimum-scale=1.0, maximum-scale=1.0">
+ <link rel="profile" href="http://gmpg.org/xfn/11">
+ <title>Coding Guidelines</title>
+
+ <style type="text/css">
+ img.wp-smiley,
+ img.emoji {
+ display: inline !important;
+ border: none !important;
+ box-shadow: none !important;
+ height: 1em !important;
+ width: 1em !important;
+ margin: 0 .07em !important;
+ vertical-align: -0.1em !important;
+ background: none !important;
+ padding: 0 !important;
+ }
+ </style>
+
+ <script>
+ (function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
+ (i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),
+ m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
+ })(window,document,'script','https://www.google-analytics.com/analytics.js','ga');
+
+ ga('create', 'UA-78972211-1', 'auto');
+ ga('send', 'pageview');
+
+ </script>
+
+ <link rel="stylesheet" id="default-css" href="/css/default.css" type="text/css" media="all">
+ <link rel="stylesheet" id="legacy-css" href="/css/legacy.css" type="text/css" media="all">
+ <link rel="stylesheet" id="gravityforms-css" href="/css/gravityforms.css" type="text/css" media="all">
+ <link rel="stylesheet" id="selectboxit-css" href="/css/selectBoxIt.css" type="text/css" media="all">
+ <link href='http://fonts.googleapis.com/css?family=Roboto%20Condensed' rel='stylesheet' type='text/css'>
+
+ <script type="text/javascript" src="/js/jquery.js"></script>
+ <script type="text/javascript" src="/js/jquery-migrate.min.js"></script>
+ <script type="text/javascript" src="/js/jquery.mobile.custom.min.js"></script>
+ <script type="text/javascript" src="/js/isotope.pkgd.min.js"></script>
+ <script type="text/javascript" src="/js/navigation.js"></script>
+ <script type="text/javascript" src="/js/slider.js"></script>
+ <script type="text/javascript" src="/js/jquery.classyloader.js"></script>
+ <script type="text/javascript" src="/js/hero-slider.js"></script>
+ <script type="text/javascript" src="/js/footer.js"></script>
+ <script type="text/javascript" src="/js/top-form.js"></script>
+ <script type="text/javascript" src="/js/local6.js"></script>
+ <script type="text/javascript" src="/js/select-list.js"></script>
+ <script type="text/javascript" src="/js/legacy.js"></script>
+ <script type="text/javascript" src="/js/paralax-scroll.js"></script>
+ <script type="text/javascript" src="/js/jquery-ui.min.js"></script>
+ <script type="text/javascript" src="/js/jquery.selectBoxIt.min.js"></script>
+ <script type="text/javascript" src="/js/scroll-slider.js"></script>
+ <script type="text/javascript" src="/js/charts-animate.js"></script>
+ <script type="text/javascript" src="/js/dotdotdot.js"></script>
+ <script type="text/javascript" src="/js/products_scroll.js"></script>
+ <script type="text/javascript" src="/js/parallax.min.js"></script>
+ <script type="text/javascript" src="/js/parallax.js"></script>
+ <script type="text/javascript" src="/js/shb0wjr.js"></script>
+
+ <script>
+ window.lsfDataLayer = window.lsfDataLayer || [];
+ </script>
+
+ <style>
+ body {
+ font-family: "roboto-condensed", sans-serif;
+ }
+ </style>
+
+ <!-- Angular js files -->
+ <script src="/js/angular.min.js"></script>
+ <script src="/js/angular-animate.min.js"></script>
+ <script src="/js/app.js"></script>
+ <script src="/js/pressReleaseController.js"></script>
+ <script src="/js/eventCtrl.js"></script>
+ <!-- End Angular js files -->
+ <link rel="stylesheet" type="text/css" href="/css/prettify.css">
+ <script type="text/javascript" async="" src="/js/munchkin-beta.js"></script>
+ <link type="text/css" href="/css/jquery-ui-insightera-custom-1.8.css" rel="stylesheet">
+ <style id="style-1-cropbar-clipper">
+ /* Copyright 2014 Evernote Corporation. All rights reserved. */
+
+ .en-markup-crop-options {
+ top: 18px !important;
+ left: 50% !important;
+ margin-left: -100px !important;
+ width: 200px !important;
+ border: 2px rgba(255, 255, 255, .38) solid !important;
+ border-radius: 4px !important;
+ }
+
+ .en-markup-crop-options div div:first-of-type {
+ margin-left: 0px !important;
+ }
+ </style>
+</head>
+
+
+<body class="wrap">
+ <header>
+ <div class="main-navigation" style="height: 85px;">
+ <div class="main-nav fixed">
+ <div class="wrapper">
+ <div class="logo">
+ <a href=""><img src="/img/logo.png" alt="Logo"></a>
+ </div>
+ <nav>
+ <div class="toggle">
+ <span></span>
+ <span></span>
+ <span></span>
+ </div>
+ <div class="menu-wrapper" style="height: auto; display: block;">
+ <ul class="main-menu">
+ <li class="products-menu"><a href="/">Home</a>
+ </li>
+ <li class="solutions-menu"><a href="/about/">About</a>
+ </li>
+ <li class="services-menu"><a href="/documentation/">Documentation</a>
+ </li>
+ <li class="services-menu"><a href="/community/">Community</a>
+ </li>
+ <li class="training-menu"><a href="/news/">News</a>
+ </li>
+ <li>
+ <button class="button-default button-green" style="line-height:43px;"> <a>Download </a></button>
+ </li>
+</ul>
+
+ </div>
+ </nav>
+ </div>
+ </div>
+ </div>
+</header>
+
+
+ <section class="standalone">
+ <div class="grid">
+
+ <div class="unit whole">
+ <article>
+ <h1>Coding Guidelines</h1>
+ <h2 id="general-rules">General rules</h2>
+
+<ul>
+ <li>All files must have an Apache copyright header at the top of the file.</li>
+ <li>Code should be removed rather than commented out.</li>
+ <li>All public functions should have javadoc comments.</li>
+ <li>Always use braces to surround branches.</li>
+ <li>try-finally should be avoided.</li>
+</ul>
+
+<h2 id="formatting">Formatting</h2>
+
+<ul>
+ <li>All files must have an 80 character maximum line length.</li>
+ <li>Indentation should be 2 spaces.</li>
+ <li>Files should use spaces instead of tabs.</li>
+ <li>Wrapping lines
+ <ul>
+ <li>Break after a comma.</li>
+ <li>Break before an operator.</li>
+ <li>Prefer higher-level breaks to lower-level breaks.</li>
+ <li>Align the new line with beginning of the expression at the same level
+on the previous line.</li>
+ <li>If the above rules lead to confusing code, just indent 8 spaces.</li>
+ </ul>
+ </li>
+ <li>One variable declaration per a line.</li>
+</ul>
+
+<h2 id="naming">Naming</h2>
+
+<ul>
+ <li>Packages should be all lowercase.
+ <ul>
+ <li>Java code should be in <code class="highlighter-rouge">org.apache.,metron</code>, except for compatibility classes</li>
+ </ul>
+ </li>
+ <li>Classes should be in mixed case.</li>
+ <li>Variables should be in camel case.</li>
+ <li>Constants should be in upper case.</li>
+</ul>
+
+ </article>
+ </div>
+
+ <div class="clear"></div>
+
+ </div>
+</section>
+
+
+ </section>
+<section class="intro-block darken" id="docshome">
+ <div class="content-960 hover-btn text-center disclaimer">
+ The contents of this website are © 2016 Apache Software Foundation under the terms of the Apache License v2. Apache Metron and its logo are trademarks of the Apache Software Foundation.
+ </div>
+</section>
+
+ <script>
+ var anchorForId = function (id) {
+ var anchor = document.createElement("a");
+ anchor.className = "header-link";
+ anchor.href = "#" + id;
+ anchor.innerHTML = "<span class=\"sr-only\">Permalink</span><i class=\"fa fa-link\"></i>";
+ anchor.title = "Permalink";
+ return anchor;
+ };
+
+ var linkifyAnchors = function (level, containingElement) {
+ var headers = containingElement.getElementsByTagName("h" + level);
+ for (var h = 0; h < headers.length; h++) {
+ var header = headers[h];
+
+ if (typeof header.id !== "undefined" && header.id !== "") {
+ header.appendChild(anchorForId(header.id));
+ }
+ }
+ };
+
+ document.onreadystatechange = function () {
+ if (this.readyState === "complete") {
+ var contentBlock = document.getElementsByClassName("docs")[0] || document.getElementsByClassName("news")[0];
+ if (!contentBlock) {
+ return;
+ }
+ for (var level = 1; level <= 6; level++) {
+ linkifyAnchors(level, contentBlock);
+ }
+ }
+ };
+</script>
+
+
+</body>
+</html>
http://git-wip-us.apache.org/repos/asf/metron/blob/cf4c2ecd/develop/index.html
----------------------------------------------------------------------
diff --git a/develop/index.html b/develop/index.html
index f752d38..95fc00f 100644
--- a/develop/index.html
+++ b/develop/index.html
@@ -234,12 +234,12 @@ developers working on the project. The project has created
<p>There are several development mailing lists for Metron</p>
<ul>
- <li><a href="mailto:dev@metron.apache.org">dev@metron.apache.org</a> - Development discussions
-with archive <a href="https://mail-archives.apache.org/mod_mbox/incubator-metron-dev/">here</a></li>
- <li><a href="mailto:issues@metron.apache.org">issues@metron.apache.org</a> - Bug tracking
-with archive <a href="https://mail-archives.apache.org/mod_mbox/incubator-metron-issues/">here</a></li>
- <li><a href="mailto:commits@metron.apache.org">commits@metron.apache.org</a> - Git tracking
-with archive <a href="https://mail-archives.apache.org/mod_mbox/incubator-metron-commits/">here</a></li>
+ <li><a href="mailto:dev@metron.apache.org">dev@metron.apache.org</a> - Development discussions
+with archive <a href="https://mail-archives.apache.org/mod_mbox/metron-dev/">here</a></li>
+ <li><a href="mailto:issues@metron.apache.org">issues@metron.apache.org</a> - Bug tracking
+with archive <a href="https://mail-archives.apache.org/mod_mbox/metron-issues/">here</a></li>
+ <li><a href="mailto:commits@metron.apache.org">commits@metron.apache.org</a> - Git tracking
+with archive <a href="https://mail-archives.apache.org/mod_mbox/metron-commits/">here</a></li>
</ul>
<p>You can subscribe to the lists by sending email to
@@ -250,14 +250,14 @@ with archive <a href="https://mail-archives.apache.org/mod_mbox/incubator-metron
<p>Metron uses git for version control. Get the source code:</p>
-<p><code>% git clone https://git-wip-us.apache.org/repos/asf/incubator-metron.git</code></p>
+<p><code class="highlighter-rouge">% git clone https://git-wip-us.apache.org/repos/asf/metron.git</code></p>
<p>The important branches are:</p>
<ul>
- <li><a href="https://github.com/apache/incubator-metron/tree/master">master</a> -
+ <li><a href="https://github.com/apache/metron/tree/master">master</a> -
The trunk for all developement</li>
- <li><a href="https://github.com/apache/incubator-metron/tree/asf-site">asf-site</a> -
+ <li><a href="https://github.com/apache/metron/tree/asf-site">asf-site</a> -
The pages that are deployed to https://metron.apache.org/</li>
</ul>
http://git-wip-us.apache.org/repos/asf/metron/blob/cf4c2ecd/docs/hive-ddl.html
----------------------------------------------------------------------
diff --git a/docs/hive-ddl.html b/docs/hive-ddl.html
index f7a99b1..0c66416 100644
--- a/docs/hive-ddl.html
+++ b/docs/hive-ddl.html
@@ -155,7 +155,7 @@
- <option value="/docs/arch.html">Metron Architecture</option>
+ <option value="http://localhost:4000/docs/arch.html">Metron Architecture</option>
@@ -231,12 +231,10 @@
-
-
- <li class=""><a href="/docs/arch.html">Metron Architecture</a></li>
+ <li class=""><a href="http://localhost:4000/docs/arch.html">Metron Architecture</a></li>
@@ -251,8 +249,6 @@
-
-
http://git-wip-us.apache.org/repos/asf/metron/blob/cf4c2ecd/docs/index.html
----------------------------------------------------------------------
diff --git a/docs/index.html b/docs/index.html
index e36da17..53c3493 100644
--- a/docs/index.html
+++ b/docs/index.html
@@ -155,7 +155,7 @@
- <option value="/docs/arch.html">Metron Architecture</option>
+ <option value="http://localhost:4000/docs/arch.html">Metron Architecture</option>
@@ -231,12 +231,10 @@
-
-
- <li class=""><a href="/docs/arch.html">Metron Architecture</a></li>
+ <li class=""><a href="http://localhost:4000/docs/arch.html">Metron Architecture</a></li>
@@ -251,8 +249,6 @@
-
-
http://git-wip-us.apache.org/repos/asf/metron/blob/cf4c2ecd/docs/sensor.html
----------------------------------------------------------------------
diff --git a/docs/sensor.html b/docs/sensor.html
index c460cdf..0f61d4b 100644
--- a/docs/sensor.html
+++ b/docs/sensor.html
@@ -155,7 +155,7 @@
- <option value="/docs/arch.html">Metron Architecture</option>
+ <option value="http://localhost:4000/docs/arch.html">Metron Architecture</option>
@@ -231,12 +231,10 @@
-
-
- <li class=""><a href="/docs/arch.html">Metron Architecture</a></li>
+ <li class=""><a href="http://localhost:4000/docs/arch.html">Metron Architecture</a></li>
@@ -251,8 +249,6 @@
-
-
http://git-wip-us.apache.org/repos/asf/metron/blob/cf4c2ecd/documentation/index.html
----------------------------------------------------------------------
diff --git a/documentation/index.html b/documentation/index.html
index 466ed75..aa486b4 100644
--- a/documentation/index.html
+++ b/documentation/index.html
@@ -211,21 +211,21 @@
<section class="events-section info-block darken" id="releases">
<div class="wrapper downloads">
<div class="text-center">
- <h4>Current Release: 0.3.1</h4>
+ <h4>Current Release: 0.4.0</h4>
</div>
<div class="content-960 hover-btn text-center">
<ul>
<li>
- <a href="http://www.apache.org/dyn/closer.cgi/incubator/metron/0.3.1/apache-metron-0.3.1-incubating.tar.gz" target="new">
- apache-metron-0.3.1-incubating.tar.gz
+ <a href="http://www.apache.org/dyn/closer.cgi/metron/0.4.0/apache-metron-0.4.0.tar.gz" target="new">
+ apache-metron-0.4.0.tar.gz
</a>
- [ <a href="https://dist.apache.org/repos/dist/release/incubator/metron/0.3.1/apache-metron-0.3.1-incubating.tar.gz.asc" target="new">
+ [ <a href="https://dist.apache.org/repos/dist/release/metron/0.4.0/apache-metron-0.4.0.tar.gz.asc" target="new">
PGP
</a>]
- [ <a href="https://dist.apache.org/repos/dist/release/incubator/metron/0.3.1/apache-metron-0.3.1-incubating.tar.gz.sha" target="new">
+ [ <a href="https://dist.apache.org/repos/dist/release/metron/0.4.0/apache-metron-0.4.0.tar.gz.sha" target="new">
SHA512
</a>]
- [ <a href="https://dist.apache.org/repos/dist/release/incubator/metron/0.3.1/apache-metron-0.3.1-incubating.tar.gz.md5" target="new">
+ [ <a href="https://dist.apache.org/repos/dist/release/metron/0.4.0/apache-metron-0.4.0.tar.gz.md5" target="new">
MD5
</a>]
[ <a href="/current-book/index.html" target="new"> Book </a>]
@@ -239,7 +239,7 @@
<h4>Older Releases</h4>
</div>
<div class="content-960 hover-btn text-center">
- If you need access to older releases they can be found in the <a href="https://archive.apache.org/dist/incubator/metron/">release archives</a>.
+ If you need access to older releases they can be found in the <a href="https://archive.apache.org/dist/metron/">release archives</a>.
</div>
</div>
</section>
http://git-wip-us.apache.org/repos/asf/metron/blob/cf4c2ecd/feed.xml
----------------------------------------------------------------------
diff --git a/feed.xml b/feed.xml
index a2812b4..910d107 100644
--- a/feed.xml
+++ b/feed.xml
@@ -3,11 +3,11 @@
<channel>
<title></title>
<description></description>
- <link>/</link>
- <atom:link href="/feed.xml" rel="self" type="application/rss+xml"/>
- <pubDate>Sun, 23 Apr 2017 21:17:52 -0400</pubDate>
- <lastBuildDate>Sun, 23 Apr 2017 21:17:52 -0400</lastBuildDate>
- <generator>Jekyll v2.4.0</generator>
+ <link>http://localhost:4000/</link>
+ <atom:link href="http://localhost:4000/feed.xml" rel="self" type="application/rss+xml"/>
+ <pubDate>Wed, 05 Jul 2017 11:58:22 -0700</pubDate>
+ <lastBuildDate>Wed, 05 Jul 2017 11:58:22 -0700</lastBuildDate>
+ <generator>Jekyll v3.4.5</generator>
<item>
<title>Debo Dutta joins PPMC</title>
@@ -21,9 +21,9 @@ the Metron website.</p>
<p>Please join me to welcome Debo.</p>
</description>
- <pubDate>Fri, 08 Jan 2016 11:23:00 -0500</pubDate>
- <link>/news/2016/01/08/debo-committer/</link>
- <guid isPermaLink="true">/news/2016/01/08/debo-committer/</guid>
+ <pubDate>Fri, 08 Jan 2016 08:23:00 -0800</pubDate>
+ <link>http://localhost:4000/news/2016/01/08/debo-committer/</link>
+ <guid isPermaLink="true">http://localhost:4000/news/2016/01/08/debo-committer/</guid>
<category>team</category>
@@ -34,12 +34,12 @@ the Metron website.</p>
<title>Metron adopts new logo</title>
<description><p>The Metron project has adopted a new logo. We hope you like it.</p>
-<p><img src="/img/logo.png" alt="metron logo" title="metron logo" /></p>
+<p><img src="http://localhost:4000/img/logo.png" alt="metron logo" title="metron logo" /></p>
</description>
- <pubDate>Sat, 12 Dec 2015 07:00:00 -0500</pubDate>
- <link>/news/2015/12/12/new-logo/</link>
- <guid isPermaLink="true">/news/2015/12/12/new-logo/</guid>
+ <pubDate>Sat, 12 Dec 2015 04:00:00 -0800</pubDate>
+ <link>http://localhost:4000/news/2015/12/12/new-logo/</link>
+ <guid isPermaLink="true">http://localhost:4000/news/2015/12/12/new-logo/</guid>
<category>project</category>
@@ -50,9 +50,9 @@ the Metron website.</p>
<title>Metron accepted into Apache Incubator</title>
<description><p>Congratulations! Metron accepted into the Apache Incubator.</p>
</description>
- <pubDate>Sun, 06 Dec 2015 15:48:00 -0500</pubDate>
- <link>/news/2015/12/06/metron-incubator/</link>
- <guid isPermaLink="true">/news/2015/12/06/metron-incubator/</guid>
+ <pubDate>Sun, 06 Dec 2015 12:48:00 -0800</pubDate>
+ <link>http://localhost:4000/news/2015/12/06/metron-incubator/</link>
+ <guid isPermaLink="true">http://localhost:4000/news/2015/12/06/metron-incubator/</guid>
<category>project</category>
http://git-wip-us.apache.org/repos/asf/metron/blob/cf4c2ecd/help/index.html
----------------------------------------------------------------------
diff --git a/help/index.html b/help/index.html
index d27f442..3b75885 100644
--- a/help/index.html
+++ b/help/index.html
@@ -151,11 +151,11 @@
<h2 id="mailing-lists">Mailing Lists</h2>
<p>The best option is to send email to the user list
-<a href="mailto:user@metron.apache.org">user@metron.apache.org</a>, which is for user
+<a href="mailto:user@metron.apache.org">user@metron.apache.org</a>, which is for user
discussions. All of the historic traffic is available in the
-<a href="http://mail-archives.apache.org/mod_mbox/incubator-metron-user/">archive</a>. To
+<a href="http://mail-archives.apache.org/mod_mbox/metron-user/">archive</a>. To
subscribe to the user list, please send email to
-<a href="mailto:user-subscribe@metron.apache.org">user-subscribe@metron.apache.org</a>.</p>
+<a href="mailto:user-subscribe@metron.apache.org">user-subscribe@metron.apache.org</a>.</p>
<h2 id="bug-reports">Bug Reports</h2>
@@ -172,7 +172,7 @@ your question.</p>
<h2 id="browse-the-code">Browse the code</h2>
<p>One of the advantages of open source software is that you can browse the code.
-The code is available on <a href="https://github.com/apache/incubator-metron/tree/master">github</a>.</p>
+The code is available on <a href="https://github.com/apache/metron/tree/master">github</a>.</p>
</article>
</div>
http://git-wip-us.apache.org/repos/asf/metron/blob/cf4c2ecd/index.html
----------------------------------------------------------------------
diff --git a/index.html b/index.html
index c9dca83..193f71d 100644
--- a/index.html
+++ b/index.html
@@ -148,7 +148,7 @@
<div class="v-middle-inner">
<div class="v-middle">
<h1>real-time big data security </h1>
- <a href="https://github.com/apache/incubator-metron" target="_blank"><button type="button" class="button-default">GitHub</button></a>
+ <a href="https://github.com/apache/metron" target="_blank"><button type="button" class="button-default">GitHub</button></a>
<a href="https://cwiki.apache.org/confluence/display/METRON/Community+Resources" target="_blank"><button type="button" class="button-default">Community Home</button></a>
http://git-wip-us.apache.org/repos/asf/metron/blob/cf4c2ecd/news/2015/12/12/new-logo/index.html
----------------------------------------------------------------------
diff --git a/news/2015/12/12/new-logo/index.html b/news/2015/12/12/new-logo/index.html
index fb1671c..752746e 100644
--- a/news/2015/12/12/new-logo/index.html
+++ b/news/2015/12/12/new-logo/index.html
@@ -185,7 +185,7 @@
<div class="post-content">
<p>The Metron project has adopted a new logo. We hope you like it.</p>
-<p><img src="/img/logo.png" alt="metron logo" title="metron logo" /></p>
+<p><img src="http://localhost:4000/img/logo.png" alt="metron logo" title="metron logo" /></p>
</div>
[06/10] metron git commit: METRON-1012 Update Metron public web site
for 0.4.0 release - push to asf-site from the generated code on master
Posted by ma...@apache.org.
http://git-wip-us.apache.org/repos/asf/metron/blob/cf4c2ecd/current-book/metron-deployment/roles/sensor-test-mode/index.html
----------------------------------------------------------------------
diff --git a/current-book/metron-deployment/roles/sensor-test-mode/index.html b/current-book/metron-deployment/roles/sensor-test-mode/index.html
index a55f54f..e3f2b2d 100644
--- a/current-book/metron-deployment/roles/sensor-test-mode/index.html
+++ b/current-book/metron-deployment/roles/sensor-test-mode/index.html
@@ -1,13 +1,13 @@
<!DOCTYPE html>
<!--
- | Generated by Apache Maven Doxia at 2017-02-23
+ | Generated by Apache Maven Doxia at 2017-06-27
| Rendered using Apache Maven Fluido Skin 1.3.0
-->
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
- <meta name="Date-Revision-yyyymmdd" content="20170223" />
+ <meta name="Date-Revision-yyyymmdd" content="20170627" />
<meta http-equiv="Content-Language" content="en" />
<title>Metron – Sensor Test Mode</title>
<link rel="stylesheet" href="../../../css/apache-maven-fluido-1.3.0.min.css" />
@@ -30,14 +30,11 @@
<div class="container-fluid">
<div id="banner">
<div class="pull-left">
- <a href="http://metron.incubator.apache.org/" id="bannerLeft">
- <img src="../../../images/metron-logo.png" alt="Apache Metron - Incubating" width="148px" height="48px"/>
+ <a href="http://metron.apache.org/" id="bannerLeft">
+ <img src="../../../images/metron-logo.png" alt="Apache Metron" width="148px" height="48px"/>
</a>
</div>
- <div class="pull-right"> <a href="http://incubator.apache.org/" id="bannerRight">
- <img src="../../../images/ApacheIncubating_Logo.png" alt="Apache Incubating" width="192px" height="48px"/>
- </a>
- </div>
+ <div class="pull-right"> </div>
<div class="clear"><hr/></div>
</div>
@@ -51,8 +48,8 @@
</li>
<li class="divider ">/</li>
<li class="">
- <a href="http://metron.incubator.apache.org/" class="externalLink" title="Metron-Incubating">
- Metron-Incubating</a>
+ <a href="http://metron.apache.org/" class="externalLink" title="Metron">
+ Metron</a>
</li>
<li class="divider ">/</li>
<li class="">
@@ -64,8 +61,8 @@
- <li id="publishDate" class="pull-right">Last Published: 2017-02-23</li> <li class="divider pull-right">|</li>
- <li id="projectVersion" class="pull-right">Version: 0.3.1</li>
+ <li id="publishDate" class="pull-right">Last Published: 2017-06-27</li> <li class="divider pull-right">|</li>
+ <li id="projectVersion" class="pull-right">Version: 0.4.0</li>
</ul>
</div>
@@ -78,7 +75,7 @@
<ul class="nav nav-list">
<li class="nav-header">User Documentation</li>
-
+
<li>
<a href="../../../index.html" title="Metron">
@@ -99,7 +96,7 @@
<i class="icon-chevron-right"></i>
Analytics</a>
</li>
-
+
<li>
<a href="../../../metron-deployment/index.html" title="Deployment">
@@ -109,6 +106,20 @@
<li>
+ <a href="../../../metron-deployment/Kerberos-ambari-setup.html" title="Kerberos-ambari-setup">
+ <i class="none"></i>
+ Kerberos-ambari-setup</a>
+ </li>
+
+ <li>
+
+ <a href="../../../metron-deployment/Kerberos-manual-setup.html" title="Kerberos-manual-setup">
+ <i class="none"></i>
+ Kerberos-manual-setup</a>
+ </li>
+
+ <li>
+
<a href="../../../metron-deployment/amazon-ec2/index.html" title="Amazon-ec2">
<i class="none"></i>
Amazon-ec2</a>
@@ -130,11 +141,11 @@
<li>
- <a href="../../../metron-deployment/packer-build/index.html" title="Packer-build">
+ <a href="../../../metron-deployment/packaging/packer-build/index.html" title="Packer-build">
<i class="none"></i>
Packer-build</a>
</li>
-
+
<li>
<a href="../../../metron-deployment/roles/index.html" title="Roles">
@@ -144,13 +155,6 @@
<li>
- <a href="../../../metron-deployment/roles/kibana/index.html" title="Kibana">
- <i class="none"></i>
- Kibana</a>
- </li>
-
- <li>
-
<a href="../../../metron-deployment/roles/monit/index.html" title="Monit">
<i class="none"></i>
Monit</a>
@@ -199,14 +203,28 @@
<i class="none"></i>
Docker</a>
</li>
-
+
+ <li>
+
+ <a href="../../../metron-interface/metron-config/index.html" title="Config">
+ <i class="none"></i>
+ Config</a>
+ </li>
+
+ <li>
+
+ <a href="../../../metron-interface/metron-rest/index.html" title="Rest">
+ <i class="none"></i>
+ Rest</a>
+ </li>
+
<li>
<a href="../../../metron-platform/index.html" title="Platform">
<i class="icon-chevron-right"></i>
Platform</a>
</li>
-
+
<li>
<a href="../../../metron-sensors/index.html" title="Sensors">
@@ -271,8 +289,9 @@
<footer>
<div class="container-fluid">
- <div class="row span12">Copyright © 2017.
- All Rights Reserved.
+ <div class="row span12">Copyright © 2017
+ <a href="https://www.apache.org">The Apache Software Foundation</a>.
+ All Rights Reserved.
</div>
http://git-wip-us.apache.org/repos/asf/metron/blob/cf4c2ecd/current-book/metron-deployment/vagrant/codelab-platform/index.html
----------------------------------------------------------------------
diff --git a/current-book/metron-deployment/vagrant/codelab-platform/index.html b/current-book/metron-deployment/vagrant/codelab-platform/index.html
index 39ba07b..341af0e 100644
--- a/current-book/metron-deployment/vagrant/codelab-platform/index.html
+++ b/current-book/metron-deployment/vagrant/codelab-platform/index.html
@@ -1,13 +1,13 @@
<!DOCTYPE html>
<!--
- | Generated by Apache Maven Doxia at 2017-02-23
+ | Generated by Apache Maven Doxia at 2017-06-27
| Rendered using Apache Maven Fluido Skin 1.3.0
-->
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
- <meta name="Date-Revision-yyyymmdd" content="20170223" />
+ <meta name="Date-Revision-yyyymmdd" content="20170627" />
<meta http-equiv="Content-Language" content="en" />
<title>Metron – Developer Image for Apache Metron on Virtualbox</title>
<link rel="stylesheet" href="../../../css/apache-maven-fluido-1.3.0.min.css" />
@@ -30,14 +30,11 @@
<div class="container-fluid">
<div id="banner">
<div class="pull-left">
- <a href="http://metron.incubator.apache.org/" id="bannerLeft">
- <img src="../../../images/metron-logo.png" alt="Apache Metron - Incubating" width="148px" height="48px"/>
+ <a href="http://metron.apache.org/" id="bannerLeft">
+ <img src="../../../images/metron-logo.png" alt="Apache Metron" width="148px" height="48px"/>
</a>
</div>
- <div class="pull-right"> <a href="http://incubator.apache.org/" id="bannerRight">
- <img src="../../../images/ApacheIncubating_Logo.png" alt="Apache Incubating" width="192px" height="48px"/>
- </a>
- </div>
+ <div class="pull-right"> </div>
<div class="clear"><hr/></div>
</div>
@@ -51,8 +48,8 @@
</li>
<li class="divider ">/</li>
<li class="">
- <a href="http://metron.incubator.apache.org/" class="externalLink" title="Metron-Incubating">
- Metron-Incubating</a>
+ <a href="http://metron.apache.org/" class="externalLink" title="Metron">
+ Metron</a>
</li>
<li class="divider ">/</li>
<li class="">
@@ -64,8 +61,8 @@
- <li id="publishDate" class="pull-right">Last Published: 2017-02-23</li> <li class="divider pull-right">|</li>
- <li id="projectVersion" class="pull-right">Version: 0.3.1</li>
+ <li id="publishDate" class="pull-right">Last Published: 2017-06-27</li> <li class="divider pull-right">|</li>
+ <li id="projectVersion" class="pull-right">Version: 0.4.0</li>
</ul>
</div>
@@ -78,7 +75,7 @@
<ul class="nav nav-list">
<li class="nav-header">User Documentation</li>
-
+
<li>
<a href="../../../index.html" title="Metron">
@@ -99,7 +96,7 @@
<i class="icon-chevron-right"></i>
Analytics</a>
</li>
-
+
<li>
<a href="../../../metron-deployment/index.html" title="Deployment">
@@ -109,6 +106,20 @@
<li>
+ <a href="../../../metron-deployment/Kerberos-ambari-setup.html" title="Kerberos-ambari-setup">
+ <i class="none"></i>
+ Kerberos-ambari-setup</a>
+ </li>
+
+ <li>
+
+ <a href="../../../metron-deployment/Kerberos-manual-setup.html" title="Kerberos-manual-setup">
+ <i class="none"></i>
+ Kerberos-manual-setup</a>
+ </li>
+
+ <li>
+
<a href="../../../metron-deployment/amazon-ec2/index.html" title="Amazon-ec2">
<i class="none"></i>
Amazon-ec2</a>
@@ -130,11 +141,11 @@
<li>
- <a href="../../../metron-deployment/packer-build/index.html" title="Packer-build">
+ <a href="../../../metron-deployment/packaging/packer-build/index.html" title="Packer-build">
<i class="none"></i>
Packer-build</a>
</li>
-
+
<li>
<a href="../../../metron-deployment/roles/index.html" title="Roles">
@@ -185,14 +196,28 @@
<i class="none"></i>
Docker</a>
</li>
-
+
+ <li>
+
+ <a href="../../../metron-interface/metron-config/index.html" title="Config">
+ <i class="none"></i>
+ Config</a>
+ </li>
+
+ <li>
+
+ <a href="../../../metron-interface/metron-rest/index.html" title="Rest">
+ <i class="none"></i>
+ Rest</a>
+ </li>
+
<li>
<a href="../../../metron-platform/index.html" title="Platform">
<i class="icon-chevron-right"></i>
Platform</a>
</li>
-
+
<li>
<a href="../../../metron-sensors/index.html" title="Sensors">
@@ -231,7 +256,7 @@
<ul>
-<li><a class="externalLink" href="https://github.com/ansible/ansible">Ansible</a> 2.0.0.2</li>
+<li><a class="externalLink" href="https://github.com/ansible/ansible">Ansible</a> (2.0.0.2 or 2.2.2.0)</li>
<li><a class="externalLink" href="https://www.vagrantup.com">Vagrant</a> 1.8.1</li>
@@ -302,8 +327,9 @@
<footer>
<div class="container-fluid">
- <div class="row span12">Copyright © 2017.
- All Rights Reserved.
+ <div class="row span12">Copyright © 2017
+ <a href="https://www.apache.org">The Apache Software Foundation</a>.
+ All Rights Reserved.
</div>
http://git-wip-us.apache.org/repos/asf/metron/blob/cf4c2ecd/current-book/metron-deployment/vagrant/fastcapa-test-platform/index.html
----------------------------------------------------------------------
diff --git a/current-book/metron-deployment/vagrant/fastcapa-test-platform/index.html b/current-book/metron-deployment/vagrant/fastcapa-test-platform/index.html
index 47d7aed..fc2b4e3 100644
--- a/current-book/metron-deployment/vagrant/fastcapa-test-platform/index.html
+++ b/current-book/metron-deployment/vagrant/fastcapa-test-platform/index.html
@@ -1,13 +1,13 @@
<!DOCTYPE html>
<!--
- | Generated by Apache Maven Doxia at 2017-02-23
+ | Generated by Apache Maven Doxia at 2017-06-27
| Rendered using Apache Maven Fluido Skin 1.3.0
-->
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
- <meta name="Date-Revision-yyyymmdd" content="20170223" />
+ <meta name="Date-Revision-yyyymmdd" content="20170627" />
<meta http-equiv="Content-Language" content="en" />
<title>Metron – Fastcapa Test Environment</title>
<link rel="stylesheet" href="../../../css/apache-maven-fluido-1.3.0.min.css" />
@@ -30,14 +30,11 @@
<div class="container-fluid">
<div id="banner">
<div class="pull-left">
- <a href="http://metron.incubator.apache.org/" id="bannerLeft">
- <img src="../../../images/metron-logo.png" alt="Apache Metron - Incubating" width="148px" height="48px"/>
+ <a href="http://metron.apache.org/" id="bannerLeft">
+ <img src="../../../images/metron-logo.png" alt="Apache Metron" width="148px" height="48px"/>
</a>
</div>
- <div class="pull-right"> <a href="http://incubator.apache.org/" id="bannerRight">
- <img src="../../../images/ApacheIncubating_Logo.png" alt="Apache Incubating" width="192px" height="48px"/>
- </a>
- </div>
+ <div class="pull-right"> </div>
<div class="clear"><hr/></div>
</div>
@@ -51,8 +48,8 @@
</li>
<li class="divider ">/</li>
<li class="">
- <a href="http://metron.incubator.apache.org/" class="externalLink" title="Metron-Incubating">
- Metron-Incubating</a>
+ <a href="http://metron.apache.org/" class="externalLink" title="Metron">
+ Metron</a>
</li>
<li class="divider ">/</li>
<li class="">
@@ -64,8 +61,8 @@
- <li id="publishDate" class="pull-right">Last Published: 2017-02-23</li> <li class="divider pull-right">|</li>
- <li id="projectVersion" class="pull-right">Version: 0.3.1</li>
+ <li id="publishDate" class="pull-right">Last Published: 2017-06-27</li> <li class="divider pull-right">|</li>
+ <li id="projectVersion" class="pull-right">Version: 0.4.0</li>
</ul>
</div>
@@ -78,7 +75,7 @@
<ul class="nav nav-list">
<li class="nav-header">User Documentation</li>
-
+
<li>
<a href="../../../index.html" title="Metron">
@@ -99,7 +96,7 @@
<i class="icon-chevron-right"></i>
Analytics</a>
</li>
-
+
<li>
<a href="../../../metron-deployment/index.html" title="Deployment">
@@ -109,6 +106,20 @@
<li>
+ <a href="../../../metron-deployment/Kerberos-ambari-setup.html" title="Kerberos-ambari-setup">
+ <i class="none"></i>
+ Kerberos-ambari-setup</a>
+ </li>
+
+ <li>
+
+ <a href="../../../metron-deployment/Kerberos-manual-setup.html" title="Kerberos-manual-setup">
+ <i class="none"></i>
+ Kerberos-manual-setup</a>
+ </li>
+
+ <li>
+
<a href="../../../metron-deployment/amazon-ec2/index.html" title="Amazon-ec2">
<i class="none"></i>
Amazon-ec2</a>
@@ -130,11 +141,11 @@
<li>
- <a href="../../../metron-deployment/packer-build/index.html" title="Packer-build">
+ <a href="../../../metron-deployment/packaging/packer-build/index.html" title="Packer-build">
<i class="none"></i>
Packer-build</a>
</li>
-
+
<li>
<a href="../../../metron-deployment/roles/index.html" title="Roles">
@@ -185,14 +196,28 @@
<i class="none"></i>
Docker</a>
</li>
-
+
+ <li>
+
+ <a href="../../../metron-interface/metron-config/index.html" title="Config">
+ <i class="none"></i>
+ Config</a>
+ </li>
+
+ <li>
+
+ <a href="../../../metron-interface/metron-rest/index.html" title="Rest">
+ <i class="none"></i>
+ Rest</a>
+ </li>
+
<li>
<a href="../../../metron-platform/index.html" title="Platform">
<i class="icon-chevron-right"></i>
Platform</a>
</li>
-
+
<li>
<a href="../../../metron-sensors/index.html" title="Sensors">
@@ -300,8 +325,9 @@ service fastcapa status
<footer>
<div class="container-fluid">
- <div class="row span12">Copyright © 2017.
- All Rights Reserved.
+ <div class="row span12">Copyright © 2017
+ <a href="https://www.apache.org">The Apache Software Foundation</a>.
+ All Rights Reserved.
</div>
http://git-wip-us.apache.org/repos/asf/metron/blob/cf4c2ecd/current-book/metron-deployment/vagrant/full-dev-platform/index.html
----------------------------------------------------------------------
diff --git a/current-book/metron-deployment/vagrant/full-dev-platform/index.html b/current-book/metron-deployment/vagrant/full-dev-platform/index.html
index 26a1336..9233051 100644
--- a/current-book/metron-deployment/vagrant/full-dev-platform/index.html
+++ b/current-book/metron-deployment/vagrant/full-dev-platform/index.html
@@ -1,13 +1,13 @@
<!DOCTYPE html>
<!--
- | Generated by Apache Maven Doxia at 2017-02-23
+ | Generated by Apache Maven Doxia at 2017-06-27
| Rendered using Apache Maven Fluido Skin 1.3.0
-->
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
- <meta name="Date-Revision-yyyymmdd" content="20170223" />
+ <meta name="Date-Revision-yyyymmdd" content="20170627" />
<meta http-equiv="Content-Language" content="en" />
<title>Metron – Full Development Platform</title>
<link rel="stylesheet" href="../../../css/apache-maven-fluido-1.3.0.min.css" />
@@ -30,14 +30,11 @@
<div class="container-fluid">
<div id="banner">
<div class="pull-left">
- <a href="http://metron.incubator.apache.org/" id="bannerLeft">
- <img src="../../../images/metron-logo.png" alt="Apache Metron - Incubating" width="148px" height="48px"/>
+ <a href="http://metron.apache.org/" id="bannerLeft">
+ <img src="../../../images/metron-logo.png" alt="Apache Metron" width="148px" height="48px"/>
</a>
</div>
- <div class="pull-right"> <a href="http://incubator.apache.org/" id="bannerRight">
- <img src="../../../images/ApacheIncubating_Logo.png" alt="Apache Incubating" width="192px" height="48px"/>
- </a>
- </div>
+ <div class="pull-right"> </div>
<div class="clear"><hr/></div>
</div>
@@ -51,8 +48,8 @@
</li>
<li class="divider ">/</li>
<li class="">
- <a href="http://metron.incubator.apache.org/" class="externalLink" title="Metron-Incubating">
- Metron-Incubating</a>
+ <a href="http://metron.apache.org/" class="externalLink" title="Metron">
+ Metron</a>
</li>
<li class="divider ">/</li>
<li class="">
@@ -64,8 +61,8 @@
- <li id="publishDate" class="pull-right">Last Published: 2017-02-23</li> <li class="divider pull-right">|</li>
- <li id="projectVersion" class="pull-right">Version: 0.3.1</li>
+ <li id="publishDate" class="pull-right">Last Published: 2017-06-27</li> <li class="divider pull-right">|</li>
+ <li id="projectVersion" class="pull-right">Version: 0.4.0</li>
</ul>
</div>
@@ -78,7 +75,7 @@
<ul class="nav nav-list">
<li class="nav-header">User Documentation</li>
-
+
<li>
<a href="../../../index.html" title="Metron">
@@ -99,7 +96,7 @@
<i class="icon-chevron-right"></i>
Analytics</a>
</li>
-
+
<li>
<a href="../../../metron-deployment/index.html" title="Deployment">
@@ -109,6 +106,20 @@
<li>
+ <a href="../../../metron-deployment/Kerberos-ambari-setup.html" title="Kerberos-ambari-setup">
+ <i class="none"></i>
+ Kerberos-ambari-setup</a>
+ </li>
+
+ <li>
+
+ <a href="../../../metron-deployment/Kerberos-manual-setup.html" title="Kerberos-manual-setup">
+ <i class="none"></i>
+ Kerberos-manual-setup</a>
+ </li>
+
+ <li>
+
<a href="../../../metron-deployment/amazon-ec2/index.html" title="Amazon-ec2">
<i class="none"></i>
Amazon-ec2</a>
@@ -130,11 +141,11 @@
<li>
- <a href="../../../metron-deployment/packer-build/index.html" title="Packer-build">
+ <a href="../../../metron-deployment/packaging/packer-build/index.html" title="Packer-build">
<i class="none"></i>
Packer-build</a>
</li>
-
+
<li>
<a href="../../../metron-deployment/roles/index.html" title="Roles">
@@ -185,14 +196,28 @@
<i class="none"></i>
Docker</a>
</li>
-
+
+ <li>
+
+ <a href="../../../metron-interface/metron-config/index.html" title="Config">
+ <i class="none"></i>
+ Config</a>
+ </li>
+
+ <li>
+
+ <a href="../../../metron-interface/metron-rest/index.html" title="Rest">
+ <i class="none"></i>
+ Rest</a>
+ </li>
+
<li>
<a href="../../../metron-platform/index.html" title="Platform">
<i class="icon-chevron-right"></i>
Platform</a>
</li>
-
+
<li>
<a href="../../../metron-sensors/index.html" title="Sensors">
@@ -232,13 +257,17 @@
<ul>
-<li><a class="externalLink" href="https://github.com/ansible/ansible">Ansible</a> 2.0.0.2</li>
+<li><a class="externalLink" href="https://github.com/ansible/ansible">Ansible</a> (2.0.0.2 or 2.2.2.0)</li>
-<li><a class="externalLink" href="https://www.vagrantup.com">Vagrant</a> 1.8.1</li>
+<li><a class="externalLink" href="https://www.docker.com/community-edition">Docker</a></li>
-<li><a class="externalLink" href="https://virtualbox.org">Virtualbox</a> 5.0.16</li>
+<li><a class="externalLink" href="https://www.vagrantup.com">Vagrant</a> 1.8+</li>
-<li>Python 2.7.11</li>
+<li><a class="externalLink" href="https://github.com/devopsgroup-io/vagrant-hostmanager">Vagrant Hostmanager Plugin</a></li>
+
+<li><a class="externalLink" href="https://virtualbox.org">Virtualbox</a> 5.0+</li>
+
+<li>Python 2.7</li>
<li>Maven 3.3.9</li>
</ul>
@@ -256,10 +285,11 @@
<div class="source">
<div class="source">
-<pre> brew cask install vagrant virtualbox java
- brew install maven git
- brew install https://raw.githubusercontent.com/Homebrew/homebrew-core/ee1273bf919a5e4e50838513a9e55ea423e1d7ce/Formula/ansible.rb
- brew switch ansible 2.0.0.2
+<pre>brew cask install vagrant virtualbox java docker
+brew install maven@3.3 git
+pip install ansible==2.2.2.0
+vagrant plugin install vagrant-hostmanager
+open /Applications/Docker.app
</pre></div></div></li>
</ol></div></div>
<div class="section">
@@ -268,35 +298,21 @@
<ol style="list-style-type: decimal">
<li>
-<p>Build Metron</p>
-
-<div class="source">
-<div class="source">
-<pre> cd incubator-metron
- mvn clean package -DskipTests
-</pre></div></div></li>
-
-<li>
-<p>Install Vagrant Hostmanager.</p>
-
-<div class="source">
-<div class="source">
-<pre> vagrant plugin install vagrant-hostmanager
-</pre></div></div></li>
+<p>Ensure that the Docker service is running.</p></li>
<li>
<p>Deploy Metron</p>
<div class="source">
<div class="source">
-<pre> cd metron-deployment/vagrant/full-dev-platform
- vagrant up
+<pre>cd metron-deployment/vagrant/full-dev-platform
+vagrant up
</pre></div></div>
<p>Should the process fail before completing the deployment, the following command will continue the deployment process without re-instantiating the host.</p>
<div class="source">
<div class="source">
-<pre> vagrant provision
+<pre>vagrant provision
</pre></div></div></li>
</ol></div>
<div class="section">
@@ -361,8 +377,9 @@
<footer>
<div class="container-fluid">
- <div class="row span12">Copyright © 2017.
- All Rights Reserved.
+ <div class="row span12">Copyright © 2017
+ <a href="https://www.apache.org">The Apache Software Foundation</a>.
+ All Rights Reserved.
</div>
http://git-wip-us.apache.org/repos/asf/metron/blob/cf4c2ecd/current-book/metron-deployment/vagrant/index.html
----------------------------------------------------------------------
diff --git a/current-book/metron-deployment/vagrant/index.html b/current-book/metron-deployment/vagrant/index.html
index 373405e..4ab6194 100644
--- a/current-book/metron-deployment/vagrant/index.html
+++ b/current-book/metron-deployment/vagrant/index.html
@@ -1,13 +1,13 @@
<!DOCTYPE html>
<!--
- | Generated by Apache Maven Doxia at 2017-02-23
+ | Generated by Apache Maven Doxia at 2017-06-27
| Rendered using Apache Maven Fluido Skin 1.3.0
-->
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
- <meta name="Date-Revision-yyyymmdd" content="20170223" />
+ <meta name="Date-Revision-yyyymmdd" content="20170627" />
<meta http-equiv="Content-Language" content="en" />
<title>Metron – Vagrant Deployment</title>
<link rel="stylesheet" href="../../css/apache-maven-fluido-1.3.0.min.css" />
@@ -30,14 +30,11 @@
<div class="container-fluid">
<div id="banner">
<div class="pull-left">
- <a href="http://metron.incubator.apache.org/" id="bannerLeft">
- <img src="../../images/metron-logo.png" alt="Apache Metron - Incubating" width="148px" height="48px"/>
+ <a href="http://metron.apache.org/" id="bannerLeft">
+ <img src="../../images/metron-logo.png" alt="Apache Metron" width="148px" height="48px"/>
</a>
</div>
- <div class="pull-right"> <a href="http://incubator.apache.org/" id="bannerRight">
- <img src="../../images/ApacheIncubating_Logo.png" alt="Apache Incubating" width="192px" height="48px"/>
- </a>
- </div>
+ <div class="pull-right"> </div>
<div class="clear"><hr/></div>
</div>
@@ -51,8 +48,8 @@
</li>
<li class="divider ">/</li>
<li class="">
- <a href="http://metron.incubator.apache.org/" class="externalLink" title="Metron-Incubating">
- Metron-Incubating</a>
+ <a href="http://metron.apache.org/" class="externalLink" title="Metron">
+ Metron</a>
</li>
<li class="divider ">/</li>
<li class="">
@@ -64,8 +61,8 @@
- <li id="publishDate" class="pull-right">Last Published: 2017-02-23</li> <li class="divider pull-right">|</li>
- <li id="projectVersion" class="pull-right">Version: 0.3.1</li>
+ <li id="publishDate" class="pull-right">Last Published: 2017-06-27</li> <li class="divider pull-right">|</li>
+ <li id="projectVersion" class="pull-right">Version: 0.4.0</li>
</ul>
</div>
@@ -78,7 +75,7 @@
<ul class="nav nav-list">
<li class="nav-header">User Documentation</li>
-
+
<li>
<a href="../../index.html" title="Metron">
@@ -99,7 +96,7 @@
<i class="icon-chevron-right"></i>
Analytics</a>
</li>
-
+
<li>
<a href="../../metron-deployment/index.html" title="Deployment">
@@ -109,6 +106,20 @@
<li>
+ <a href="../../metron-deployment/Kerberos-ambari-setup.html" title="Kerberos-ambari-setup">
+ <i class="none"></i>
+ Kerberos-ambari-setup</a>
+ </li>
+
+ <li>
+
+ <a href="../../metron-deployment/Kerberos-manual-setup.html" title="Kerberos-manual-setup">
+ <i class="none"></i>
+ Kerberos-manual-setup</a>
+ </li>
+
+ <li>
+
<a href="../../metron-deployment/amazon-ec2/index.html" title="Amazon-ec2">
<i class="none"></i>
Amazon-ec2</a>
@@ -130,11 +141,11 @@
<li>
- <a href="../../metron-deployment/packer-build/index.html" title="Packer-build">
+ <a href="../../metron-deployment/packaging/packer-build/index.html" title="Packer-build">
<i class="none"></i>
Packer-build</a>
</li>
-
+
<li>
<a href="../../metron-deployment/roles/index.html" title="Roles">
@@ -185,14 +196,28 @@
<i class="none"></i>
Docker</a>
</li>
-
+
+ <li>
+
+ <a href="../../metron-interface/metron-config/index.html" title="Config">
+ <i class="none"></i>
+ Config</a>
+ </li>
+
+ <li>
+
+ <a href="../../metron-interface/metron-rest/index.html" title="Rest">
+ <i class="none"></i>
+ Rest</a>
+ </li>
+
<li>
<a href="../../metron-platform/index.html" title="Platform">
<i class="icon-chevron-right"></i>
Platform</a>
</li>
-
+
<li>
<a href="../../metron-sensors/index.html" title="Sensors">
@@ -242,8 +267,9 @@
<footer>
<div class="container-fluid">
- <div class="row span12">Copyright © 2017.
- All Rights Reserved.
+ <div class="row span12">Copyright © 2017
+ <a href="https://www.apache.org">The Apache Software Foundation</a>.
+ All Rights Reserved.
</div>
http://git-wip-us.apache.org/repos/asf/metron/blob/cf4c2ecd/current-book/metron-deployment/vagrant/quick-dev-platform/index.html
----------------------------------------------------------------------
diff --git a/current-book/metron-deployment/vagrant/quick-dev-platform/index.html b/current-book/metron-deployment/vagrant/quick-dev-platform/index.html
index 123efc6..227ccd4 100644
--- a/current-book/metron-deployment/vagrant/quick-dev-platform/index.html
+++ b/current-book/metron-deployment/vagrant/quick-dev-platform/index.html
@@ -1,13 +1,13 @@
<!DOCTYPE html>
<!--
- | Generated by Apache Maven Doxia at 2017-02-23
+ | Generated by Apache Maven Doxia at 2017-06-27
| Rendered using Apache Maven Fluido Skin 1.3.0
-->
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
- <meta name="Date-Revision-yyyymmdd" content="20170223" />
+ <meta name="Date-Revision-yyyymmdd" content="20170627" />
<meta http-equiv="Content-Language" content="en" />
<title>Metron – Quick Development Platform</title>
<link rel="stylesheet" href="../../../css/apache-maven-fluido-1.3.0.min.css" />
@@ -30,14 +30,11 @@
<div class="container-fluid">
<div id="banner">
<div class="pull-left">
- <a href="http://metron.incubator.apache.org/" id="bannerLeft">
- <img src="../../../images/metron-logo.png" alt="Apache Metron - Incubating" width="148px" height="48px"/>
+ <a href="http://metron.apache.org/" id="bannerLeft">
+ <img src="../../../images/metron-logo.png" alt="Apache Metron" width="148px" height="48px"/>
</a>
</div>
- <div class="pull-right"> <a href="http://incubator.apache.org/" id="bannerRight">
- <img src="../../../images/ApacheIncubating_Logo.png" alt="Apache Incubating" width="192px" height="48px"/>
- </a>
- </div>
+ <div class="pull-right"> </div>
<div class="clear"><hr/></div>
</div>
@@ -51,8 +48,8 @@
</li>
<li class="divider ">/</li>
<li class="">
- <a href="http://metron.incubator.apache.org/" class="externalLink" title="Metron-Incubating">
- Metron-Incubating</a>
+ <a href="http://metron.apache.org/" class="externalLink" title="Metron">
+ Metron</a>
</li>
<li class="divider ">/</li>
<li class="">
@@ -64,8 +61,8 @@
- <li id="publishDate" class="pull-right">Last Published: 2017-02-23</li> <li class="divider pull-right">|</li>
- <li id="projectVersion" class="pull-right">Version: 0.3.1</li>
+ <li id="publishDate" class="pull-right">Last Published: 2017-06-27</li> <li class="divider pull-right">|</li>
+ <li id="projectVersion" class="pull-right">Version: 0.4.0</li>
</ul>
</div>
@@ -78,7 +75,7 @@
<ul class="nav nav-list">
<li class="nav-header">User Documentation</li>
-
+
<li>
<a href="../../../index.html" title="Metron">
@@ -99,7 +96,7 @@
<i class="icon-chevron-right"></i>
Analytics</a>
</li>
-
+
<li>
<a href="../../../metron-deployment/index.html" title="Deployment">
@@ -109,6 +106,20 @@
<li>
+ <a href="../../../metron-deployment/Kerberos-ambari-setup.html" title="Kerberos-ambari-setup">
+ <i class="none"></i>
+ Kerberos-ambari-setup</a>
+ </li>
+
+ <li>
+
+ <a href="../../../metron-deployment/Kerberos-manual-setup.html" title="Kerberos-manual-setup">
+ <i class="none"></i>
+ Kerberos-manual-setup</a>
+ </li>
+
+ <li>
+
<a href="../../../metron-deployment/amazon-ec2/index.html" title="Amazon-ec2">
<i class="none"></i>
Amazon-ec2</a>
@@ -130,11 +141,11 @@
<li>
- <a href="../../../metron-deployment/packer-build/index.html" title="Packer-build">
+ <a href="../../../metron-deployment/packaging/packer-build/index.html" title="Packer-build">
<i class="none"></i>
Packer-build</a>
</li>
-
+
<li>
<a href="../../../metron-deployment/roles/index.html" title="Roles">
@@ -185,14 +196,28 @@
<i class="none"></i>
Docker</a>
</li>
-
+
+ <li>
+
+ <a href="../../../metron-interface/metron-config/index.html" title="Config">
+ <i class="none"></i>
+ Config</a>
+ </li>
+
+ <li>
+
+ <a href="../../../metron-interface/metron-rest/index.html" title="Rest">
+ <i class="none"></i>
+ Rest</a>
+ </li>
+
<li>
<a href="../../../metron-platform/index.html" title="Platform">
<i class="icon-chevron-right"></i>
Platform</a>
</li>
-
+
<li>
<a href="../../../metron-sensors/index.html" title="Sensors">
@@ -233,51 +258,62 @@
<ul>
-<li><a class="externalLink" href="https://github.com/ansible/ansible">Ansible</a> 2.0.0.2</li>
+<li><a class="externalLink" href="https://github.com/ansible/ansible">Ansible</a> (2.0.0.2 or 2.2.2.0)</li>
+
+<li><a class="externalLink" href="https://www.docker.com/community-edition">Docker</a></li>
<li><a class="externalLink" href="https://www.vagrantup.com">Vagrant</a> 1.8.1</li>
+<li><a class="externalLink" href="https://github.com/devopsgroup-io/vagrant-hostmanager">Vagrant Hostmanager Plugin</a> <tt>vagrant plugin install vagrant-hostmanager</tt></li>
+
<li><a class="externalLink" href="https://virtualbox.org">Virtualbox</a> 5.0.16</li>
<li>Python 2.7.11</li>
<li>Maven 3.3.9</li>
-</ul></div>
+</ul>
<div class="section">
-<h3><a name="Deploy_Metron"></a>Deploy Metron</h3>
+<h4><a name="macOS"></a>macOS</h4>
+<p>Any platform that supports these tools is suitable, but the following instructions cover installation on macOS. The easiest means of installing these tools on a Mac is to use the excellent <a class="externalLink" href="http://brew.sh/">Homebrew</a> project.</p>
<ol style="list-style-type: decimal">
<li>
-<p>Build Metron</p>
+<p>Install Homebrew by following the instructions at <a class="externalLink" href="http://brew.sh/">Homebrew</a>.</p></li>
+
+<li>
+<p>Run the following command in a terminal to install all of the required tools.</p>
<div class="source">
<div class="source">
-<pre> cd incubator-metron
- mvn clean package -DskipTests
+<pre> brew cask install vagrant virtualbox java docker
+ brew install maven git
</pre></div></div></li>
<li>
-<p>Install Vagrant Hostmanager.</p>
+<p>Install Ansible by following the instructions <a class="externalLink" href="http://docs.ansible.com/ansible/intro_installation.html#latest-releases-via-pip">here</a>.</p></li>
+</ol></div></div>
+<div class="section">
+<h3><a name="Deploy_Metron"></a>Deploy Metron</h3>
+
+<ol style="list-style-type: decimal">
-<div class="source">
-<div class="source">
-<pre> vagrant plugin install vagrant-hostmanager
-</pre></div></div></li>
+<li>
+<p>Ensure that the Docker service is running.</p></li>
<li>
<p>Deploy Metron</p>
<div class="source">
<div class="source">
-<pre> cd metron-deployment/vagrant/quick-dev-platform
- vagrant up
+<pre>cd metron-deployment/vagrant/quick-dev-platform
+vagrant up
</pre></div></div>
<p>Should the process fail before completing the deployment, the following command will continue the deployment process without re-instantiating the host.</p>
<div class="source">
<div class="source">
-<pre> vagrant provision
+<pre>vagrant provision
</pre></div></div></li>
</ol></div>
<div class="section">
@@ -314,8 +350,9 @@
<footer>
<div class="container-fluid">
- <div class="row span12">Copyright © 2017.
- All Rights Reserved.
+ <div class="row span12">Copyright © 2017
+ <a href="https://www.apache.org">The Apache Software Foundation</a>.
+ All Rights Reserved.
</div>
http://git-wip-us.apache.org/repos/asf/metron/blob/cf4c2ecd/current-book/metron-docker/index.html
----------------------------------------------------------------------
diff --git a/current-book/metron-docker/index.html b/current-book/metron-docker/index.html
index a87945b..5591c9a 100644
--- a/current-book/metron-docker/index.html
+++ b/current-book/metron-docker/index.html
@@ -1,13 +1,13 @@
<!DOCTYPE html>
<!--
- | Generated by Apache Maven Doxia at 2017-02-23
+ | Generated by Apache Maven Doxia at 2017-06-27
| Rendered using Apache Maven Fluido Skin 1.3.0
-->
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
- <meta name="Date-Revision-yyyymmdd" content="20170223" />
+ <meta name="Date-Revision-yyyymmdd" content="20170627" />
<meta http-equiv="Content-Language" content="en" />
<title>Metron – Metron Docker</title>
<link rel="stylesheet" href="../css/apache-maven-fluido-1.3.0.min.css" />
@@ -30,14 +30,11 @@
<div class="container-fluid">
<div id="banner">
<div class="pull-left">
- <a href="http://metron.incubator.apache.org/" id="bannerLeft">
- <img src="../images/metron-logo.png" alt="Apache Metron - Incubating" width="148px" height="48px"/>
+ <a href="http://metron.apache.org/" id="bannerLeft">
+ <img src="../images/metron-logo.png" alt="Apache Metron" width="148px" height="48px"/>
</a>
</div>
- <div class="pull-right"> <a href="http://incubator.apache.org/" id="bannerRight">
- <img src="../images/ApacheIncubating_Logo.png" alt="Apache Incubating" width="192px" height="48px"/>
- </a>
- </div>
+ <div class="pull-right"> </div>
<div class="clear"><hr/></div>
</div>
@@ -51,8 +48,8 @@
</li>
<li class="divider ">/</li>
<li class="">
- <a href="http://metron.incubator.apache.org/" class="externalLink" title="Metron-Incubating">
- Metron-Incubating</a>
+ <a href="http://metron.apache.org/" class="externalLink" title="Metron">
+ Metron</a>
</li>
<li class="divider ">/</li>
<li class="">
@@ -64,8 +61,8 @@
- <li id="publishDate" class="pull-right">Last Published: 2017-02-23</li> <li class="divider pull-right">|</li>
- <li id="projectVersion" class="pull-right">Version: 0.3.1</li>
+ <li id="publishDate" class="pull-right">Last Published: 2017-06-27</li> <li class="divider pull-right">|</li>
+ <li id="projectVersion" class="pull-right">Version: 0.4.0</li>
</ul>
</div>
@@ -78,7 +75,7 @@
<ul class="nav nav-list">
<li class="nav-header">User Documentation</li>
-
+
<li>
<a href="../index.html" title="Metron">
@@ -99,7 +96,7 @@
<i class="icon-chevron-right"></i>
Analytics</a>
</li>
-
+
<li>
<a href="../metron-deployment/index.html" title="Deployment">
@@ -111,14 +108,28 @@
<a href="#"><i class="none"></i>Docker</a>
</li>
-
+
+ <li>
+
+ <a href="../metron-interface/metron-config/index.html" title="Config">
+ <i class="none"></i>
+ Config</a>
+ </li>
+
+ <li>
+
+ <a href="../metron-interface/metron-rest/index.html" title="Rest">
+ <i class="none"></i>
+ Rest</a>
+ </li>
+
<li>
<a href="../metron-platform/index.html" title="Platform">
<i class="icon-chevron-right"></i>
Platform</a>
</li>
-
+
<li>
<a href="../metron-sensors/index.html" title="Sensors">
@@ -172,6 +183,8 @@
<li>Elasticsearch</li>
<li>Kibana</li>
+
+<li>HDFS</li>
</ul>
<div class="section">
<h2><a name="Setup"></a>Setup</h2>
@@ -249,9 +262,11 @@ metron-machine * virtualbox Running tcp://192.168.99.100:2376
<li>HBase - <a class="externalLink" href="http://192.168.99.100:16010/">http://192.168.99.100:16010/</a></li>
-<li>Elasticsearch - <a class="externalLink" href="http://192.168.99.100:9200/">http://192.168.99.100:9200/</a></li>
+<li>Elasticsearch - <a class="externalLink" href="http://192.168.99.100:9200/_plugin/head/">http://192.168.99.100:9200/_plugin/head/</a></li>
<li>Kibana - <a class="externalLink" href="http://192.168.99.100:5601/">http://192.168.99.100:5601/</a></li>
+
+<li>HDFS (Namenode) - <a class="externalLink" href="http://192.168.99.100:50070/">http://192.168.99.100:50070/</a></li>
</ul>
<p>The Storm logs can be useful when troubleshooting topologies. They can be found on the Storm container in <tt>/usr/share/apache-storm/logs</tt>.</p>
<p>When done using the machine, shut it down with:</p>
@@ -457,8 +472,9 @@ yellow open bro_index_2016.12.19.18 5 1 180 0 475kb
<footer>
<div class="container-fluid">
- <div class="row span12">Copyright © 2017.
- All Rights Reserved.
+ <div class="row span12">Copyright © 2017
+ <a href="https://www.apache.org">The Apache Software Foundation</a>.
+ All Rights Reserved.
</div>
http://git-wip-us.apache.org/repos/asf/metron/blob/cf4c2ecd/current-book/metron-interface/metron-config/index.html
----------------------------------------------------------------------
diff --git a/current-book/metron-interface/metron-config/index.html b/current-book/metron-interface/metron-config/index.html
new file mode 100644
index 0000000..ad4b320
--- /dev/null
+++ b/current-book/metron-interface/metron-config/index.html
@@ -0,0 +1,328 @@
+<!DOCTYPE html>
+<!--
+ | Generated by Apache Maven Doxia at 2017-06-27
+ | Rendered using Apache Maven Fluido Skin 1.3.0
+-->
+<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
+ <head>
+ <meta charset="UTF-8" />
+ <meta name="viewport" content="width=device-width, initial-scale=1.0" />
+ <meta name="Date-Revision-yyyymmdd" content="20170627" />
+ <meta http-equiv="Content-Language" content="en" />
+ <title>Metron – Metron Management UI</title>
+ <link rel="stylesheet" href="../../css/apache-maven-fluido-1.3.0.min.css" />
+ <link rel="stylesheet" href="../../css/site.css" />
+ <link rel="stylesheet" href="../../css/print.css" media="print" />
+
+
+ <script type="text/javascript" src="../../js/apache-maven-fluido-1.3.0.min.js"></script>
+
+
+
+<script type="text/javascript">$( document ).ready( function() { $( '.carousel' ).carousel( { interval: 3500 } ) } );</script>
+
+ </head>
+ <body class="topBarDisabled">
+
+
+
+
+ <div class="container-fluid">
+ <div id="banner">
+ <div class="pull-left">
+ <a href="http://metron.apache.org/" id="bannerLeft">
+ <img src="../../images/metron-logo.png" alt="Apache Metron" width="148px" height="48px"/>
+ </a>
+ </div>
+ <div class="pull-right"> </div>
+ <div class="clear"><hr/></div>
+ </div>
+
+ <div id="breadcrumbs">
+ <ul class="breadcrumb">
+
+
+ <li class="">
+ <a href="http://www.apache.org" class="externalLink" title="Apache">
+ Apache</a>
+ </li>
+ <li class="divider ">/</li>
+ <li class="">
+ <a href="http://metron.apache.org/" class="externalLink" title="Metron">
+ Metron</a>
+ </li>
+ <li class="divider ">/</li>
+ <li class="">
+ <a href="../../index.html" title="Documentation">
+ Documentation</a>
+ </li>
+ <li class="divider ">/</li>
+ <li class="">Metron Management UI</li>
+
+
+
+ <li id="publishDate" class="pull-right">Last Published: 2017-06-27</li> <li class="divider pull-right">|</li>
+ <li id="projectVersion" class="pull-right">Version: 0.4.0</li>
+
+ </ul>
+ </div>
+
+
+ <div class="row-fluid">
+ <div id="leftColumn" class="span3">
+ <div class="well sidebar-nav">
+
+
+ <ul class="nav nav-list">
+ <li class="nav-header">User Documentation</li>
+
+ <li>
+
+ <a href="../../index.html" title="Metron">
+ <i class="icon-chevron-down"></i>
+ Metron</a>
+ <ul class="nav nav-list">
+
+ <li>
+
+ <a href="../../Upgrading.html" title="Upgrading">
+ <i class="none"></i>
+ Upgrading</a>
+ </li>
+
+ <li>
+
+ <a href="../../metron-analytics/index.html" title="Analytics">
+ <i class="icon-chevron-right"></i>
+ Analytics</a>
+ </li>
+
+ <li>
+
+ <a href="../../metron-deployment/index.html" title="Deployment">
+ <i class="icon-chevron-right"></i>
+ Deployment</a>
+ </li>
+
+ <li>
+
+ <a href="../../metron-docker/index.html" title="Docker">
+ <i class="none"></i>
+ Docker</a>
+ </li>
+
+ <li class="active">
+
+ <a href="#"><i class="none"></i>Config</a>
+ </li>
+
+ <li>
+
+ <a href="../../metron-interface/metron-rest/index.html" title="Rest">
+ <i class="none"></i>
+ Rest</a>
+ </li>
+
+ <li>
+
+ <a href="../../metron-platform/index.html" title="Platform">
+ <i class="icon-chevron-right"></i>
+ Platform</a>
+ </li>
+
+ <li>
+
+ <a href="../../metron-sensors/index.html" title="Sensors">
+ <i class="icon-chevron-right"></i>
+ Sensors</a>
+ </li>
+ </ul>
+ </li>
+ </ul>
+
+
+
+ <hr class="divider" />
+
+ <div id="poweredBy">
+ <div class="clear"></div>
+ <div class="clear"></div>
+ <div class="clear"></div>
+ <a href="http://maven.apache.org/" title="Built by Maven" class="poweredBy">
+ <img class="builtBy" alt="Built by Maven" src="../../images/logos/maven-feather.png" />
+ </a>
+ </div>
+ </div>
+ </div>
+
+
+ <div id="bodyColumn" class="span9" >
+
+ <h1>Metron Management UI</h1>
+<p><a name="Metron_Management_UI"></a></p>
+<p>This module provides a user interface for management functions in Metron.</p>
+<div class="section">
+<h2><a name="Prerequisites"></a>Prerequisites</h2>
+
+<ul>
+
+<li>A network accessible Metron REST application</li>
+
+<li>nodejs v6.9+ (nodejs can be installed on quick dev with <tt>curl --silent --location https://rpm.nodesource.com/setup_6.x | bash - && yum install -y nodejs</tt>)</li>
+</ul></div>
+<div class="section">
+<h2><a name="Installation"></a>Installation</h2>
+<div class="section">
+<h3><a name="From_Source"></a>From Source</h3>
+
+<ol style="list-style-type: decimal">
+
+<li>
+<p>Package the application with Maven:</p>
+
+<div class="source">
+<div class="source">
+<pre>cd metron-interface/metron-config
+mvn clean package
+</pre></div></div></li>
+
+<li>
+<p>Untar the archive in the $METRON_HOME directory. The directory structure will look like:</p>
+
+<div class="source">
+<div class="source">
+<pre>bin
+ metron-management-ui
+web
+ expressjs
+ package.json
+ server.js
+ management-ui
+ web assets (html, css, js, ...)
+</pre></div></div></li>
+
+<li>
+<p>Copy the <tt>$METRON_HOME/bin/metron-management-ui</tt> script to <tt>/etc/init.d/metron-management-ui</tt></p></li>
+
+<li>
+<p>Install the <a class="externalLink" href="https://expressjs.com/">Express</a> web framework from the <tt>package.json</tt> file in <tt>$METRON_HOME/web/expressjs</tt>:</p>
+
+<div class="source">
+<div class="source">
+<pre>npm --prefix $METRON_HOME/web/expressjs/ install
+</pre></div></div></li>
+</ol></div>
+<div class="section">
+<h3><a name="From_Package_Manager"></a>From Package Manager</h3>
+
+<ol style="list-style-type: decimal">
+
+<li>
+<p>Deploy the RPM at <tt>/metron/metron-deployment/packaging/docker/rpm-docker/target/RPMS/noarch/metron-config-$METRON_VERSION-*.noarch.rpm</tt></p></li>
+
+<li>
+<p>Install the RPM with:</p>
+
+<div class="source">
+<div class="source">
+<pre>rpm -ih metron-config-$METRON_VERSION-*.noarch.rpm
+</pre></div></div></li>
+
+<li>
+<p>Install the <a class="externalLink" href="https://expressjs.com/">Express</a> web framework from the <tt>package.json</tt> file in <tt>$METRON_HOME/web/expressjs</tt>:</p>
+
+<div class="source">
+<div class="source">
+<pre>npm --prefix $METRON_HOME/web/expressjs/ install
+</pre></div></div></li>
+</ol></div></div>
+<div class="section">
+<h2><a name="Configuration"></a>Configuration</h2>
+<p>The Managment UI is configured in the <tt>$METRON_HOME/config/management_ui.yml</tt> file. Create this file and set the values to match your environment:</p>
+
+<div class="source">
+<div class="source">
+<pre>port: port the managment UI will run on
+
+rest:
+ host: REST application host
+ port: REST applciation port
+</pre></div></div></div>
+<div class="section">
+<h2><a name="Usage"></a>Usage</h2>
+<p>After configuration is complete, the Management UI can be managed as a service:</p>
+
+<div class="source">
+<div class="source">
+<pre>service metron-management-ui start
+</pre></div></div>
+<p>The application will be available at <a class="externalLink" href="http://host:4200">http://host:4200</a> assuming the port is set to <tt>4200</tt>. Logs can be found at <tt>/var/log/metron/metron-management-ui.log</tt>.</p></div>
+<div class="section">
+<h2><a name="Development"></a>Development</h2>
+<p>The Management UI can also be started in development mode. This allows changes to web assets to be seen interactively.</p>
+
+<ol style="list-style-type: decimal">
+
+<li>
+<p>Install the application with dev dependencies:</p>
+
+<div class="source">
+<div class="source">
+<pre>cd metron-interface/metron-config
+npm install
+</pre></div></div></li>
+
+<li>
+<p>Start the application:</p>
+
+<div class="source">
+<div class="source">
+<pre>./scripts/start_dev.sh
+</pre></div></div></li>
+</ol>
+<p>The application will be available at <a class="externalLink" href="http://localhost:4200/">http://localhost:4200/</a>. The REST application url defaults to <tt>http://localhost:8080</tt> but can be changed in the <tt>proxy.conf.json</tt> file.</p></div>
+<div class="section">
+<h2><a name="Testing"></a>Testing</h2>
+
+<ol style="list-style-type: decimal">
+
+<li>
+<p>Install the application with dev dependencies:</p>
+
+<div class="source">
+<div class="source">
+<pre>cd metron-interface/metron-config
+npm install
+</pre></div></div></li>
+
+<li>
+<p>Unit tests can be run with:</p>
+
+<div class="source">
+<div class="source">
+<pre>npm test
+</pre></div></div></li>
+</ol></div>
+<div class="section">
+<h2><a name="License"></a>License</h2>
+<p>This projects bundles Font Awesome which is available under the SIL Open Font License. See <a class="externalLink" href="http://fontawesome.io/license/">http://fontawesome.io/license/</a> for more details.</p></div>
+ </div>
+ </div>
+ </div>
+
+ <hr/>
+
+ <footer>
+ <div class="container-fluid">
+ <div class="row span12">Copyright © 2017
+ <a href="https://www.apache.org">The Apache Software Foundation</a>.
+ All Rights Reserved.
+
+ </div>
+
+
+
+ </div>
+ </footer>
+ </body>
+</html>
[08/10] metron git commit: METRON-1012 Update Metron public web site
for 0.4.0 release - push to asf-site from the generated code on master
Posted by ma...@apache.org.
http://git-wip-us.apache.org/repos/asf/metron/blob/cf4c2ecd/current-book/metron-deployment/Kerberos-ambari-setup.html
----------------------------------------------------------------------
diff --git a/current-book/metron-deployment/Kerberos-ambari-setup.html b/current-book/metron-deployment/Kerberos-ambari-setup.html
new file mode 100644
index 0000000..cd22919
--- /dev/null
+++ b/current-book/metron-deployment/Kerberos-ambari-setup.html
@@ -0,0 +1,275 @@
+<!DOCTYPE html>
+<!--
+ | Generated by Apache Maven Doxia at 2017-06-27
+ | Rendered using Apache Maven Fluido Skin 1.3.0
+-->
+<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
+ <head>
+ <meta charset="UTF-8" />
+ <meta name="viewport" content="width=device-width, initial-scale=1.0" />
+ <meta name="Date-Revision-yyyymmdd" content="20170627" />
+ <meta http-equiv="Content-Language" content="en" />
+ <title>Metron – Setting Up Kerberos in Vagrant Full Dev</title>
+ <link rel="stylesheet" href="../css/apache-maven-fluido-1.3.0.min.css" />
+ <link rel="stylesheet" href="../css/site.css" />
+ <link rel="stylesheet" href="../css/print.css" media="print" />
+
+
+ <script type="text/javascript" src="../js/apache-maven-fluido-1.3.0.min.js"></script>
+
+
+
+<script type="text/javascript">$( document ).ready( function() { $( '.carousel' ).carousel( { interval: 3500 } ) } );</script>
+
+ </head>
+ <body class="topBarDisabled">
+
+
+
+
+ <div class="container-fluid">
+ <div id="banner">
+ <div class="pull-left">
+ <a href="http://metron.apache.org/" id="bannerLeft">
+ <img src="../images/metron-logo.png" alt="Apache Metron" width="148px" height="48px"/>
+ </a>
+ </div>
+ <div class="pull-right"> </div>
+ <div class="clear"><hr/></div>
+ </div>
+
+ <div id="breadcrumbs">
+ <ul class="breadcrumb">
+
+
+ <li class="">
+ <a href="http://www.apache.org" class="externalLink" title="Apache">
+ Apache</a>
+ </li>
+ <li class="divider ">/</li>
+ <li class="">
+ <a href="http://metron.apache.org/" class="externalLink" title="Metron">
+ Metron</a>
+ </li>
+ <li class="divider ">/</li>
+ <li class="">
+ <a href="../index.html" title="Documentation">
+ Documentation</a>
+ </li>
+ <li class="divider ">/</li>
+ <li class="">Setting Up Kerberos in Vagrant Full Dev</li>
+
+
+
+ <li id="publishDate" class="pull-right">Last Published: 2017-06-27</li> <li class="divider pull-right">|</li>
+ <li id="projectVersion" class="pull-right">Version: 0.4.0</li>
+
+ </ul>
+ </div>
+
+
+ <div class="row-fluid">
+ <div id="leftColumn" class="span3">
+ <div class="well sidebar-nav">
+
+
+ <ul class="nav nav-list">
+ <li class="nav-header">User Documentation</li>
+
+ <li>
+
+ <a href="../index.html" title="Metron">
+ <i class="icon-chevron-down"></i>
+ Metron</a>
+ <ul class="nav nav-list">
+
+ <li>
+
+ <a href="../Upgrading.html" title="Upgrading">
+ <i class="none"></i>
+ Upgrading</a>
+ </li>
+
+ <li>
+
+ <a href="../metron-analytics/index.html" title="Analytics">
+ <i class="icon-chevron-right"></i>
+ Analytics</a>
+ </li>
+
+ <li>
+
+ <a href="../metron-deployment/index.html" title="Deployment">
+ <i class="icon-chevron-down"></i>
+ Deployment</a>
+ <ul class="nav nav-list">
+
+ <li class="active">
+
+ <a href="#"><i class="none"></i>Kerberos-ambari-setup</a>
+ </li>
+
+ <li>
+
+ <a href="../metron-deployment/Kerberos-manual-setup.html" title="Kerberos-manual-setup">
+ <i class="none"></i>
+ Kerberos-manual-setup</a>
+ </li>
+
+ <li>
+
+ <a href="../metron-deployment/amazon-ec2/index.html" title="Amazon-ec2">
+ <i class="none"></i>
+ Amazon-ec2</a>
+ </li>
+
+ <li>
+
+ <a href="../metron-deployment/packaging/docker/ansible-docker/index.html" title="Ansible-docker">
+ <i class="none"></i>
+ Ansible-docker</a>
+ </li>
+
+ <li>
+
+ <a href="../metron-deployment/packaging/docker/rpm-docker/index.html" title="Rpm-docker">
+ <i class="none"></i>
+ Rpm-docker</a>
+ </li>
+
+ <li>
+
+ <a href="../metron-deployment/packaging/packer-build/index.html" title="Packer-build">
+ <i class="none"></i>
+ Packer-build</a>
+ </li>
+
+ <li>
+
+ <a href="../metron-deployment/roles/index.html" title="Roles">
+ <i class="icon-chevron-right"></i>
+ Roles</a>
+ </li>
+
+ <li>
+
+ <a href="../metron-deployment/vagrant/index.html" title="Vagrant">
+ <i class="icon-chevron-right"></i>
+ Vagrant</a>
+ </li>
+ </ul>
+ </li>
+
+ <li>
+
+ <a href="../metron-docker/index.html" title="Docker">
+ <i class="none"></i>
+ Docker</a>
+ </li>
+
+ <li>
+
+ <a href="../metron-interface/metron-config/index.html" title="Config">
+ <i class="none"></i>
+ Config</a>
+ </li>
+
+ <li>
+
+ <a href="../metron-interface/metron-rest/index.html" title="Rest">
+ <i class="none"></i>
+ Rest</a>
+ </li>
+
+ <li>
+
+ <a href="../metron-platform/index.html" title="Platform">
+ <i class="icon-chevron-right"></i>
+ Platform</a>
+ </li>
+
+ <li>
+
+ <a href="../metron-sensors/index.html" title="Sensors">
+ <i class="icon-chevron-right"></i>
+ Sensors</a>
+ </li>
+ </ul>
+ </li>
+ </ul>
+
+
+
+ <hr class="divider" />
+
+ <div id="poweredBy">
+ <div class="clear"></div>
+ <div class="clear"></div>
+ <div class="clear"></div>
+ <a href="http://maven.apache.org/" title="Built by Maven" class="poweredBy">
+ <img class="builtBy" alt="Built by Maven" src="../images/logos/maven-feather.png" />
+ </a>
+ </div>
+ </div>
+ </div>
+
+
+ <div id="bodyColumn" class="span9" >
+
+ <h1>Setting Up Kerberos in Vagrant Full Dev</h1>
+<p><a name="Setting_Up_Kerberos_in_Vagrant_Full_Dev"></a></p>
+<p><b>Note:</b> These are instructions for Kerberizing Metron Storm topologies from Kafka to Kafka. This does not cover the sensor connections or MAAS. General Kerberization notes can be found in the metron-deployment <a href="../index.html">README.md</a></p>
+<div class="section">
+<h2><a name="Setup_a_KDC"></a>Setup a KDC</h2>
+<p>See <a href="Kerberos-manual-setup.html#Setup_a_KDC">Setup a KDC</a> and <a href="Kerberos-manual-setup.html#Verify_KDC">Verify KDC</a></p></div>
+<div class="section">
+<h2><a name="Ambari_Setup"></a>Ambari Setup</h2>
+
+<ol style="list-style-type: decimal">
+
+<li>
+<p>Kerberize the cluster via Ambari. More detailed documentation can be found <a class="externalLink" href="http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.5.3/bk_security/content/_enabling_kerberos_security_in_ambari.html">here</a>.</p>
+<p>a. For this exercise, choose existing MIT KDC (this is what we setup and installed in the previous steps.)</p>
+<p><img src="../images/enable-kerberos.png" alt="enable keberos" /></p>
+<p><img src="../images/enable-kerberos-started.png" alt="enable keberos get started" /></p>
+<p>b. Setup Kerberos configuration. Realm is EXAMPLE.COM. The admin principal will end up as admin/admin@EXAMPLE.COM when testing the KDC. Use the password you entered during the step for adding the admin principal.</p>
+<p><img src="../images/enable-kerberos-configure-kerberos.png" alt="enable keberos configure" /></p>
+<p>c. Click through to “Start and Test Services.” Let the cluster spin up.</p></li>
+</ol></div>
+<div class="section">
+<h2><a name="Push_Data"></a>Push Data</h2>
+
+<ol style="list-style-type: decimal">
+
+<li>
+<p>Kinit with the metron user</p>
+
+<div class="source">
+<div class="source">
+<pre>kinit -kt /etc/security/keytabs/metron.headless.keytab metron@EXAMPLE.COM
+</pre></div></div></li>
+</ol>
+<p>See <a href="Kerberos-manual-setup.html#Push_Data">Push Data</a></p>
+<div class="section">
+<h3><a name="More_Information"></a>More Information</h3>
+<p>See <a href="Kerberos-manual-setup.html#More_Information">More Information</a></p></div></div>
+ </div>
+ </div>
+ </div>
+
+ <hr/>
+
+ <footer>
+ <div class="container-fluid">
+ <div class="row span12">Copyright © 2017
+ <a href="https://www.apache.org">The Apache Software Foundation</a>.
+ All Rights Reserved.
+
+ </div>
+
+
+
+ </div>
+ </footer>
+ </body>
+</html>
http://git-wip-us.apache.org/repos/asf/metron/blob/cf4c2ecd/current-book/metron-deployment/Kerberos-manual-setup.html
----------------------------------------------------------------------
diff --git a/current-book/metron-deployment/Kerberos-manual-setup.html b/current-book/metron-deployment/Kerberos-manual-setup.html
new file mode 100644
index 0000000..49d003b
--- /dev/null
+++ b/current-book/metron-deployment/Kerberos-manual-setup.html
@@ -0,0 +1,803 @@
+<!DOCTYPE html>
+<!--
+ | Generated by Apache Maven Doxia at 2017-06-27
+ | Rendered using Apache Maven Fluido Skin 1.3.0
+-->
+<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
+ <head>
+ <meta charset="UTF-8" />
+ <meta name="viewport" content="width=device-width, initial-scale=1.0" />
+ <meta name="Date-Revision-yyyymmdd" content="20170627" />
+ <meta http-equiv="Content-Language" content="en" />
+ <title>Metron – Kerberos Setup</title>
+ <link rel="stylesheet" href="../css/apache-maven-fluido-1.3.0.min.css" />
+ <link rel="stylesheet" href="../css/site.css" />
+ <link rel="stylesheet" href="../css/print.css" media="print" />
+
+
+ <script type="text/javascript" src="../js/apache-maven-fluido-1.3.0.min.js"></script>
+
+
+
+<script type="text/javascript">$( document ).ready( function() { $( '.carousel' ).carousel( { interval: 3500 } ) } );</script>
+
+ </head>
+ <body class="topBarDisabled">
+
+
+
+
+ <div class="container-fluid">
+ <div id="banner">
+ <div class="pull-left">
+ <a href="http://metron.apache.org/" id="bannerLeft">
+ <img src="../images/metron-logo.png" alt="Apache Metron" width="148px" height="48px"/>
+ </a>
+ </div>
+ <div class="pull-right"> </div>
+ <div class="clear"><hr/></div>
+ </div>
+
+ <div id="breadcrumbs">
+ <ul class="breadcrumb">
+
+
+ <li class="">
+ <a href="http://www.apache.org" class="externalLink" title="Apache">
+ Apache</a>
+ </li>
+ <li class="divider ">/</li>
+ <li class="">
+ <a href="http://metron.apache.org/" class="externalLink" title="Metron">
+ Metron</a>
+ </li>
+ <li class="divider ">/</li>
+ <li class="">
+ <a href="../index.html" title="Documentation">
+ Documentation</a>
+ </li>
+ <li class="divider ">/</li>
+ <li class="">Kerberos Setup</li>
+
+
+
+ <li id="publishDate" class="pull-right">Last Published: 2017-06-27</li> <li class="divider pull-right">|</li>
+ <li id="projectVersion" class="pull-right">Version: 0.4.0</li>
+
+ </ul>
+ </div>
+
+
+ <div class="row-fluid">
+ <div id="leftColumn" class="span3">
+ <div class="well sidebar-nav">
+
+
+ <ul class="nav nav-list">
+ <li class="nav-header">User Documentation</li>
+
+ <li>
+
+ <a href="../index.html" title="Metron">
+ <i class="icon-chevron-down"></i>
+ Metron</a>
+ <ul class="nav nav-list">
+
+ <li>
+
+ <a href="../Upgrading.html" title="Upgrading">
+ <i class="none"></i>
+ Upgrading</a>
+ </li>
+
+ <li>
+
+ <a href="../metron-analytics/index.html" title="Analytics">
+ <i class="icon-chevron-right"></i>
+ Analytics</a>
+ </li>
+
+ <li>
+
+ <a href="../metron-deployment/index.html" title="Deployment">
+ <i class="icon-chevron-down"></i>
+ Deployment</a>
+ <ul class="nav nav-list">
+
+ <li>
+
+ <a href="../metron-deployment/Kerberos-ambari-setup.html" title="Kerberos-ambari-setup">
+ <i class="none"></i>
+ Kerberos-ambari-setup</a>
+ </li>
+
+ <li class="active">
+
+ <a href="#"><i class="none"></i>Kerberos-manual-setup</a>
+ </li>
+
+ <li>
+
+ <a href="../metron-deployment/amazon-ec2/index.html" title="Amazon-ec2">
+ <i class="none"></i>
+ Amazon-ec2</a>
+ </li>
+
+ <li>
+
+ <a href="../metron-deployment/packaging/docker/ansible-docker/index.html" title="Ansible-docker">
+ <i class="none"></i>
+ Ansible-docker</a>
+ </li>
+
+ <li>
+
+ <a href="../metron-deployment/packaging/docker/rpm-docker/index.html" title="Rpm-docker">
+ <i class="none"></i>
+ Rpm-docker</a>
+ </li>
+
+ <li>
+
+ <a href="../metron-deployment/packaging/packer-build/index.html" title="Packer-build">
+ <i class="none"></i>
+ Packer-build</a>
+ </li>
+
+ <li>
+
+ <a href="../metron-deployment/roles/index.html" title="Roles">
+ <i class="icon-chevron-right"></i>
+ Roles</a>
+ </li>
+
+ <li>
+
+ <a href="../metron-deployment/vagrant/index.html" title="Vagrant">
+ <i class="icon-chevron-right"></i>
+ Vagrant</a>
+ </li>
+ </ul>
+ </li>
+
+ <li>
+
+ <a href="../metron-docker/index.html" title="Docker">
+ <i class="none"></i>
+ Docker</a>
+ </li>
+
+ <li>
+
+ <a href="../metron-interface/metron-config/index.html" title="Config">
+ <i class="none"></i>
+ Config</a>
+ </li>
+
+ <li>
+
+ <a href="../metron-interface/metron-rest/index.html" title="Rest">
+ <i class="none"></i>
+ Rest</a>
+ </li>
+
+ <li>
+
+ <a href="../metron-platform/index.html" title="Platform">
+ <i class="icon-chevron-right"></i>
+ Platform</a>
+ </li>
+
+ <li>
+
+ <a href="../metron-sensors/index.html" title="Sensors">
+ <i class="icon-chevron-right"></i>
+ Sensors</a>
+ </li>
+ </ul>
+ </li>
+ </ul>
+
+
+
+ <hr class="divider" />
+
+ <div id="poweredBy">
+ <div class="clear"></div>
+ <div class="clear"></div>
+ <div class="clear"></div>
+ <a href="http://maven.apache.org/" title="Built by Maven" class="poweredBy">
+ <img class="builtBy" alt="Built by Maven" src="../images/logos/maven-feather.png" />
+ </a>
+ </div>
+ </div>
+ </div>
+
+
+ <div id="bodyColumn" class="span9" >
+
+ <h1>Kerberos Setup</h1>
+<p>This document provides instructions for kerberizing Metron’s Vagrant-based development environments; “Quick Dev” and “Full Dev”. These instructions do not cover the Ambari MPack or sensors. General Kerberization notes can be found in the metron-deployment <a href="../index.html">README.md</a>.</p>
+
+<ul>
+
+<li><a href="#Setup">Setup</a></li>
+
+<li><a href="#Setup_a_KDC">Setup a KDC</a></li>
+
+<li><a href="#Verify_KDC">Verify KDC</a></li>
+
+<li><a href="#Enable_Kerberos">Enable Kerberos</a></li>
+
+<li><a href="#Kafka_Authorization">Kafka Authorization</a></li>
+
+<li><a href="#HBase_Authorization">HBase Authorization</a></li>
+
+<li><a href="#Storm_Authorization">Storm Authorization</a></li>
+
+<li><a href="#Start_Metron">Start Metron</a></li>
+
+<li><a href="#Push_Data">Push Data</a></li>
+
+<li><a href="#More_Information">More Information</a></li>
+</ul>
+<div class="section">
+<h2><a name="Setup"></a>Setup</h2>
+
+<ol style="list-style-type: decimal">
+
+<li>
+<p>Deploy a Vagrant development environment; either <a href="vagrant/full-dev-platform/index.html">Full Dev</a> or <a href="vagrant/quick-dev-platform/index.html">Quick Dev</a>.</p></li>
+
+<li>
+<p>Export the following environment variables. These need to be set for the remainder of the instructions. Replace <tt>node1</tt> with the appropriate hosts, if you are running Metron anywhere other than Vagrant.</p>
+
+<div class="source">
+<div class="source">
+<pre># execute as root
+sudo su -
+export KAFKA_HOME="/usr/hdp/current/kafka-broker"
+export ZOOKEEPER=node1:2181
+export ELASTICSEARCH=node1:9200
+export BROKERLIST=node1:6667
+export HDP_HOME="/usr/hdp/current"
+export KAFKA_HOME="${HDP_HOME}/kafka-broker"
+export METRON_VERSION="0.4.0"
+export METRON_HOME="/usr/metron/${METRON_VERSION}"
+</pre></div></div></li>
+
+<li>
+<p>Execute the following commands as root.</p>
+
+<div class="source">
+<div class="source">
+<pre>sudo su -
+</pre></div></div></li>
+
+<li>
+<p>Stop all Metron topologies. They will be restarted again once Kerberos has been enabled.</p>
+
+<div class="source">
+<div class="source">
+<pre>for topology in bro snort enrichment indexing; do
+ storm kill $topology;
+done
+</pre></div></div></li>
+
+<li>
+<p>Create the <tt>metron</tt> user’s home directory in HDFS.</p>
+
+<div class="source">
+<div class="source">
+<pre>sudo -u hdfs hdfs dfs -mkdir /user/metron
+sudo -u hdfs hdfs dfs -chown metron:hdfs /user/metron
+sudo -u hdfs hdfs dfs -chmod 770 /user/metron
+</pre></div></div></li>
+</ol></div>
+<div class="section">
+<h2><a name="Setup_a_KDC"></a>Setup a KDC</h2>
+
+<ol style="list-style-type: decimal">
+
+<li>
+<p>Install dependencies.</p>
+
+<div class="source">
+<div class="source">
+<pre>yum -y install krb5-server krb5-libs krb5-workstation
+</pre></div></div></li>
+
+<li>
+<p>Define the host, <tt>node1</tt>, as the KDC.</p>
+
+<div class="source">
+<div class="source">
+<pre>sed -i 's/kerberos.example.com/node1/g' /etc/krb5.conf
+cp -f /etc/krb5.conf /var/lib/ambari-server/resources/scripts
+</pre></div></div></li>
+
+<li>
+<p>Ensure the KDC can issue renewable tickets. This can be necessary on a real cluster, but should not be on full-dev. In /var/kerberos/krb5kdc/kdc.conf ensure the following is in the realm section</p>
+
+<div class="source">
+<div class="source">
+<pre>max_renewable_life = 7d
+</pre></div></div></li>
+
+<li>
+<p>Do not copy/paste this full set of commands as the <tt>kdb5_util</tt> command will not run as expected. Run the commands individually to ensure they all execute. This step takes a moment. It creates the kerberos database.</p>
+
+<div class="source">
+<div class="source">
+<pre>kdb5_util create -s
+/etc/rc.d/init.d/krb5kdc start
+chkconfig krb5kdc on
+/etc/rc.d/init.d/kadmin start
+chkconfig kadmin on
+</pre></div></div></li>
+
+<li>
+<p>Setup the <tt>admin</tt> and <tt>metron</tt> principals. You’ll <tt>kinit</tt> as the <tt>metron</tt> principal when running topologies. Make sure to remember the passwords.</p>
+
+<div class="source">
+<div class="source">
+<pre>kadmin.local -q "addprinc admin/admin"
+kadmin.local -q "addprinc metron"
+</pre></div></div></li>
+</ol></div>
+<div class="section">
+<h2><a name="Verify_KDC"></a>Verify KDC</h2>
+<p>Ticket renewal is by default disallowed in many linux distributions. If the KDC cannot issue renewable tickets, an error will be thrown when starting Metron’s Storm topologies:</p>
+
+<div class="source">
+<div class="source">
+<pre>Exception in thread "main" java.lang.RuntimeException: java.lang.RuntimeException: The TGT found is not renewable
+</pre></div></div>
+<p>Ensure the Metron keytab is renewable. Look for the ‘R’ flag from the following command</p>
+
+<div class="source">
+<div class="source">
+<pre>klist -f
+</pre></div></div>
+<p>If the ‘R’ flags are present, you may skip to next section.</p>
+<p>If the ‘R’ flags are absent, you will need to follow the below steps: If the KDC is already setup, then editing max_life and max_renewable_life in <tt>/var/kerberos/krb5kdc/kdc.conf</tt>, and restarting kadmin and krb5kdc services will not change the policies for existing users. </p>
+<p>You need to set the renew lifetime for existing users and krbtgt realm. Modify the appropriate principals to allow renewable tickets using the following commands. Adjust the parameters to match your desired KDC parameters:</p>
+
+<div class="source">
+<div class="source">
+<pre>kadmin.local -q "modprinc -maxlife 1days -maxrenewlife 7days +allow_renewable krbtgt/EXAMPLE.COM@EXAMPLE.COM"
+kadmin.local -q "modprinc -maxlife 1days -maxrenewlife 7days +allow_renewable metron@EXAMPLE.COM"
+</pre></div></div></div>
+<div class="section">
+<h2><a name="Enable_Kerberos"></a>Enable Kerberos</h2>
+
+<ol style="list-style-type: decimal">
+
+<li>
+<p>In <a class="externalLink" href="http://node1:8080">Ambari</a>, setup Storm to use Kerberos and run worker jobs as the submitting user.</p>
+<p>a. Add the following properties to the custom storm-site:</p>
+
+<div class="source">
+<div class="source">
+<pre>topology.auto-credentials=['org.apache.storm.security.auth.kerberos.AutoTGT']
+nimbus.credential.renewers.classes=['org.apache.storm.security.auth.kerberos.AutoTGT']
+supervisor.run.worker.as.user=true
+</pre></div></div>
+<p>b. In the Storm config section in Ambari, choose “Add Property” under custom storm-site:</p>
+<p><img src="../images/ambari-storm-site.png" alt="custom storm-site" /></p>
+<p>c. In the dialog window, choose the “bulk property add mode” toggle button and add the below values:</p>
+<p><img src="../images/ambari-storm-site-properties.png" alt="custom storm-site properties" /></p></li>
+
+<li>
+<p>Kerberize the cluster via Ambari. More detailed documentation can be found <a class="externalLink" href="http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.5.3/bk_security/content/_enabling_kerberos_security_in_ambari.html">here</a>.</p>
+<p>a. For this exercise, choose existing MIT KDC (this is what we setup and installed in the previous steps.)</p>
+<p><img src="../images/enable-kerberos.png" alt="enable keberos" /></p>
+<p><img src="../images/enable-kerberos-started.png" alt="enable keberos get started" /></p>
+<p>b. Setup Kerberos configuration. Realm is EXAMPLE.COM. The admin principal will end up as admin/admin@EXAMPLE.COM when testing the KDC. Use the password you entered during the step for adding the admin principal.</p>
+<p><img src="../images/enable-kerberos-configure-kerberos.png" alt="enable keberos configure" /></p>
+<p>c. Click through to “Start and Test Services.” Let the cluster spin up, but don’t worry about starting up Metron via Ambari - we’re going to run the parsers manually against the rest of the Hadoop cluster Kerberized. The wizard will fail at starting Metron, but this is OK. Click “continue.” When you’re finished, the custom storm-site should look similar to the following:</p>
+<p><img src="../images/custom-storm-site-final.png" alt="enable keberos configure" /></p></li>
+
+<li>
+<p>Create a Metron keytab</p>
+
+<div class="source">
+<div class="source">
+<pre>kadmin.local -q "ktadd -k metron.headless.keytab metron@EXAMPLE.COM"
+cp metron.headless.keytab /etc/security/keytabs
+chown metron:hadoop /etc/security/keytabs/metron.headless.keytab
+chmod 440 /etc/security/keytabs/metron.headless.keytab
+</pre></div></div></li>
+</ol></div>
+<div class="section">
+<h2><a name="Kafka_Authorization"></a>Kafka Authorization</h2>
+
+<ol style="list-style-type: decimal">
+
+<li>
+<p>Acquire a Kerberos ticket using the <tt>metron</tt> principal.</p>
+
+<div class="source">
+<div class="source">
+<pre>kinit -kt /etc/security/keytabs/metron.headless.keytab metron@EXAMPLE.COM
+</pre></div></div></li>
+
+<li>
+<p>Create any additional Kafka topics that you will need. We need to create the topics before adding the required ACLs. The current full dev installation will deploy bro, snort, enrichments, and indexing only. For example, you may want to add a topic for ‘yaf’ telemetry.</p>
+
+<div class="source">
+<div class="source">
+<pre>${KAFKA_HOME}/bin/kafka-topics.sh \
+ --zookeeper ${ZOOKEEPER} \
+ --create \
+ --topic yaf \
+ --partitions 1 \
+ --replication-factor 1
+</pre></div></div></li>
+
+<li>
+<p>Setup Kafka ACLs for the <tt>bro</tt>, <tt>snort</tt>, <tt>enrichments</tt>, and <tt>indexing</tt> topics. Run the same command against any additional topics that you might be using; for example <tt>yaf</tt>.</p>
+
+<div class="source">
+<div class="source">
+<pre>export KERB_USER=metron
+for topic in bro snort enrichments indexing; do
+ ${KAFKA_HOME}/bin/kafka-acls.sh \
+ --authorizer kafka.security.auth.SimpleAclAuthorizer \
+ --authorizer-properties zookeeper.connect=${ZOOKEEPER} \
+ --add \
+ --allow-principal User:${KERB_USER} \
+ --topic ${topic}
+done
+</pre></div></div></li>
+
+<li>
+<p>Setup Kafka ACLs for the consumer groups. This command sets the ACLs for Bro, Snort, YAF, Enrichments, Indexing, and the Profiler. Execute the same command for any additional Parsers that you may be running.</p>
+
+<div class="source">
+<div class="source">
+<pre>export KERB_USER=metron
+for group in bro_parser snort_parser yaf_parser enrichments indexing profiler; do
+ ${KAFKA_HOME}/bin/kafka-acls.sh \
+ --authorizer kafka.security.auth.SimpleAclAuthorizer \
+ --authorizer-properties zookeeper.connect=${ZOOKEEPER} \
+ --add \
+ --allow-principal User:${KERB_USER} \
+ --group ${group}
+done
+</pre></div></div></li>
+
+<li>
+<p>Add the <tt>metron</tt> principal to the <tt>kafka-cluster</tt> ACL.</p>
+
+<div class="source">
+<div class="source">
+<pre>${KAFKA_HOME}/bin/kafka-acls.sh \
+ --authorizer kafka.security.auth.SimpleAclAuthorizer \
+ --authorizer-properties zookeeper.connect=${ZOOKEEPER} \
+ --add \
+ --allow-principal User:${KERB_USER} \
+ --cluster kafka-cluster
+</pre></div></div></li>
+</ol></div>
+<div class="section">
+<h2><a name="HBase_Authorization"></a>HBase Authorization</h2>
+
+<ol style="list-style-type: decimal">
+
+<li>
+<p>Acquire a Kerberos ticket using the <tt>hbase</tt> principal</p>
+
+<div class="source">
+<div class="source">
+<pre>kinit -kt /etc/security/keytabs/hbase.headless.keytab hbase-metron_cluster@EXAMPLE.COM
+</pre></div></div></li>
+
+<li>
+<p>Grant permissions for the HBase tables used in Metron.</p>
+
+<div class="source">
+<div class="source">
+<pre>echo "grant 'metron', 'RW', 'threatintel'" | hbase shell
+echo "grant 'metron', 'RW', 'enrichment'" | hbase shell
+</pre></div></div></li>
+
+<li>
+<p>If you are using the Profiler, do the same for its HBase table.</p>
+
+<div class="source">
+<div class="source">
+<pre>echo "create 'profiler', 'P'" | hbase shell
+echo "grant 'metron', 'RW', 'profiler', 'P'" | hbase shell
+</pre></div></div></li>
+</ol></div>
+<div class="section">
+<h2><a name="Storm_Authorization"></a>Storm Authorization</h2>
+
+<ol style="list-style-type: decimal">
+
+<li>
+<p>Switch to the <tt>metron</tt> user and acquire a Kerberos ticket for the <tt>metron</tt> principal.</p>
+
+<div class="source">
+<div class="source">
+<pre>su metron
+kinit -kt /etc/security/keytabs/metron.headless.keytab metron@EXAMPLE.COM
+</pre></div></div></li>
+
+<li>
+<p>Create the directory <tt>/home/metron/.storm</tt> and switch to that directory.</p>
+
+<div class="source">
+<div class="source">
+<pre>mkdir /home/metron/.storm
+cd /home/metron/.storm
+</pre></div></div></li>
+
+<li>
+<p>Ensure the Metron keytab is renewable. See <a href="#Verify_KDC">Verify KDC</a> above.</p></li>
+
+<li>
+<p>Create a client JAAS file at <tt>/home/metron/.storm/client_jaas.conf</tt>. This should look identical to the Storm client JAAS file located at <tt>/etc/storm/conf/client_jaas.conf</tt> except for the addition of a <tt>Client</tt> stanza. The <tt>Client</tt> stanza is used for Zookeeper. All quotes and semicolons are necessary.</p>
+
+<div class="source">
+<div class="source">
+<pre>cat << EOF > client_jaas.conf
+StormClient {
+ com.sun.security.auth.module.Krb5LoginModule required
+ useTicketCache=true
+ renewTicket=true
+ serviceName="nimbus";
+};
+Client {
+ com.sun.security.auth.module.Krb5LoginModule required
+ useKeyTab=true
+ keyTab="/etc/security/keytabs/metron.headless.keytab"
+ storeKey=true
+ useTicketCache=false
+ serviceName="zookeeper"
+ principal="metron@EXAMPLE.COM";
+};
+KafkaClient {
+ com.sun.security.auth.module.Krb5LoginModule required
+ useKeyTab=true
+ keyTab="/etc/security/keytabs/metron.headless.keytab"
+ storeKey=true
+ useTicketCache=false
+ serviceName="kafka"
+ principal="metron@EXAMPLE.COM";
+};
+EOF
+</pre></div></div></li>
+
+<li>
+<p>Create a YAML file at <tt>/home/metron/.storm/storm.yaml</tt>. This should point to the client JAAS file. Set the array of nimbus hosts accordingly.</p>
+
+<div class="source">
+<div class="source">
+<pre>cat << EOF > /home/metron/.storm/storm.yaml
+nimbus.seeds : ['node1']
+java.security.auth.login.config : '/home/metron/.storm/client_jaas.conf'
+storm.thrift.transport : 'org.apache.storm.security.auth.kerberos.KerberosSaslTransportPlugin'
+EOF
+</pre></div></div></li>
+
+<li>
+<p>Create an auxiliary storm configuration file at <tt>/home/metron/storm-config.json</tt>. Note the login config option in the file points to the client JAAS file.</p>
+
+<div class="source">
+<div class="source">
+<pre>cat << EOF > /home/metron/storm-config.json
+{
+ "topology.worker.childopts" : "-Djava.security.auth.login.config=/home/metron/.storm/client_jaas.conf"
+}
+EOF
+</pre></div></div></li>
+
+<li>
+<p>Configure the Enrichment, Indexing and Profiler topologies to use the client JAAS file. To do this, the following key-value pairs:</p>
+
+<ul>
+
+<li><tt>kafka.security.protocol=PLAINTEXTSASL</tt></li>
+
+<li><tt>topology.worker.childopts=-Djava.security.auth.login.config=/home/metron/.storm/client_jaas.conf</tt></li>
+ </ul>
+<p>must be added to each of the topology properties files:</p>
+
+<ul>
+
+<li><tt>${METRON_HOME}/config/enrichment.properties</tt></li>
+
+<li><tt>${METRON_HOME}/config/elasticsearch.properties</tt></li>
+
+<li><tt>${METRON_HOME}/config/profiler.properties</tt></li>
+ </ul>
+<p>You may use the following command to automate this step:</p>
+
+<div class="source">
+<div class="source">
+<pre>for file in enrichment.properties elasticsearch.properties profiler.properties; do
+ echo ${file}
+ sed -i "s/^kafka.security.protocol=.*/kafka.security.protocol=PLAINTEXTSASL/" "${METRON_HOME}/config/${file}"
+ sed -i "s/^topology.worker.childopts=.*/topology.worker.childopts=-Djava.security.auth.login.config=\/home\/metron\/.storm\/client_jaas.conf/" "${METRON_HOME}/config/${file}"
+done
+</pre></div></div></li>
+</ol></div>
+<div class="section">
+<h2><a name="Start_Metron"></a>Start Metron</h2>
+
+<ol style="list-style-type: decimal">
+
+<li>
+<p>Switch to the <tt>metron</tt> user and acquire a Kerberos ticket for the <tt>metron</tt> principal.</p>
+
+<div class="source">
+<div class="source">
+<pre>su metron
+kinit -kt /etc/security/keytabs/metron.headless.keytab metron@EXAMPLE.COM
+</pre></div></div></li>
+
+<li>
+<p>Restart the parser topologies. Be sure to pass in the new parameter, <tt>-ksp</tt> or <tt>--kafka_security_protocol</tt>. The following command will start only the Bro and Snort topologies. Execute the same command for any other Parsers that you may need, for example <tt>yaf</tt>.</p>
+
+<div class="source">
+<div class="source">
+<pre>for parser in bro snort; do
+ ${METRON_HOME}/bin/start_parser_topology.sh \
+ -z ${ZOOKEEPER} \
+ -s ${parser} \
+ -ksp SASL_PLAINTEXT \
+ -e /home/metron/storm-config.json;
+done
+</pre></div></div></li>
+
+<li>
+<p>Restart the Enrichment and Indexing topologies.</p>
+
+<div class="source">
+<div class="source">
+<pre>${METRON_HOME}/bin/start_enrichment_topology.sh
+${METRON_HOME}/bin/start_elasticsearch_topology.sh
+</pre></div></div></li>
+</ol>
+<p>Metron should be ready to receive data.</p></div>
+<div class="section">
+<h2><a name="Push_Data"></a>Push Data</h2>
+
+<ol style="list-style-type: decimal">
+
+<li>
+<p>Push some sample data to one of the parser topics. E.g for Bro we took raw data from <a href="../metron-platform/metron-integration-test/src/main/sample/data/bro/raw/BroExampleOutput/index.html">metron/metron-platform/metron-integration-test/src/main/sample/data/bro/raw/BroExampleOutput</a></p>
+
+<div class="source">
+<div class="source">
+<pre>cat sample-bro.txt | ${KAFKA_HOME}/kafka-broker/bin/kafka-console-producer.sh \
+ --broker-list ${BROKERLIST} \
+ --security-protocol SASL_PLAINTEXT \
+ --topic bro
+</pre></div></div></li>
+
+<li>
+<p>Wait a few moments for data to flow through the system and then check for data in the Elasticsearch indices. Replace yaf with whichever parser type you’ve chosen.</p>
+
+<div class="source">
+<div class="source">
+<pre>curl -XGET "${ELASTICSEARCH}/bro*/_search"
+curl -XGET "${ELASTICSEARCH}/bro*/_count"
+</pre></div></div></li>
+
+<li>
+<p>You should have data flowing from the parsers all the way through to the indexes. This completes the Kerberization instructions</p></li>
+</ol></div>
+<div class="section">
+<h2><a name="More_Information"></a>More Information</h2>
+<div class="section">
+<h3><a name="Kerberos"></a>Kerberos</h3>
+<p>Unsure of your Kerberos principal associated with a keytab? There are a couple ways to get this. One is via the list of principals that Ambari provides via downloadable csv. If you didn’t download this list, you can also check the principal manually by running the following against the keytab.</p>
+
+<div class="source">
+<div class="source">
+<pre>klist -kt /etc/security/keytabs/<keytab-file-name>
+</pre></div></div>
+<p>E.g.</p>
+
+<div class="source">
+<div class="source">
+<pre>klist -kt /etc/security/keytabs/hbase.headless.keytab
+Keytab name: FILE:/etc/security/keytabs/hbase.headless.keytab
+KVNO Timestamp Principal
+---- ----------------- --------------------------------------------------------
+ 1 03/28/17 19:29:36 hbase-metron_cluster@EXAMPLE.COM
+ 1 03/28/17 19:29:36 hbase-metron_cluster@EXAMPLE.COM
+ 1 03/28/17 19:29:36 hbase-metron_cluster@EXAMPLE.COM
+ 1 03/28/17 19:29:36 hbase-metron_cluster@EXAMPLE.COM
+ 1 03/28/17 19:29:36 hbase-metron_cluster@EXAMPLE.COM
+</pre></div></div></div>
+<div class="section">
+<h3><a name="Kafka_with_Kerberos_enabled"></a>Kafka with Kerberos enabled</h3>
+<div class="section">
+<h4><a name="Running_Sensors"></a>Running Sensors</h4>
+<p>A couple steps are required to produce data to a Kerberized Kafka topic. On the host you’ll be setting up your sensor(s), switch to the metron user and create a client_jaas.conf file in the metron home directory if one doesn’t already exist. It should be owned by metron:metron and contain at least the following stanza that tells the Kafka client how to interact with Kerberos:</p>
+
+<div class="source">
+<div class="source">
+<pre>su - metron
+cat ${METRON_HOME}/client_jaas.conf
+...
+KafkaClient {
+ com.sun.security.auth.module.Krb5LoginModule required
+ useKeyTab=true
+ keyTab="/etc/security/keytabs/metron.headless.keytab"
+ storeKey=true
+ useTicketCache=false
+ serviceName="kafka"
+ principal="metron@EXAMPLE.COM";
+};
+</pre></div></div>
+<p>You’ll also need to set KAFKA_OPTS to tell the Kafka client how to interact with Kerberos.</p>
+
+<div class="source">
+<div class="source">
+<pre>export KAFKA_OPTS="-Djava.security.auth.login.config=${METRON_HOME}/client_jaas.conf"
+</pre></div></div>
+<p>For sensors that leverage the Kafka console producer to pipe data into Metron, e.g. Snort and Yaf, you will need to modify the corresponding sensor shell scripts or config to append the SASL security protocol property. <tt>--security-protocol SASL_PLAINTEXT</tt>. Be sure to kinit with the metron user’s keytab before executing the script that starts the sensor.</p>
+<p>More notes can be found in <a href="../metron-sensors/index.html">metron/metron-sensors/README.md</a></p></div>
+<div class="section">
+<h4><a name="Write_data_to_a_topic_with_SASL"></a>Write data to a topic with SASL</h4>
+
+<div class="source">
+<div class="source">
+<pre>cat sample-yaf.txt | ${KAFKA_HOME}/bin/kafka-console-producer.sh \
+ --broker-list ${BROKERLIST} \
+ --security-protocol PLAINTEXTSASL \
+ --topic yaf
+</pre></div></div></div>
+<div class="section">
+<h4><a name="View_topic_data_from_latest_offset_with_SASL"></a>View topic data from latest offset with SASL</h4>
+
+<div class="source">
+<div class="source">
+<pre>${KAFKA_HOME}/bin/kafka-console-consumer.sh \
+ --zookeeper ${ZOOKEEPER} \
+ --security-protocol PLAINTEXTSASL \
+ --topic yaf
+</pre></div></div></div>
+<div class="section">
+<h4><a name="Modify_the_sensor-stubs_to_send_logs_via_SASL"></a>Modify the sensor-stubs to send logs via SASL</h4>
+
+<div class="source">
+<div class="source">
+<pre>sed -i 's/node1:6667 --topic/node1:6667 --security-protocol PLAINTEXTSASL --topic/' /opt/sensor-stubs/bin/start-*-stub
+for sensorstub in bro snort; do
+ service sensor-stubs stop ${sensorstub};
+ service sensor-stubs start ${sensorstub};
+done
+</pre></div></div></div></div>
+<div class="section">
+<h3><a name="References"></a>References</h3>
+
+<ul>
+
+<li><a class="externalLink" href="https://github.com/apache/storm/blob/master/SECURITY.md">https://github.com/apache/storm/blob/master/SECURITY.md</a></li>
+</ul></div></div>
+ </div>
+ </div>
+ </div>
+
+ <hr/>
+
+ <footer>
+ <div class="container-fluid">
+ <div class="row span12">Copyright © 2017
+ <a href="https://www.apache.org">The Apache Software Foundation</a>.
+ All Rights Reserved.
+
+ </div>
+
+
+
+ </div>
+ </footer>
+ </body>
+</html>
http://git-wip-us.apache.org/repos/asf/metron/blob/cf4c2ecd/current-book/metron-deployment/amazon-ec2/index.html
----------------------------------------------------------------------
diff --git a/current-book/metron-deployment/amazon-ec2/index.html b/current-book/metron-deployment/amazon-ec2/index.html
index 8eab79a..0b155b0 100644
--- a/current-book/metron-deployment/amazon-ec2/index.html
+++ b/current-book/metron-deployment/amazon-ec2/index.html
@@ -1,13 +1,13 @@
<!DOCTYPE html>
<!--
- | Generated by Apache Maven Doxia at 2017-02-23
+ | Generated by Apache Maven Doxia at 2017-06-27
| Rendered using Apache Maven Fluido Skin 1.3.0
-->
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
- <meta name="Date-Revision-yyyymmdd" content="20170223" />
+ <meta name="Date-Revision-yyyymmdd" content="20170627" />
<meta http-equiv="Content-Language" content="en" />
<title>Metron – Apache Metron on Amazon EC2</title>
<link rel="stylesheet" href="../../css/apache-maven-fluido-1.3.0.min.css" />
@@ -30,14 +30,11 @@
<div class="container-fluid">
<div id="banner">
<div class="pull-left">
- <a href="http://metron.incubator.apache.org/" id="bannerLeft">
- <img src="../../images/metron-logo.png" alt="Apache Metron - Incubating" width="148px" height="48px"/>
+ <a href="http://metron.apache.org/" id="bannerLeft">
+ <img src="../../images/metron-logo.png" alt="Apache Metron" width="148px" height="48px"/>
</a>
</div>
- <div class="pull-right"> <a href="http://incubator.apache.org/" id="bannerRight">
- <img src="../../images/ApacheIncubating_Logo.png" alt="Apache Incubating" width="192px" height="48px"/>
- </a>
- </div>
+ <div class="pull-right"> </div>
<div class="clear"><hr/></div>
</div>
@@ -51,8 +48,8 @@
</li>
<li class="divider ">/</li>
<li class="">
- <a href="http://metron.incubator.apache.org/" class="externalLink" title="Metron-Incubating">
- Metron-Incubating</a>
+ <a href="http://metron.apache.org/" class="externalLink" title="Metron">
+ Metron</a>
</li>
<li class="divider ">/</li>
<li class="">
@@ -64,8 +61,8 @@
- <li id="publishDate" class="pull-right">Last Published: 2017-02-23</li> <li class="divider pull-right">|</li>
- <li id="projectVersion" class="pull-right">Version: 0.3.1</li>
+ <li id="publishDate" class="pull-right">Last Published: 2017-06-27</li> <li class="divider pull-right">|</li>
+ <li id="projectVersion" class="pull-right">Version: 0.4.0</li>
</ul>
</div>
@@ -78,7 +75,7 @@
<ul class="nav nav-list">
<li class="nav-header">User Documentation</li>
-
+
<li>
<a href="../../index.html" title="Metron">
@@ -99,7 +96,7 @@
<i class="icon-chevron-right"></i>
Analytics</a>
</li>
-
+
<li>
<a href="../../metron-deployment/index.html" title="Deployment">
@@ -107,6 +104,20 @@
Deployment</a>
<ul class="nav nav-list">
+ <li>
+
+ <a href="../../metron-deployment/Kerberos-ambari-setup.html" title="Kerberos-ambari-setup">
+ <i class="none"></i>
+ Kerberos-ambari-setup</a>
+ </li>
+
+ <li>
+
+ <a href="../../metron-deployment/Kerberos-manual-setup.html" title="Kerberos-manual-setup">
+ <i class="none"></i>
+ Kerberos-manual-setup</a>
+ </li>
+
<li class="active">
<a href="#"><i class="none"></i>Amazon-ec2</a>
@@ -128,11 +139,11 @@
<li>
- <a href="../../metron-deployment/packer-build/index.html" title="Packer-build">
+ <a href="../../metron-deployment/packaging/packer-build/index.html" title="Packer-build">
<i class="none"></i>
Packer-build</a>
</li>
-
+
<li>
<a href="../../metron-deployment/roles/index.html" title="Roles">
@@ -155,14 +166,28 @@
<i class="none"></i>
Docker</a>
</li>
-
+
+ <li>
+
+ <a href="../../metron-interface/metron-config/index.html" title="Config">
+ <i class="none"></i>
+ Config</a>
+ </li>
+
+ <li>
+
+ <a href="../../metron-interface/metron-rest/index.html" title="Rest">
+ <i class="none"></i>
+ Rest</a>
+ </li>
+
<li>
<a href="../../metron-platform/index.html" title="Platform">
<i class="icon-chevron-right"></i>
Platform</a>
</li>
-
+
<li>
<a href="../../metron-sensors/index.html" title="Sensors">
@@ -202,7 +227,7 @@
<ul>
-<li>Ansible 2.0.0.2</li>
+<li>Ansible 2.0.0.2 or 2.2.2.0</li>
<li>Python 2.7.11</li>
@@ -227,12 +252,13 @@
<div class="source">
<pre> brew cask install java
brew install maven git
- brew install https://raw.githubusercontent.com/Homebrew/homebrew-core/ee1273bf919a5e4e50838513a9e55ea423e1d7ce/Formula/ansible.rb
- brew switch ansible 2.0.0.2
</pre></div></div></li>
<li>
-<p>Ensure that a public SSH key is located at <tt>~/.ssh/id_rsa.pub</tt>. </p>
+<p>Install Ansible by following the instructions <a class="externalLink" href="http://docs.ansible.com/ansible/intro_installation.html#latest-releases-via-pip">here</a>.</p></li>
+
+<li>
+<p>Ensure that a public SSH key is located at <tt>~/.ssh/id_rsa.pub</tt>.</p>
<div class="source">
<div class="source">
@@ -315,7 +341,7 @@
" Metron @ http://ec2-52-37-255-142.us-west-2.compute.amazonaws.com:5000",
" Ambari @ http://ec2-52-37-225-202.us-west-2.compute.amazonaws.com:8080",
" Sensors @ ec2-52-37-225-202.us-west-2.compute.amazonaws.com on tap0",
- "For additional information, see https://metron.incubator.apache.org/'"
+ "For additional information, see https://metron.apache.org/'"
]
}
</pre></div></div></li>
@@ -372,36 +398,49 @@ Enter same passphrase again:
<div class="section">
<h4><a name="Solution"></a>Solution</h4>
<p>If you see this error, please report your findings by creating a JIRA or dropping an email to the Metron Users mailing list. Follow these steps to work around the problem.</p>
-<p>(1) Define which Elasticsearch host to interact with. Any Elasticsearch host should work.</p>
+<ol style="list-style-type: decimal">
+
+<li>
+<p>Define which Elasticsearch host to interact with. Any Elasticsearch host should work.</p>
+
<div class="source">
<div class="source">
<pre>export ES_HOST="http://ec2-52-25-237-20.us-west-2.compute.amazonaws.com:9200"
-</pre></div></div>
-<p>(2) Confirm the index templates are in fact missing. </p>
-
+</pre></div></div></li>
+
+<li>
+<p>Confirm the index templates are in fact missing. </p>
+
<div class="source">
<div class="source">
<pre>curl -s -XGET $ES_HOST/_template
-</pre></div></div>
-<p>(3) Manually load the index templates.</p>
-
+</pre></div></div></li>
+
+<li>
+<p>Manually load the index templates.</p>
+
<div class="source">
<div class="source">
<pre>cd metron-deployment
curl -s -XPOST $ES_HOST/_template/bro_index -d @roles/metron_elasticsearch_templates/files/es_templates/bro_index.template
curl -s -XPOST $ES_HOST/_template/snort_index -d @roles/metron_elasticsearch_templates/files/es_templates/snort_index.template
curl -s -XPOST $ES_HOST/_template/yaf_index -d @roles/metron_elasticsearch_templates/files/es_templates/yaf_index.template
-</pre></div></div>
-<p>(4) Delete the existing indexes. Only a new index will use the templates defined in the previous step.</p>
-
+</pre></div></div></li>
+
+<li>
+<p>Delete the existing indexes. Only a new index will use the templates defined in the previous step.</p>
+
<div class="source">
<div class="source">
<pre>curl -s -XDELETE "$ES_HOST/yaf_index*"
curl -s -XDELETE "$ES_HOST/bro_index*"
curl -s -XDELETE "$ES_HOST/snort_index*"
-</pre></div></div>
-<p>(5) Open up Kibana and wait for the new indexes to be created. The dashboard should now work.</p></div></div>
+</pre></div></div></li>
+
+<li>
+<p>Open up Kibana and wait for the new indexes to be created. The dashboard should now work.</p></li>
+</ol></div></div>
<div class="section">
<h3><a name="Error:_No_handler_was_ready_to_authenticateCheck_your_credentials"></a>Error: ‘No handler was ready to authenticate…Check your credentials’</h3>
@@ -508,8 +547,9 @@ fatal: [ec2-52-26-113-221.us-west-2.compute.amazonaws.com]: UNREACHABLE! => {
<footer>
<div class="container-fluid">
- <div class="row span12">Copyright © 2017.
- All Rights Reserved.
+ <div class="row span12">Copyright © 2017
+ <a href="https://www.apache.org">The Apache Software Foundation</a>.
+ All Rights Reserved.
</div>
http://git-wip-us.apache.org/repos/asf/metron/blob/cf4c2ecd/current-book/metron-deployment/index.html
----------------------------------------------------------------------
diff --git a/current-book/metron-deployment/index.html b/current-book/metron-deployment/index.html
index 43e6678..7865a52 100644
--- a/current-book/metron-deployment/index.html
+++ b/current-book/metron-deployment/index.html
@@ -1,13 +1,13 @@
<!DOCTYPE html>
<!--
- | Generated by Apache Maven Doxia at 2017-02-23
+ | Generated by Apache Maven Doxia at 2017-06-27
| Rendered using Apache Maven Fluido Skin 1.3.0
-->
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
- <meta name="Date-Revision-yyyymmdd" content="20170223" />
+ <meta name="Date-Revision-yyyymmdd" content="20170627" />
<meta http-equiv="Content-Language" content="en" />
<title>Metron – Overview</title>
<link rel="stylesheet" href="../css/apache-maven-fluido-1.3.0.min.css" />
@@ -30,14 +30,11 @@
<div class="container-fluid">
<div id="banner">
<div class="pull-left">
- <a href="http://metron.incubator.apache.org/" id="bannerLeft">
- <img src="../images/metron-logo.png" alt="Apache Metron - Incubating" width="148px" height="48px"/>
+ <a href="http://metron.apache.org/" id="bannerLeft">
+ <img src="../images/metron-logo.png" alt="Apache Metron" width="148px" height="48px"/>
</a>
</div>
- <div class="pull-right"> <a href="http://incubator.apache.org/" id="bannerRight">
- <img src="../images/ApacheIncubating_Logo.png" alt="Apache Incubating" width="192px" height="48px"/>
- </a>
- </div>
+ <div class="pull-right"> </div>
<div class="clear"><hr/></div>
</div>
@@ -51,8 +48,8 @@
</li>
<li class="divider ">/</li>
<li class="">
- <a href="http://metron.incubator.apache.org/" class="externalLink" title="Metron-Incubating">
- Metron-Incubating</a>
+ <a href="http://metron.apache.org/" class="externalLink" title="Metron">
+ Metron</a>
</li>
<li class="divider ">/</li>
<li class="">
@@ -64,8 +61,8 @@
- <li id="publishDate" class="pull-right">Last Published: 2017-02-23</li> <li class="divider pull-right">|</li>
- <li id="projectVersion" class="pull-right">Version: 0.3.1</li>
+ <li id="publishDate" class="pull-right">Last Published: 2017-06-27</li> <li class="divider pull-right">|</li>
+ <li id="projectVersion" class="pull-right">Version: 0.4.0</li>
</ul>
</div>
@@ -78,7 +75,7 @@
<ul class="nav nav-list">
<li class="nav-header">User Documentation</li>
-
+
<li>
<a href="../index.html" title="Metron">
@@ -99,7 +96,7 @@
<i class="icon-chevron-right"></i>
Analytics</a>
</li>
-
+
<li class="active">
<a href="#"><i class="icon-chevron-down"></i>Deployment</a>
@@ -107,6 +104,20 @@
<li>
+ <a href="../metron-deployment/Kerberos-ambari-setup.html" title="Kerberos-ambari-setup">
+ <i class="none"></i>
+ Kerberos-ambari-setup</a>
+ </li>
+
+ <li>
+
+ <a href="../metron-deployment/Kerberos-manual-setup.html" title="Kerberos-manual-setup">
+ <i class="none"></i>
+ Kerberos-manual-setup</a>
+ </li>
+
+ <li>
+
<a href="../metron-deployment/amazon-ec2/index.html" title="Amazon-ec2">
<i class="none"></i>
Amazon-ec2</a>
@@ -128,11 +139,11 @@
<li>
- <a href="../metron-deployment/packer-build/index.html" title="Packer-build">
+ <a href="../metron-deployment/packaging/packer-build/index.html" title="Packer-build">
<i class="none"></i>
Packer-build</a>
</li>
-
+
<li>
<a href="../metron-deployment/roles/index.html" title="Roles">
@@ -155,14 +166,28 @@
<i class="none"></i>
Docker</a>
</li>
-
+
+ <li>
+
+ <a href="../metron-interface/metron-config/index.html" title="Config">
+ <i class="none"></i>
+ Config</a>
+ </li>
+
+ <li>
+
+ <a href="../metron-interface/metron-rest/index.html" title="Rest">
+ <i class="none"></i>
+ Rest</a>
+ </li>
+
<li>
<a href="../metron-platform/index.html" title="Platform">
<i class="icon-chevron-right"></i>
Platform</a>
</li>
-
+
<li>
<a href="../metron-sensors/index.html" title="Sensors">
@@ -193,8 +218,30 @@
<h1>Overview</h1>
<p><a name="Overview"></a></p>
-<p>This set of playbooks can be used to deploy an Ambari-managed Hadoop cluster, Metron services, or both using ansible playbooks. These playbooks currently only target RHEL/CentOS 6.x operating systems.</p>
-<p>In addition, an Ambari Management Pack can be built which can be deployed in conjuction with RPMs detailed in this README.</p>
+<p>This set of playbooks can be used to deploy an Ambari-managed Hadoop cluster containing Metron services using Ansible. These playbooks target RHEL/CentOS 6.x operating systems.</p>
+<p>Installation consists of -</p>
+
+<ul>
+
+<li>Building Metron tarballs, RPMs and the Ambari MPack</li>
+
+<li>Deploying Ambari</li>
+
+<li>Leveraging Ambari to install:
+
+<ul>
+
+<li>The required Hadoop Components</li>
+
+<li>Core Metron (Parsing, Enrichment, Indexing)</li>
+
+<li>Elasticsearch</li>
+
+<li>Kibana</li>
+ </ul></li>
+
+<li>Starting All Services</li>
+</ul>
<div class="section">
<h2><a name="Prerequisites"></a>Prerequisites</h2>
<p>The following tools are required to run these scripts:</p>
@@ -205,16 +252,10 @@
<li><a class="externalLink" href="https://git-scm.com/">Git</a></li>
-<li><a class="externalLink" href="http://www.ansible.com/">Ansible</a> (version 2.0 or greater)</li>
-</ul>
-<p>Currently Metron must be built from source. Before running these scripts perform the following steps:</p>
-
-<ol style="list-style-type: decimal">
+<li><a class="externalLink" href="http://www.ansible.com/">Ansible</a> (2.0.0.2 or 2.2.2.0)</li>
-<li>Clone the Metron git repository with <tt>git clone git@github.com:apache/incubator-metron.git</tt></li>
-
-<li>Navigate to <tt>incubator-metron</tt> and run <tt>mvn clean package</tt></li>
-</ol>
+<li><a class="externalLink" href="https://www.docker.com/">Docker</a> (Docker for Mac on OSX)</li>
+</ul>
<p>These scripts depend on two files for configuration:</p>
<ul>
@@ -223,75 +264,29 @@
<li>group_vars/all - various configuration settings needed to install Metron</li>
</ul>
-<p>Examples can be found in the <tt>incubator-metron/metron-deployment/inventory/metron_example</tt> directory and are a good starting point. Copy this directory into <tt>incubator-metron/metron-deployment/inventory/</tt> and rename it to your <tt>project_name</tt>. More information about Ansible files and directory structure can be found at <a class="externalLink" href="http://docs.ansible.com/ansible/playbooks_best_practices.html">http://docs.ansible.com/ansible/playbooks_best_practices.html</a>.</p></div>
+<p>For production use, it is recommended that Metron be installed on an existing cluster managed by Ambari as described in the Installing Management Pack section below.</p></div>
<div class="section">
<h2><a name="Ambari"></a>Ambari</h2>
-<p>The Ambari playbook will install a Hadoop cluster with all the services and configuration required by Metron. This section can be skipped if installing Metron on a pre-existing cluster.</p>
-<p>Currently, this playbook supports building a local development cluster running on one node but options for other types of clusters will be added in the future.</p>
+<p>The Ambari playbook will install a Hadoop cluster including the Metron Services (Parsing, Enrichment, Indexing). Ambari will also install Elasticsearch and Kibana.</p>
+<p>Currently, the playbooks supports building a local development cluster running on one node or deploying to a 10 node cluster on AWS EC2.</p></div>
<div class="section">
-<h3><a name="Setting_up_your_inventory"></a>Setting up your inventory</h3>
-<p>Make sure to update the hosts file in <tt>incubator-metron/metron-deployment/inventory/project_name/hosts</tt> or provide an alternate inventory file when you launch the playbooks, including the ssh user(s) and ssh keyfile location(s). These playbooks expect two host groups:</p>
-
-<ul>
-
-<li>ambari_master</li>
-
-<li>ambari_slaves</li>
-</ul></div>
-<div class="section">
-<h3><a name="Running_the_playbook"></a>Running the playbook</h3>
-<p>This playbook will install the Ambari server on the ambari_master, install the ambari agents on the ambari_slaves, and create a cluster in Ambari with a blueprint for the required Metron components.</p>
-<p>Navigate to <tt>incubator-metron/metron-deployment/playbooks</tt> and run: <tt>ansible-playbook -i ../inventory/project_name ambari_install.yml</tt></p></div></div>
-<div class="section">
-<h2><a name="Metron"></a>Metron</h2>
-<p>The Metron playbook will gather the necessary cluster settings from Ambari and install the Metron services.</p>
+<h2><a name="Vagrant"></a>Vagrant</h2>
+<p>There are current two Vagrant modes, full-dev and quick-dev. Full-dev installs the entire Ambari/Metron stack. This is useful in testing out changes to the installation procedure. Quick-dev re-installs the core Metron Services (Parsing, Enrichment, and Indexing)on a pre-built instance. Use quick-dev for testing out changes to core Metron services.</p>
<div class="section">
-<h3><a name="Setting_up_your_inventory"></a>Setting up your inventory</h3>
-<p>Edit the hosts file at <tt>incubator-metron/metron-deployment/inventory/project_name/hosts</tt>. Declare where which hosts the Metron services will be installed on by updating these groups:</p>
-
-<ul>
-
-<li>enrichment - submits the topology code to Storm and requires a storm client</li>
-
-<li>search - host where Elasticsearch will be run</li>
-
-<li>web - host where the Metron UI and underlying services will run</li>
-
-<li>sensors - host where network data will be collected and published to Kafka</li>
-</ul>
-<p>The Metron topologies depend on Kafka topics and HBase tables being created beforehand. Declare a host that has Kafka and HBase clients installed by updating these groups:</p>
+<h3><a name="Prerequsities"></a>Prerequsities</h3>
<ul>
-<li>metron_kafka_topics</li>
+<li>Install <a class="externalLink" href="https://www.vagrantup.com/">Vagrant</a> (5.0.16+)</li>
-<li>metron_hbase_tables</li>
-</ul>
-<p>If only installing Metron, these groups can be ignored:</p>
-
-<ul>
-
-<li>ambari_master</li>
-
-<li>ambari_slaves</li>
+<li>Install the Hostmanager plugin for vagrant - Run <tt>vagrant plugin install vagrant-hostmanager</tt> on the machine where Vagrant is installed</li>
</ul></div>
<div class="section">
-<h3><a name="Configuring_group_variables"></a>Configuring group variables</h3>
-<p>The Metron Ansible scripts depend on a set of variables. These variables can be found in the file at <tt>incubator-metron/metron-deployment/inventory/project_name/group_vars/all</tt>. Edit the ambari* variables to match your Ambari instance and update the java_home variable to match the java path on your hosts.</p></div>
-<div class="section">
-<h3><a name="Running_the_playbook"></a>Running the playbook</h3>
-<p>Navigate to <tt>incubator-metron/metron-deployment/playbooks</tt> and run: <tt>ansible-playbook -i ../inventory/project_name metron_install.yml</tt></p></div></div>
+<h3><a name="Full-Dev"></a>Full-Dev</h3>
+<p>Navigate to <tt>metron/metron-deployment/vagrant/full-dev-platform</tt> and run <tt>vagrant up</tt>.</p></div>
<div class="section">
-<h2><a name="Vagrant"></a>Vagrant</h2>
-<p>A VagrantFile is included and will install a working version of the entire Metron stack. The following is required to run this:</p>
-
-<ul>
-
-<li><a class="externalLink" href="https://www.vagrantup.com/">Vagrant</a></li>
-
-<li>Hostmanager plugin for vagrant - Run <tt>vagrant plugin install vagrant-hostmanager</tt> on the machine where Vagrant is installed</li>
-</ul>
-<p>Navigate to <tt>incubator-metron/metron-deployment/vagrant/full-dev-platform</tt> and run <tt>vagrant up</tt>. This also provides a good example of how to run a full end-to-end Metron install.</p></div>
+<h3><a name="Quick-Dev"></a>Quick-Dev</h3>
+<p>Navigate to <tt>metron/metron-deployment/vagrant/quick-dev-platform</tt> and run <tt>vagrant up</tt>.</p></div></div>
<div class="section">
<h2><a name="Ambari_Management_Pack"></a>Ambari Management Pack</h2>
<p>An Ambari Management Pack can be built in order to make the Metron service available on top of an existing stack, rather than needing a direct stack update.</p>
@@ -316,9 +311,11 @@
<ul>
-<li>A cluster managed by Ambari 2.4</li>
+<li>A cluster managed by Ambari 2.4.2+</li>
<li>Metron RPMs available on the cluster in the /localrepo directory. See <a href="#RPM">RPM</a> for further information.</li>
+
+<li><a class="externalLink" href="https://nodejs.org/en/download/package-manager/">Node.js</a> repository installed on the Management UI host</li>
</ul></div>
<div class="section">
<h3><a name="Building_Management_Pack"></a>Building Management Pack</h3>
@@ -391,6 +388,30 @@
<li>Docker. The image detailed in: <tt>metron-deployment/packaging/docker/rpm-docker/README.md</tt> will automatically be built (or rebuilt if necessary).</li>
<li>Artifacts for metron-platform have been produced. E.g. <tt>mvn clean package -DskipTests</tt> in <tt>metron-platform</tt></li>
+</ul>
+<p>The artifacts are required because there is a dependency on modules not expressed via Maven (we grab the resulting assemblies, but don’t need the jars). These are</p>
+
+<ul>
+
+<li>metron-common</li>
+
+<li>metron-data-management</li>
+
+<li>metron-elasticsearch</li>
+
+<li>metron-enrichment</li>
+
+<li>metron-indexing</li>
+
+<li>metron-parsers</li>
+
+<li>metron-pcap-backend</li>
+
+<li>metron-solr</li>
+
+<li>metron-profiler</li>
+
+<li>metron-config</li>
</ul></div>
<div class="section">
<h3><a name="Building_RPMs"></a>Building RPMs</h3>
@@ -407,6 +428,49 @@
<pre>rpm -i <package>
</pre></div></div></div></div>
<div class="section">
+<h2><a name="Kibana_Dashboards"></a>Kibana Dashboards</h2>
+<p>The dashboards installed by the Kibana custom action are managed by the dashboard.p file. This file is created by exporting existing dashboards from a running Kibana instance.</p>
+<p>To create a new version of the file, make any necessary changes to Kibana (e.g. on quick-dev), and export with the appropriate script.</p>
+
+<div class="source">
+<div class="source">
+<pre>python packaging/ambari/metron-mpack/src/main/resources/common-services/KIBANA/4.5.1/package/scripts/dashboard/dashboardindex.py \
+$ES_HOST 9200 \
+packaging/ambari/metron-mpack/src/main/resources/common-services/KIBANA/4.5.1/package/scripts/dashboard/dashboard.p -s
+</pre></div></div>
+<p>Build the Ambari Mpack to get the dashboard updated appropriately.</p>
+<p>Once the MPack is installed, run the Kibana service’s action “Load Template” to install dashboards. This will completely overwrite the .kibana in Elasticsearch, so use with caution.</p></div>
+<div class="section">
+<h2><a name="Kerberos"></a>Kerberos</h2>
+<p>The MPack can allow Metron to be installed and then Kerberized, or installed on top of an already Kerberized cluster. This is done through Ambari’s standard Kerberization setup.</p>
+<div class="section">
+<h3><a name="Caveats"></a>Caveats</h3>
+
+<ul>
+
+<li>For nodes using a Metron client and a local repo, the repo must exist on all nodes (e.g via createrepo). This repo can be empty; only the main Metron services need the RPMs.</li>
+
+<li>A Metron client must be installed on each supervisor node in a secured cluster. This is to ensure that the Metron keytab and client_jaas.conf get distributed in order to allow reading and writing from Kafka.
+
+<ul>
+
+<li>When Metron is already installed on the cluster, this should be done before Kerberizing.</li>
+
+<li>When addding Metron to an already Kerberized cluster, ensure that all supervisor nodes receive a Metron client.</li>
+ </ul></li>
+
+<li>Storm (and Metron) must be restarted after Metron is installed on an already Kerberized cluster. Several Storm configs get updated, and Metron will be unable to write to Kafka without a restart.
+
+<ul>
+
+<li>Kerberizing a cluster with an existing Metron already has restarts of all services during Kerberization, so it’s unneeded.</li>
+ </ul></li>
+</ul>
+<p>Instructions for setup on Full Dev can be found at <a href="Kerberos-ambari-setup.html">Kerberos-ambari-setup.md</a>. These instructions reference the manual install instructions.</p></div>
+<div class="section">
+<h3><a name="Kerberos_Without_an_MPack"></a>Kerberos Without an MPack</h3>
+<p>Using the MPack is preferred, but instructions for Kerberizing manually can be found at <a href="Kerberos-manual-setup.html">Kerberos-manual-setup.md</a>. These instructions are reference by the Ambari Kerberos install instructions and include commands for setting up a KDC.</p></div></div>
+<div class="section">
<h2><a name="TODO"></a>TODO</h2>
<ul>
@@ -421,8 +485,9 @@
<footer>
<div class="container-fluid">
- <div class="row span12">Copyright © 2017.
- All Rights Reserved.
+ <div class="row span12">Copyright © 2017
+ <a href="https://www.apache.org">The Apache Software Foundation</a>.
+ All Rights Reserved.
</div>
[02/10] metron git commit: METRON-1012 Update Metron public web site
for 0.4.0 release - push to asf-site from the generated code on master
Posted by ma...@apache.org.
http://git-wip-us.apache.org/repos/asf/metron/blob/cf4c2ecd/current-book/metron-platform/metron-writer/index.html
----------------------------------------------------------------------
diff --git a/current-book/metron-platform/metron-writer/index.html b/current-book/metron-platform/metron-writer/index.html
new file mode 100644
index 0000000..ad36a4f
--- /dev/null
+++ b/current-book/metron-platform/metron-writer/index.html
@@ -0,0 +1,321 @@
+<!DOCTYPE html>
+<!--
+ | Generated by Apache Maven Doxia at 2017-06-27
+ | Rendered using Apache Maven Fluido Skin 1.3.0
+-->
+<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
+ <head>
+ <meta charset="UTF-8" />
+ <meta name="viewport" content="width=device-width, initial-scale=1.0" />
+ <meta name="Date-Revision-yyyymmdd" content="20170627" />
+ <meta http-equiv="Content-Language" content="en" />
+ <title>Metron – Writer</title>
+ <link rel="stylesheet" href="../../css/apache-maven-fluido-1.3.0.min.css" />
+ <link rel="stylesheet" href="../../css/site.css" />
+ <link rel="stylesheet" href="../../css/print.css" media="print" />
+
+
+ <script type="text/javascript" src="../../js/apache-maven-fluido-1.3.0.min.js"></script>
+
+
+
+<script type="text/javascript">$( document ).ready( function() { $( '.carousel' ).carousel( { interval: 3500 } ) } );</script>
+
+ </head>
+ <body class="topBarDisabled">
+
+
+
+
+ <div class="container-fluid">
+ <div id="banner">
+ <div class="pull-left">
+ <a href="http://metron.apache.org/" id="bannerLeft">
+ <img src="../../images/metron-logo.png" alt="Apache Metron" width="148px" height="48px"/>
+ </a>
+ </div>
+ <div class="pull-right"> </div>
+ <div class="clear"><hr/></div>
+ </div>
+
+ <div id="breadcrumbs">
+ <ul class="breadcrumb">
+
+
+ <li class="">
+ <a href="http://www.apache.org" class="externalLink" title="Apache">
+ Apache</a>
+ </li>
+ <li class="divider ">/</li>
+ <li class="">
+ <a href="http://metron.apache.org/" class="externalLink" title="Metron">
+ Metron</a>
+ </li>
+ <li class="divider ">/</li>
+ <li class="">
+ <a href="../../index.html" title="Documentation">
+ Documentation</a>
+ </li>
+ <li class="divider ">/</li>
+ <li class="">Writer</li>
+
+
+
+ <li id="publishDate" class="pull-right">Last Published: 2017-06-27</li> <li class="divider pull-right">|</li>
+ <li id="projectVersion" class="pull-right">Version: 0.4.0</li>
+
+ </ul>
+ </div>
+
+
+ <div class="row-fluid">
+ <div id="leftColumn" class="span3">
+ <div class="well sidebar-nav">
+
+
+ <ul class="nav nav-list">
+ <li class="nav-header">User Documentation</li>
+
+ <li>
+
+ <a href="../../index.html" title="Metron">
+ <i class="icon-chevron-down"></i>
+ Metron</a>
+ <ul class="nav nav-list">
+
+ <li>
+
+ <a href="../../Upgrading.html" title="Upgrading">
+ <i class="none"></i>
+ Upgrading</a>
+ </li>
+
+ <li>
+
+ <a href="../../metron-analytics/index.html" title="Analytics">
+ <i class="icon-chevron-right"></i>
+ Analytics</a>
+ </li>
+
+ <li>
+
+ <a href="../../metron-deployment/index.html" title="Deployment">
+ <i class="icon-chevron-right"></i>
+ Deployment</a>
+ </li>
+
+ <li>
+
+ <a href="../../metron-docker/index.html" title="Docker">
+ <i class="none"></i>
+ Docker</a>
+ </li>
+
+ <li>
+
+ <a href="../../metron-interface/metron-config/index.html" title="Config">
+ <i class="none"></i>
+ Config</a>
+ </li>
+
+ <li>
+
+ <a href="../../metron-interface/metron-rest/index.html" title="Rest">
+ <i class="none"></i>
+ Rest</a>
+ </li>
+
+ <li>
+
+ <a href="../../metron-platform/index.html" title="Platform">
+ <i class="icon-chevron-down"></i>
+ Platform</a>
+ <ul class="nav nav-list">
+
+ <li>
+
+ <a href="../../metron-platform/metron-api/index.html" title="Api">
+ <i class="none"></i>
+ Api</a>
+ </li>
+
+ <li>
+
+ <a href="../../metron-platform/metron-common/index.html" title="Common">
+ <i class="icon-chevron-right"></i>
+ Common</a>
+ </li>
+
+ <li>
+
+ <a href="../../metron-platform/metron-data-management/index.html" title="Data-management">
+ <i class="none"></i>
+ Data-management</a>
+ </li>
+
+ <li>
+
+ <a href="../../metron-platform/metron-enrichment/index.html" title="Enrichment">
+ <i class="none"></i>
+ Enrichment</a>
+ </li>
+
+ <li>
+
+ <a href="../../metron-platform/metron-indexing/index.html" title="Indexing">
+ <i class="none"></i>
+ Indexing</a>
+ </li>
+
+ <li>
+
+ <a href="../../metron-platform/metron-management/index.html" title="Management">
+ <i class="none"></i>
+ Management</a>
+ </li>
+
+ <li>
+
+ <a href="../../metron-platform/metron-parsers/index.html" title="Parsers">
+ <i class="none"></i>
+ Parsers</a>
+ </li>
+
+ <li>
+
+ <a href="../../metron-platform/metron-pcap-backend/index.html" title="Pcap-backend">
+ <i class="none"></i>
+ Pcap-backend</a>
+ </li>
+
+ <li class="active">
+
+ <a href="#"><i class="none"></i>Writer</a>
+ </li>
+ </ul>
+ </li>
+
+ <li>
+
+ <a href="../../metron-sensors/index.html" title="Sensors">
+ <i class="icon-chevron-right"></i>
+ Sensors</a>
+ </li>
+ </ul>
+ </li>
+ </ul>
+
+
+
+ <hr class="divider" />
+
+ <div id="poweredBy">
+ <div class="clear"></div>
+ <div class="clear"></div>
+ <div class="clear"></div>
+ <a href="http://maven.apache.org/" title="Built by Maven" class="poweredBy">
+ <img class="builtBy" alt="Built by Maven" src="../../images/logos/maven-feather.png" />
+ </a>
+ </div>
+ </div>
+ </div>
+
+
+ <div id="bodyColumn" class="span9" >
+
+ <!-- Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to You under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License. --><h1>Writer</h1>
+<p><a name="Writer"></a></p>
+<div class="section">
+<h2><a name="Introduction"></a>Introduction</h2>
+<p>The writer module provides some utilties for writing to outside components from within Storm. This includes managing bulk writing. An implemention is included for writing to HDFS in this module. Other writers can be found in their own modules.</p></div>
+<div class="section">
+<h2><a name="HDFS_Writer"></a>HDFS Writer</h2>
+<p>The HDFS writer included here expands on what Storm has in several ways. There’s customization in syncing to HDFS, rotation policy, etc. In addition, the writer allows for users to define output paths based on the fields in the provided JSON message. This can be defined using Stellar.</p>
+<p>To manage the output path, a base path argument is provided by the Flux file, with the FileNameFormat as follows</p>
+
+<div class="source">
+<div class="source">
+<pre> - id: "fileNameFormat"
+ className: "org.apache.storm.hdfs.bolt.format.DefaultFileNameFormat"
+ configMethods:
+ - name: "withPrefix"
+ args:
+ - "enrichment-"
+ - name: "withExtension"
+ args:
+ - ".json"
+ - name: "withPath"
+ args:
+ - "/apps/metron/"
+</pre></div></div>
+<p>This means that all output will land in <tt>/apps/metron/</tt>. With no further adjustment, it will be <tt>/apps/metron/<sensor>/</tt>. However, by modifying the sensor’s JSON config, it is possible to provide additional pathing based on the the message itself.</p>
+<p>E.g.</p>
+
+<div class="source">
+<div class="source">
+<pre>{
+ "index": "bro",
+ "batchSize": 5,
+ "outputPathFunction": "FORMAT('uid-%s', uid)"
+}
+</pre></div></div>
+<p>will land data in <tt>/apps/metron/uid-<uid>/</tt>.</p>
+<p>For example, if the data contains uid’s 1, 3, and 5, there will be 3 output folders in HDFS:</p>
+
+<div class="source">
+<div class="source">
+<pre>/apps/metron/uid-1/
+/apps/metron/uid-3/
+/apps/metron/uid-5/
+</pre></div></div>
+<p>The Stellar function must return a String, but is not limited to FORMAT functions. Other functions, such as <tt>TO_LOWER</tt>, <tt>TO_UPPER</tt>, etc. are all available for use. Typically, it’s preferable to do nontrivial transformations as part of enrichment and simply reference the output here.</p>
+<p>If no Stellar function is provided, it will default to putting the sensor in a folder, as above.</p>
+<p>A caveat is that the writer will only allow a certain number of files to be created at once. HdfsWriter has a function <tt>withMaxOpenFiles</tt> allowing this to be set. The default is 500. This can be set in Flux:</p>
+
+<div class="source">
+<div class="source">
+<pre> - id: "hdfsWriter"
+ className: "org.apache.metron.writer.hdfs.HdfsWriter"
+ configMethods:
+ - name: "withFileNameFormat"
+ args:
+ - ref: "fileNameFormat"
+ - name: "withRotationPolicy"
+ args:
+ - ref: "hdfsRotationPolicy"
+ - name: "withMaxOpenFiles"
+ args: 500
+</pre></div></div></div>
+ </div>
+ </div>
+ </div>
+
+ <hr/>
+
+ <footer>
+ <div class="container-fluid">
+ <div class="row span12">Copyright © 2017
+ <a href="https://www.apache.org">The Apache Software Foundation</a>.
+ All Rights Reserved.
+
+ </div>
+
+
+
+ </div>
+ </footer>
+ </body>
+</html>
http://git-wip-us.apache.org/repos/asf/metron/blob/cf4c2ecd/current-book/metron-sensors/bro-plugin-kafka/index.html
----------------------------------------------------------------------
diff --git a/current-book/metron-sensors/bro-plugin-kafka/index.html b/current-book/metron-sensors/bro-plugin-kafka/index.html
new file mode 100644
index 0000000..6a9654a
--- /dev/null
+++ b/current-book/metron-sensors/bro-plugin-kafka/index.html
@@ -0,0 +1,491 @@
+<!DOCTYPE html>
+<!--
+ | Generated by Apache Maven Doxia at 2017-06-27
+ | Rendered using Apache Maven Fluido Skin 1.3.0
+-->
+<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
+ <head>
+ <meta charset="UTF-8" />
+ <meta name="viewport" content="width=device-width, initial-scale=1.0" />
+ <meta name="Date-Revision-yyyymmdd" content="20170627" />
+ <meta http-equiv="Content-Language" content="en" />
+ <title>Metron – Logging Bro Output to Kafka</title>
+ <link rel="stylesheet" href="../../css/apache-maven-fluido-1.3.0.min.css" />
+ <link rel="stylesheet" href="../../css/site.css" />
+ <link rel="stylesheet" href="../../css/print.css" media="print" />
+
+
+ <script type="text/javascript" src="../../js/apache-maven-fluido-1.3.0.min.js"></script>
+
+
+
+<script type="text/javascript">$( document ).ready( function() { $( '.carousel' ).carousel( { interval: 3500 } ) } );</script>
+
+ </head>
+ <body class="topBarDisabled">
+
+
+
+
+ <div class="container-fluid">
+ <div id="banner">
+ <div class="pull-left">
+ <a href="http://metron.apache.org/" id="bannerLeft">
+ <img src="../../images/metron-logo.png" alt="Apache Metron" width="148px" height="48px"/>
+ </a>
+ </div>
+ <div class="pull-right"> </div>
+ <div class="clear"><hr/></div>
+ </div>
+
+ <div id="breadcrumbs">
+ <ul class="breadcrumb">
+
+
+ <li class="">
+ <a href="http://www.apache.org" class="externalLink" title="Apache">
+ Apache</a>
+ </li>
+ <li class="divider ">/</li>
+ <li class="">
+ <a href="http://metron.apache.org/" class="externalLink" title="Metron">
+ Metron</a>
+ </li>
+ <li class="divider ">/</li>
+ <li class="">
+ <a href="../../index.html" title="Documentation">
+ Documentation</a>
+ </li>
+ <li class="divider ">/</li>
+ <li class="">Logging Bro Output to Kafka</li>
+
+
+
+ <li id="publishDate" class="pull-right">Last Published: 2017-06-27</li> <li class="divider pull-right">|</li>
+ <li id="projectVersion" class="pull-right">Version: 0.4.0</li>
+
+ </ul>
+ </div>
+
+
+ <div class="row-fluid">
+ <div id="leftColumn" class="span3">
+ <div class="well sidebar-nav">
+
+
+ <ul class="nav nav-list">
+ <li class="nav-header">User Documentation</li>
+
+ <li>
+
+ <a href="../../index.html" title="Metron">
+ <i class="icon-chevron-down"></i>
+ Metron</a>
+ <ul class="nav nav-list">
+
+ <li>
+
+ <a href="../../Upgrading.html" title="Upgrading">
+ <i class="none"></i>
+ Upgrading</a>
+ </li>
+
+ <li>
+
+ <a href="../../metron-analytics/index.html" title="Analytics">
+ <i class="icon-chevron-right"></i>
+ Analytics</a>
+ </li>
+
+ <li>
+
+ <a href="../../metron-deployment/index.html" title="Deployment">
+ <i class="icon-chevron-right"></i>
+ Deployment</a>
+ </li>
+
+ <li>
+
+ <a href="../../metron-docker/index.html" title="Docker">
+ <i class="none"></i>
+ Docker</a>
+ </li>
+
+ <li>
+
+ <a href="../../metron-interface/metron-config/index.html" title="Config">
+ <i class="none"></i>
+ Config</a>
+ </li>
+
+ <li>
+
+ <a href="../../metron-interface/metron-rest/index.html" title="Rest">
+ <i class="none"></i>
+ Rest</a>
+ </li>
+
+ <li>
+
+ <a href="../../metron-platform/index.html" title="Platform">
+ <i class="icon-chevron-right"></i>
+ Platform</a>
+ </li>
+
+ <li>
+
+ <a href="../../metron-sensors/index.html" title="Sensors">
+ <i class="icon-chevron-down"></i>
+ Sensors</a>
+ <ul class="nav nav-list">
+
+ <li class="active">
+
+ <a href="#"><i class="none"></i>Bro-plugin-kafka</a>
+ </li>
+
+ <li>
+
+ <a href="../../metron-sensors/fastcapa/index.html" title="Fastcapa">
+ <i class="none"></i>
+ Fastcapa</a>
+ </li>
+
+ <li>
+
+ <a href="../../metron-sensors/pycapa/index.html" title="Pycapa">
+ <i class="none"></i>
+ Pycapa</a>
+ </li>
+ </ul>
+ </li>
+ </ul>
+ </li>
+ </ul>
+
+
+
+ <hr class="divider" />
+
+ <div id="poweredBy">
+ <div class="clear"></div>
+ <div class="clear"></div>
+ <div class="clear"></div>
+ <a href="http://maven.apache.org/" title="Built by Maven" class="poweredBy">
+ <img class="builtBy" alt="Built by Maven" src="../../images/logos/maven-feather.png" />
+ </a>
+ </div>
+ </div>
+ </div>
+
+
+ <div id="bodyColumn" class="span9" >
+
+ <h1>Logging Bro Output to Kafka</h1>
+<p>A Bro log writer that sends logging output to Kafka. This provides a convenient means for tools in the Hadoop ecosystem, such as Storm, Spark, and others, to process the data generated by Bro.</p>
+<div class="section">
+<h2><a name="Installation"></a>Installation</h2>
+
+<ol style="list-style-type: decimal">
+
+<li>
+<p>Install <a class="externalLink" href="https://github.com/edenhill/librdkafka">librdkafka</a>, a native client library for Kafka. This plugin has been tested against the latest release of librdkafka, which at the time of this writing is v0.9.4.</p>
+<p>In order to use this plugin within a kerberized Kafka environment, you will also need <tt>libsasl2</tt> installed and will need to pass <tt>--enable-sasl</tt> to the <tt>configure</tt> script.</p>
+
+<div class="source">
+<div class="source">
+<pre>curl -L https://github.com/edenhill/librdkafka/archive/v0.9.4.tar.gz | tar xvz
+cd librdkafka-0.9.4/
+./configure --enable-sasl
+make
+sudo make install
+</pre></div></div></li>
+
+<li>
+<p>Build the plugin using the following commands.</p>
+
+<div class="source">
+<div class="source">
+<pre>./configure --bro-dist=$BRO_SRC
+make
+sudo make install
+</pre></div></div></li>
+
+<li>
+<p>Run the following command to ensure that the plugin was installed successfully.</p>
+
+<div class="source">
+<div class="source">
+<pre>$ bro -N Bro::Kafka
+Bro::Kafka - Writes logs to Kafka (dynamic, version 0.1)
+</pre></div></div></li>
+</ol></div>
+<div class="section">
+<h2><a name="Activation"></a>Activation</h2>
+<p>The following examples highlight different ways that the plugin can be used. Simply add the Bro script language to your <tt>local.bro</tt> file (for example, <tt>/usr/share/bro/site/local.bro</tt>) as shown to demonstrate the example.</p>
+<div class="section">
+<h3><a name="Example_1"></a>Example 1</h3>
+<p>The goal in this example is to send all HTTP and DNS records to a Kafka topic named <tt>bro</tt>. </p>
+
+<ul>
+
+<li>Any configuration value accepted by librdkafka can be added to the <tt>kafka_conf</tt> configuration table.</li>
+
+<li>By defining <tt>topic_name</tt> all records will be sent to the same Kafka topic.</li>
+
+<li>Defining <tt>logs_to_send</tt> will ensure that only HTTP and DNS records are sent.</li>
+</ul>
+
+<div class="source">
+<div class="source">
+<pre>@load Bro/Kafka/logs-to-kafka.bro
+redef Kafka::logs_to_send = set(HTTP::LOG, DNS::LOG);
+redef Kafka::topic_name = "bro";
+redef Kafka::kafka_conf = table(
+ ["metadata.broker.list"] = "localhost:9092"
+);
+</pre></div></div></div>
+<div class="section">
+<h3><a name="Example_2"></a>Example 2</h3>
+<p>It is also possible to send each log stream to a uniquely named topic. The goal in this example is to send all HTTP records to a Kafka topic named <tt>http</tt> and all DNS records to a separate Kafka topic named <tt>dns</tt>.</p>
+
+<ul>
+
+<li>The <tt>topic_name</tt> value must be set to an empty string.</li>
+
+<li>The <tt>$path</tt> value of Bro’s Log Writer mechanism is used to define the topic name.</li>
+
+<li>Any configuration value accepted by librdkafka can be added to the <tt>$config</tt> configuration table.</li>
+
+<li>Each log writer accepts a separate configuration table.</li>
+</ul>
+
+<div class="source">
+<div class="source">
+<pre>@load Bro/Kafka/logs-to-kafka.bro
+redef Kafka::topic_name = "";
+redef Kafka::tag_json = T;
+
+event bro_init()
+{
+ # handles HTTP
+ local http_filter: Log::Filter = [
+ $name = "kafka-http",
+ $writer = Log::WRITER_KAFKAWRITER,
+ $config = table(
+ ["metadata.broker.list"] = "localhost:9092"
+ ),
+ $path = "http"
+ ];
+ Log::add_filter(HTTP::LOG, http_filter);
+
+ # handles DNS
+ local dns_filter: Log::Filter = [
+ $name = "kafka-dns",
+ $writer = Log::WRITER_KAFKAWRITER,
+ $config = table(
+ ["metadata.broker.list"] = "localhost:9092"
+ ),
+ $path = "dns"
+ ];
+ Log::add_filter(DNS::LOG, dns_filter);
+}
+</pre></div></div></div>
+<div class="section">
+<h3><a name="Example_3"></a>Example 3</h3>
+<p>You may want to configure bro to filter log messages with certain characteristics from being sent to your kafka topics. For instance, Metron currently doesn’t support IPv6 source or destination IPs in the default enrichments, so it may be helpful to filter those log messages from being sent to kafka (although there are <a href="#notes">multiple ways</a> to approach this). In this example we will do that that, and are assuming a somewhat standard bro kafka plugin configuration, such that:</p>
+
+<ul>
+
+<li>All bro logs are sent to the <tt>bro</tt> topic, by configuring <tt>Kafka::topic_name</tt>.</li>
+
+<li>Each JSON message is tagged with the appropriate log type (such as <tt>http</tt>, <tt>dns</tt>, or <tt>conn</tt>), by setting <tt>tag_json</tt> to true.</li>
+
+<li>If the log message contains a 128 byte long source or destination IP address, the log is not sent to kafka.</li>
+</ul>
+
+<div class="source">
+<div class="source">
+<pre>@load Bro/Kafka/logs-to-kafka.bro
+redef Kafka::topic_name = "bro";
+redef Kafka::tag_json = T;
+
+event bro_init() &priority=-5
+{
+ # handles HTTP
+ Log::add_filter(HTTP::LOG, [
+ $name = "kafka-http",
+ $writer = Log::WRITER_KAFKAWRITER,
+ $pred(rec: HTTP::Info) = { return ! (( |rec$id$orig_h| == 128 || |rec$id$resp_h| == 128 )); },
+ $config = table(
+ ["metadata.broker.list"] = "localhost:9092"
+ )
+ ]);
+
+ # handles DNS
+ Log::add_filter(DNS::LOG, [
+ $name = "kafka-dns",
+ $writer = Log::WRITER_KAFKAWRITER,
+ $pred(rec: DNS::Info) = { return ! (( |rec$id$orig_h| == 128 || |rec$id$resp_h| == 128 )); },
+ $config = table(
+ ["metadata.broker.list"] = "localhost:9092"
+ )
+ ]);
+
+ # handles Conn
+ Log::add_filter(Conn::LOG, [
+ $name = "kafka-conn",
+ $writer = Log::WRITER_KAFKAWRITER,
+ $pred(rec: Conn::Info) = { return ! (( |rec$id$orig_h| == 128 || |rec$id$resp_h| == 128 )); },
+ $config = table(
+ ["metadata.broker.list"] = "localhost:9092"
+ )
+ ]);
+}
+</pre></div></div>
+<div class="section">
+<h4><a name="Notes"></a>Notes</h4>
+
+<ul>
+
+<li><tt>logs_to_send</tt> is mutually exclusive with <tt>$pred</tt>, thus for each log you want to set <tt>$pred</tt> on, you must individually setup a <tt>Log::add_filter</tt> and refrain from including that log in <tt>logs_to_send</tt>.</li>
+
+<li>You can also filter IPv6 logs from within your Metron cluster <a href="../../metron-platform/metron-common/index.html#IS_IP">using Stellar</a>. In that case, you wouldn’t apply a predicate in your bro configuration, and instead Stellar would filter the logs out before they were processed by the enrichment layer of Metron.</li>
+
+<li>It is also possible to use the <tt>is_v6_subnet()</tt> bro function in your predicate, as of their <a class="externalLink" href="https://www.bro.org/sphinx-git/install/release-notes.html#bro-2-5">2.5 release</a>, however the above example should work on <a class="externalLink" href="https://www.bro.org/sphinx-git/install/release-notes.html#bro-2-4">bro 2.4</a> and newer, which has been the focus of the kafka plugin.</li>
+</ul></div></div></div>
+<div class="section">
+<h2><a name="Settings"></a>Settings</h2>
+<div class="section">
+<h3><a name="kafka_conf"></a><tt>kafka_conf</tt></h3>
+<p>The global configuration settings for Kafka. These values are passed through directly to librdkafka. Any valid librdkafka settings can be defined in this table. The full set of valid librdkafka settings are available <a class="externalLink" href="https://github.com/edenhill/librdkafka/blob/v0.9.4/CONFIGURATION.md">here</a>.</p>
+
+<div class="source">
+<div class="source">
+<pre>redef Kafka::kafka_conf = table(
+ ["metadata.broker.list"] = "localhost:9092",
+ ["client.id"] = "bro"
+);
+</pre></div></div></div>
+<div class="section">
+<h3><a name="topic_name"></a><tt>topic_name</tt></h3>
+<p>The name of the topic in Kafka where all Bro logs will be sent to.</p>
+
+<div class="source">
+<div class="source">
+<pre>redef Kafka::topic_name = "bro";
+</pre></div></div></div>
+<div class="section">
+<h3><a name="max_wait_on_shutdown"></a><tt>max_wait_on_shutdown</tt></h3>
+<p>The maximum number of milliseconds that the plugin will wait for any backlog of queued messages to be sent to Kafka before forced shutdown.</p>
+
+<div class="source">
+<div class="source">
+<pre>redef Kafka::max_wait_on_shutdown = 3000;
+</pre></div></div></div>
+<div class="section">
+<h3><a name="tag_json"></a><tt>tag_json</tt></h3>
+<p>If true, a log stream identifier is appended to each JSON-formatted message. For example, a Conn::LOG message will look like <tt>{ 'conn' : { ... }}</tt>.</p>
+
+<div class="source">
+<div class="source">
+<pre>redef Kafka::tag_json = T;
+</pre></div></div></div>
+<div class="section">
+<h3><a name="debug"></a><tt>debug</tt></h3>
+<p>A comma separated list of debug contexts in librdkafka which you want to enable. The available contexts are:</p>
+
+<ul>
+
+<li>generic</li>
+
+<li>broker</li>
+
+<li>topic</li>
+
+<li>metadata</li>
+
+<li>queue</li>
+
+<li>msg</li>
+
+<li>protocol</li>
+
+<li>cgrp</li>
+
+<li>security</li>
+
+<li>fetch</li>
+
+<li>feature</li>
+
+<li>all</li>
+</ul></div></div>
+<div class="section">
+<h2><a name="Kerberos"></a>Kerberos</h2>
+<p>This plugin supports producing messages from a kerberized kafka. There are a couple of prerequisites and a couple of settings to set. </p>
+<div class="section">
+<h3><a name="SASL"></a>SASL</h3>
+<p>If you are using SASL as a security protocol for kafka, then you must have libsasl or libsasl2 installed. You can tell if sasl is enabled by running the following from the directory in which you have build librdkafka:</p>
+
+<div class="source">
+<div class="source">
+<pre>examples/rdkafka_example -X builtin.features
+builtin.features = gzip,snappy,ssl,sasl,regex
+</pre></div></div></div>
+<div class="section">
+<h3><a name="Producer_Config"></a>Producer Config</h3>
+<p>As stated above, you can configure the producer kafka configs in <tt>${BRO_HOME}/share/bro/site/local.bro</tt>. There are a few configs necessary to set, which are described <a class="externalLink" href="https://github.com/edenhill/librdkafka/wiki/Using-SASL-with-librdkafka">here</a>. For an environment where the following is true:</p>
+
+<ul>
+
+<li>The broker is <tt>node1:6667</tt></li>
+
+<li>This kafka is using <tt>SASL_PLAINTEXT</tt> as the security protocol</li>
+
+<li>The keytab used is the <tt>metron</tt> keytab</li>
+
+<li>The service principal for <tt>metron</tt> is <tt>metron@EXAMPLE.COM</tt></li>
+</ul>
+<p>The kafka topic <tt>bro</tt> has been given permission for the <tt>metron</tt> user to write:</p>
+
+<div class="source">
+<div class="source">
+<pre># login using the metron user
+kinit -kt /etc/security/keytabs/metron.headless.keytab metron@EXAMPLE.COM
+${KAFKA_HOME}/kafka-broker/bin/kafka-acls.sh --authorizer kafka.security.auth.SimpleAclAuthorizer --authorizer-properties zookeeper.connect=node1:2181 --add --allow-principal User:metron --topic bro
+</pre></div></div>
+<p>The following is how the <tt>${BRO_HOME}/share/bro/site/local.bro</tt> looks:</p>
+
+<div class="source">
+<div class="source">
+<pre>@load Bro/Kafka/logs-to-kafka.bro
+redef Kafka::logs_to_send = set(HTTP::LOG, DNS::LOG);
+redef Kafka::topic_name = "bro";
+redef Kafka::tag_json = T;
+redef Kafka::kafka_conf = table( ["metadata.broker.list"] = "node1:6667"
+ , ["security.protocol"] = "SASL_PLAINTEXT"
+ , ["sasl.kerberos.keytab"] = "/etc/security/keytabs/metron.headless.keytab"
+ , ["sasl.kerberos.principal"] = "metron@EXAMPLE.COM"
+ );
+</pre></div></div></div></div>
+ </div>
+ </div>
+ </div>
+
+ <hr/>
+
+ <footer>
+ <div class="container-fluid">
+ <div class="row span12">Copyright © 2017
+ <a href="https://www.apache.org">The Apache Software Foundation</a>.
+ All Rights Reserved.
+
+ </div>
+
+
+
+ </div>
+ </footer>
+ </body>
+</html>
http://git-wip-us.apache.org/repos/asf/metron/blob/cf4c2ecd/current-book/metron-sensors/fastcapa/index.html
----------------------------------------------------------------------
diff --git a/current-book/metron-sensors/fastcapa/index.html b/current-book/metron-sensors/fastcapa/index.html
index a12f6e5..68d74d0 100644
--- a/current-book/metron-sensors/fastcapa/index.html
+++ b/current-book/metron-sensors/fastcapa/index.html
@@ -1,13 +1,13 @@
<!DOCTYPE html>
<!--
- | Generated by Apache Maven Doxia at 2017-02-23
+ | Generated by Apache Maven Doxia at 2017-06-27
| Rendered using Apache Maven Fluido Skin 1.3.0
-->
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
- <meta name="Date-Revision-yyyymmdd" content="20170223" />
+ <meta name="Date-Revision-yyyymmdd" content="20170627" />
<meta http-equiv="Content-Language" content="en" />
<title>Metron – Fastcapa</title>
<link rel="stylesheet" href="../../css/apache-maven-fluido-1.3.0.min.css" />
@@ -30,14 +30,11 @@
<div class="container-fluid">
<div id="banner">
<div class="pull-left">
- <a href="http://metron.incubator.apache.org/" id="bannerLeft">
- <img src="../../images/metron-logo.png" alt="Apache Metron - Incubating" width="148px" height="48px"/>
+ <a href="http://metron.apache.org/" id="bannerLeft">
+ <img src="../../images/metron-logo.png" alt="Apache Metron" width="148px" height="48px"/>
</a>
</div>
- <div class="pull-right"> <a href="http://incubator.apache.org/" id="bannerRight">
- <img src="../../images/ApacheIncubating_Logo.png" alt="Apache Incubating" width="192px" height="48px"/>
- </a>
- </div>
+ <div class="pull-right"> </div>
<div class="clear"><hr/></div>
</div>
@@ -51,8 +48,8 @@
</li>
<li class="divider ">/</li>
<li class="">
- <a href="http://metron.incubator.apache.org/" class="externalLink" title="Metron-Incubating">
- Metron-Incubating</a>
+ <a href="http://metron.apache.org/" class="externalLink" title="Metron">
+ Metron</a>
</li>
<li class="divider ">/</li>
<li class="">
@@ -64,8 +61,8 @@
- <li id="publishDate" class="pull-right">Last Published: 2017-02-23</li> <li class="divider pull-right">|</li>
- <li id="projectVersion" class="pull-right">Version: 0.3.1</li>
+ <li id="publishDate" class="pull-right">Last Published: 2017-06-27</li> <li class="divider pull-right">|</li>
+ <li id="projectVersion" class="pull-right">Version: 0.4.0</li>
</ul>
</div>
@@ -78,7 +75,7 @@
<ul class="nav nav-list">
<li class="nav-header">User Documentation</li>
-
+
<li>
<a href="../../index.html" title="Metron">
@@ -99,7 +96,7 @@
<i class="icon-chevron-right"></i>
Analytics</a>
</li>
-
+
<li>
<a href="../../metron-deployment/index.html" title="Deployment">
@@ -113,14 +110,28 @@
<i class="none"></i>
Docker</a>
</li>
-
+
+ <li>
+
+ <a href="../../metron-interface/metron-config/index.html" title="Config">
+ <i class="none"></i>
+ Config</a>
+ </li>
+
+ <li>
+
+ <a href="../../metron-interface/metron-rest/index.html" title="Rest">
+ <i class="none"></i>
+ Rest</a>
+ </li>
+
<li>
<a href="../../metron-platform/index.html" title="Platform">
<i class="icon-chevron-right"></i>
Platform</a>
</li>
-
+
<li>
<a href="../../metron-sensors/index.html" title="Sensors">
@@ -128,6 +139,13 @@
Sensors</a>
<ul class="nav nav-list">
+ <li>
+
+ <a href="../../metron-sensors/bro-plugin-kafka/index.html" title="Bro-plugin-kafka">
+ <i class="none"></i>
+ Bro-plugin-kafka</a>
+ </li>
+
<li class="active">
<a href="#"><i class="none"></i>Fastcapa</a>
@@ -164,16 +182,859 @@
<div id="bodyColumn" class="span9" >
<h1>Fastcapa</h1>
-<p>Fastcapa is an Apache Metron sensor that performs fast network packet capture by leveraging Linux kernel-bypass and user space networking technology. </p>
-<p>The sensor will bind to a network interface, capture network packets, and send the raw packet data to Kafka. This provides a scalable mechanism for ingesting high-volumes of network packet data into a Hadoop-y cluster.</p>
+<p>Fastcapa is a probe that performs fast network packet capture by leveraging Linux kernel-bypass and user space networking technology. The probe will bind to a network interface, capture network packets, and send the raw packet data to Kafka. This provides a scalable mechanism for ingesting high-volumes of network packet data into a Hadoop-y cluster.</p>
<p>Fastcapa leverages the Data Plane Development Kit (<a class="externalLink" href="http://dpdk.org/">DPDK</a>). DPDK is a set of libraries and drivers to perform fast packet processing in Linux user space. </p>
+
+<ul>
+
+<li><a href="#Quick_Start">Quick Start</a></li>
+
+<li><a href="#Requirements">Requirements</a></li>
+
+<li><a href="#Installation">Installation</a></li>
+
+<li><a href="#Usage">Usage</a>
+
+<ul>
+
+<li><a href="#Parameters">Parameters</a></li>
+
+<li><a href="#Output">Output</a></li>
+
+<li><a href="#Kerberos">Kerberos</a></li>
+ </ul></li>
+
+<li><a href="#How_It_Works">How It Works</a></li>
+
+<li><a href="#Performance">Performance</a></li>
+
+<li><a href="#FAQs">FAQs</a></li>
+</ul>
<div class="section">
-<h2><a name="Getting_Started"></a>Getting Started</h2>
-<p>The quickest way to get up and running is to use a Virtualbox environment on your local machine. The necessary files and instructions to do this are located at <a href="../../metron-deployment/vagrant/fastcapa-test-platform/index.html"><tt>metron-deployment/vagrant/fastcapa-vagrant</tt></a>. </p></div>
+<h2><a name="Quick_Start"></a>Quick Start</h2>
+<p>The quickest way to see Fastcapa in action is to use a Virtualbox environment on your local machine. The necessary files and instructions to do this are located at <a href="../../metron-deployment/vagrant/fastcapa-test-platform/index.html"><tt>metron-deployment/vagrant/fastcapa-vagrant</tt></a>. All you need to do is execute the following.</p>
+
+<div class="source">
+<div class="source">
+<pre>cd metron-deployment/vagrant/fastcapa-test-platform
+vagrant up
+</pre></div></div>
+<p>This environment sets up two nodes. One node produces network packets over a network interface. The second node uses Fastcapa to capture those packets and write them to a Kafka broker running on the first node. Basic validation is performed to ensure that Fastcapa is able to land packet data in Kafka.</p></div>
+<div class="section">
+<h2><a name="Requirements"></a>Requirements</h2>
+<p>The following system requirements must be met to run the Fastcapa probe.</p>
+
+<ul>
+
+<li>Linux kernel >= 2.6.34</li>
+
+<li>A <a class="externalLink" href="http://dpdk.org/doc/nics">DPDK supported ethernet device; NIC</a>.</li>
+
+<li>Port(s) on the ethernet device that can be dedicated for exclusive use by Fastcapa</li>
+</ul></div>
<div class="section">
<h2><a name="Installation"></a>Installation</h2>
<p>The process of installing Fastcapa has a fair number of steps and involves building DPDK, loading specific kernel modules, enabling huge page memory, and binding compatible network interface cards.</p>
-<p>The best documentation is code that actually does this for you. An Ansible role that performs the entire installation procedure can be found at <a href="../../metron-deployment/roles/fastcapa/index.html"><tt>metron-deployment/roles/fastcapa</tt></a>. Use this to install Fastcapa or as a guide for manual installation.</p></div>
+<div class="section">
+<h3><a name="Automated_Installation"></a>Automated Installation</h3>
+<p>The best documentation is code that actually does this for you. An Ansible role that performs the entire installation procedure can be found at <a href="../../metron-deployment/roles/fastcapa/index.html"><tt>metron-deployment/roles/fastcapa</tt></a>. Use this to install Fastcapa or as a guide for manual installation. The automated installation assumes CentOS 7.1 and is directly tested against <a class="externalLink" href="https://atlas.hashicorp.com/bento/boxes/centos-7.1">bento/centos-7.1</a>.</p></div>
+<div class="section">
+<h3><a name="Manual_Installation"></a>Manual Installation</h3>
+<p>The following manual installation steps assume that they are executed on CentOS 7.1. Some minor differences may result if you use a different Linux distribution.</p>
+
+<ul>
+
+<li><a href="#Enable_Transparent_Huge_Pages">Enable Transparent Huge Pages</a></li>
+
+<li><a href="#Install_DPDK">Install DPDK</a></li>
+
+<li><a href="#Install_Librdkafka">Install Librdkafka</a></li>
+
+<li><a href="#Install_Fastcapa">Install Fastcapa</a></li>
+</ul>
+<div class="section">
+<h4><a name="Enable_Transparent_Huge_Pages"></a>Enable Transparent Huge Pages</h4>
+<p>The probe performs its own memory management by leveraging transparent huge pages. In Linux, Transparent Huge Pages (THP) can be enabled either dynamically or on boot. It is recommended that these be allocated on boot to increase the chance that a larger, physically contiguous chunk of memory can be allocated.</p>
+<p>The size of THPs that are supported will vary based on your CPU. These typically include 2 MB and 1 GB THPs. For better performance, allocate 1 GB THPs if supported by your CPU.</p>
+
+<ol style="list-style-type: decimal">
+
+<li>
+<p>Ensure that your CPU supports 1 GB THPs. A CPU flag <tt>pdpe1gb</tt> indicates whether or not the CPU supports 1 GB THPs.</p>
+
+<div class="source">
+<div class="source">
+<pre>grep --color=always pdpe1gb /proc/cpuinfo | uniq
+</pre></div></div></li>
+
+<li>
+<p>Add the following boot parameters to the Linux kernel. Edit <tt>/etc/default/grub</tt> and add the additional kernel parameters to the line starting with <tt>GRUB_CMDLINE_LINUX</tt>.</p>
+
+<div class="source">
+<div class="source">
+<pre>GRUB_CMDLINE_LINUX=... default_hugepagesz=1G hugepagesz=1G hugepages=16
+</pre></div></div></li>
+
+<li>
+<p>Rebuild the grub configuration then reboot. The location of the Grub configuration file will differ across Linux distributions.</p>
+
+<div class="source">
+<div class="source">
+<pre>cp /etc/grub2-efi.cfg /etc/grub2-efi.cfg.orig
+/sbin/grub2-mkconfig -o /etc/grub2-efi.cfg
+</pre></div></div></li>
+
+<li>
+<p>Once the host has been rebooted, ensure that the THPs were successfully allocated.</p>
+
+<div class="source">
+<div class="source">
+<pre>$ grep HugePage /proc/meminfo
+AnonHugePages: 933888 kB
+HugePages_Total: 16
+HugePages_Free: 0
+HugePages_Rsvd: 0
+HugePages_Surp: 0
+</pre></div></div>
+<p>The total number of huge pages that you have been allocated should be distributed fairly evenly across each NUMA node. In this example, a total of 16 were requested and 8 have been assigned on each of the 2 NUMA nodes.</p>
+
+<div class="source">
+<div class="source">
+<pre>$ cat /sys/devices/system/node/node*/hugepages/hugepages-1048576kB/nr_hugepages
+8
+8
+</pre></div></div></li>
+
+<li>
+<p>Once the THPs have been reserved, they need to be mounted to make them available to the probe.</p>
+
+<div class="source">
+<div class="source">
+<pre>cp /etc/fstab /etc/fstab.orig
+mkdir -p /mnt/huge_1GB
+echo "nodev /mnt/huge_1GB hugetlbfs pagesize=1GB 0 0" >> /etc/fstab
+mount -fav
+</pre></div></div></li>
+</ol></div>
+<div class="section">
+<h4><a name="Install_DPDK"></a>Install DPDK</h4>
+
+<ol style="list-style-type: decimal">
+
+<li>
+<p>Install the required dependencies.</p>
+
+<div class="source">
+<div class="source">
+<pre>yum -y install "@Development tools"
+yum -y install pciutils net-tools glib2 glib2-devel git
+yum -y install kernel kernel-devel kernel-headers
+</pre></div></div></li>
+
+<li>
+<p>Decide where DPDK will be installed.</p>
+
+<div class="source">
+<div class="source">
+<pre>export DPDK_HOME=/usr/local/dpdk/
+</pre></div></div></li>
+
+<li>
+<p>Download, build and install DPDK.</p>
+
+<div class="source">
+<div class="source">
+<pre>wget http://fast.dpdk.org/rel/dpdk-16.11.1.tar.xz -O - | tar -xJ
+cd dpdk-stable-16.11.1/
+make config install T=x86_64-native-linuxapp-gcc DESTDIR=$DPDK_HOME
+</pre></div></div></li>
+
+<li>
+<p>Find the PCI address of the ethernet device that you plan on using to capture network packets. In the following example I plan on binding <tt>enp9s0f0</tt> which has a PCI address of <tt>09:00.0</tt>.</p>
+
+<div class="source">
+<div class="source">
+<pre>$ lspci | grep "VIC Ethernet"
+09:00.0 Ethernet controller: Cisco Systems Inc VIC Ethernet NIC (rev a2)
+0a:00.0 Ethernet controller: Cisco Systems Inc VIC Ethernet NIC (rev a2)
+</pre></div></div></li>
+
+<li>
+<p>Bind the device. Replace the device name and PCI address with what is appropriate for your environment.</p>
+
+<div class="source">
+<div class="source">
+<pre>ifdown enp9s0f0
+modprobe uio_pci_generic
+$DPDK_HOME/sbin/dpdk-devbind --bind=uio_pci_generic "09:00.0"
+</pre></div></div></li>
+
+<li>
+<p>Ensure that the device was bound. It should be shown as a ‘network device using DPDK-compatible driver.’</p>
+
+<div class="source">
+<div class="source">
+<pre>$ dpdk-devbind --status
+Network devices using DPDK-compatible driver
+============================================
+0000:09:00.0 'VIC Ethernet NIC' drv=uio_pci_generic unused=enic
+Network devices using kernel driver
+===================================
+0000:01:00.0 'I350 Gigabit Network Connection' if=eno1 drv=igb unused=uio_pci_generic
+</pre></div></div></li>
+</ol></div>
+<div class="section">
+<h4><a name="Install_Librdkafka"></a>Install Librdkafka</h4>
+<p>The probe has been tested with <a class="externalLink" href="https://github.com/edenhill/librdkafka/releases/tag/v0.9.4">Librdkafka 0.9.4</a>.</p>
+
+<ol style="list-style-type: decimal">
+
+<li>
+<p>Choose an installation path. In this example, the libs will actually be installed at <tt>/usr/local/lib</tt>; note that <tt>lib</tt> is appended to the prefix.</p>
+
+<div class="source">
+<div class="source">
+<pre>export RDK_PREFIX=/usr/local
+</pre></div></div></li>
+
+<li>
+<p>Download, build and install.</p>
+
+<div class="source">
+<div class="source">
+<pre>wget https://github.com/edenhill/librdkafka/archive/v0.9.4.tar.gz -O - | tar -xz
+cd librdkafka-0.9.4/
+./configure --prefix=$RDK_PREFIX
+make
+make install
+</pre></div></div></li>
+
+<li>
+<p>Ensure that the installation location is on the search path for the runtime shared library loader.</p>
+
+<div class="source">
+<div class="source">
+<pre>export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$RDK_PREFIX/lib
+</pre></div></div></li>
+</ol></div>
+<div class="section">
+<h4><a name="Install_Fastcapa"></a>Install Fastcapa</h4>
+
+<ol style="list-style-type: decimal">
+
+<li>
+<p>Set the required environment variables.</p>
+
+<div class="source">
+<div class="source">
+<pre>export RTE_SDK=$DPDK_HOME/share/dpdk/
+export RTE_TARGET=x86_64-native-linuxapp-gcc
+export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$RDK_HOME
+</pre></div></div></li>
+
+<li>
+<p>Build Fastcapa. The resulting binary will be placed at <tt>build/app/fastcapa</tt>.</p>
+
+<div class="source">
+<div class="source">
+<pre>cd metron/metron-sensors/fastcapa
+make
+</pre></div></div></li>
+</ol></div></div></div>
+<div class="section">
+<h2><a name="Usage"></a>Usage</h2>
+<p>Follow these steps to run Fastcapa.</p>
+
+<ol style="list-style-type: decimal">
+
+<li>
+<p>Create a configuration file that at a minimum specifies your Kafka broker. An example configuration file, <tt>conf/fastcapa.conf</tt>, is available that documents other useful parameters.</p>
+
+<div class="source">
+<div class="source">
+<pre>[kafka-global]
+metadata.broker.list = kafka-broker1:9092
+</pre></div></div></li>
+
+<li>
+<p>Bind the capture device. This is only needed if the device is not already bound. In this example, the device <tt>enp9s0f0</tt> with a PCI address of <tt>09:00:0</tt> is bound. Use values specific to your environment.</p>
+
+<div class="source">
+<div class="source">
+<pre>ifdown enp9s0f0
+modprobe uio_pci_generic
+$DPDK_HOME/sbin/dpdk-devbind --bind=uio_pci_generic "09:00.0"
+</pre></div></div></li>
+
+<li>
+<p>Run Fastcapa.</p>
+
+<div class="source">
+<div class="source">
+<pre>fastcapa -c 0x03 --huge-dir /mnt/huge_1GB -- -p 0x01 -t pcap -c /etc/fastcapa.conf
+</pre></div></div></li>
+
+<li>
+<p>Terminate Fastcapa with <tt>SIGINT</tt> or by entering <tt>CTRL-C</tt>. The probe will cleanly shut down all of the workers and allow the backlog of packets to drain. To terminate the process without clearing the queue, send a <tt>SIGKILL</tt> or be entering <tt>killall -9 fastcapa</tt>.</p></li>
+</ol>
+<div class="section">
+<h3><a name="Parameters"></a>Parameters</h3>
+<p>Fastcapa accepts three sets of parameters. </p>
+
+<ol style="list-style-type: decimal">
+
+<li>Command-line parameters passed directly to DPDK’s Environmental Abstraction Layer (EAL)</li>
+
+<li>Command-line parameters that define how Fastcapa will interact with DPDK. These parametera are separated on the command line by a double-dash (<tt>--</tt>).</li>
+
+<li>A configuration file that define how Fastcapa interacts with Librdkafka.</li>
+</ol>
+<div class="section">
+<h4><a name="Environmental_Abstraction_Layer_Parameters"></a>Environmental Abstraction Layer Parameters</h4>
+<p>The most commonly used EAL parameter involves specifying which logical CPU cores should be used for processing. This can be specified in any of the following ways.</p>
+
+<div class="source">
+<div class="source">
+<pre> -c COREMASK Hexadecimal bitmask of cores to run on
+ -l CORELIST List of cores to run on
+ The argument format is <c1>[-c2][,c3[-c4],...]
+ where c1, c2, etc are core indexes between 0 and 128
+ --lcores COREMAP Map lcore set to physical cpu set
+ The argument format is
+ '<lcores[@cpus]>[<,lcores[@cpus]>...]'
+ lcores and cpus list are grouped by '(' and ')'
+ Within the group, '-' is used for range separator,
+ ',' is used for single number separator.
+ '( )' can be omitted for single element group,
+ '@' can be omitted if cpus and lcores have the same value
+</pre></div></div>
+<p>To get more information about other EAL parameters, run the following.</p>
+
+<div class="source">
+<div class="source">
+<pre>fastcapa -h
+</pre></div></div></div>
+<div class="section">
+<h4><a name="Fastcapa-Core_Parameters"></a>Fastcapa-Core Parameters</h4>
+
+<table border="0" class="table table-striped">
+ <thead>
+
+<tr class="a">
+
+<th>Name </th>
+
+<th>Command </th>
+
+<th>Description </th>
+
+<th>Default </th>
+ </tr>
+ </thead>
+ <tbody>
+
+<tr class="b">
+
+<td>Port Mask </td>
+
+<td>-p PORT_MASK </td>
+
+<td>A bit mask identifying which ports to bind. </td>
+
+<td>0x01 </td>
+ </tr>
+
+<tr class="a">
+
+<td>Burst Size </td>
+
+<td>-b BURST_SIZE </td>
+
+<td>Maximum number of packets to receive at one time. </td>
+
+<td>32 </td>
+ </tr>
+
+<tr class="b">
+
+<td>Receive Descriptors </td>
+
+<td>-r NB_RX_DESC </td>
+
+<td>The number of descriptors for each receive queue (the size of the receive queue.) Limited by the ethernet device in use. </td>
+
+<td>1024 </td>
+ </tr>
+
+<tr class="a">
+
+<td>Transmission Ring Size </td>
+
+<td>-x TX_RING_SIZE </td>
+
+<td>The size of each transmission ring. This must be a power of 2. </td>
+
+<td>2048 </td>
+ </tr>
+
+<tr class="b">
+
+<td>Number Receive Queues </td>
+
+<td>-q NB_RX_QUEUE </td>
+
+<td>Number of receive queues to use for each port. Limited by the ethernet device in use. </td>
+
+<td>2 </td>
+ </tr>
+
+<tr class="a">
+
+<td>Kafka Topic </td>
+
+<td>-t KAFKA_TOPIC </td>
+
+<td>The name of the Kafka topic. </td>
+
+<td>pcap </td>
+ </tr>
+
+<tr class="b">
+
+<td>Configuration File </td>
+
+<td>-c KAFKA_CONF </td>
+
+<td>Path to a file containing configuration values. </td>
+
+<td> </td>
+ </tr>
+
+<tr class="a">
+
+<td>Stats </td>
+
+<td>-s KAFKA_STATS </td>
+
+<td>Appends performance metrics in the form of JSON strings to the specified file. </td>
+
+<td> </td>
+ </tr>
+ </tbody>
+</table>
+<p>To get more information about the Fastcapa specific parameters, run the following. Note that this puts the <tt>-h</tt> after the double-dash <tt>--</tt>.</p>
+
+<div class="source">
+<div class="source">
+<pre>fastcapa -- -h
+</pre></div></div></div>
+<div class="section">
+<h4><a name="Fastcapa-Kafka_Configuration_File"></a>Fastcapa-Kafka Configuration File</h4>
+<p>The path to the configuration file is specified with the <tt>-c</tt> command line argument. The file can contain any global or topic-specific, producer-focused <a class="externalLink" href="https://github.com/edenhill/librdkafka/blob/master/CONFIGURATION.md">configuration values accepted by Librdkafka</a>. </p>
+<p>The configuration file is a <tt>.ini</tt>-like Glib configuration file. The global configuration values should be placed under a <tt>[kafka-global]</tt> header and topic-specific values should be placed under <tt>[kafka-topic]</tt>.</p>
+<p>A minimally viable configuration file would only need to include the Kafka broker to connect to.</p>
+
+<div class="source">
+<div class="source">
+<pre>[kafka-global]
+metadata.broker.list = kafka-broker1:9092, kafka-broker2:9092
+</pre></div></div>
+<p>The configuration parameters that are important for either basic functioning or performance tuning of Fastcapa include the following.</p>
+<p>Global configuration values that should be located under the <tt>[kafka-global]</tt> header.</p>
+
+<table border="0" class="table table-striped">
+ <thead>
+
+<tr class="a">
+
+<th><i>Name</i> </th>
+
+<th><i>Description</i> </th>
+
+<th><i>Default</i> </th>
+ </tr>
+ </thead>
+ <tbody>
+
+<tr class="b">
+
+<td>metadata.broker.list </td>
+
+<td>Initial list of brokers as a CSV list of broker host or host:port </td>
+
+<td> </td>
+ </tr>
+
+<tr class="a">
+
+<td>client.id </td>
+
+<td>Client identifier. </td>
+
+<td> </td>
+ </tr>
+
+<tr class="b">
+
+<td>queue.buffering.max.messages </td>
+
+<td>Maximum number of messages allowed on the producer queue </td>
+
+<td>100000 </td>
+ </tr>
+
+<tr class="a">
+
+<td>queue.buffering.max.ms </td>
+
+<td>Maximum time, in milliseconds, for buffering data on the producer queue </td>
+
+<td>1000 </td>
+ </tr>
+
+<tr class="b">
+
+<td>message.copy.max.bytes </td>
+
+<td>Maximum size for message to be copied to buffer. Messages larger than this will be passed by reference (zero-copy) at the expense of larger iovecs. </td>
+
+<td>65535 </td>
+ </tr>
+
+<tr class="a">
+
+<td>batch.num.messages </td>
+
+<td>Maximum number of messages batched in one MessageSet </td>
+
+<td>10000 </td>
+ </tr>
+
+<tr class="b">
+
+<td>statistics.interval.ms </td>
+
+<td>How often statistics are emitted; 0 = never </td>
+
+<td>0 </td>
+ </tr>
+
+<tr class="a">
+
+<td>compression.codec </td>
+
+<td>Compression codec to use for compressing message sets; {none, gzip, snappy, lz4 } </td>
+
+<td>none </td>
+ </tr>
+ </tbody>
+</table>
+<p>Topic configuration values that should be located under the <tt>[kafka-topic]</tt> header.</p>
+
+<table border="0" class="table table-striped">
+ <thead>
+
+<tr class="a">
+
+<th><i>Name</i> </th>
+
+<th><i>Description</i> </th>
+
+<th><i>Default</i> </th>
+ </tr>
+ </thead>
+ <tbody>
+
+<tr class="b">
+
+<td>compression.codec </td>
+
+<td>Compression codec to use for compressing message sets; {none, gzip, snappy, lz4 } </td>
+
+<td>none </td>
+ </tr>
+
+<tr class="a">
+
+<td>request.required.acks </td>
+
+<td>How many acknowledgements the leader broker must receive from ISR brokers before responding to the request; { 0 = no ack, 1 = leader ack, -1 = all ISRs } </td>
+
+<td>1 </td>
+ </tr>
+
+<tr class="b">
+
+<td>message.timeout.ms </td>
+
+<td>Local message timeout. This value is only enforced locally and limits the time a produced message waits for successful delivery. A time of 0 is infinite. </td>
+
+<td>300000 </td>
+ </tr>
+
+<tr class="a">
+
+<td>queue.buffering.max.kbytes </td>
+
+<td>Maximum total message size sum allowed on the producer queue </td>
+
+<td> </td>
+ </tr>
+ </tbody>
+</table></div></div>
+<div class="section">
+<h3><a name="Output"></a>Output</h3>
+<p>When running the probe some basic counters are output to stdout. Of course during normal operation these values will be much larger.</p>
+
+<div class="source">
+<div class="source">
+<pre> ------ in ------ --- queued --- ----- out ----- ---- drops ----
+[nic] 8 - - -
+[rx] 8 0 8 0
+[tx] 8 0 8 0
+[kaf] 8 1 7 0
+</pre></div></div>
+
+<ul>
+
+<li><tt>[nic]</tt> + <tt>in</tt>: The ethernet device is reporting that it has seen 8 packets.</li>
+
+<li><tt>[rx]</tt> + <tt>in</tt>: The receive workers have consumed 8 packets from the device.</li>
+
+<li><tt>[rx]</tt> + <tt>out</tt>: The receive workers have enqueued 8 packets onto the transmission rings.</li>
+
+<li><tt>[rx]</tt> + <tt>drops</tt>: If the transmission rings become full it will prevent the receive workers from enqueuing additional packets. The excess packets are dropped. This value will never decrease.</li>
+
+<li><tt>[tx]</tt> + <tt>in</tt>: The transmission workers have consumed 8 packets.</li>
+
+<li><tt>[tx]</tt> + <tt>out</tt>: The transmission workers have packaged 8 packets into Kafka messages.</li>
+
+<li><tt>[tx]</tt> + <tt>drops</tt>: If the Kafka client library accepted fewer packets than expected. This value can increase or decrease over time as additional packets are acknowledged by the Kafka client library at a later point in time.</li>
+
+<li><tt>[kaf]</tt> + <tt>in</tt>: The Kafka client library has received 8 packets.</li>
+
+<li><tt>[kaf]</tt> + <tt>out</tt>: A total of 7 packets has successfully reached Kafka.</li>
+
+<li><tt>[kaf]</tt> + <tt>queued</tt>: There is 1 packet within the <tt>rdkafka</tt> queue waiting to be sent.</li>
+</ul></div>
+<div class="section">
+<h3><a name="Kerberos"></a>Kerberos</h3>
+<p>The probe can be used in a Kerberized environment. Follow these additional steps to use Fastcapa with Kerberos. The following assumptions have been made. These may need altered to fit your environment.</p>
+
+<ul>
+
+<li>The Kafka broker is at <tt>kafka1:6667</tt></li>
+
+<li>Zookeeper is at <tt>zookeeper1:2181</tt></li>
+
+<li>The Kafka security protocol is <tt>SASL_PLAINTEXT</tt></li>
+
+<li>The keytab used is located at <tt>/etc/security/keytabs/metron.headless.keytab</tt></li>
+
+<li>The service principal is <tt>metron@EXAMPLE.COM</tt></li>
+</ul>
+
+<ol style="list-style-type: decimal">
+
+<li>
+<p>Build Librdkafka with SASL support (<tt>--enable-sasl</tt>).</p>
+
+<div class="source">
+<div class="source">
+<pre>wget https://github.com/edenhill/librdkafka/archive/v0.9.4.tar.gz -O - | tar -xz
+cd librdkafka-0.9.4/
+./configure --prefix=$RDK_PREFIX --enable-sasl
+make
+make install
+</pre></div></div></li>
+
+<li>
+<p>Validate Librdkafka does indeed support SASL. Run the following command and ensure that <tt>sasl</tt> is returned as a built-in feature.</p>
+
+<div class="source">
+<div class="source">
+<pre>$ examples/rdkafka_example -X builtin.features
+builtin.features = gzip,snappy,ssl,sasl,regex
+</pre></div></div>
+<p>If it is not, ensure that you have <tt>libsasl</tt> or <tt>libsasl2</tt> installed. On CentOS, this can be installed with the following command.</p>
+
+<div class="source">
+<div class="source">
+<pre>yum install -y cyrus-sasl cyrus-sasl-devel cyrus-sasl-gssapi
+</pre></div></div></li>
+
+<li>
+<p>Grant access to your Kafka topic. In this example, it is simply named <tt>pcap</tt>.</p>
+
+<div class="source">
+<div class="source">
+<pre>$KAFKA_HOME/bin/kafka-acls.sh --authorizer kafka.security.auth.SimpleAclAuthorizer \
+ --authorizer-properties zookeeper.connect=zookeeper1:2181 \
+ --add --allow-principal User:metron --topic pcap
+</pre></div></div></li>
+
+<li>
+<p>Obtain a Kerberos ticket.</p>
+
+<div class="source">
+<div class="source">
+<pre>kinit -kt /etc/security/keytabs/metron.headless.keytab metron@EXAMPLE.COM
+</pre></div></div></li>
+
+<li>
+<p>Add the following additional configuration values to your Fastcapa configuration file.</p>
+
+<div class="source">
+<div class="source">
+<pre>security.protocol = SASL_PLAINTEXT
+sasl.kerberos.keytab = /etc/security/keytabs/metron.headless.keytab
+sasl.kerberos.principal = metron@EXAMPLE.COM
+</pre></div></div></li>
+
+<li>
+<p>Now run Fastcapa as you normally would. It should have no problem landing packets in your kerberized Kafka broker.</p></li>
+</ol></div></div>
+<div class="section">
+<h2><a name="How_It_Works"></a>How It Works</h2>
+<p>The probe leverages a poll-mode, burst-oriented mechanism to capture packets from a network interface and transmit them efficiently to a Kafka topic. Each packet is wrapped within a single Kafka message and the current timestamp, as epoch microseconds in network byte order, is attached as the message’s key.</p>
+<p>The probe leverages Receive Side Scaling (RSS), a feature provided by some ethernet devices that allows processing of received data to occur across multiple processes and logical cores. It does this by running a hash function on each packet, whose value assigns the packet to one, of possibly many, receive queues. The total number and size of these receive queues are limited by the ethernet device in use. More capable ethernet devices will offer a greater number and greater sized receive queues. </p>
+
+<ul>
+
+<li>Increasing the number of receive queues allows for greater parallelization of receive side processing.</li>
+
+<li>Increasing the size of each receive queue can allow the probe to handle larger, temporary spikes of network packets that can often occur.</li>
+</ul>
+<p>A set of receive workers, each assigned to a unique logical core, are responsible for fetching packets from the receive queues. There can only be one receive worker for each receive queue. The receive workers continually poll the receive queues and attempt to fetch multiple packets on each request. The maximum number of packets fetched in one request is known as the ‘burst size’. If the receive worker actually receives ‘burst size’ packets, then it is likely that the queue is under pressure and more packets are available. In this case the worker immediately fetches another ‘burst size’ set of packets. It repeats this process up to a fixed number of times while the queue is under pressure.</p>
+<p>The receive workers then enqueue the received packets into a fixed size ring buffer known as a transmit ring. There is always one transmit ring for each receive queue. A set of transmit workers then dequeue packets from the transmit rings. There can be one or more transmit workers assigned to any single transmit ring. Each transmit worker has its own unique connection to Kafka.</p>
+
+<ul>
+
+<li>Increasing the number of transmit workers allows for greater parallelization when writing data to Kafka.</li>
+
+<li>Increasing the size of the transmit rings allows the probe to better handle temporary interruptions and latency when writing to Kafka.</li>
+</ul>
+<p>After receiving the network packets from the transmit worker, the Kafka client library internally maintains its own send queue of messages. Multiple threads are then responsible for managing this queue and creating batches of messages that are sent in bulk to a Kafka broker. No control is exercised over this additional send queue and its worker threads, which can be an impediment to performance. This is an opportunity for improvement that can be addressed as follow-on work.</p></div>
+<div class="section">
+<h2><a name="Performance"></a>Performance</h2>
+<p>Beyond tuning the parameters previously described, the following should be carefully considered to achieve maximum performance.</p>
+<div class="section">
+<h3><a name="Kafka_Partitions"></a>Kafka Partitions</h3>
+<p>Parallelizing access to a topic in Kafka is achieved by defining multiple partitions. The greater the number of partitions, the more parallelizable access to that topic becomes. To achieve high throughput it is important to ensure that the Kafka topic in use has a large number of partitions, evenly distributed across each of the nodes in your Kafka cluster.</p>
+<p>The specific number of partitions needed will differ for each environment, but at least 128 partitions has been shown to significantly increase performance in some environments.</p></div>
+<div class="section">
+<h3><a name="Physical_Layout"></a>Physical Layout</h3>
+<p>It is important to note the physical layout of the hardware when assigning worker cores to the probe. The worker cores should be on the same NUMA node or socket as the ethernet device itself. Assigning logical cores across NUMA boundaries can significantly impede performance.</p>
+<p>The following commands can help identify logical cores that are located on the same NUMA node or socket as the ethernet device itself. These commands should be run when the device is still managed by the kernel itself; before binding the interface.</p>
+
+<div class="source">
+<div class="source">
+<pre>$ cat /sys/class/net/enp9s0f0/device/local_cpulist
+0-7,16-23
+</pre></div></div>
+<p>The following command can be used to better understand the physical layout of the CPU in relation to NUMA nodes.</p>
+
+<div class="source">
+<div class="source">
+<pre>$ lscpu
+...
+NUMA node0 CPU(s): 0-7,16-23
+NUMA node1 CPU(s): 8-15,24-31
+</pre></div></div>
+<p>In this example <tt>enp9s0f0</tt> is located on NUMA node 0 and is local to the logical cores 0-7 and 16-23. You should choose worker cores from this list.</p></div>
+<div class="section">
+<h3><a name="CPU_Isolation"></a>CPU Isolation</h3>
+<p>Once you have chosen the logical cores to use that are local to the ethernet device, it also beneficial to isolate those cores so that the Linux kernel scheduler does not attempt to run tasks on them. This can be done using the <tt>isolcpus</tt> kernel boot parameter.</p>
+
+<div class="source">
+<div class="source">
+<pre>isolcpus=0,1,2,3,4,5,6,7
+</pre></div></div></div>
+<div class="section">
+<h3><a name="Device_Limitations"></a>Device Limitations</h3>
+<p>Check the output of running the probe to ensure that there are no device limitations that you are not aware of. While you may have specified 16 receive queues on the command line, your device may not support that number. This is especially true for the number of receive queues and descriptors.</p>
+<p>The following example shows the output when the number of receive descriptors requested is greater than what can be supported by the device. In many cases the probe will not terminate, but will choose the maximum allowable value and continue. This behavior is specific to the underlying device driver in use.</p>
+
+<div class="source">
+<div class="source">
+<pre>PMD: rte_enic_pmd: Rq 0 Scatter rx mode enabled
+PMD: rte_enic_pmd: Rq 0 Scatter rx mode not being used
+PMD: rte_enic_pmd: Number of rx_descs too high, adjusting to maximum
+PMD: rte_enic_pmd: Using 512 rx descriptors (sop 512, data 0)
+PMD: rte_enic_pmd: Rq 1 Scatter rx mode enabled
+PMD: rte_enic_pmd: Rq 1 Scatter rx mode not being used
+PMD: rte_enic_pmd: Number of rx_descs too high, adjusting to maximum
+PMD: rte_enic_pmd: Using 512 rx descriptors (sop 512, data 0)
+PMD: rte_enic_pmd: TX Queues - effective number of descs:32
+PMD: rte_enic_pmd: vNIC resources used: wq 1 rq 4 cq 3 intr 0
+</pre></div></div></div>
+<div class="section">
+<h3><a name="More_Information"></a>More Information</h3>
+<p>More information on this topic can be found in <a class="externalLink" href="http://dpdk.org/doc/guides/linux_gsg/nic_perf_intel_platform.html">DPDK’s Getting Started Guide</a>.</p></div></div>
+<div class="section">
+<h2><a name="FAQs"></a>FAQs</h2>
+<div class="section">
+<h3><a name="No_free_hugepages_reported"></a>No free hugepages reported</h3>
+<p>Problem: When executing <tt>fastcapa</tt> it fails with the following error message.</p>
+
+<div class="source">
+<div class="source">
+<pre>EAL: No free hugepages reported in hugepages-1048576kB
+PANIC in rte_eal_init():
+Cannot get hugepage information
+</pre></div></div>
+<p>Solution: This can occur if any process that has been allocated THPs crashes and fails to return the resources. </p>
+
+<ul>
+
+<li>
+<p>Delete the THP files that are not in use.</p>
+
+<div class="source">
+<div class="source">
+<pre>rm -f /mnt/huge_1GB/rtemap_*
+</pre></div></div></li>
+
+<li>
+<p>If the first option does not work, re-mount the <tt>hugetlbfs</tt> file system.</p>
+
+<div class="source">
+<div class="source">
+<pre>umount -a -t hugetlbfs
+mount -a
+</pre></div></div></li>
+</ul></div>
+<div class="section">
+<h3><a name="No_ethernet_ports_detected"></a>No ethernet ports detected</h3>
+<p>Problem: When executing <tt>fastcapa</tt> it fails with the following error message.</p>
+
+<div class="source">
+<div class="source">
+<pre>EAL: Error - exiting with code: 1
+ Cause: No ethernet ports detected.
+</pre></div></div>
+
+<ul>
+
+<li>Solution 1: The <tt>uio_pci_generic</tt> kernel module has not been loaded.</li>
+</ul>
+
+<div class="source">
+<div class="source">
+<pre>modprobe uio_pci_generic
+</pre></div></div>
+
+<ul>
+
+<li>Solution 2: Ensure that the ethernet device is bound. Re-bind if necessary.</li>
+</ul>
+
+<div class="source">
+<div class="source">
+<pre> dpdk-devbind --unbind "09:00.0"
+ dpdk-devbind --bind=uio_pci_generic "09:00.0"
+ dpdk-devbind --status
+</pre></div></div></div></div>
</div>
</div>
</div>
@@ -182,8 +1043,9 @@
<footer>
<div class="container-fluid">
- <div class="row span12">Copyright © 2017.
- All Rights Reserved.
+ <div class="row span12">Copyright © 2017
+ <a href="https://www.apache.org">The Apache Software Foundation</a>.
+ All Rights Reserved.
</div>
http://git-wip-us.apache.org/repos/asf/metron/blob/cf4c2ecd/current-book/metron-sensors/index.html
----------------------------------------------------------------------
diff --git a/current-book/metron-sensors/index.html b/current-book/metron-sensors/index.html
index a340bbe..e750460 100644
--- a/current-book/metron-sensors/index.html
+++ b/current-book/metron-sensors/index.html
@@ -1,13 +1,13 @@
<!DOCTYPE html>
<!--
- | Generated by Apache Maven Doxia at 2017-02-23
+ | Generated by Apache Maven Doxia at 2017-06-27
| Rendered using Apache Maven Fluido Skin 1.3.0
-->
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
- <meta name="Date-Revision-yyyymmdd" content="20170223" />
+ <meta name="Date-Revision-yyyymmdd" content="20170627" />
<meta http-equiv="Content-Language" content="en" />
<title>Metron – Metron Sensors</title>
<link rel="stylesheet" href="../css/apache-maven-fluido-1.3.0.min.css" />
@@ -30,14 +30,11 @@
<div class="container-fluid">
<div id="banner">
<div class="pull-left">
- <a href="http://metron.incubator.apache.org/" id="bannerLeft">
- <img src="../images/metron-logo.png" alt="Apache Metron - Incubating" width="148px" height="48px"/>
+ <a href="http://metron.apache.org/" id="bannerLeft">
+ <img src="../images/metron-logo.png" alt="Apache Metron" width="148px" height="48px"/>
</a>
</div>
- <div class="pull-right"> <a href="http://incubator.apache.org/" id="bannerRight">
- <img src="../images/ApacheIncubating_Logo.png" alt="Apache Incubating" width="192px" height="48px"/>
- </a>
- </div>
+ <div class="pull-right"> </div>
<div class="clear"><hr/></div>
</div>
@@ -51,8 +48,8 @@
</li>
<li class="divider ">/</li>
<li class="">
- <a href="http://metron.incubator.apache.org/" class="externalLink" title="Metron-Incubating">
- Metron-Incubating</a>
+ <a href="http://metron.apache.org/" class="externalLink" title="Metron">
+ Metron</a>
</li>
<li class="divider ">/</li>
<li class="">
@@ -64,8 +61,8 @@
- <li id="publishDate" class="pull-right">Last Published: 2017-02-23</li> <li class="divider pull-right">|</li>
- <li id="projectVersion" class="pull-right">Version: 0.3.1</li>
+ <li id="publishDate" class="pull-right">Last Published: 2017-06-27</li> <li class="divider pull-right">|</li>
+ <li id="projectVersion" class="pull-right">Version: 0.4.0</li>
</ul>
</div>
@@ -78,7 +75,7 @@
<ul class="nav nav-list">
<li class="nav-header">User Documentation</li>
-
+
<li>
<a href="../index.html" title="Metron">
@@ -99,7 +96,7 @@
<i class="icon-chevron-right"></i>
Analytics</a>
</li>
-
+
<li>
<a href="../metron-deployment/index.html" title="Deployment">
@@ -113,14 +110,28 @@
<i class="none"></i>
Docker</a>
</li>
-
+
+ <li>
+
+ <a href="../metron-interface/metron-config/index.html" title="Config">
+ <i class="none"></i>
+ Config</a>
+ </li>
+
+ <li>
+
+ <a href="../metron-interface/metron-rest/index.html" title="Rest">
+ <i class="none"></i>
+ Rest</a>
+ </li>
+
<li>
<a href="../metron-platform/index.html" title="Platform">
<i class="icon-chevron-right"></i>
Platform</a>
</li>
-
+
<li class="active">
<a href="#"><i class="icon-chevron-down"></i>Sensors</a>
@@ -128,6 +139,13 @@
<li>
+ <a href="../metron-sensors/bro-plugin-kafka/index.html" title="Bro-plugin-kafka">
+ <i class="none"></i>
+ Bro-plugin-kafka</a>
+ </li>
+
+ <li>
+
<a href="../metron-sensors/fastcapa/index.html" title="Fastcapa">
<i class="none"></i>
Fastcapa</a>
@@ -163,15 +181,20 @@
<div id="bodyColumn" class="span9" >
- <h1>Metron Sensors</h1>
-<p><a name="Metron_Sensors"></a></p>
+ <div class="section">
+<h2><a name="Metron_Sensors"></a>Metron Sensors</h2>
<ul>
-<li>Fast CAPA</li>
+<li>
+<p><a href="bro-plugin-kafka/index.html"><tt>bro-plugin-kafka</tt></a>: Provides integration between <a class="externalLink" href="https://www.bro.org/">Bro</a> and Kafka. A Bro plugin that sends logging output to Kafka. This provides a convenient means for tools in the Hadoop ecosystem, such as Storm, Spark, and others to process the data generated by Bro.</p></li>
+
+<li>
+<p><a href="fastcapa/index.html"><tt>fastcapa</tt></a>: Performs fast network packet capture by leveraging Linux kernel-bypass and user space networking technology. The probe will bind to a network interface, capture network packets, and send the raw packet data to Kafka. This provides a scalable mechanism for ingesting high-volumes of network packet data.</p></li>
-<li>Py CAPA</li>
-</ul>
+<li>
+<p><a href="pycapa/index.html"><tt>pycapa</tt></a>: Performs lightweight network packet capture, retrieves network packets from Kafka, generates <tt>libpcap</tt>-compliant files, and enables integration with third-party tools like Wireshark.</p></li>
+</ul></div>
</div>
</div>
</div>
@@ -180,8 +203,9 @@
<footer>
<div class="container-fluid">
- <div class="row span12">Copyright © 2017.
- All Rights Reserved.
+ <div class="row span12">Copyright © 2017
+ <a href="https://www.apache.org">The Apache Software Foundation</a>.
+ All Rights Reserved.
</div>
[04/10] metron git commit: METRON-1012 Update Metron public web site
for 0.4.0 release - push to asf-site from the generated code on master
Posted by ma...@apache.org.
http://git-wip-us.apache.org/repos/asf/metron/blob/cf4c2ecd/current-book/metron-platform/metron-common/3rdPartyStellar.html
----------------------------------------------------------------------
diff --git a/current-book/metron-platform/metron-common/3rdPartyStellar.html b/current-book/metron-platform/metron-common/3rdPartyStellar.html
new file mode 100644
index 0000000..3e7b190
--- /dev/null
+++ b/current-book/metron-platform/metron-common/3rdPartyStellar.html
@@ -0,0 +1,398 @@
+<!DOCTYPE html>
+<!--
+ | Generated by Apache Maven Doxia at 2017-06-27
+ | Rendered using Apache Maven Fluido Skin 1.3.0
+-->
+<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
+ <head>
+ <meta charset="UTF-8" />
+ <meta name="viewport" content="width=device-width, initial-scale=1.0" />
+ <meta name="Date-Revision-yyyymmdd" content="20170627" />
+ <meta http-equiv="Content-Language" content="en" />
+ <title>Metron – Custom Stellar Functions</title>
+ <link rel="stylesheet" href="../../css/apache-maven-fluido-1.3.0.min.css" />
+ <link rel="stylesheet" href="../../css/site.css" />
+ <link rel="stylesheet" href="../../css/print.css" media="print" />
+
+
+ <script type="text/javascript" src="../../js/apache-maven-fluido-1.3.0.min.js"></script>
+
+
+
+<script type="text/javascript">$( document ).ready( function() { $( '.carousel' ).carousel( { interval: 3500 } ) } );</script>
+
+ </head>
+ <body class="topBarDisabled">
+
+
+
+
+ <div class="container-fluid">
+ <div id="banner">
+ <div class="pull-left">
+ <a href="http://metron.apache.org/" id="bannerLeft">
+ <img src="../../images/metron-logo.png" alt="Apache Metron" width="148px" height="48px"/>
+ </a>
+ </div>
+ <div class="pull-right"> </div>
+ <div class="clear"><hr/></div>
+ </div>
+
+ <div id="breadcrumbs">
+ <ul class="breadcrumb">
+
+
+ <li class="">
+ <a href="http://www.apache.org" class="externalLink" title="Apache">
+ Apache</a>
+ </li>
+ <li class="divider ">/</li>
+ <li class="">
+ <a href="http://metron.apache.org/" class="externalLink" title="Metron">
+ Metron</a>
+ </li>
+ <li class="divider ">/</li>
+ <li class="">
+ <a href="../../index.html" title="Documentation">
+ Documentation</a>
+ </li>
+ <li class="divider ">/</li>
+ <li class="">Custom Stellar Functions</li>
+
+
+
+ <li id="publishDate" class="pull-right">Last Published: 2017-06-27</li> <li class="divider pull-right">|</li>
+ <li id="projectVersion" class="pull-right">Version: 0.4.0</li>
+
+ </ul>
+ </div>
+
+
+ <div class="row-fluid">
+ <div id="leftColumn" class="span3">
+ <div class="well sidebar-nav">
+
+
+ <ul class="nav nav-list">
+ <li class="nav-header">User Documentation</li>
+
+ <li>
+
+ <a href="../../index.html" title="Metron">
+ <i class="icon-chevron-down"></i>
+ Metron</a>
+ <ul class="nav nav-list">
+
+ <li>
+
+ <a href="../../Upgrading.html" title="Upgrading">
+ <i class="none"></i>
+ Upgrading</a>
+ </li>
+
+ <li>
+
+ <a href="../../metron-analytics/index.html" title="Analytics">
+ <i class="icon-chevron-right"></i>
+ Analytics</a>
+ </li>
+
+ <li>
+
+ <a href="../../metron-deployment/index.html" title="Deployment">
+ <i class="icon-chevron-right"></i>
+ Deployment</a>
+ </li>
+
+ <li>
+
+ <a href="../../metron-docker/index.html" title="Docker">
+ <i class="none"></i>
+ Docker</a>
+ </li>
+
+ <li>
+
+ <a href="../../metron-interface/metron-config/index.html" title="Config">
+ <i class="none"></i>
+ Config</a>
+ </li>
+
+ <li>
+
+ <a href="../../metron-interface/metron-rest/index.html" title="Rest">
+ <i class="none"></i>
+ Rest</a>
+ </li>
+
+ <li>
+
+ <a href="../../metron-platform/index.html" title="Platform">
+ <i class="icon-chevron-down"></i>
+ Platform</a>
+ <ul class="nav nav-list">
+
+ <li>
+
+ <a href="../../metron-platform/metron-api/index.html" title="Api">
+ <i class="none"></i>
+ Api</a>
+ </li>
+
+ <li>
+
+ <a href="../../metron-platform/metron-common/index.html" title="Common">
+ <i class="icon-chevron-down"></i>
+ Common</a>
+ <ul class="nav nav-list">
+
+ <li class="active">
+
+ <a href="#"><i class="none"></i>3rdPartyStellar</a>
+ </li>
+ </ul>
+ </li>
+
+ <li>
+
+ <a href="../../metron-platform/metron-data-management/index.html" title="Data-management">
+ <i class="none"></i>
+ Data-management</a>
+ </li>
+
+ <li>
+
+ <a href="../../metron-platform/metron-enrichment/index.html" title="Enrichment">
+ <i class="none"></i>
+ Enrichment</a>
+ </li>
+
+ <li>
+
+ <a href="../../metron-platform/metron-indexing/index.html" title="Indexing">
+ <i class="none"></i>
+ Indexing</a>
+ </li>
+
+ <li>
+
+ <a href="../../metron-platform/metron-management/index.html" title="Management">
+ <i class="none"></i>
+ Management</a>
+ </li>
+
+ <li>
+
+ <a href="../../metron-platform/metron-parsers/index.html" title="Parsers">
+ <i class="none"></i>
+ Parsers</a>
+ </li>
+
+ <li>
+
+ <a href="../../metron-platform/metron-pcap-backend/index.html" title="Pcap-backend">
+ <i class="none"></i>
+ Pcap-backend</a>
+ </li>
+
+ <li>
+
+ <a href="../../metron-platform/metron-writer/index.html" title="Writer">
+ <i class="none"></i>
+ Writer</a>
+ </li>
+ </ul>
+ </li>
+
+ <li>
+
+ <a href="../../metron-sensors/index.html" title="Sensors">
+ <i class="icon-chevron-right"></i>
+ Sensors</a>
+ </li>
+ </ul>
+ </li>
+ </ul>
+
+
+
+ <hr class="divider" />
+
+ <div id="poweredBy">
+ <div class="clear"></div>
+ <div class="clear"></div>
+ <div class="clear"></div>
+ <a href="http://maven.apache.org/" title="Built by Maven" class="poweredBy">
+ <img class="builtBy" alt="Built by Maven" src="../../images/logos/maven-feather.png" />
+ </a>
+ </div>
+ </div>
+ </div>
+
+
+ <div id="bodyColumn" class="span9" >
+
+ <h1>Custom Stellar Functions</h1>
+<p><a name="Custom_Stellar_Functions"></a></p>
+<p>Metron is fundamentally a programmable, extensible system and Stellar is the extension language. We have some great Stellar functions available out of the box and we’ll be adding more over time, but they may not quite scratch quite your particular itch. </p>
+<p>Of course, we’d love to have your contribution inside of Metron if you think it general purpose enough, but not every function is general-purpose or it may rely on libraries those licenses aren’t acceptable for an Apache project. In that case, then you will be wondering how to add your custom function to a running instance of Metron.</p>
+<div class="section">
+<h2><a name="Building_Your_Own_Function"></a>Building Your Own Function</h2>
+<p>Let’s say that I need a function that returns the current time in milliseconds since the epoch. I notice that there’s nothing like that currently in Metron, so I embark on the adventure of adding it for my cluster.</p>
+<p>I will presume that you have an installed Metron into your local maven repo via <tt>mvn install</tt> . In the future, when we publish to a maven repo, you will not need this. I will depend on 0.4.0 for the purpose of this demonstration</p>
+<div class="section">
+<h3><a name="Hack_Hack_Hack"></a>Hack, Hack, Hack</h3>
+<p>I like to use Maven, so we’ll use that for this demonstration, but you can use whatever build system that you like. Here’s my favorite way to build a project with groupId <tt>com.mycompany.stellar</tt> and artifactId of <tt>tempus</tt> <tt>mvn archetype:create -DgroupId=com.mycompany.stellar -DartifactId=tempus -DarchetypeArtifactId=maven-archetype-quickstart</tt></p>
+<p>First, we should depend on <tt>metron-common</tt> and we can do that by adjusting the <tt>pom.xml</tt> just created:</p>
+
+<div class="source">
+<div class="source">
+<pre><project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+ <modelVersion>4.0.0</modelVersion>
+
+ <groupId>com.mycompany.stellar</groupId>
+ <artifactId>tempus</artifactId>
+ <version>1.0-SNAPSHOT</version>
+ <packaging>jar</packaging>
+
+ <name>Stellar Time Functions</name>
+ <url>http://mycompany.com</url>
+
+ <properties>
+ <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
+ </properties>
+
+ <dependencies>
+ <dependency>
+ <groupId>org.apache.metron</groupId>
+ <artifactId>metron-common</artifactId>
+ <version>0.4.0</version>
+ <!-- NOTE: We will want to depend on the deployed common on the classpath. -->
+ <scope>provided</scope>
+ </dependency>
+ <dependency>
+ <groupId>junit</groupId>
+ <artifactId>junit</artifactId>
+ <version>3.8.1</version>
+ <scope>test</scope>
+ </dependency>
+ </dependencies>
+</project>
+</pre></div></div>
+<p>Let’s add our implementation in <tt>src/main/java/com/mycompany/stellar/TimeFunctions.java</tt> with the following content:</p>
+
+<div class="source">
+<div class="source">
+<pre>package com.notmetron.stellar;
+
+import org.apache.metron.common.dsl.Context;
+import org.apache.metron.common.dsl.ParseException;
+import org.apache.metron.common.dsl.Stellar;
+import org.apache.metron.common.dsl.StellarFunction;
+
+import java.util.List;
+
+public class TimeFunction {
+ @Stellar( name="NOW",
+ description = "Right now!",
+ params = {},
+ returns="Timestamp"
+ )
+ public static class Now implements StellarFunction {
+
+ public Object apply(List<Object> list, Context context) throws ParseException {
+ return System.currentTimeMillis();
+ }
+
+ public void initialize(Context context) { }
+
+ public boolean isInitialized() {
+ return true;
+ }
+ }
+}
+</pre></div></div>
+<p>Now we can build the project via <tt>mvn package</tt> which will create a <tt>target/tempus-1.0-SNAPSHOT.jar</tt> file.</p></div></div>
+<div class="section">
+<h2><a name="Install_the_Function"></a>Install the Function</h2>
+<p>Now that we have a jar with our custom function, we must make Metron aware of it.</p>
+<div class="section">
+<h3><a name="Deploy_the_Jar"></a>Deploy the Jar</h3>
+<p>First you need to place the jar in HDFS, if we have it on an access node, one way to do that is:</p>
+
+<ul>
+
+<li><tt>hadoop fs -put tempus-1.0-SNAPSHOT.jar /apps/metron/stellar</tt> This presumes that:</li>
+
+<li>you’ve standardized on <tt>/apps/metron/stellar</tt> as the location for custom jars</li>
+
+<li>you are running the command from an access node with the <tt>hadoop</tt> command installed</li>
+
+<li>you are running from a user that has write access to <tt>/apps/metron/stellar</tt></li>
+</ul></div>
+<div class="section">
+<h3><a name="Set_Global_Config"></a>Set Global Config</h3>
+<p>You may not need this if your Metron administrator already has this setup.</p>
+<p>With that dispensed with, we need to ensure that Metron knows to look at that location. We need to ensure that the <tt>stellar.function.paths</tt> property in the <tt>global.json</tt> is in place that makes Metron aware to look for Stellar functions in <tt>/apps/metron/stellar</tt> on HDFS. </p>
+<p>This property looks like, the following for a vagrant install</p>
+
+<div class="source">
+<div class="source">
+<pre>{
+ "es.clustername": "metron",
+ "es.ip": "node1",
+ "es.port": "9300",
+ "es.date.format": "yyyy.MM.dd.HH",
+ "stellar.function.paths" : "hdfs://node1:8020/apps/metron/stellar/.*.jar",
+}
+</pre></div></div>
+<p>The <tt>stellar.function.paths</tt> property takes a comma separated list of URIs or URIs with regex expressions at the end. Also, note path is prefaced by the HDFS default name, which, if you do not know, can be found by executing, <tt>hdfs getconf -confKey fs.default.name</tt>, such as</p>
+
+<div class="source">
+<div class="source">
+<pre>[root@node1 ~]# hdfs getconf -confKey fs.default.name
+hdfs://node1:8020
+</pre></div></div></div>
+<div class="section">
+<h3><a name="Use_the_Function"></a>Use the Function</h3>
+<p>Now that we have deployed the function, if we want to use it, any running topologies that use Stellar will need to be restarted.</p>
+<p>Beyond that, let’s take a look at it in the REPL:</p>
+
+<div class="source">
+<div class="source">
+<pre>Stellar, Go!
+Please note that functions are loading lazily in the background and will be unavailable until loaded fully.
+{es.clustername=metron, es.ip=node1, es.port=9300, es.date.format=yyyy.MM.dd.HH, stellar.function.paths=hdfs://node1:8020/apps/metron/stellar/.*.jar, profiler.client.period.duration=1, profiler.client.period.duration.units=MINUTES}
+[Stellar]>>> # Get the help for NOW
+[Stellar]>>> ?NOW
+Functions loaded, you may refer to functions now...
+NOW
+Description: Right now!
+
+Returns: Timestamp
+[Stellar]>>> # Try to run the NOW function, which we added:
+[Stellar]>>> NOW()
+1488400515655
+[Stellar]>>> # Looks like I got a timestamp, success!
+</pre></div></div></div></div>
+ </div>
+ </div>
+ </div>
+
+ <hr/>
+
+ <footer>
+ <div class="container-fluid">
+ <div class="row span12">Copyright © 2017
+ <a href="https://www.apache.org">The Apache Software Foundation</a>.
+ All Rights Reserved.
+
+ </div>
+
+
+
+ </div>
+ </footer>
+ </body>
+</html>
http://git-wip-us.apache.org/repos/asf/metron/blob/cf4c2ecd/current-book/metron-platform/metron-common/index.html
----------------------------------------------------------------------
diff --git a/current-book/metron-platform/metron-common/index.html b/current-book/metron-platform/metron-common/index.html
index c116276..7ddbc27 100644
--- a/current-book/metron-platform/metron-common/index.html
+++ b/current-book/metron-platform/metron-common/index.html
@@ -1,13 +1,13 @@
<!DOCTYPE html>
<!--
- | Generated by Apache Maven Doxia at 2017-02-23
+ | Generated by Apache Maven Doxia at 2017-06-27
| Rendered using Apache Maven Fluido Skin 1.3.0
-->
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
- <meta name="Date-Revision-yyyymmdd" content="20170223" />
+ <meta name="Date-Revision-yyyymmdd" content="20170627" />
<meta http-equiv="Content-Language" content="en" />
<title>Metron – Contents</title>
<link rel="stylesheet" href="../../css/apache-maven-fluido-1.3.0.min.css" />
@@ -30,14 +30,11 @@
<div class="container-fluid">
<div id="banner">
<div class="pull-left">
- <a href="http://metron.incubator.apache.org/" id="bannerLeft">
- <img src="../../images/metron-logo.png" alt="Apache Metron - Incubating" width="148px" height="48px"/>
+ <a href="http://metron.apache.org/" id="bannerLeft">
+ <img src="../../images/metron-logo.png" alt="Apache Metron" width="148px" height="48px"/>
</a>
</div>
- <div class="pull-right"> <a href="http://incubator.apache.org/" id="bannerRight">
- <img src="../../images/ApacheIncubating_Logo.png" alt="Apache Incubating" width="192px" height="48px"/>
- </a>
- </div>
+ <div class="pull-right"> </div>
<div class="clear"><hr/></div>
</div>
@@ -51,8 +48,8 @@
</li>
<li class="divider ">/</li>
<li class="">
- <a href="http://metron.incubator.apache.org/" class="externalLink" title="Metron-Incubating">
- Metron-Incubating</a>
+ <a href="http://metron.apache.org/" class="externalLink" title="Metron">
+ Metron</a>
</li>
<li class="divider ">/</li>
<li class="">
@@ -64,8 +61,8 @@
- <li id="publishDate" class="pull-right">Last Published: 2017-02-23</li> <li class="divider pull-right">|</li>
- <li id="projectVersion" class="pull-right">Version: 0.3.1</li>
+ <li id="publishDate" class="pull-right">Last Published: 2017-06-27</li> <li class="divider pull-right">|</li>
+ <li id="projectVersion" class="pull-right">Version: 0.4.0</li>
</ul>
</div>
@@ -78,7 +75,7 @@
<ul class="nav nav-list">
<li class="nav-header">User Documentation</li>
-
+
<li>
<a href="../../index.html" title="Metron">
@@ -99,7 +96,7 @@
<i class="icon-chevron-right"></i>
Analytics</a>
</li>
-
+
<li>
<a href="../../metron-deployment/index.html" title="Deployment">
@@ -113,7 +110,21 @@
<i class="none"></i>
Docker</a>
</li>
-
+
+ <li>
+
+ <a href="../../metron-interface/metron-config/index.html" title="Config">
+ <i class="none"></i>
+ Config</a>
+ </li>
+
+ <li>
+
+ <a href="../../metron-interface/metron-rest/index.html" title="Rest">
+ <i class="none"></i>
+ Rest</a>
+ </li>
+
<li>
<a href="../../metron-platform/index.html" title="Platform">
@@ -127,11 +138,20 @@
<i class="none"></i>
Api</a>
</li>
-
+
<li class="active">
- <a href="#"><i class="none"></i>Common</a>
- </li>
+ <a href="#"><i class="icon-chevron-down"></i>Common</a>
+ <ul class="nav nav-list">
+
+ <li>
+
+ <a href="../../metron-platform/metron-common/3rdPartyStellar.html" title="3rdPartyStellar">
+ <i class="none"></i>
+ 3rdPartyStellar</a>
+ </li>
+ </ul>
+ </li>
<li>
@@ -174,9 +194,16 @@
<i class="none"></i>
Pcap-backend</a>
</li>
+
+ <li>
+
+ <a href="../../metron-platform/metron-writer/index.html" title="Writer">
+ <i class="none"></i>
+ Writer</a>
+ </li>
</ul>
</li>
-
+
<li>
<a href="../../metron-sensors/index.html" title="Sensors">
@@ -218,6 +245,8 @@
<li><a href="#Stellar_Core_Functions">Stellar Core Functions</a></li>
+<li><a href="#Stellar_Benchmarks">Stellar Benchmarks</a></li>
+
<li><a href="#Stellar_Shell">Stellar Shell</a></li>
</ul></li>
@@ -234,7 +263,25 @@
<li>Referencing fields in the enriched JSON</li>
-<li>Simple boolean operations: <tt>and</tt>, <tt>not</tt>, <tt>or</tt></li>
+<li>String literals are quoted with either <tt>'</tt> or <tt>"</tt>.</li>
+
+<li>String literals support escaping for <tt>'</tt>, <tt>"</tt>, <tt>\t</tt>, <tt>\r</tt>, <tt>\n</tt>, and backslash
+
+<ul>
+
+<li>The literal <tt>'\'foo\''</tt> would represent <tt>'foo'</tt></li>
+
+<li>The literal <tt>"\"foo\""</tt> would represent <tt>"foo"</tt></li>
+
+<li>The literal <tt>'foo \\ bar'</tt> would represent <tt>foo \ bar</tt></li>
+ </ul></li>
+
+<li>Simple boolean operations: <tt>and</tt>, <tt>not</tt>, <tt>or</tt>
+
+<ul>
+
+<li>Boolean expressions are short-circuited (e.g. <tt>true or FUNC()</tt> would never execute <tt>FUNC</tt>)</li>
+ </ul></li>
<li>Simple arithmetic operations: <tt>*</tt>, <tt>/</tt>, <tt>+</tt>, <tt>-</tt> on real numbers or integers</li>
@@ -250,7 +297,7 @@
<li>The ability to have parenthesis to make order of operations explicit</li>
-<li>User defined functions</li>
+<li>User defined functions, including Lambda expressions</li>
</ul>
<div class="section">
<h2><a name="Stellar_Language_Keywords"></a>Stellar Language Keywords</h2>
@@ -390,6 +437,51 @@
</ol>
<p>The <tt>!=</tt> operator is the negation of the above.</p></div>
<div class="section">
+<h2><a name="Stellar_Language_Lambda_Expressions"></a>Stellar Language Lambda Expressions</h2>
+<p>Stellar provides the capability to pass lambda expressions to functions which wish to support that layer of indirection. The syntax is:</p>
+
+<ul>
+
+<li><tt>(named_variables) -> stellar_expression</tt> : Lambda expression with named variables
+
+<ul>
+
+<li>For instance, the lambda expression which calls <tt>TO_UPPER</tt> on a named argument <tt>x</tt> could be be expressed as <tt>(x) -> TO_UPPER(x)</tt>.</li>
+ </ul></li>
+
+<li><tt>var -> stellar_expression</tt> : Lambda expression with a single named variable, <tt>var</tt>
+
+<ul>
+
+<li>For instance, the lambda expression which calls <tt>TO_UPPER</tt> on a named argument <tt>x</tt> could be expressed as <tt>x -> TO_UPPER(x)</tt>. Note, this is more succinct but equivalent to the example directly above.</li>
+ </ul></li>
+
+<li><tt>() -> stellar_expression</tt> : Lambda expression with no named variables.
+
+<ul>
+
+<li>If no named variables are needed, you may omit the named variable section. For instance, the lambda expression which returns a constant <tt>false</tt> would be <tt>() -> false</tt></li>
+ </ul></li>
+</ul>
+<p>where </p>
+
+<ul>
+
+<li><tt>named_variables</tt> is a comma separated list of variables to use in the Stellar expression</li>
+
+<li><tt>stellar_expression</tt> is an arbitrary stellar expression</li>
+</ul>
+<p>In the core language functions, we support basic functional programming primitives such as</p>
+
+<ul>
+
+<li><tt>MAP</tt> - Applies a lambda expression over a list of input. For instance <tt>MAP([ 'foo', 'bar'], (x) -> TO_UPPER(x) )</tt> returns <tt>[ 'FOO', 'BAR' ]</tt></li>
+
+<li><tt>FILTER</tt> - Filters a list by a predicate in the form of a lambda expression. For instance <tt>FILTER([ 'foo', 'bar'], (x ) -> x == 'foo' )</tt> returns <tt>[ 'foo' ]</tt></li>
+
+<li><tt>REDUCE</tt> - Applies a function over a list of input. For instance <tt>REDUCE([ 1, 2, 3], (sum, x) -> sum + x, 0 )</tt> returns <tt>6</tt></li>
+</ul></div>
+<div class="section">
<h2><a name="Stellar_Core_Functions"></a>Stellar Core Functions</h2>
<table border="0" class="table table-striped">
@@ -409,29 +501,49 @@
<tr class="a">
-<td><a href="../../metron-analytics/metron-statistics/index.html#BIN"> <tt>BIN</tt></a> </td>
+<td><a href="#APPEND_IF_MISSING"> <tt>APPEND_IF_MISSING</tt></a> </td>
</tr>
<tr class="b">
-<td><a href="#BLOOM_ADD"> <tt>BLOOM_ADD</tt></a> </td>
+<td><a href="../../metron-analytics/metron-statistics/index.html#BIN"> <tt>BIN</tt></a> </td>
</tr>
<tr class="a">
-<td><a href="#BLOOM_EXISTS"> <tt>BLOOM_EXISTS</tt></a> </td>
+<td><a href="#BLOOM_ADD"> <tt>BLOOM_ADD</tt></a> </td>
</tr>
<tr class="b">
-<td><a href="#BLOOM_INIT"> <tt>BLOOM_INIT</tt></a> </td>
+<td><a href="#BLOOM_EXISTS"> <tt>BLOOM_EXISTS</tt></a> </td>
</tr>
<tr class="a">
+<td><a href="#BLOOM_INIT"> <tt>BLOOM_INIT</tt></a> </td>
+ </tr>
+
+<tr class="b">
+
<td><a href="#BLOOM_MERGE"> <tt>BLOOM_MERGE</tt></a> </td>
</tr>
+<tr class="a">
+
+<td><a href="#CHOP"> <tt>CHOP</tt></a> </td>
+ </tr>
+
+<tr class="b">
+
+<td><a href="#CHOMP"> <tt>CHOMP</tt></a> </td>
+ </tr>
+
+<tr class="a">
+
+<td><a href="#COUNT_MATCHES"> <tt>COUNT_MATCHES</tt></a> </td>
+ </tr>
+
<tr class="b">
<td><a href="#DAY_OF_MONTH"> <tt>DAY_OF_MONTH</tt></a> </td>
@@ -489,119 +601,129 @@
<tr class="a">
-<td><a href="#FORMAT"> <tt>FORMAT</tt></a> </td>
+<td><a href="#FILTER"> <tt>FILTER</tt></a> </td>
</tr>
<tr class="b">
-<td><a href="../../metron-analytics/metron-statistics/index.html#HLLP_CARDINALITY"> <tt>HLLP_CARDINALITY</tt></a> </td>
+<td><a href="#FORMAT"> <tt>FORMAT</tt></a> </td>
</tr>
<tr class="a">
-<td><a href="../../metron-analytics/metron-statistics/index.html#HLLP_INIT"> <tt>HLLP_INIT</tt></a> </td>
+<td><a href="../../metron-analytics/metron-statistics/index.html#HLLP_CARDINALITY"> <tt>HLLP_CARDINALITY</tt></a> </td>
</tr>
<tr class="b">
-<td><a href="../../metron-analytics/metron-statistics/index.html#HLLP_MERGE"> <tt>HLLP_MERGE</tt></a> </td>
+<td><a href="../../metron-analytics/metron-statistics/index.html#HLLP_INIT"> <tt>HLLP_INIT</tt></a> </td>
</tr>
<tr class="a">
-<td><a href="../../metron-analytics/metron-statistics/index.html#HLLP_OFFER"> <tt>HLLP_OFFER</tt></a> </td>
+<td><a href="../../metron-analytics/metron-statistics/index.html#HLLP_MERGE"> <tt>HLLP_MERGE</tt></a> </td>
</tr>
<tr class="b">
-<td><a href="#GEO_GET"> <tt>GEO_GET</tt></a> </td>
+<td><a href="../../metron-analytics/metron-statistics/index.html#HLLP_OFFER"> <tt>HLLP_OFFER</tt></a> </td>
</tr>
<tr class="a">
-<td><a href="#GET"> <tt>GET</tt></a> </td>
+<td><a href="#GEO_GET"> <tt>GEO_GET</tt></a> </td>
</tr>
<tr class="b">
-<td><a href="#GET_FIRST"> <tt>GET_FIRST</tt></a> </td>
+<td><a href="#GET"> <tt>GET</tt></a> </td>
</tr>
<tr class="a">
-<td><a href="#GET_LAST"> <tt>GET_LAST</tt></a> </td>
+<td><a href="#GET_FIRST"> <tt>GET_FIRST</tt></a> </td>
</tr>
<tr class="b">
-<td><a href="#IN_SUBNET"> <tt>IN_SUBNET</tt></a> </td>
+<td><a href="#GET_LAST"> <tt>GET_LAST</tt></a> </td>
</tr>
<tr class="a">
-<td><a href="#IS_DATE"> <tt>IS_DATE</tt></a> </td>
+<td><a href="#IN_SUBNET"> <tt>IN_SUBNET</tt></a> </td>
</tr>
<tr class="b">
-<td><a href="#IS_DOMAIN"> <tt>IS_DOMAIN</tt></a> </td>
+<td><a href="#IS_DATE"> <tt>IS_DATE</tt></a> </td>
</tr>
<tr class="a">
-<td><a href="#IS_EMAIL"> <tt>IS_EMAIL</tt></a> </td>
+<td><a href="#IS_DOMAIN"> <tt>IS_DOMAIN</tt></a> </td>
</tr>
<tr class="b">
-<td><a href="#IS_EMPTY"> <tt>IS_EMPTY</tt></a> </td>
+<td><a href="#IS_EMAIL"> <tt>IS_EMAIL</tt></a> </td>
</tr>
<tr class="a">
-<td><a href="#IS_INTEGER"> <tt>IS_INTEGER</tt></a> </td>
+<td><a href="#IS_EMPTY"> <tt>IS_EMPTY</tt></a> </td>
</tr>
<tr class="b">
-<td><a href="#IS_IP"> <tt>IS_IP</tt></a> </td>
+<td><a href="#IS_INTEGER"> <tt>IS_INTEGER</tt></a> </td>
</tr>
<tr class="a">
-<td><a href="#IS_URL"> <tt>IS_URL</tt></a> </td>
+<td><a href="#IS_IP"> <tt>IS_IP</tt></a> </td>
</tr>
<tr class="b">
-<td><a href="#JOIN"> <tt>JOIN</tt></a> </td>
+<td><a href="#IS_URL"> <tt>IS_URL</tt></a> </td>
</tr>
<tr class="a">
-<td><a href="#KAFKA_GET"> <tt>KAFKA_GET</tt></a> </td>
+<td><a href="#JOIN"> <tt>JOIN</tt></a> </td>
</tr>
<tr class="b">
-<td><a href="#KAFKA_PROPS"> <tt>KAFKA_PROPS</tt></a> </td>
+<td><a href="#KAFKA_GET"> <tt>KAFKA_GET</tt></a> </td>
</tr>
<tr class="a">
-<td><a href="#KAFKA_PUT"> <tt>KAFKA_PUT</tt></a> </td>
+<td><a href="#KAFKA_PROPS"> <tt>KAFKA_PROPS</tt></a> </td>
</tr>
<tr class="b">
-<td><a href="#KAFKA_TAIL"> <tt>KAFKA_TAIL</tt></a> </td>
+<td><a href="#KAFKA_PUT"> <tt>KAFKA_PUT</tt></a> </td>
</tr>
<tr class="a">
+<td><a href="#KAFKA_TAIL"> <tt>KAFKA_TAIL</tt></a> </td>
+ </tr>
+
+<tr class="b">
+
<td><a href="#LENGTH"> <tt>LENGTH</tt></a> </td>
</tr>
+<tr class="a">
+
+<td><a href="#LIST_ADD"> <tt>LIST_ADD</tt></a> </td>
+ </tr>
+
<tr class="b">
<td><a href="#MAAS_GET_ENDPOINT"> <tt>MAAS_GET_ENDPOINT</tt></a> </td>
@@ -614,14 +736,24 @@
<tr class="b">
-<td><a href="#MAP_EXISTS"> <tt>MAP_EXISTS</tt></a> </td>
+<td><a href="#MAP"> <tt>MAP</tt></a> </td>
</tr>
<tr class="a">
+<td><a href="#MAP_EXISTS"> <tt>MAP_EXISTS</tt></a> </td>
+ </tr>
+
+<tr class="b">
+
<td><a href="#MONTH"> <tt>MONTH</tt></a> </td>
</tr>
+<tr class="a">
+
+<td><a href="#PREPEND_IF_MISSING"> <tt>PREPEND_IF_MISSING</tt></a> </td>
+ </tr>
+
<tr class="b">
<td><a href="#PROFILE_GET"> <tt>PROFILE_GET</tt></a> </td>
@@ -634,9 +766,19 @@
<tr class="b">
+<td><a href="#PROFILE_WINDOW"> <tt>PROFILE_WINDOW</tt></a> </td>
+ </tr>
+
+<tr class="a">
+
<td><a href="#PROTOCOL_TO_NAME"> <tt>PROTOCOL_TO_NAME</tt></a> </td>
</tr>
+<tr class="b">
+
+<td><a href="#REDUCE"> <tt>REDUCE</tt></a> </td>
+ </tr>
+
<tr class="a">
<td><a href="#REGEXP_MATCH"> <tt>REGEXP_MATCH</tt></a> </td>
@@ -844,6 +986,26 @@
</tbody>
</table>
<div class="section">
+<h3><a name="APPEND_IF_MISSING"></a><tt>APPEND_IF_MISSING</tt></h3>
+
+<ul>
+
+<li>Description: Appends the suffix to the end of the string if the string does not already end with any of the suffixes.</li>
+
+<li>Input:
+
+<ul>
+
+<li>string - The string to be appended.</li>
+
+<li>suffix - The string suffix to append to the end of the string.</li>
+
+<li>additionalsuffix - Optional - Additional string suffix that is a valid terminator.</li>
+ </ul></li>
+
+<li>Returns: A new String if prefix was prepended, the same string otherwise.</li>
+</ul></div>
+<div class="section">
<h3><a name="BLOOM_ADD"></a><tt>BLOOM_ADD</tt></h3>
<ul>
@@ -914,6 +1076,56 @@
<li>Returns: Bloom Filter or null if the list is empty</li>
</ul></div>
<div class="section">
+<h3><a name="CHOP"></a><tt>CHOP</tt></h3>
+
+<ul>
+
+<li>Description: Remove the last character from a String</li>
+
+<li>Input:
+
+<ul>
+
+<li>string - the String to chop last character from, may be null</li>
+ </ul></li>
+
+<li>Returns: String without last character, null if null String input</li>
+</ul></div>
+<div class="section">
+<h3><a name="CHOMP"></a><tt>CHOMP</tt></h3>
+
+<ul>
+
+<li>Description: Removes one newline from end of a String if it’s there, otherwise leave it alone. A newline is “\n”, “\r”, or “\r\n”</li>
+
+<li>Input:
+
+<ul>
+
+<li>string - the String to chomp a newline from, may be null</li>
+ </ul></li>
+
+<li>Returns: String without newline, null if null String input</li>
+</ul></div>
+<div class="section">
+<h3><a name="COUNT_MATCHES"></a><tt>COUNT_MATCHES</tt></h3>
+
+<ul>
+
+<li>Description: Counts how many times the substring appears in the larger string.</li>
+
+<li>Input:
+
+<ul>
+
+<li>string - the CharSequence to check, may be null.</li>
+
+<li>substring/character - the substring or character to count, may be null.</li>
+ </ul></li>
+
+<li>Returns: the number of non-overlapping occurrences, 0 if either CharSequence is null.</li>
+</ul></div>
+<div class="section">
<h3><a name="DAY_OF_MONTH"></a><tt>DAY_OF_MONTH</tt></h3>
<ul>
@@ -1112,6 +1324,24 @@
<li>Returns: Last element of the list</li>
</ul></div>
<div class="section">
+<h3><a name="FILTER"></a><tt>FILTER</tt></h3>
+
+<ul>
+
+<li>Description: Applies a filter in the form of a lambda expression to a list. e.g. <tt>FILTER( [ 'foo', 'bar' ] , (x) -> x == 'foo')</tt> would yield <tt>[ 'foo']</tt></li>
+
+<li>Input:
+
+<ul>
+
+<li>list - List of arguments.</li>
+
+<li>predicate - The lambda expression to apply. This expression is assumed to take one argument and return a boolean.</li>
+ </ul></li>
+
+<li>Returns: The input list filtered by the predicate.</li>
+</ul></div>
+<div class="section">
<h3><a name="FORMAT"></a><tt>FORMAT</tt></h3>
<ul>
@@ -1442,6 +1672,24 @@
<li>Returns: Integer</li>
</ul></div>
<div class="section">
+<h3><a name="LIST_ADD"></a><tt>LIST_ADD</tt></h3>
+
+<ul>
+
+<li>Description: Adds an element to a list.</li>
+
+<li>Input:
+
+<ul>
+
+<li>list - List to add element to.</li>
+
+<li>element - Element to add to list</li>
+ </ul></li>
+
+<li>Returns: Resulting list with the item added at the end.</li>
+</ul></div>
+<div class="section">
<h3><a name="MAAS_GET_ENDPOINT"></a><tt>MAAS_GET_ENDPOINT</tt></h3>
<ul>
@@ -1480,6 +1728,24 @@
<li>Returns: The output of the model deployed as a REST endpoint in Map form. Assumes REST endpoint returns a JSON Map.</li>
</ul></div>
<div class="section">
+<h3><a name="MAP"></a><tt>MAP</tt></h3>
+
+<ul>
+
+<li>Description: Applies lambda expression to a list of arguments. e.g. <tt>MAP( [ 'foo', 'bar' ] , (x) -> TO_UPPER(x) )</tt> would yield <tt>[ 'FOO', 'BAR' ]</tt></li>
+
+<li>Input:
+
+<ul>
+
+<li>list - List of arguments.</li>
+
+<li>transform_expression - The lambda expression to apply. This expression is assumed to take one argument.</li>
+ </ul></li>
+
+<li>Returns: The input list transformed item-wise by the lambda expression.</li>
+</ul></div>
+<div class="section">
<h3><a name="MAP_EXISTS"></a><tt>MAP_EXISTS</tt></h3>
<ul>
@@ -1534,6 +1800,26 @@
<li>Returns: The current month (0-based).</li>
</ul></div>
<div class="section">
+<h3><a name="PREPEND_IF_MISSING"></a><tt>PREPEND_IF_MISSING</tt></h3>
+
+<ul>
+
+<li>Description: Prepends the prefix to the start of the string if the string does not already start with any of the prefixes.</li>
+
+<li>Input:
+
+<ul>
+
+<li>string - The string to be prepended.</li>
+
+<li>prefix - The string prefix to prepend to the start of the string.</li>
+
+<li>additionalprefix - Optional - Additional string prefix that is valid.</li>
+ </ul></li>
+
+<li>Returns: A new String if prefix was prepended, the same string otherwise.</li>
+</ul></div>
+<div class="section">
<h3><a name="PROFILE_GET"></a><tt>PROFILE_GET</tt></h3>
<ul>
@@ -1578,6 +1864,26 @@
<li>Returns: The selected profile measurement timestamps. These are ProfilePeriod objects.</li>
</ul></div>
<div class="section">
+<h3><a name="PROFILE_WINDOW"></a><tt>PROFILE_WINDOW</tt></h3>
+
+<ul>
+
+<li>Description: The profiler periods associated with a window selector statement from an optional reference timestamp.</li>
+
+<li>Input:
+
+<ul>
+
+<li>windowSelector - The statement specifying the window to select.</li>
+
+<li>now - Optional - The timestamp to use for now.</li>
+
+<li>config_overrides - Optional - Map (in curly braces) of name:value pairs, each overriding the global config parameter of the same name. Default is the empty Map, meaning no overrides.</li>
+ </ul></li>
+
+<li>Returns: The selected profile measurement periods. These are ProfilePeriod objects.</li>
+</ul></div>
+<div class="section">
<h3><a name="PROTOCOL_TO_NAME"></a><tt>PROTOCOL_TO_NAME</tt></h3>
<ul>
@@ -1594,6 +1900,26 @@
<li>Returns: The protocol name associated with the IANA number.</li>
</ul></div>
<div class="section">
+<h3><a name="REDUCE"></a><tt>REDUCE</tt></h3>
+
+<ul>
+
+<li>Description: Reduces a list by a binary lambda expression. That is, the expression takes two arguments. Usage example: <tt>REDUCE( [ 1, 2, 3 ] , (x, y) -> x + y, 0)</tt> would sum the input list, yielding <tt>6</tt>.</li>
+
+<li>Input:
+
+<ul>
+
+<li>list - List of arguments.</li>
+
+<li>binary_operation - The lambda expression function to apply to reduce the list. It is assumed that this takes two arguments, the first being the running total and the second being an item from the list.</li>
+
+<li>initial_value - The initial value to use.</li>
+ </ul></li>
+
+<li>Returns: The reduction of the list.</li>
+</ul></div>
+<div class="section">
<h3><a name="REGEXP_MATCH"></a><tt>REGEXP_MATCH</tt></h3>
<ul>
@@ -1979,8 +2305,71 @@
<li>The value in <tt>dc2tz</tt> associated with the value associated with field <tt>dc</tt>, defaulting to <tt>UTC</tt></li>
</ul></div></div>
<div class="section">
+<h2><a name="Stellar_Benchmarks"></a>Stellar Benchmarks</h2>
+<p>A microbenchmarking utility is included to assist in executing microbenchmarks for Stellar functions. The utility can be executed via maven using the <tt>exec</tt> plugin, like so, from the <tt>metron-common</tt> directory:</p>
+
+<div class="source">
+<div class="source">
+<pre>mvn -DskipTests clean package && \
+mvn exec:java -Dexec.mainClass="org.apache.metron.common.stellar.benchmark.StellarMicrobenchmark" -Dexec.args="..."
+</pre></div></div>
+<p>where <tt>exec.args</tt> can be one of the following:</p>
+
+<div class="source">
+<div class="source">
+<pre> -e,--expressions <FILE> Stellar expressions
+ -h,--help Generate Help screen
+ -n,--num_times <NUM> Number of times to run per expression (after
+ warmup). Default: 1000
+ -o,--output <FILE> File to write output.
+ -p,--percentiles <NUM> Percentiles to calculate per run. Default:
+ 50.0,75.0,95.0,99.0
+ -v,--variables <FILE> File containing a JSON Map of variables to use
+ -w,--warmup <NUM> Number of times for warmup per expression.
+ Default: 100
+</pre></div></div>
+<p>For instance, to run with a set of Stellar expression in file <tt>/tmp/expressions.txt</tt>:</p>
+
+<div class="source">
+<div class="source">
+<pre> # simple functions
+ TO_UPPER('casey')
+ TO_LOWER(name)
+ # math functions
+ 1 + 2*(3 + int_num) / 10.0
+ 1.5 + 2*(3 + double_num) / 10.0
+ # conditionals
+ if ('foo' in ['foo']) OR one == very_nearly_one then 'one' else 'two'
+ 1 + 2*(3 + int_num) / 10.0
+ #Network funcs
+ DOMAIN_TO_TLD(domain)
+ DOMAIN_REMOVE_SUBDOMAINS(domain)
+</pre></div></div>
+<p>And variables in file <tt>/tmp/variables.json</tt>:</p>
+
+<div class="source">
+<div class="source">
+<pre>{
+ "name" : "casey",
+ "int_num" : 1,
+ "double_num" : 17.5,
+ "one" : 1,
+ "very_nearly_one" : 1.000001,
+ "domain" : "www.google.com"
+}
+</pre></div></div>
+<p>Written to file <tt>/tmp/output.txt</tt> would be the following command:</p>
+
+<div class="source">
+<div class="source">
+<pre>mvn -DskipTests clean package && \
+mvn exec:java -Dexec.mainClass="org.apache.metron.common.stellar.benchmark.StellarMicrobenchmark" \
+-Dexec.args="-e /tmp/expressions.txt -v /tmp/variables.json -o ./output.json"
+</pre></div></div></div>
+<div class="section">
<h2><a name="Stellar_Shell"></a>Stellar Shell</h2>
-<p>A REPL (Read Eval Print Loop) for the Stellar language that helps in debugging, troubleshooting and learning Stellar. The Stellar DSL (domain specific language) is used to act upon streaming data within Apache Storm. It is difficult to troubleshoot Stellar when it can only be executed within a Storm topology. This REPL is intended to help mitigate that problem by allowing a user to replicate data encountered in production, isolate initialization errors, or understand function resolution problems.</p>
+<p>The Stellar Shell is a REPL (Read Eval Print Loop) for the Stellar language that helps troubleshooting, learning Stellar or even interacting with a live Metron cluster. </p>
+<p>The Stellar DSL (domain specific language) is used to act upon streaming data within Apache Storm. It is difficult to troubleshoot Stellar when it can only be executed within a Storm topology. This REPL is intended to help mitigate that problem by allowing a user to replicate data encountered in production, isolate initialization errors, or understand function resolution problems.</p>
<p>The shell supports customization via <tt>~/.inputrc</tt> as it is backed by a proper readline implementation. </p>
<p>Shell-like operations are supported such as </p>
@@ -1997,8 +2386,10 @@
<li>emacs or vi keybindings for edit mode</li>
</ul>
+<p>Note: Stellar classpath configuration from the global config is honored here if the REPL knows about zookeeper.</p>
<div class="section">
<h3><a name="Getting_Started"></a>Getting Started</h3>
+<p>To run the Stellar Shell from within a deployed Metron cluster, run the following command on the host where Metron is installed.</p>
<div class="source">
<div class="source">
@@ -2042,7 +2433,7 @@ usage: stellar
<div class="section">
<h4><a name="a-z_--zookeeper"></a><tt>-z, --zookeeper</tt></h4>
<p><i>Optional</i></p>
-<p>Attempts to connect to Zookeeper and read the Metron global configuration. Stellar functions may require the global configuration to work properly. If found, the global configuration values are printed to the console.</p>
+<p>Attempts to connect to Zookeeper and read the Metron global configuration. Stellar functions may require the global configuration to work properly. If found, the global configuration values are printed to the console. If specified, then the classpath may be augmented by the paths specified in the stellar config in the global config.</p>
<div class="source">
<div class="source">
@@ -2115,6 +2506,37 @@ IS_EMAIL
args: address - The String to test
ret: True if the string is a valid email address and false otherwise.
[Stellar]>>>
+</pre></div></div></div></div>
+<div class="section">
+<h3><a name="Advanced_Usage"></a>Advanced Usage</h3>
+<p>To run the Stellar Shell directly from the Metron source code, run a command like the following. Ensure that Metron has already been built and installed with <tt>mvn clean install -DskipTests</tt>.</p>
+
+<div class="source">
+<div class="source">
+<pre>$ mvn exec:java \
+ -Dexec.mainClass="org.apache.metron.common.stellar.shell.StellarShell" \
+ -pl metron-platform/metron-enrichment
+...
+Stellar, Go!
+Please note that functions are loading lazily in the background and will be unavailable until loaded fully.
+[Stellar]>>> Functions loaded, you may refer to functions now...
+[Stellar]>>> %functions
+ABS, APPEND_IF_MISSING, BIN, BLOOM_ADD, BLOOM_EXISTS, BLOOM_INIT, BLOOM_MERGE, CHOMP, CHOP, COUNT_MATCHES, DAY_OF_MONTH, DAY_OF_WEEK, DAY_OF_YEAR, DOMAIN_REMOVE_SUBDOMAINS, DOMAIN_REMOVE_TLD, DOMAIN_TO_TLD, ENDS_WITH, ENRICHMENT_EXISTS, ENRICHMENT_GET, FILL_LEFT, FILL_RIGHT, FILTER, FORMAT, GEO_GET, GET, GET_FIRST, GET_LAST, HLLP_ADD, HLLP_CARDINALITY, HLLP_INIT, HLLP_MERGE, IN_SUBNET, IS_DATE, IS_DOMAIN, IS_EMAIL, IS_EMPTY, IS_INTEGER, IS_IP, IS_URL, JOIN, LENGTH, LIST_ADD, MAAS_GET_ENDPOINT, MAAS_MODEL_APPLY, MAP, MAP_EXISTS, MAP_GET, MONTH, OUTLIER_MAD_ADD, OUTLIER_MAD_SCORE, OUTLIER_MAD_STATE_MERGE, PREPEND_IF_MISSING, PROFILE_FIXED, PROFILE_GET, PROFILE_WINDOW, PROTOCOL_TO_NAME, REDUCE, REGEXP_MATCH, SPLIT, STARTS_WITH, STATS_ADD, STATS_BIN, STATS_COUNT, STATS_GEOMETRIC_MEAN, STATS_INIT, STATS_KURTOSIS, STATS_MAX, STATS_MEAN, STATS_MERGE, STATS_MIN, STATS_PERCENTILE, STATS_POPULATION_VARIANCE, STATS_QUADRATIC_MEAN, STATS_SD, STATS_SKEWNESS, STATS_SUM, STATS_SUM_LOGS, STATS_SUM_
SQUARES, STATS_VARIANCE, STRING_ENTROPY, SYSTEM_ENV_GET, SYSTEM_PROPERTY_GET, TO_DOUBLE, TO_EPOCH_TIMESTAMP, TO_FLOAT, TO_INTEGER, TO_LONG, TO_LOWER, TO_STRING, TO_UPPER, TRIM, URL_TO_HOST, URL_TO_PATH, URL_TO_PORT, URL_TO_PROTOCOL, WEEK_OF_MONTH, WEEK_OF_YEAR, YEAR
+</pre></div></div>
+<p>Changing the project passed to the <tt>-pl</tt> argument will define which dependencies are included and ultimately which Stellar functions are available within the shell environment. </p>
+<p>This can be useful for troubleshooting function resolution problems. The previous example defines which functions are available during Enrichment. For example, to determine which functions are available within the Profiler run the following.</p>
+
+<div class="source">
+<div class="source">
+<pre> $ mvn exec:java \
+ -Dexec.mainClass="org.apache.metron.common.stellar.shell.StellarShell" \
+ -pl metron-analytics/metron-profiler
+...
+Stellar, Go!
+Please note that functions are loading lazily in the background and will be unavailable until loaded fully.
+[Stellar]>>> Functions loaded, you may refer to functions now...
+%functions
+ABS, APPEND_IF_MISSING, BIN, BLOOM_ADD, BLOOM_EXISTS, BLOOM_INIT, BLOOM_MERGE, CHOMP, CHOP, COUNT_MATCHES, DAY_OF_MONTH, DAY_OF_WEEK, DAY_OF_YEAR, DOMAIN_REMOVE_SUBDOMAINS, DOMAIN_REMOVE_TLD, DOMAIN_TO_TLD, ENDS_WITH, FILL_LEFT, FILL_RIGHT, FILTER, FORMAT, GET, GET_FIRST, GET_LAST, HLLP_ADD, HLLP_CARDINALITY, HLLP_INIT, HLLP_MERGE, IN_SUBNET, IS_DATE, IS_DOMAIN, IS_EMAIL, IS_EMPTY, IS_INTEGER, IS_IP, IS_URL, JOIN, LENGTH, LIST_ADD, MAAS_GET_ENDPOINT, MAAS_MODEL_APPLY, MAP, MAP_EXISTS, MAP_GET, MONTH, OUTLIER_MAD_ADD, OUTLIER_MAD_SCORE, OUTLIER_MAD_STATE_MERGE, PREPEND_IF_MISSING, PROFILE_FIXED, PROFILE_GET, PROFILE_WINDOW, PROTOCOL_TO_NAME, REDUCE, REGEXP_MATCH, SPLIT, STARTS_WITH, STATS_ADD, STATS_BIN, STATS_COUNT, STATS_GEOMETRIC_MEAN, STATS_INIT, STATS_KURTOSIS, STATS_MAX, STATS_MEAN, STATS_MERGE, STATS_MIN, STATS_PERCENTILE, STATS_POPULATION_VARIANCE, STATS_QUADRATIC_MEAN, STATS_SD, STATS_SKEWNESS, STATS_SUM, STATS_SUM_LOGS, STATS_SUM_SQUARES, STATS_VARIANCE, STRING_ENTROPY, SYS
TEM_ENV_GET, SYSTEM_PROPERTY_GET, TO_DOUBLE, TO_EPOCH_TIMESTAMP, TO_FLOAT, TO_INTEGER, TO_LONG, TO_LOWER, TO_STRING, TO_UPPER, TRIM, URL_TO_HOST, URL_TO_PATH, URL_TO_PORT, URL_TO_PROTOCOL, WEEK_OF_MONTH, WEEK_OF_YEAR, YEAR
</pre></div></div>
<p><a name="Global_Configuration"></a></p>
<h1>Global Configuration</h1>
@@ -2128,6 +2550,7 @@ IS_EMAIL
"es.ip": "node1",
"es.port": "9300",
"es.date.format": "yyyy.MM.dd.HH",
+ "parser.error.topic": "indexing"
"fieldValidations" : [
{
"input" : [ "ip_src_addr", "ip_dst_addr" ],
@@ -2138,9 +2561,70 @@ IS_EMAIL
}
]
}
-</pre></div></div></div></div></div>
+</pre></div></div>
+<p><a name="Stellar_Configuration"></a></p>
+<h1>Stellar Configuration</h1>
+<p>Stellar can be configured in a variety of ways from the global config. In particular, there are three main configuration parameters around configuring Stellar:</p>
+
+<ul>
+
+<li><tt>stellar.function.paths</tt></li>
+
+<li><tt>stellar.function.resolver.includes</tt></li>
+
+<li><tt>stellar.function.resolver.excludes</tt></li>
+</ul></div></div>
+<div class="section">
+<h2><a name="stellar.function.paths"></a><tt>stellar.function.paths</tt></h2>
+<p>If specified, Stellar will use a custom classloader which will wrap the context classloader and allow for the resolution of classes stored in jars not shipped with Metron and stored in a variety of mediums:</p>
+
+<ul>
+
+<li>On HDFS</li>
+
+<li>In tar.gz files</li>
+
+<li>At http/s locations</li>
+
+<li>At ftp locations</li>
+</ul>
+<p>This path is a comma separated list of </p>
+
+<ul>
+
+<li>URIs</li>
+
+<li>URIs with a regex pattern ending it for matching within a directory</li>
+</ul>
+
+<div class="source">
+<div class="source">
+<pre>{
+ ...
+ "stellar.function.paths" : "hdfs://node1:8020/apps/metron/stellar/metron-management-0.4.0.jar, hdfs://node1:8020/apps/metron/3rdparty/.*.jar"
+}
+</pre></div></div>
+<p>Please be aware that this classloader does not reload functions dynamically and the classpath specified here in the global config is read on topology start. A change in classpath, to be picked up, would necessitate a topology restart at the moment</p></div>
<div class="section">
-<h2><a name="Validation_Framework"></a>Validation Framework</h2>
+<h2><a name="stellar.function.resolver.includesexcludes"></a><tt>stellar.function.resolver.{includes,excludes}</tt></h2>
+<p>If specified, this defines one or more regular expressions applied to the classes implementing the Stellar function that specify what should be included when searching for Stellar functions.</p>
+
+<ul>
+
+<li><tt>stellar.function.resolver.includes</tt> defines the list of classes to include.</li>
+
+<li><tt>stellar.function.resolver.excludes</tt> defines the list of classes to exclude.</li>
+</ul>
+
+<div class="source">
+<div class="source">
+<pre>{
+ ...
+ "stellar.function.resolver.includes" : "org.apache.metron.*,com.myorg.stellar.*"
+}
+</pre></div></div>
+<p><a name="Validation_Framework"></a></p>
+<h1>Validation Framework</h1>
<p>Inside of the global configuration, there is a validation framework in place that enables the validation that messages coming from all parsers are valid. This is done in the form of validation plugins where assertions about fields or whole messages can be made. </p>
<p>The format for this is a <tt>fieldValidations</tt> field inside of global config. This is associated with an array of field validation objects structured like so:</p>
@@ -2211,7 +2695,41 @@ IS_EMAIL
<li>To push the configs into zookeeper on the singlenode vagrant machine: <tt>$METRON_HOME/bin/zk_load_configs.sh -z node1:2181 -m PUSH -i $METRON_HOME/config/zookeeper</tt></li>
<li>To pull the configs from zookeeper to the singlenode vagrant machine disk: <tt>$METRON_HOME/bin/zk_load_configs.sh -z node1:2181 -m PULL -o $METRON_HOME/config/zookeeper -f</tt></li>
-</ul></div>
+</ul>
+<p><a name="Topology_Errors"></a></p>
+<h1>Topology Errors</h1>
+<p>Errors generated in Metron topologies are transformed into JSON format and follow this structure:</p>
+
+<div class="source">
+<div class="source">
+<pre>{
+ "exception": "java.lang.IllegalStateException: Unable to parse Message: ...",
+ "failed_sensor_type": "bro",
+ "stack": "java.lang.IllegalStateException: Unable to parse Message: ...",
+ "hostname": "node1",
+ "source:type": "error",
+ "raw_message": "{\"http\": {\"ts\":1488809627.000000.31915,\"uid\":\"C9JpSd2vFAWo3mXKz1\", ...",
+ "error_hash": "f7baf053f2d3c801a01d196f40f3468e87eea81788b2567423030100865c5061",
+ "error_type": "parser_error",
+ "message": "Unable to parse Message: {\"http\": {\"ts\":1488809627.000000.31915,\"uid\":\"C9JpSd2vFAWo3mXKz1\", ...",
+ "timestamp": 1488809630698
+}
+</pre></div></div>
+<p>Each topology can be configured to send error messages to a specific Kafka topic. The parser topologies retrieve this setting from the the <tt>parser.error.topic</tt> setting in the global config:</p>
+
+<div class="source">
+<div class="source">
+<pre>{
+ "es.clustername": "metron",
+ "es.ip": "node1",
+ "es.port": "9300",
+ "es.date.format": "yyyy.MM.dd.HH",
+ "parser.error.topic": "indexing"
+}
+</pre></div></div>
+<p>Error topics for enrichment and threat intel errors are passed into the enrichment topology as flux properties named <tt>enrichment.error.topic</tt> and <tt>threat.intel.error.topic</tt>. These properties can be found in <tt>$METRON_HOME/config/enrichment.properties</tt>.</p>
+<p>The error topic for indexing errors is passed into the indexing topology as a flux property named <tt>index.error.topic</tt>. This property can be found in either <tt>$METRON_HOME/config/elasticsearch.properties</tt> or <tt>$METRON_HOME/config/solr.properties</tt> depending on the search engine selected.</p>
+<p>By default all error messages are sent to the <tt>indexing</tt> topic so that they are indexed and archived, just like other messages. The indexing config for error messages can be found at <tt>$METRON_HOME/config/zookeeper/indexing/error.json</tt>.</p></div>
</div>
</div>
</div>
@@ -2220,8 +2738,9 @@ IS_EMAIL
<footer>
<div class="container-fluid">
- <div class="row span12">Copyright © 2017.
- All Rights Reserved.
+ <div class="row span12">Copyright © 2017
+ <a href="https://www.apache.org">The Apache Software Foundation</a>.
+ All Rights Reserved.
</div>
http://git-wip-us.apache.org/repos/asf/metron/blob/cf4c2ecd/current-book/metron-platform/metron-data-management/index.html
----------------------------------------------------------------------
diff --git a/current-book/metron-platform/metron-data-management/index.html b/current-book/metron-platform/metron-data-management/index.html
index 8029ef2..d73ea2e 100644
--- a/current-book/metron-platform/metron-data-management/index.html
+++ b/current-book/metron-platform/metron-data-management/index.html
@@ -1,13 +1,13 @@
<!DOCTYPE html>
<!--
- | Generated by Apache Maven Doxia at 2017-02-23
+ | Generated by Apache Maven Doxia at 2017-06-27
| Rendered using Apache Maven Fluido Skin 1.3.0
-->
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
- <meta name="Date-Revision-yyyymmdd" content="20170223" />
+ <meta name="Date-Revision-yyyymmdd" content="20170627" />
<meta http-equiv="Content-Language" content="en" />
<title>Metron – Resource Data Management</title>
<link rel="stylesheet" href="../../css/apache-maven-fluido-1.3.0.min.css" />
@@ -30,14 +30,11 @@
<div class="container-fluid">
<div id="banner">
<div class="pull-left">
- <a href="http://metron.incubator.apache.org/" id="bannerLeft">
- <img src="../../images/metron-logo.png" alt="Apache Metron - Incubating" width="148px" height="48px"/>
+ <a href="http://metron.apache.org/" id="bannerLeft">
+ <img src="../../images/metron-logo.png" alt="Apache Metron" width="148px" height="48px"/>
</a>
</div>
- <div class="pull-right"> <a href="http://incubator.apache.org/" id="bannerRight">
- <img src="../../images/ApacheIncubating_Logo.png" alt="Apache Incubating" width="192px" height="48px"/>
- </a>
- </div>
+ <div class="pull-right"> </div>
<div class="clear"><hr/></div>
</div>
@@ -51,8 +48,8 @@
</li>
<li class="divider ">/</li>
<li class="">
- <a href="http://metron.incubator.apache.org/" class="externalLink" title="Metron-Incubating">
- Metron-Incubating</a>
+ <a href="http://metron.apache.org/" class="externalLink" title="Metron">
+ Metron</a>
</li>
<li class="divider ">/</li>
<li class="">
@@ -64,8 +61,8 @@
- <li id="publishDate" class="pull-right">Last Published: 2017-02-23</li> <li class="divider pull-right">|</li>
- <li id="projectVersion" class="pull-right">Version: 0.3.1</li>
+ <li id="publishDate" class="pull-right">Last Published: 2017-06-27</li> <li class="divider pull-right">|</li>
+ <li id="projectVersion" class="pull-right">Version: 0.4.0</li>
</ul>
</div>
@@ -78,7 +75,7 @@
<ul class="nav nav-list">
<li class="nav-header">User Documentation</li>
-
+
<li>
<a href="../../index.html" title="Metron">
@@ -99,7 +96,7 @@
<i class="icon-chevron-right"></i>
Analytics</a>
</li>
-
+
<li>
<a href="../../metron-deployment/index.html" title="Deployment">
@@ -113,7 +110,21 @@
<i class="none"></i>
Docker</a>
</li>
-
+
+ <li>
+
+ <a href="../../metron-interface/metron-config/index.html" title="Config">
+ <i class="none"></i>
+ Config</a>
+ </li>
+
+ <li>
+
+ <a href="../../metron-interface/metron-rest/index.html" title="Rest">
+ <i class="none"></i>
+ Rest</a>
+ </li>
+
<li>
<a href="../../metron-platform/index.html" title="Platform">
@@ -127,13 +138,13 @@
<i class="none"></i>
Api</a>
</li>
-
+
<li>
<a href="../../metron-platform/metron-common/index.html" title="Common">
- <i class="none"></i>
+ <i class="icon-chevron-right"></i>
Common</a>
- </li>
+ </li>
<li class="active">
@@ -174,9 +185,16 @@
<i class="none"></i>
Pcap-backend</a>
</li>
+
+ <li>
+
+ <a href="../../metron-platform/metron-writer/index.html" title="Writer">
+ <i class="none"></i>
+ Writer</a>
+ </li>
</ul>
</li>
-
+
<li>
<a href="../../metron-sensors/index.html" title="Sensors">
@@ -442,7 +460,7 @@
<th>Property </th>
-<th>Description </th>
+<th>Description</th>
</tr>
</thead>
<tbody>
@@ -451,28 +469,28 @@
<td>value_transform </td>
-<td>Transform fields defined in the “columns” mapping with Stellar transformations. New keys introduced in the transform will be added to the key metadata. </td>
+<td>Transform fields defined in the “columns” mapping with Stellar transformations. New keys introduced in the transform will be added to the key metadata.</td>
</tr>
<tr class="a">
<td>value_filter </td>
-<td>Allows additional filtering with Stellar predicates based on results from the value transformations. In this example, records whose domain property is empty after removing the TLD will be omitted. </td>
+<td>Allows additional filtering with Stellar predicates based on results from the value transformations. In this example, records whose domain property is empty after removing the TLD will be omitted.</td>
</tr>
<tr class="b">
<td>indicator_transform </td>
-<td>Transform the indicator column independent of the value transformations. You can refer to the original indicator value by using “indicator” as the variable name, as shown in the example above. In addition, if you prefer to piggyback your transformations, you can refer to the variable “domain”, which will allow your indicator transforms to inherit transformations done to this value during the value transformations. </td>
+<td>Transform the indicator column independent of the value transformations. You can refer to the original indicator value by using “indicator” as the variable name, as shown in the example above. In addition, if you prefer to piggyback your transformations, you can refer to the variable “domain”, which will allow your indicator transforms to inherit transformations done to this value during the value transformations.</td>
</tr>
<tr class="a">
<td>indicator_filter </td>
-<td>Allows additional filtering with Stellar predicates based on results from the value transformations. In this example, records whose indicator value is empty after removing the TLD will be omitted. </td>
+<td>Allows additional filtering with Stellar predicates based on results from the value transformations. In this example, records whose indicator value is empty after removing the TLD will be omitted.</td>
</tr>
</tbody>
</table>
@@ -537,7 +555,6 @@
"a-new-prop" : "global_property"
},
...
-
</pre></div></div>
<p>The resulting value data would look like the following:</p>
@@ -767,8 +784,6 @@
<th>Is Required? </th>
<th>Description </th>
-
-<th> </th>
</tr>
</thead>
<tbody>
@@ -782,8 +797,6 @@
<td>No </td>
<td>Generate the help screen/set of options </td>
-
-<td> </td>
</tr>
<tr class="a">
@@ -794,7 +807,7 @@
<td>No </td>
-<td>Do not update progress</td>
+<td>Do not update progress </td>
</tr>
<tr class="b">
@@ -806,8 +819,6 @@
<td>Yes </td>
<td>JSON Document describing the extractor for this input data source </td>
-
-<td> </td>
</tr>
<tr class="a">
@@ -819,8 +830,6 @@
<td>No </td>
<td>The Import mode to use: LOCAL, MR. Default: LOCAL </td>
-
-<td> </td>
</tr>
<tr class="b">
@@ -832,8 +841,6 @@
<td>Yes </td>
<td>The HBase table to import into </td>
-
-<td> </td>
</tr>
<tr class="a">
@@ -845,8 +852,6 @@
<td>Yes </td>
<td>The HBase table column family to import into </td>
-
-<td> </td>
</tr>
<tr class="b">
@@ -858,8 +863,6 @@
<td>Yes </td>
<td>The input data location on local disk. If this is a file, then that file will be loaded. If this is a directory, then the files will be loaded recursively under that directory. </td>
-
-<td> </td>
</tr>
<tr class="a">
@@ -871,8 +874,6 @@
<td>No </td>
<td>The log4j properties file to load </td>
-
-<td> </td>
</tr>
<tr class="b">
@@ -884,8 +885,6 @@
<td>No </td>
<td>The JSON document describing the enrichments to configure. Unlike other loaders, this is run first if specified. </td>
-
-<td> </td>
</tr>
<tr class="a">
@@ -897,8 +896,6 @@
<td>No </td>
<td>The number of threads to use when extracting data. The default is the number of cores. </td>
-
-<td> </td>
</tr>
<tr class="b">
@@ -910,8 +907,6 @@
<td>No </td>
<td>The batch size to use for HBase puts </td>
-
-<td> </td>
</tr>
</tbody>
</table></div>
@@ -956,7 +951,7 @@
<td>No </td>
-<td>GeoIP URL - defaults to <a class="externalLink" href="http://geolite.maxmind.com/download/geoip/database/GeoLite2-City.mmdb.gz">http://geolite.maxmind.com/download/geoip/database/GeoLite2-City.mmdb.gz</a></td>
+<td>GeoIP URL - defaults to <a class="externalLink" href="http://geolite.maxmind.com/download/geoip/database/GeoLite2-City.mmdb.gz">http://geolite.maxmind.com/download/geoip/database/GeoLite2-City.mmdb.gz</a> </td>
</tr>
<tr class="b">
@@ -967,7 +962,7 @@
<td>No </td>
-<td>HDFS directory to land formatted GeoIP file - defaults to /apps/metron/geo/<epoch millis>/</td>
+<td>HDFS directory to land formatted GeoIP file - defaults to /apps/metron/geo/<epoch millis>/ </td>
</tr>
<tr class="a">
@@ -978,7 +973,7 @@
<td>No </td>
-<td>Directory for landing the temporary GeoIP data - defaults to /tmp</td>
+<td>Directory for landing the temporary GeoIP data - defaults to /tmp </td>
</tr>
<tr class="b">
@@ -989,7 +984,7 @@
<td>Yes </td>
-<td>Zookeeper Quorum URL (zk1:port,zk2:port,…)</td>
+<td>Zookeeper Quorum URL (zk1:port,zk2:port,…) </td>
</tr>
</tbody>
</table></div></div>
@@ -1001,8 +996,9 @@
<footer>
<div class="container-fluid">
- <div class="row span12">Copyright © 2017.
- All Rights Reserved.
+ <div class="row span12">Copyright © 2017
+ <a href="https://www.apache.org">The Apache Software Foundation</a>.
+ All Rights Reserved.
</div>
[10/10] metron git commit: METRON-1012 Update Metron public web site
for 0.4.0 release - push to asf-site from the generated code on master
Posted by ma...@apache.org.
METRON-1012 Update Metron public web site for 0.4.0 release - push to asf-site
from the generated code on master
Project: http://git-wip-us.apache.org/repos/asf/metron/repo
Commit: http://git-wip-us.apache.org/repos/asf/metron/commit/cf4c2ecd
Tree: http://git-wip-us.apache.org/repos/asf/metron/tree/cf4c2ecd
Diff: http://git-wip-us.apache.org/repos/asf/metron/diff/cf4c2ecd
Branch: refs/heads/asf-site
Commit: cf4c2ecdebf49ddeae618477e1144903b948193e
Parents: 7152bce
Author: mattf <ma...@apache.org>
Authored: Wed Jul 5 12:57:10 2017 -0700
Committer: mattf <ma...@apache.org>
Committed: Wed Jul 5 12:59:15 2017 -0700
----------------------------------------------------------------------
README.md | 2 +-
community/index.html | 4 +-
current-book/Upgrading.html | 83 +-
.../images/ambari-storm-site-properties.png | Bin 0 -> 49605 bytes
current-book/images/ambari-storm-site.png | Bin 0 -> 134251 bytes
current-book/images/custom-storm-site-final.png | Bin 0 -> 128117 bytes
.../enable-kerberos-configure-kerberos.png | Bin 0 -> 136196 bytes
current-book/images/enable-kerberos-started.png | Bin 0 -> 149382 bytes
current-book/images/enable-kerberos.png | Bin 0 -> 16497 bytes
current-book/index.html | 76 +-
current-book/metron-analytics/index.html | 48 +-
.../metron-maas-service/index.html | 51 +-
.../metron-profiler-client/index.html | 497 ++++-
.../metron-analytics/metron-profiler/index.html | 170 +-
.../metron-statistics/HLLP.html | 50 +-
.../metron-statistics/index.html | 71 +-
.../Kerberos-ambari-setup.html | 275 +++
.../Kerberos-manual-setup.html | 803 +++++++
.../metron-deployment/amazon-ec2/index.html | 114 +-
current-book/metron-deployment/index.html | 245 ++-
.../packaging/docker/ansible-docker/index.html | 76 +-
.../packaging/docker/rpm-docker/index.html | 68 +-
.../packaging/packer-build/index.html | 298 +++
current-book/metron-deployment/roles/index.html | 79 +-
.../metron-deployment/roles/monit/index.html | 75 +-
.../roles/opentaxii/index.html | 73 +-
.../roles/pcap_replay/index.html | 73 +-
.../roles/sensor-stubs/index.html | 73 +-
.../roles/sensor-test-mode/index.html | 73 +-
.../vagrant/codelab-platform/index.html | 68 +-
.../vagrant/fastcapa-test-platform/index.html | 66 +-
.../vagrant/full-dev-platform/index.html | 109 +-
.../metron-deployment/vagrant/index.html | 66 +-
.../vagrant/quick-dev-platform/index.html | 105 +-
current-book/metron-docker/index.html | 54 +-
.../metron-interface/metron-config/index.html | 328 +++
.../metron-interface/metron-rest/index.html | 1996 ++++++++++++++++++
current-book/metron-platform/index.html | 65 +-
.../metron-platform/metron-api/index.html | 61 +-
.../metron-common/3rdPartyStellar.html | 398 ++++
.../metron-platform/metron-common/index.html | 629 +++++-
.../metron-data-management/index.html | 104 +-
.../metron-enrichment/index.html | 150 +-
.../metron-platform/metron-indexing/index.html | 65 +-
.../metron-management/index.html | 63 +-
.../metron-platform/metron-parsers/index.html | 112 +-
.../metron-pcap-backend/index.html | 310 ++-
.../metron-platform/metron-writer/index.html | 321 +++
.../metron-sensors/bro-plugin-kafka/index.html | 491 +++++
current-book/metron-sensors/fastcapa/index.html | 908 +++++++-
current-book/metron-sensors/index.html | 70 +-
current-book/metron-sensors/pycapa/index.html | 406 +++-
develop/bylaws/index.html | 508 +++++
develop/coding/index.html | 243 +++
develop/index.html | 18 +-
docs/hive-ddl.html | 8 +-
docs/index.html | 8 +-
docs/sensor.html | 8 +-
documentation/index.html | 14 +-
feed.xml | 30 +-
help/index.html | 8 +-
index.html | 2 +-
news/2015/12/12/new-logo/index.html | 2 +-
63 files changed, 9944 insertions(+), 1227 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/metron/blob/cf4c2ecd/README.md
----------------------------------------------------------------------
diff --git a/README.md b/README.md
index fd2d8cc..2d3d943 100644
--- a/README.md
+++ b/README.md
@@ -4,4 +4,4 @@ This branch contains the generated HTML for the
[Metron website](https://metron.apache.org).
You are probably looking for the
-[Metron master branch](https://github.com/apache/incubator-metron/tree/master).
+[Metron master branch](https://github.com/apache/metron/tree/master).
http://git-wip-us.apache.org/repos/asf/metron/blob/cf4c2ecd/community/index.html
----------------------------------------------------------------------
diff --git a/community/index.html b/community/index.html
index c7cd080..1c3cbe9 100644
--- a/community/index.html
+++ b/community/index.html
@@ -254,11 +254,11 @@
<h4> General & Public Discussion </h4>
[<a href="mailto:user-subscribe@metron.apache.org">Subscribe</a>]
[<a href="mailto:user-unsubscribe@metron.apache.org">Unsubscribe</a>]
- [<a href="http://mail-archives.apache.org/mod_mbox/incubator-metron-user/">Archives</a>]
+ [<a href="http://mail-archives.apache.org/mod_mbox/metron-user/">Archives</a>]
<h4> Code & Documentation Change </h4>
[<a href="mailto:dev-subscribe@metron.apache.org">Subscribe</a>]
[<a href="mailto:dev-unsubscribe@metron.apache.org">Unsubscribe</a>]
- [<a href="http://mail-archives.apache.org/mod_mbox/incubator-metron-dev/">Archives</a>]
+ [<a href="http://mail-archives.apache.org/mod_mbox/metron-dev/">Archives</a>]
</div>
</section>
http://git-wip-us.apache.org/repos/asf/metron/blob/cf4c2ecd/current-book/Upgrading.html
----------------------------------------------------------------------
diff --git a/current-book/Upgrading.html b/current-book/Upgrading.html
index 0e083cf..dd68d32 100644
--- a/current-book/Upgrading.html
+++ b/current-book/Upgrading.html
@@ -1,13 +1,13 @@
<!DOCTYPE html>
<!--
- | Generated by Apache Maven Doxia at 2017-02-23
+ | Generated by Apache Maven Doxia at 2017-06-27
| Rendered using Apache Maven Fluido Skin 1.3.0
-->
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
- <meta name="Date-Revision-yyyymmdd" content="20170223" />
+ <meta name="Date-Revision-yyyymmdd" content="20170627" />
<meta http-equiv="Content-Language" content="en" />
<title>Metron – Upgrading</title>
<link rel="stylesheet" href="./css/apache-maven-fluido-1.3.0.min.css" />
@@ -30,14 +30,11 @@
<div class="container-fluid">
<div id="banner">
<div class="pull-left">
- <a href="http://metron.incubator.apache.org/" id="bannerLeft">
- <img src="images/metron-logo.png" alt="Apache Metron - Incubating" width="148px" height="48px"/>
+ <a href="http://metron.apache.org/" id="bannerLeft">
+ <img src="images/metron-logo.png" alt="Apache Metron" width="148px" height="48px"/>
</a>
</div>
- <div class="pull-right"> <a href="http://incubator.apache.org/" id="bannerRight">
- <img src="images/ApacheIncubating_Logo.png" alt="Apache Incubating" width="192px" height="48px"/>
- </a>
- </div>
+ <div class="pull-right"> </div>
<div class="clear"><hr/></div>
</div>
@@ -51,8 +48,8 @@
</li>
<li class="divider ">/</li>
<li class="">
- <a href="http://metron.incubator.apache.org/" class="externalLink" title="Metron-Incubating">
- Metron-Incubating</a>
+ <a href="http://metron.apache.org/" class="externalLink" title="Metron">
+ Metron</a>
</li>
<li class="divider ">/</li>
<li class="">
@@ -64,8 +61,8 @@
- <li id="publishDate" class="pull-right">Last Published: 2017-02-23</li> <li class="divider pull-right">|</li>
- <li id="projectVersion" class="pull-right">Version: 0.3.1</li>
+ <li id="publishDate" class="pull-right">Last Published: 2017-06-27</li> <li class="divider pull-right">|</li>
+ <li id="projectVersion" class="pull-right">Version: 0.4.0</li>
</ul>
</div>
@@ -78,7 +75,7 @@
<ul class="nav nav-list">
<li class="nav-header">User Documentation</li>
-
+
<li>
<a href="index.html" title="Metron">
@@ -97,7 +94,7 @@
<i class="icon-chevron-right"></i>
Analytics</a>
</li>
-
+
<li>
<a href="metron-deployment/index.html" title="Deployment">
@@ -111,14 +108,28 @@
<i class="none"></i>
Docker</a>
</li>
-
+
+ <li>
+
+ <a href="metron-interface/metron-config/index.html" title="Config">
+ <i class="none"></i>
+ Config</a>
+ </li>
+
+ <li>
+
+ <a href="metron-interface/metron-rest/index.html" title="Rest">
+ <i class="none"></i>
+ Rest</a>
+ </li>
+
<li>
<a href="metron-platform/index.html" title="Platform">
<i class="icon-chevron-right"></i>
Platform</a>
</li>
-
+
<li>
<a href="metron-sensors/index.html" title="Sensors">
@@ -151,22 +162,12 @@
<p><a name="Upgrading"></a></p>
<p>This document constitutes a per-version listing of changes of configuration which are non-backwards compatible.</p>
<div class="section">
-<h2><a name="a0.2.0BETA_to_0.3.0"></a>0.2.0BETA to 0.3.0</h2>
+<h2><a name="a0.3.1_to_0.4.0"></a>0.3.1 to 0.4.0</h2>
<div class="section">
-<h3><a name="METRON-447:_Monit_fails_to_reload_when_upgrading_from_0.2.0BETA_to_master"></a><a class="externalLink" href="https://issues.apache.org/jira/browse/METRON-447">METRON-447: Monit fails to reload when upgrading from 0.2.0BETA to master</a></h3>
+<h3><a name="METRON-671:_Refactor_existing_Ansible_deployment_to_use_Ambari_MPack"></a><a class="externalLink" href="https://issues.apache.org/jira/browse/METRON-671">METRON-671: Refactor existing Ansible deployment to use Ambari MPack</a></h3>
<div class="section">
<h4><a name="Description"></a>Description</h4>
-<p><tt>/etc/monit.d/enrichment-elasticsearch.monit</tt> was renamed to <tt>/etc/monit.d/indexing-elasticsearch.monit</tt>, however the old file isn’t removed via ansible, which causes the below error during an upgrade: <tt>Starting monit: /etc/monit.d/enrichment-elasticsearch.monit:18: Service
-name conflict, enrichment already defined
-'/usr/local/monit/status_enrichment_topology.sh'</tt></p></div></div>
-<div class="section">
-<h3><a name="METRON-448:Upgrading_via_Ansible_deployment_does_not_add_topology.classpath"></a><a class="externalLink" href="https://issues.apache.org/jira/browse/METRON-448">METRON-448:Upgrading via Ansible deployment does not add topology.classpath </a></h3>
-<div class="section">
-<h4><a name="Description"></a>Description</h4>
-<p>When using Ansible to deploy the latest Metron bits to an existing installation, storm-site is not being updated with the new 0.2.1BETA parameter <tt>topology.classpath</tt>. Topologies are unable to find the client configs as a result.</p></div>
-<div class="section">
-<h4><a name="Workaround"></a>Workaround</h4>
-<p>Set the <tt>topology.classpath</tt> property for storm in Ambari to <tt>/etc/hbase/conf:/etc/hadoop/conf</tt></p></div></div></div>
+<p>Since the Ansible Deployment uses the MPack, RPMs must be built prior to deployment. As a result, <a class="externalLink" href="https://www.docker.com/">Docker</a> is required to perform a Quick-Dev, Full-Dev or Ansible deployment. This effectively limits the build environment to Docker supported <a class="externalLink" href="https://docs.docker.com/engine/installation/#platform-support-matrix">platforms</a>.</p></div></div></div>
<div class="section">
<h2><a name="a0.3.0_to_0.3.1"></a>0.3.0 to 0.3.1</h2>
<div class="section">
@@ -182,7 +183,7 @@ name conflict, enrichment already defined
<li>Has properties which control every writers (i.e. HDFS, solr or elasticsearch).</li>
</ul>
<p>In the 0.3.1 release, this configuration has been broken out and control for individual writers are separated.</p>
-<p>Please see the description of the configurations in the indexing <a class="externalLink" href="https://github.com/apache/incubator-metron/tree/Metron_0.3.1/metron-platform/metron-indexing#sensor-indexing-configuration">README</a></p></div>
+<p>Please see the description of the configurations in the indexing <a class="externalLink" href="https://github.com/apache/metron/tree/Metron_0.3.1/metron-platform/metron-indexing#sensor-indexing-configuration">README</a></p></div>
<div class="section">
<h4><a name="Migration"></a>Migration</h4>
<p>Migrate the configurations from each sensor enrichment configuration and create appropriate configurations for indexing.</p>
@@ -272,6 +273,23 @@ name conflict, enrichment already defined
<div class="section">
<h4><a name="Migration"></a>Migration</h4>
<p>Existing calls to <tt>PROFILE_GET</tt> will need to change from <tt>PROFILE_GET('profile', 'entity', duration, 'durationUnits')</tt> to <tt>PROFILE_GET('profile', 'entity', PROFILE_FIXED(duration, 'durationUnits'))</tt></p></div></div></div>
+<div class="section">
+<h2><a name="a0.2.0BETA_to_0.3.0"></a>0.2.0BETA to 0.3.0</h2>
+<div class="section">
+<h3><a name="METRON-447:_Monit_fails_to_reload_when_upgrading_from_0.2.0BETA_to_master"></a><a class="externalLink" href="https://issues.apache.org/jira/browse/METRON-447">METRON-447: Monit fails to reload when upgrading from 0.2.0BETA to master</a></h3>
+<div class="section">
+<h4><a name="Description"></a>Description</h4>
+<p><tt>/etc/monit.d/enrichment-elasticsearch.monit</tt> was renamed to <tt>/etc/monit.d/indexing-elasticsearch.monit</tt>, however the old file isn’t removed via ansible, which causes the below error during an upgrade: <tt>Starting monit: /etc/monit.d/enrichment-elasticsearch.monit:18: Service
+name conflict, enrichment already defined
+'/usr/local/monit/status_enrichment_topology.sh'</tt></p></div></div>
+<div class="section">
+<h3><a name="METRON-448:Upgrading_via_Ansible_deployment_does_not_add_topology.classpath"></a><a class="externalLink" href="https://issues.apache.org/jira/browse/METRON-448">METRON-448:Upgrading via Ansible deployment does not add topology.classpath </a></h3>
+<div class="section">
+<h4><a name="Description"></a>Description</h4>
+<p>When using Ansible to deploy the latest Metron bits to an existing installation, storm-site is not being updated with the new 0.2.1BETA parameter <tt>topology.classpath</tt>. Topologies are unable to find the client configs as a result.</p></div>
+<div class="section">
+<h4><a name="Workaround"></a>Workaround</h4>
+<p>Set the <tt>topology.classpath</tt> property for storm in Ambari to <tt>/etc/hbase/conf:/etc/hadoop/conf</tt></p></div></div></div>
</div>
</div>
</div>
@@ -280,8 +298,9 @@ name conflict, enrichment already defined
<footer>
<div class="container-fluid">
- <div class="row span12">Copyright © 2017.
- All Rights Reserved.
+ <div class="row span12">Copyright © 2017
+ <a href="https://www.apache.org">The Apache Software Foundation</a>.
+ All Rights Reserved.
</div>
http://git-wip-us.apache.org/repos/asf/metron/blob/cf4c2ecd/current-book/images/ambari-storm-site-properties.png
----------------------------------------------------------------------
diff --git a/current-book/images/ambari-storm-site-properties.png b/current-book/images/ambari-storm-site-properties.png
new file mode 100644
index 0000000..e0050a2
Binary files /dev/null and b/current-book/images/ambari-storm-site-properties.png differ
http://git-wip-us.apache.org/repos/asf/metron/blob/cf4c2ecd/current-book/images/ambari-storm-site.png
----------------------------------------------------------------------
diff --git a/current-book/images/ambari-storm-site.png b/current-book/images/ambari-storm-site.png
new file mode 100644
index 0000000..5ff2d24
Binary files /dev/null and b/current-book/images/ambari-storm-site.png differ
http://git-wip-us.apache.org/repos/asf/metron/blob/cf4c2ecd/current-book/images/custom-storm-site-final.png
----------------------------------------------------------------------
diff --git a/current-book/images/custom-storm-site-final.png b/current-book/images/custom-storm-site-final.png
new file mode 100644
index 0000000..9b383d5
Binary files /dev/null and b/current-book/images/custom-storm-site-final.png differ
http://git-wip-us.apache.org/repos/asf/metron/blob/cf4c2ecd/current-book/images/enable-kerberos-configure-kerberos.png
----------------------------------------------------------------------
diff --git a/current-book/images/enable-kerberos-configure-kerberos.png b/current-book/images/enable-kerberos-configure-kerberos.png
new file mode 100644
index 0000000..212c64b
Binary files /dev/null and b/current-book/images/enable-kerberos-configure-kerberos.png differ
http://git-wip-us.apache.org/repos/asf/metron/blob/cf4c2ecd/current-book/images/enable-kerberos-started.png
----------------------------------------------------------------------
diff --git a/current-book/images/enable-kerberos-started.png b/current-book/images/enable-kerberos-started.png
new file mode 100644
index 0000000..96adb51
Binary files /dev/null and b/current-book/images/enable-kerberos-started.png differ
http://git-wip-us.apache.org/repos/asf/metron/blob/cf4c2ecd/current-book/images/enable-kerberos.png
----------------------------------------------------------------------
diff --git a/current-book/images/enable-kerberos.png b/current-book/images/enable-kerberos.png
new file mode 100644
index 0000000..bb46923
Binary files /dev/null and b/current-book/images/enable-kerberos.png differ
http://git-wip-us.apache.org/repos/asf/metron/blob/cf4c2ecd/current-book/index.html
----------------------------------------------------------------------
diff --git a/current-book/index.html b/current-book/index.html
index ae246ce..517703c 100644
--- a/current-book/index.html
+++ b/current-book/index.html
@@ -1,13 +1,13 @@
<!DOCTYPE html>
<!--
- | Generated by Apache Maven Doxia at 2017-02-23
+ | Generated by Apache Maven Doxia at 2017-06-27
| Rendered using Apache Maven Fluido Skin 1.3.0
-->
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
- <meta name="Date-Revision-yyyymmdd" content="20170223" />
+ <meta name="Date-Revision-yyyymmdd" content="20170627" />
<meta http-equiv="Content-Language" content="en" />
<title>Metron – </title>
<link rel="stylesheet" href="./css/apache-maven-fluido-1.3.0.min.css" />
@@ -30,14 +30,11 @@
<div class="container-fluid">
<div id="banner">
<div class="pull-left">
- <a href="http://metron.incubator.apache.org/" id="bannerLeft">
- <img src="images/metron-logo.png" alt="Apache Metron - Incubating" width="148px" height="48px"/>
+ <a href="http://metron.apache.org/" id="bannerLeft">
+ <img src="images/metron-logo.png" alt="Apache Metron" width="148px" height="48px"/>
</a>
</div>
- <div class="pull-right"> <a href="http://incubator.apache.org/" id="bannerRight">
- <img src="images/ApacheIncubating_Logo.png" alt="Apache Incubating" width="192px" height="48px"/>
- </a>
- </div>
+ <div class="pull-right"> </div>
<div class="clear"><hr/></div>
</div>
@@ -51,8 +48,8 @@
</li>
<li class="divider ">/</li>
<li class="">
- <a href="http://metron.incubator.apache.org/" class="externalLink" title="Metron-Incubating">
- Metron-Incubating</a>
+ <a href="http://metron.apache.org/" class="externalLink" title="Metron">
+ Metron</a>
</li>
<li class="divider ">/</li>
<li class="">
@@ -64,8 +61,8 @@
- <li id="publishDate" class="pull-right">Last Published: 2017-02-23</li> <li class="divider pull-right">|</li>
- <li id="projectVersion" class="pull-right">Version: 0.3.1</li>
+ <li id="publishDate" class="pull-right">Last Published: 2017-06-27</li> <li class="divider pull-right">|</li>
+ <li id="projectVersion" class="pull-right">Version: 0.4.0</li>
</ul>
</div>
@@ -78,7 +75,7 @@
<ul class="nav nav-list">
<li class="nav-header">User Documentation</li>
-
+
<li class="active">
<a href="#"><i class="icon-chevron-down"></i>Metron</a>
@@ -97,7 +94,7 @@
<i class="icon-chevron-right"></i>
Analytics</a>
</li>
-
+
<li>
<a href="metron-deployment/index.html" title="Deployment">
@@ -111,14 +108,28 @@
<i class="none"></i>
Docker</a>
</li>
-
+
+ <li>
+
+ <a href="metron-interface/metron-config/index.html" title="Config">
+ <i class="none"></i>
+ Config</a>
+ </li>
+
+ <li>
+
+ <a href="metron-interface/metron-rest/index.html" title="Rest">
+ <i class="none"></i>
+ Rest</a>
+ </li>
+
<li>
<a href="metron-platform/index.html" title="Platform">
<i class="icon-chevron-right"></i>
Platform</a>
</li>
-
+
<li>
<a href="metron-sensors/index.html" title="Sensors">
@@ -147,10 +158,11 @@
<div id="bodyColumn" class="span9" >
- <p><a class="externalLink" href="https://travis-ci.org/apache/incubator-metron"><img src="https://travis-ci.org/apache/incubator-metron.svg?branch=master" alt="Build Status" /></a></p>
-<h1>Apache Metron (Incubating)</h1>
-<p><a name="Apache_Metron_Incubating"></a></p>
+ <p><a class="externalLink" href="https://travis-ci.org/apache/metron"><img src="https://travis-ci.org/apache/metron.svg?branch=master" alt="Build Status" /></a></p>
+<h1>Apache Metron</h1>
+<p><a name="Apache_Metron"></a></p>
<p>Metron integrates a variety of open source big data technologies in order to offer a centralized tool for security monitoring and analysis. Metron provides capabilities for log aggregation, full packet capture indexing, storage, advanced behavioral analytics and data enrichment, while applying the most current threat intelligence information to security telemetry within a single platform.</p>
+<p>For the latest information, please visit our website at <a class="externalLink" href="http://metron.apache.org/">http://metron.apache.org/</a></p>
<p>Metron can be divided into 4 areas:</p>
<ol style="list-style-type: decimal">
@@ -179,13 +191,14 @@
<p>Big data is a natural fit for powerful security analytics. The Metron framework integrates a number of elements from the Hadoop ecosystem to provide a scalable platform for security analytics, incorporating such functionality as full-packet capture, stream processing, batch processing, real-time search, and telemetry aggregation. With Metron, our goal is to tie big data into security analytics and drive towards an extensible centralized platform to effectively enable rapid detection and rapid response for advanced security threats. </p>
<p><a name="Obtaining_Metron"></a></p>
<h1>Obtaining Metron</h1>
+<p>To obtain a release of Metron, please visit <a class="externalLink" href="http://metron.apache.org/documentation/#releases">http://metron.apache.org/documentation/#releases</a></p>
<p>This repository is a collection of submodules for convenience which is regularly updated to point to the latest versions. Github provides multiple ways to obtain Metron’s code:</p>
<ol style="list-style-type: decimal">
-<li>git clone –recursive <a class="externalLink" href="https://github.com/apache/incubator-metron">https://github.com/apache/incubator-metron</a></li>
+<li>git clone –recursive <a class="externalLink" href="https://github.com/apache/metron">https://github.com/apache/metron</a></li>
-<li><a class="externalLink" href="https://github.com/apache/incubator-metron/archive/master.zip">Download ZIP</a></li>
+<li><a class="externalLink" href="https://github.com/apache/metron/archive/master.zip">Download ZIP</a></li>
<li>Clone or download each repository individually</li>
</ol>
@@ -211,6 +224,22 @@
<pre>$ mvn clean install -PHDP-2.5.0.0
</pre></div></div>
<p>You can swap “install” for “package” in the commands above if you don’t want to deploy the artifacts to your local .m2 repo.</p>
+<p><a name="Build_Metron_Reporting"></a></p>
+<h1>Build Metron Reporting</h1>
+<p>To build and run reporting with code coverage:</p>
+
+<div class="source">
+<div class="source">
+<pre>$ mvn clean install
+$ mvn site site:stage-deploy site:deploy
+</pre></div></div>
+<p>Code coverage can be skipped by skipping tests:</p>
+
+<div class="source">
+<div class="source">
+<pre>$ mvn clean install -DskipTests site site:stage-deploy site:deploy
+</pre></div></div>
+<p>The staged site is deployed to /tmp/metron/site/index.html, and can be viewed by opening the file in a browser.</p>
<p><a name="Navigating_the_Architecture"></a></p>
<h1>Navigating the Architecture</h1>
<p>Metron is at its core a Kappa architecture with Apache Storm as the processing component and Apache Kafka as the unified data bus.</p>
@@ -244,8 +273,9 @@
<footer>
<div class="container-fluid">
- <div class="row span12">Copyright © 2017.
- All Rights Reserved.
+ <div class="row span12">Copyright © 2017
+ <a href="https://www.apache.org">The Apache Software Foundation</a>.
+ All Rights Reserved.
</div>
http://git-wip-us.apache.org/repos/asf/metron/blob/cf4c2ecd/current-book/metron-analytics/index.html
----------------------------------------------------------------------
diff --git a/current-book/metron-analytics/index.html b/current-book/metron-analytics/index.html
index f5a9628..61354bd 100644
--- a/current-book/metron-analytics/index.html
+++ b/current-book/metron-analytics/index.html
@@ -1,13 +1,13 @@
<!DOCTYPE html>
<!--
- | Generated by Apache Maven Doxia at 2017-02-23
+ | Generated by Apache Maven Doxia at 2017-06-27
| Rendered using Apache Maven Fluido Skin 1.3.0
-->
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
- <meta name="Date-Revision-yyyymmdd" content="20170223" />
+ <meta name="Date-Revision-yyyymmdd" content="20170627" />
<meta http-equiv="Content-Language" content="en" />
<title>Metron – Metron Analytics</title>
<link rel="stylesheet" href="../css/apache-maven-fluido-1.3.0.min.css" />
@@ -30,14 +30,11 @@
<div class="container-fluid">
<div id="banner">
<div class="pull-left">
- <a href="http://metron.incubator.apache.org/" id="bannerLeft">
- <img src="../images/metron-logo.png" alt="Apache Metron - Incubating" width="148px" height="48px"/>
+ <a href="http://metron.apache.org/" id="bannerLeft">
+ <img src="../images/metron-logo.png" alt="Apache Metron" width="148px" height="48px"/>
</a>
</div>
- <div class="pull-right"> <a href="http://incubator.apache.org/" id="bannerRight">
- <img src="../images/ApacheIncubating_Logo.png" alt="Apache Incubating" width="192px" height="48px"/>
- </a>
- </div>
+ <div class="pull-right"> </div>
<div class="clear"><hr/></div>
</div>
@@ -51,8 +48,8 @@
</li>
<li class="divider ">/</li>
<li class="">
- <a href="http://metron.incubator.apache.org/" class="externalLink" title="Metron-Incubating">
- Metron-Incubating</a>
+ <a href="http://metron.apache.org/" class="externalLink" title="Metron">
+ Metron</a>
</li>
<li class="divider ">/</li>
<li class="">
@@ -64,8 +61,8 @@
- <li id="publishDate" class="pull-right">Last Published: 2017-02-23</li> <li class="divider pull-right">|</li>
- <li id="projectVersion" class="pull-right">Version: 0.3.1</li>
+ <li id="publishDate" class="pull-right">Last Published: 2017-06-27</li> <li class="divider pull-right">|</li>
+ <li id="projectVersion" class="pull-right">Version: 0.4.0</li>
</ul>
</div>
@@ -78,7 +75,7 @@
<ul class="nav nav-list">
<li class="nav-header">User Documentation</li>
-
+
<li>
<a href="../index.html" title="Metron">
@@ -127,7 +124,7 @@
</li>
</ul>
</li>
-
+
<li>
<a href="../metron-deployment/index.html" title="Deployment">
@@ -141,14 +138,28 @@
<i class="none"></i>
Docker</a>
</li>
-
+
+ <li>
+
+ <a href="../metron-interface/metron-config/index.html" title="Config">
+ <i class="none"></i>
+ Config</a>
+ </li>
+
+ <li>
+
+ <a href="../metron-interface/metron-rest/index.html" title="Rest">
+ <i class="none"></i>
+ Rest</a>
+ </li>
+
<li>
<a href="../metron-platform/index.html" title="Platform">
<i class="icon-chevron-right"></i>
Platform</a>
</li>
-
+
<li>
<a href="../metron-sensors/index.html" title="Sensors">
@@ -197,8 +208,9 @@
<footer>
<div class="container-fluid">
- <div class="row span12">Copyright © 2017.
- All Rights Reserved.
+ <div class="row span12">Copyright © 2017
+ <a href="https://www.apache.org">The Apache Software Foundation</a>.
+ All Rights Reserved.
</div>
http://git-wip-us.apache.org/repos/asf/metron/blob/cf4c2ecd/current-book/metron-analytics/metron-maas-service/index.html
----------------------------------------------------------------------
diff --git a/current-book/metron-analytics/metron-maas-service/index.html b/current-book/metron-analytics/metron-maas-service/index.html
index 66716a1..b0f9162 100644
--- a/current-book/metron-analytics/metron-maas-service/index.html
+++ b/current-book/metron-analytics/metron-maas-service/index.html
@@ -1,13 +1,13 @@
<!DOCTYPE html>
<!--
- | Generated by Apache Maven Doxia at 2017-02-23
+ | Generated by Apache Maven Doxia at 2017-06-27
| Rendered using Apache Maven Fluido Skin 1.3.0
-->
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
- <meta name="Date-Revision-yyyymmdd" content="20170223" />
+ <meta name="Date-Revision-yyyymmdd" content="20170627" />
<meta http-equiv="Content-Language" content="en" />
<title>Metron – Model Management Infrastructure</title>
<link rel="stylesheet" href="../../css/apache-maven-fluido-1.3.0.min.css" />
@@ -30,14 +30,11 @@
<div class="container-fluid">
<div id="banner">
<div class="pull-left">
- <a href="http://metron.incubator.apache.org/" id="bannerLeft">
- <img src="../../images/metron-logo.png" alt="Apache Metron - Incubating" width="148px" height="48px"/>
+ <a href="http://metron.apache.org/" id="bannerLeft">
+ <img src="../../images/metron-logo.png" alt="Apache Metron" width="148px" height="48px"/>
</a>
</div>
- <div class="pull-right"> <a href="http://incubator.apache.org/" id="bannerRight">
- <img src="../../images/ApacheIncubating_Logo.png" alt="Apache Incubating" width="192px" height="48px"/>
- </a>
- </div>
+ <div class="pull-right"> </div>
<div class="clear"><hr/></div>
</div>
@@ -51,8 +48,8 @@
</li>
<li class="divider ">/</li>
<li class="">
- <a href="http://metron.incubator.apache.org/" class="externalLink" title="Metron-Incubating">
- Metron-Incubating</a>
+ <a href="http://metron.apache.org/" class="externalLink" title="Metron">
+ Metron</a>
</li>
<li class="divider ">/</li>
<li class="">
@@ -64,8 +61,8 @@
- <li id="publishDate" class="pull-right">Last Published: 2017-02-23</li> <li class="divider pull-right">|</li>
- <li id="projectVersion" class="pull-right">Version: 0.3.1</li>
+ <li id="publishDate" class="pull-right">Last Published: 2017-06-27</li> <li class="divider pull-right">|</li>
+ <li id="projectVersion" class="pull-right">Version: 0.4.0</li>
</ul>
</div>
@@ -78,7 +75,7 @@
<ul class="nav nav-list">
<li class="nav-header">User Documentation</li>
-
+
<li>
<a href="../../index.html" title="Metron">
@@ -127,7 +124,7 @@
</li>
</ul>
</li>
-
+
<li>
<a href="../../metron-deployment/index.html" title="Deployment">
@@ -141,14 +138,28 @@
<i class="none"></i>
Docker</a>
</li>
-
+
+ <li>
+
+ <a href="../../metron-interface/metron-config/index.html" title="Config">
+ <i class="none"></i>
+ Config</a>
+ </li>
+
+ <li>
+
+ <a href="../../metron-interface/metron-rest/index.html" title="Rest">
+ <i class="none"></i>
+ Rest</a>
+ </li>
+
<li>
<a href="../../metron-platform/index.html" title="Platform">
<i class="icon-chevron-right"></i>
Platform</a>
</li>
-
+
<li>
<a href="../../metron-sensors/index.html" title="Sensors">
@@ -246,7 +257,7 @@ usage: MaaSClient
-h,--help This screen
-j,--jar <arg> Jar file containing the application
master
- -l4j,--log_properties <arg> log4j.properties file
+ -l,--log4j <arg> The log4j properties file to load
-ma,--modify_acls <arg> Users and groups that allowed to
modify the timeline entities in the
given domain
@@ -293,6 +304,7 @@ usage: ModelSubmission
-h,--help This screen
-hmp,--hdfs_model_path <arg> Model Path (HDFS)
-lmp,--local_model_path <arg> Model Path (local)
+ -l,--log4j <arg> The log4j properties file to load
-m,--memory <arg> Memory for container
-mo,--mode <arg> ADD, LIST or REMOVE
-n,--name <arg> Model Name
@@ -461,8 +473,9 @@ usage: ModelSubmission
<footer>
<div class="container-fluid">
- <div class="row span12">Copyright © 2017.
- All Rights Reserved.
+ <div class="row span12">Copyright © 2017
+ <a href="https://www.apache.org">The Apache Software Foundation</a>.
+ All Rights Reserved.
</div>
[05/10] metron git commit: METRON-1012 Update Metron public web site
for 0.4.0 release - push to asf-site from the generated code on master
Posted by ma...@apache.org.
http://git-wip-us.apache.org/repos/asf/metron/blob/cf4c2ecd/current-book/metron-interface/metron-rest/index.html
----------------------------------------------------------------------
diff --git a/current-book/metron-interface/metron-rest/index.html b/current-book/metron-interface/metron-rest/index.html
new file mode 100644
index 0000000..5955c5f
--- /dev/null
+++ b/current-book/metron-interface/metron-rest/index.html
@@ -0,0 +1,1996 @@
+<!DOCTYPE html>
+<!--
+ | Generated by Apache Maven Doxia at 2017-06-27
+ | Rendered using Apache Maven Fluido Skin 1.3.0
+-->
+<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
+ <head>
+ <meta charset="UTF-8" />
+ <meta name="viewport" content="width=device-width, initial-scale=1.0" />
+ <meta name="Date-Revision-yyyymmdd" content="20170627" />
+ <meta http-equiv="Content-Language" content="en" />
+ <title>Metron – Metron REST</title>
+ <link rel="stylesheet" href="../../css/apache-maven-fluido-1.3.0.min.css" />
+ <link rel="stylesheet" href="../../css/site.css" />
+ <link rel="stylesheet" href="../../css/print.css" media="print" />
+
+
+ <script type="text/javascript" src="../../js/apache-maven-fluido-1.3.0.min.js"></script>
+
+
+
+<script type="text/javascript">$( document ).ready( function() { $( '.carousel' ).carousel( { interval: 3500 } ) } );</script>
+
+ </head>
+ <body class="topBarDisabled">
+
+
+
+
+ <div class="container-fluid">
+ <div id="banner">
+ <div class="pull-left">
+ <a href="http://metron.apache.org/" id="bannerLeft">
+ <img src="../../images/metron-logo.png" alt="Apache Metron" width="148px" height="48px"/>
+ </a>
+ </div>
+ <div class="pull-right"> </div>
+ <div class="clear"><hr/></div>
+ </div>
+
+ <div id="breadcrumbs">
+ <ul class="breadcrumb">
+
+
+ <li class="">
+ <a href="http://www.apache.org" class="externalLink" title="Apache">
+ Apache</a>
+ </li>
+ <li class="divider ">/</li>
+ <li class="">
+ <a href="http://metron.apache.org/" class="externalLink" title="Metron">
+ Metron</a>
+ </li>
+ <li class="divider ">/</li>
+ <li class="">
+ <a href="../../index.html" title="Documentation">
+ Documentation</a>
+ </li>
+ <li class="divider ">/</li>
+ <li class="">Metron REST</li>
+
+
+
+ <li id="publishDate" class="pull-right">Last Published: 2017-06-27</li> <li class="divider pull-right">|</li>
+ <li id="projectVersion" class="pull-right">Version: 0.4.0</li>
+
+ </ul>
+ </div>
+
+
+ <div class="row-fluid">
+ <div id="leftColumn" class="span3">
+ <div class="well sidebar-nav">
+
+
+ <ul class="nav nav-list">
+ <li class="nav-header">User Documentation</li>
+
+ <li>
+
+ <a href="../../index.html" title="Metron">
+ <i class="icon-chevron-down"></i>
+ Metron</a>
+ <ul class="nav nav-list">
+
+ <li>
+
+ <a href="../../Upgrading.html" title="Upgrading">
+ <i class="none"></i>
+ Upgrading</a>
+ </li>
+
+ <li>
+
+ <a href="../../metron-analytics/index.html" title="Analytics">
+ <i class="icon-chevron-right"></i>
+ Analytics</a>
+ </li>
+
+ <li>
+
+ <a href="../../metron-deployment/index.html" title="Deployment">
+ <i class="icon-chevron-right"></i>
+ Deployment</a>
+ </li>
+
+ <li>
+
+ <a href="../../metron-docker/index.html" title="Docker">
+ <i class="none"></i>
+ Docker</a>
+ </li>
+
+ <li>
+
+ <a href="../../metron-interface/metron-config/index.html" title="Config">
+ <i class="none"></i>
+ Config</a>
+ </li>
+
+ <li class="active">
+
+ <a href="#"><i class="none"></i>Rest</a>
+ </li>
+
+ <li>
+
+ <a href="../../metron-platform/index.html" title="Platform">
+ <i class="icon-chevron-right"></i>
+ Platform</a>
+ </li>
+
+ <li>
+
+ <a href="../../metron-sensors/index.html" title="Sensors">
+ <i class="icon-chevron-right"></i>
+ Sensors</a>
+ </li>
+ </ul>
+ </li>
+ </ul>
+
+
+
+ <hr class="divider" />
+
+ <div id="poweredBy">
+ <div class="clear"></div>
+ <div class="clear"></div>
+ <div class="clear"></div>
+ <a href="http://maven.apache.org/" title="Built by Maven" class="poweredBy">
+ <img class="builtBy" alt="Built by Maven" src="../../images/logos/maven-feather.png" />
+ </a>
+ </div>
+ </div>
+ </div>
+
+
+ <div id="bodyColumn" class="span9" >
+
+ <h1>Metron REST</h1>
+<p><a name="Metron_REST"></a></p>
+<p>This module provides a RESTful API for interacting with Metron.</p>
+<div class="section">
+<h2><a name="Prerequisites"></a>Prerequisites</h2>
+
+<ul>
+
+<li>A running Metron cluster</li>
+
+<li>Java 8 installed</li>
+
+<li>Storm CLI and Metron topology scripts (start_parser_topology.sh, start_enrichment_topology.sh, start_elasticsearch_topology.sh) installed</li>
+
+<li>A relational database</li>
+</ul></div>
+<div class="section">
+<h2><a name="Installation"></a>Installation</h2>
+<div class="section">
+<h3><a name="From_Source"></a>From Source</h3>
+
+<ol style="list-style-type: decimal">
+
+<li>
+<p>Package the application with Maven:</p>
+
+<div class="source">
+<div class="source">
+<pre>mvn clean package
+</pre></div></div></li>
+
+<li>
+<p>Untar the archive in the $METRON_HOME directory. The directory structure will look like:</p>
+
+<div class="source">
+<div class="source">
+<pre>config
+ rest_application.yml
+bin
+ metron-rest
+lib
+ metron-rest-$METRON_VERSION.jar
+</pre></div></div></li>
+
+<li>
+<p>Copy the <tt>$METRON_HOME/bin/metron-rest</tt> script to <tt>/etc/init.d/metron-rest</tt></p></li>
+</ol></div>
+<div class="section">
+<h3><a name="From_Package_Manager"></a>From Package Manager</h3>
+
+<ol style="list-style-type: decimal">
+
+<li>
+<p>Deploy the RPM at <tt>/metron/metron-deployment/packaging/docker/rpm-docker/target/RPMS/noarch/metron-rest-$METRON_VERSION-*.noarch.rpm</tt></p></li>
+
+<li>
+<p>Install the RPM with:</p>
+
+<div class="source">
+<div class="source">
+<pre>rpm -ih metron-rest-$METRON_VERSION-*.noarch.rpm
+</pre></div></div></li>
+</ol></div></div>
+<div class="section">
+<h2><a name="Configuration"></a>Configuration</h2>
+<p>The REST application depends on several configuration parameters:</p>
+<div class="section">
+<h3><a name="REQUIRED"></a>REQUIRED</h3>
+<p>No optional parameter has a default.</p>
+
+<table border="0" class="table table-striped">
+ <thead>
+
+<tr class="a">
+
+<th>Environment Variable </th>
+
+<th>Description</th>
+ </tr>
+ </thead>
+ <tbody>
+
+<tr class="b">
+
+<td>METRON_JDBC_DRIVER </td>
+
+<td>JDBC driver class</td>
+ </tr>
+
+<tr class="a">
+
+<td>METRON_JDBC_URL </td>
+
+<td>JDBC url</td>
+ </tr>
+
+<tr class="b">
+
+<td>METRON_JDBC_USERNAME </td>
+
+<td>JDBC username</td>
+ </tr>
+
+<tr class="a">
+
+<td>METRON_JDBC_PLATFORM </td>
+
+<td>JDBC platform (one of h2, mysql, postgres, oracle</td>
+ </tr>
+
+<tr class="b">
+
+<td>ZOOKEEPER </td>
+
+<td>Zookeeper quorum (ex. node1:2181,node2:2181)</td>
+ </tr>
+
+<tr class="a">
+
+<td>BROKERLIST </td>
+
+<td>Kafka Broker list (ex. node1:6667,node2:6667)</td>
+ </tr>
+
+<tr class="b">
+
+<td>HDFS_URL </td>
+
+<td>HDFS url or <tt>fs.defaultFS</tt> Hadoop setting (ex. <a class="externalLink" href="hdfs://node1:8020">hdfs://node1:8020</a>)</td>
+ </tr>
+ </tbody>
+</table></div>
+<div class="section">
+<h3><a name="Optional_-_With_Defaults"></a>Optional - With Defaults</h3>
+
+<table border="0" class="table table-striped">
+ <thead>
+
+<tr class="a">
+
+<th>Environment Variable </th>
+
+<th>Description </th>
+
+<th>Required </th>
+
+<th>Default</th>
+ </tr>
+ </thead>
+ <tbody>
+
+<tr class="b">
+
+<td>METRON_USER </td>
+
+<td>Run the application as this user </td>
+
+<td>Optional </td>
+
+<td>metron</td>
+ </tr>
+
+<tr class="a">
+
+<td>METRON_LOG_DIR </td>
+
+<td>Directory where the log file is written </td>
+
+<td>Optional </td>
+
+<td>/var/log/metron/</td>
+ </tr>
+
+<tr class="b">
+
+<td>METRON_PID_DIR </td>
+
+<td>Directory where the pid file is written </td>
+
+<td>Optional </td>
+
+<td>/var/run/metron/</td>
+ </tr>
+
+<tr class="a">
+
+<td>METRON_REST_PORT </td>
+
+<td>REST application port </td>
+
+<td>Optional </td>
+
+<td>8082</td>
+ </tr>
+
+<tr class="b">
+
+<td>METRON_JDBC_CLIENT_PATH </td>
+
+<td>Path to JDBC client jar </td>
+
+<td>Optional </td>
+
+<td>H2 is bundled</td>
+ </tr>
+
+<tr class="a">
+
+<td>METRON_TEMP_GROK_PATH </td>
+
+<td>Temporary directory used to test grok statements </td>
+
+<td>Optional </td>
+
+<td>./patterns/temp</td>
+ </tr>
+
+<tr class="b">
+
+<td>METRON_DEFAULT_GROK_PATH </td>
+
+<td>Defaults HDFS directory used to store grok statements </td>
+
+<td>Optional </td>
+
+<td>/apps/metron/patterns</td>
+ </tr>
+
+<tr class="a">
+
+<td>SECURITY_ENABLED </td>
+
+<td>Enables Kerberos support </td>
+
+<td>Optional </td>
+
+<td>false</td>
+ </tr>
+ </tbody>
+</table></div>
+<div class="section">
+<h3><a name="Optional_-_Blank_Defaults"></a>Optional - Blank Defaults</h3>
+
+<table border="0" class="table table-striped">
+ <thead>
+
+<tr class="a">
+
+<th>Environment Variable </th>
+
+<th>Description </th>
+
+<th>Required</th>
+ </tr>
+ </thead>
+ <tbody>
+
+<tr class="b">
+
+<td>METRON_JVMFLAGS </td>
+
+<td>JVM flags added to the start command </td>
+
+<td>Optional</td>
+ </tr>
+
+<tr class="a">
+
+<td>METRON_SPRING_PROFILES_ACTIVE </td>
+
+<td>Active Spring profiles (see <a href="#spring-profiles">below</a>) </td>
+
+<td>Optional</td>
+ </tr>
+
+<tr class="b">
+
+<td>METRON_SPRING_OPTIONS </td>
+
+<td>Additional Spring input parameters </td>
+
+<td>Optional</td>
+ </tr>
+
+<tr class="a">
+
+<td>METRON_PRINCIPAL_NAME </td>
+
+<td>Kerberos principal for the metron user </td>
+
+<td>Optional</td>
+ </tr>
+
+<tr class="b">
+
+<td>METRON_SERVICE_KEYTAB </td>
+
+<td>Path to the Kerberos keytab for the metron user </td>
+
+<td>Optional</td>
+ </tr>
+ </tbody>
+</table>
+<p>These are set in the <tt>/etc/sysconfig/metron</tt> file.</p></div></div>
+<div class="section">
+<h2><a name="Database_setup"></a>Database setup</h2>
+<p>The REST application persists data in a relational database and requires a dedicated database user and database (see <a class="externalLink" href="https://docs.spring.io/spring-boot/docs/current/reference/html/boot-features-sql.html">https://docs.spring.io/spring-boot/docs/current/reference/html/boot-features-sql.html</a> for more detail).</p>
+<div class="section">
+<h3><a name="Development"></a>Development</h3>
+<p>The REST application comes with embedded database support for development purposes (<a class="externalLink" href="https://docs.spring.io/spring-boot/docs/current/reference/html/boot-features-sql.html#boot-features-embedded-database-support)">https://docs.spring.io/spring-boot/docs/current/reference/html/boot-features-sql.html#boot-features-embedded-database-support)</a>.</p>
+<p>For example, edit these variables in <tt>/etc/sysconfig/metron</tt> before starting the application to configure H2:</p>
+
+<div class="source">
+<div class="source">
+<pre>METRON_JDBC_DRIVER="org.h2.Driver"
+METRON_JDBC_URL="jdbc:h2:file:~/metrondb"
+METRON_JDBC_USERNAME="root"
+METRON_JDBC_PASSWORD='root"
+METRON_JDBC_PLATFORM="h2"
+</pre></div></div></div>
+<div class="section">
+<h3><a name="Production"></a>Production</h3>
+<p>The REST application should be configured with a production-grade database outside of development.</p>
+<p>For example, the following configures the application for MySQL:</p>
+
+<ol style="list-style-type: decimal">
+
+<li>
+<p>Install MySQL if not already available (this example uses version 5.7, installation instructions can be found <a class="externalLink" href="https://dev.mysql.com/doc/refman/5.7/en/linux-installation-yum-repo.html">here</a>)</p></li>
+
+<li>
+<p>Create a metron user and REST database and permission the user for that database:</p>
+
+<div class="source">
+<div class="source">
+<pre>CREATE USER 'metron'@'node1' IDENTIFIED BY 'Myp@ssw0rd';
+CREATE DATABASE IF NOT EXISTS metronrest;
+GRANT ALL PRIVILEGES ON metronrest.* TO 'metron'@'node1';
+</pre></div></div></li>
+
+<li>
+<p>Install the MySQL JDBC client onto the REST application host and configurate the METRON_JDBC_CLIENT_PATH variable:</p>
+
+<div class="source">
+<div class="source">
+<pre>cd $METRON_HOME/lib
+wget https://dev.mysql.com/get/Downloads/Connector-J/mysql-connector-java-5.1.41.tar.gz
+tar xf mysql-connector-java-5.1.41.tar.gz
+</pre></div></div></li>
+
+<li>
+<p>Edit these variables in <tt>/etc/sysconfig/metron</tt> to configure the REST application for MySQL:</p>
+
+<div class="source">
+<div class="source">
+<pre>METRON_JDBC_DRIVER="com.mysql.jdbc.Driver"
+METRON_JDBC_URL="jdbc:mysql://mysql_host:3306/metronrest"
+METRON_JDBC_USERNAME="metron"
+METRON_JDBC_PLATFORM="mysql"
+METRON_JDBC_CLIENT_PATH=$METRON_HOME/lib/mysql-connector-java-5.1.41/mysql-connector-java-5.1.41-bin.jar
+</pre></div></div></li>
+</ol></div></div>
+<div class="section">
+<h2><a name="Usage"></a>Usage</h2>
+<p>After configuration is complete, the REST application can be managed as a service:</p>
+
+<div class="source">
+<div class="source">
+<pre>service metron-rest start
+</pre></div></div>
+<p>If a production database is configured, the JDBC password should be passed in as the first argument on startup:</p>
+
+<div class="source">
+<div class="source">
+<pre>service metron-rest start Myp@ssw0rd
+</pre></div></div>
+<p>The REST application can be accessed with the Swagger UI at <a class="externalLink" href="http://host:port/swagger-ui.html#/">http://host:port/swagger-ui.html#/</a>. The default port is 8082.</p></div>
+<div class="section">
+<h2><a name="Security"></a>Security</h2>
+<div class="section">
+<h3><a name="Authentication"></a>Authentication</h3>
+<p>The metron-rest module uses <a class="externalLink" href="http://projects.spring.io/spring-security/">Spring Security</a> for authentication and stores user credentials in the relational database configured above. The required tables are created automatically the first time the application is started so that should be done first. For example (continuing the MySQL example above), users can be added by connecting to MySQL and running:</p>
+
+<div class="source">
+<div class="source">
+<pre>use metronrest;
+insert into users (username, password, enabled) values ('your_username','your_password',1);
+insert into authorities (username, authority) values ('your_username', 'ROLE_USER');
+</pre></div></div></div>
+<div class="section">
+<h3><a name="Kerberos"></a>Kerberos</h3>
+<p>Metron REST can be configured for a cluster with Kerberos enabled. A client JAAS file is required for Kafka and Zookeeper and a Kerberos keytab for the metron user principal is required for all other services. Configure these settings in the <tt>/etc/sysconfig/metron</tt> file:</p>
+
+<div class="source">
+<div class="source">
+<pre>SECURITY_ENABLED=true
+METRON_JVMFLAGS="-Djava.security.auth.login.config=$METRON_HOME/client_jaas.conf"
+METRON_PRINCIPAL_NAME="metron@EXAMPLE.COM"
+METRON_SERVICE_KEYTAB="/etc/security/keytabs/metron.keytab"
+</pre></div></div></div></div>
+<div class="section">
+<h2><a name="Spring_Profiles"></a>Spring Profiles</h2>
+<p>The REST application comes with a few <a class="externalLink" href="http://docs.spring.io/autorepo/docs/spring-boot/current/reference/html/boot-features-profiles.html">Spring Profiles</a> to aid in testing and development.</p>
+
+<table border="0" class="table table-striped">
+ <thead>
+
+<tr class="a">
+
+<th>Profile </th>
+
+<th>Description </th>
+ </tr>
+ </thead>
+ <tbody>
+
+<tr class="b">
+
+<td>test </td>
+
+<td>sets variables to in-memory services, only used for integration testing </td>
+ </tr>
+
+<tr class="a">
+
+<td>dev </td>
+
+<td>adds a test user to the database with credentials <tt>user/password</tt> </td>
+ </tr>
+
+<tr class="b">
+
+<td>vagrant </td>
+
+<td>sets configuration variables to match the Metron vagrant environment </td>
+ </tr>
+
+<tr class="a">
+
+<td>docker </td>
+
+<td>sets configuration variables to match the Metron dcoker environment </td>
+ </tr>
+ </tbody>
+</table>
+<p>Setting active profiles is done with the METRON_SPRING_PROFILES_ACTIVE variable. For example, set this variable in <tt>/etc/sysconfig/metron</tt> to configure the REST application for the Vagrant environment and add a test user:</p>
+
+<div class="source">
+<div class="source">
+<pre>METRON_SPRING_PROFILES_ACTIVE="vagrant,dev"
+</pre></div></div></div>
+<div class="section">
+<h2><a name="API"></a>API</h2>
+<p>Request and Response objects are JSON formatted. The JSON schemas are available in the Swagger UI.</p>
+
+<table border="0" class="table table-striped">
+ <thead>
+
+<tr class="a">
+
+<th> </th>
+ </tr>
+ </thead>
+ <tbody>
+
+<tr class="b">
+
+<td><a href="#GET_apiv1globalconfig"> <tt>GET /api/v1/global/config</tt></a></td>
+ </tr>
+
+<tr class="a">
+
+<td><a href="#DELETE_apiv1globalconfig"> <tt>DELETE /api/v1/global/config</tt></a></td>
+ </tr>
+
+<tr class="b">
+
+<td><a href="#POST_apiv1globalconfig"> <tt>POST /api/v1/global/config</tt></a></td>
+ </tr>
+
+<tr class="a">
+
+<td><a href="#GET_apiv1grokgetstatement"> <tt>GET /api/v1/grok/get/statement</tt></a></td>
+ </tr>
+
+<tr class="b">
+
+<td><a href="#GET_apiv1groklist"> <tt>GET /api/v1/grok/list</tt></a></td>
+ </tr>
+
+<tr class="a">
+
+<td><a href="#POST_apiv1grokvalidate"> <tt>POST /api/v1/grok/validate</tt></a></td>
+ </tr>
+
+<tr class="b">
+
+<td><a href="#POST_apiv1hdfs"> <tt>POST /api/v1/hdfs</tt></a></td>
+ </tr>
+
+<tr class="a">
+
+<td><a href="#GET_apiv1hdfs"> <tt>GET /api/v1/hdfs</tt></a></td>
+ </tr>
+
+<tr class="b">
+
+<td><a href="#DELETE_apiv1hdfs"> <tt>DELETE /api/v1/hdfs</tt></a></td>
+ </tr>
+
+<tr class="a">
+
+<td><a href="#GET_apiv1hdfslist"> <tt>GET /api/v1/hdfs/list</tt></a></td>
+ </tr>
+
+<tr class="b">
+
+<td><a href="#GET_apiv1kafkatopic"> <tt>GET /api/v1/kafka/topic</tt></a></td>
+ </tr>
+
+<tr class="a">
+
+<td><a href="#POST_apiv1kafkatopic"> <tt>POST /api/v1/kafka/topic</tt></a></td>
+ </tr>
+
+<tr class="b">
+
+<td><a href="#GET_apiv1kafkatopicname"> <tt>GET /api/v1/kafka/topic/{name}</tt></a></td>
+ </tr>
+
+<tr class="a">
+
+<td><a href="#DELETE_apiv1kafkatopicname"> <tt>DELETE /api/v1/kafka/topic/{name}</tt></a></td>
+ </tr>
+
+<tr class="b">
+
+<td><a href="#GET_apiv1kafkatopicnamesample"> <tt>GET /api/v1/kafka/topic/{name}/sample</tt></a></td>
+ </tr>
+
+<tr class="a">
+
+<td><a href="#GET_apiv1sensorenrichmentconfig"> <tt>GET /api/v1/sensor/enrichment/config</tt></a></td>
+ </tr>
+
+<tr class="b">
+
+<td><a href="#GET_apiv1sensorenrichmentconfiglistavailableenrichments"> <tt>GET /api/v1/sensor/enrichment/config/list/available/enrichments</tt></a></td>
+ </tr>
+
+<tr class="a">
+
+<td><a href="#GET_apiv1sensorenrichmentconfiglistavailablethreattriageaggregators"> <tt>GET /api/v1/sensor/enrichment/config/list/available/threat/triage/aggregators</tt></a></td>
+ </tr>
+
+<tr class="b">
+
+<td><a href="#DELETE_apiv1sensorenrichmentconfigname"> <tt>DELETE /api/v1/sensor/enrichment/config/{name}</tt></a></td>
+ </tr>
+
+<tr class="a">
+
+<td><a href="#POST_apiv1sensorenrichmentconfigname"> <tt>POST /api/v1/sensor/enrichment/config/{name}</tt></a></td>
+ </tr>
+
+<tr class="b">
+
+<td><a href="#GET_apiv1sensorenrichmentconfigname"> <tt>GET /api/v1/sensor/enrichment/config/{name}</tt></a></td>
+ </tr>
+
+<tr class="a">
+
+<td><a href="#GET_apiv1sensorindexingconfig"> <tt>GET /api/v1/sensor/indexing/config</tt></a></td>
+ </tr>
+
+<tr class="b">
+
+<td><a href="#DELETE_apiv1sensorindexingconfigname"> <tt>DELETE /api/v1/sensor/indexing/config/{name}</tt></a></td>
+ </tr>
+
+<tr class="a">
+
+<td><a href="#POST_apiv1sensorindexingconfigname"> <tt>POST /api/v1/sensor/indexing/config/{name}</tt></a></td>
+ </tr>
+
+<tr class="b">
+
+<td><a href="#GET_apiv1sensorindexingconfigname"> <tt>GET /api/v1/sensor/indexing/config/{name}</tt></a></td>
+ </tr>
+
+<tr class="a">
+
+<td><a href="#POST_apiv1sensorparserconfig"> <tt>POST /api/v1/sensor/parser/config</tt></a></td>
+ </tr>
+
+<tr class="b">
+
+<td><a href="#GET_apiv1sensorparserconfig"> <tt>GET /api/v1/sensor/parser/config</tt></a></td>
+ </tr>
+
+<tr class="a">
+
+<td><a href="#GET_apiv1sensorparserconfiglistavailable"> <tt>GET /api/v1/sensor/parser/config/list/available</tt></a></td>
+ </tr>
+
+<tr class="b">
+
+<td><a href="#POST_apiv1sensorparserconfigparseMessage"> <tt>POST /api/v1/sensor/parser/config/parseMessage</tt></a></td>
+ </tr>
+
+<tr class="a">
+
+<td><a href="#GET_apiv1sensorparserconfigreloadavailable"> <tt>GET /api/v1/sensor/parser/config/reload/available</tt></a></td>
+ </tr>
+
+<tr class="b">
+
+<td><a href="#DELETE_apiv1sensorparserconfigname"> <tt>DELETE /api/v1/sensor/parser/config/{name}</tt></a></td>
+ </tr>
+
+<tr class="a">
+
+<td><a href="#GET_apiv1sensorparserconfigname"> <tt>GET /api/v1/sensor/parser/config/{name}</tt></a></td>
+ </tr>
+
+<tr class="b">
+
+<td><a href="#POST_apiv1stellarapplytransformations"> <tt>POST /api/v1/stellar/apply/transformations</tt></a></td>
+ </tr>
+
+<tr class="a">
+
+<td><a href="#GET_apiv1stellarlist"> <tt>GET /api/v1/stellar/list</tt></a></td>
+ </tr>
+
+<tr class="b">
+
+<td><a href="#GET_apiv1stellarlistfunctions"> <tt>GET /api/v1/stellar/list/functions</tt></a></td>
+ </tr>
+
+<tr class="a">
+
+<td><a href="#GET_apiv1stellarlistsimplefunctions"> <tt>GET /api/v1/stellar/list/simple/functions</tt></a></td>
+ </tr>
+
+<tr class="b">
+
+<td><a href="#POST_apiv1stellarvalidaterules"> <tt>POST /api/v1/stellar/validate/rules</tt></a></td>
+ </tr>
+
+<tr class="a">
+
+<td><a href="#GET_apiv1storm"> <tt>GET /api/v1/storm</tt></a></td>
+ </tr>
+
+<tr class="b">
+
+<td><a href="#GET_apiv1stormclientstatus"> <tt>GET /api/v1/storm/client/status</tt></a></td>
+ </tr>
+
+<tr class="a">
+
+<td><a href="#GET_apiv1stormenrichment"> <tt>GET /api/v1/storm/enrichment</tt></a></td>
+ </tr>
+
+<tr class="b">
+
+<td><a href="#GET_apiv1stormenrichmentactivate"> <tt>GET /api/v1/storm/enrichment/activate</tt></a></td>
+ </tr>
+
+<tr class="a">
+
+<td><a href="#GET_apiv1stormenrichmentdeactivate"> <tt>GET /api/v1/storm/enrichment/deactivate</tt></a></td>
+ </tr>
+
+<tr class="b">
+
+<td><a href="#GET_apiv1stormenrichmentstart"> <tt>GET /api/v1/storm/enrichment/start</tt></a></td>
+ </tr>
+
+<tr class="a">
+
+<td><a href="#GET_apiv1stormenrichmentstop"> <tt>GET /api/v1/storm/enrichment/stop</tt></a></td>
+ </tr>
+
+<tr class="b">
+
+<td><a href="#GET_apiv1stormindexing"> <tt>GET /api/v1/storm/indexing</tt></a></td>
+ </tr>
+
+<tr class="a">
+
+<td><a href="#GET_apiv1stormindexingactivate"> <tt>GET /api/v1/storm/indexing/activate</tt></a></td>
+ </tr>
+
+<tr class="b">
+
+<td><a href="#GET_apiv1stormindexingdeactivate"> <tt>GET /api/v1/storm/indexing/deactivate</tt></a></td>
+ </tr>
+
+<tr class="a">
+
+<td><a href="#GET_apiv1stormindexingstart"> <tt>GET /api/v1/storm/indexing/start</tt></a></td>
+ </tr>
+
+<tr class="b">
+
+<td><a href="#GET_apiv1stormindexingstop"> <tt>GET /api/v1/storm/indexing/stop</tt></a></td>
+ </tr>
+
+<tr class="a">
+
+<td><a href="#GET_apiv1stormparseractivatename"> <tt>GET /api/v1/storm/parser/activate/{name}</tt></a></td>
+ </tr>
+
+<tr class="b">
+
+<td><a href="#GET_apiv1stormparserdeactivatename"> <tt>GET /api/v1/storm/parser/deactivate/{name}</tt></a></td>
+ </tr>
+
+<tr class="a">
+
+<td><a href="#GET_apiv1stormparserstartname"> <tt>GET /api/v1/storm/parser/start/{name}</tt></a></td>
+ </tr>
+
+<tr class="b">
+
+<td><a href="#GET_apiv1stormparserstopname"> <tt>GET /api/v1/storm/parser/stop/{name}</tt></a></td>
+ </tr>
+
+<tr class="a">
+
+<td><a href="#GET_apiv1stormname"> <tt>GET /api/v1/storm/{name}</tt></a></td>
+ </tr>
+
+<tr class="b">
+
+<td><a href="#GET_apiv1user"> <tt>GET /api/v1/user</tt></a></td>
+ </tr>
+ </tbody>
+</table>
+<div class="section">
+<h3><a name="GET_apiv1globalconfig"></a><tt>GET /api/v1/global/config</tt></h3>
+
+<ul>
+
+<li>Description: Retrieves the current Global Config from Zookeeper</li>
+
+<li>Returns:
+
+<ul>
+
+<li>200 - Returns current Global Config JSON in Zookeeper</li>
+
+<li>404 - Global Config JSON was not found in Zookeeper</li>
+ </ul></li>
+</ul></div>
+<div class="section">
+<h3><a name="DELETE_apiv1globalconfig"></a><tt>DELETE /api/v1/global/config</tt></h3>
+
+<ul>
+
+<li>Description: Deletes the current Global Config from Zookeeper</li>
+
+<li>Returns:
+
+<ul>
+
+<li>200 - Global Config JSON was deleted</li>
+
+<li>404 - Global Config JSON was not found in Zookeeper</li>
+ </ul></li>
+</ul></div>
+<div class="section">
+<h3><a name="POST_apiv1globalconfig"></a><tt>POST /api/v1/global/config</tt></h3>
+
+<ul>
+
+<li>Description: Creates or updates the Global Config in Zookeeper</li>
+
+<li>Input:
+
+<ul>
+
+<li>globalConfig - The Global Config JSON to be saved</li>
+ </ul></li>
+
+<li>Returns:
+
+<ul>
+
+<li>200 - Global Config updated. Returns saved Global Config JSON</li>
+
+<li>201 - Global Config created. Returns saved Global Config JSON</li>
+ </ul></li>
+</ul></div>
+<div class="section">
+<h3><a name="GET_apiv1grokgetstatement"></a><tt>GET /api/v1/grok/get/statement</tt></h3>
+
+<ul>
+
+<li>Description: Retrieves a Grok statement from the classpath</li>
+
+<li>Input:
+
+<ul>
+
+<li>path - Path to classpath resource</li>
+ </ul></li>
+
+<li>Returns:
+
+<ul>
+
+<li>200 - Grok statement</li>
+ </ul></li>
+</ul></div>
+<div class="section">
+<h3><a name="GET_apiv1groklist"></a><tt>GET /api/v1/grok/list</tt></h3>
+
+<ul>
+
+<li>Description: Lists the common Grok statements available in Metron</li>
+
+<li>Returns:
+
+<ul>
+
+<li>200 - JSON object containing pattern label/Grok statements key value pairs</li>
+ </ul></li>
+</ul></div>
+<div class="section">
+<h3><a name="POST_apiv1grokvalidate"></a><tt>POST /api/v1/grok/validate</tt></h3>
+
+<ul>
+
+<li>Description: Applies a Grok statement to a sample message</li>
+
+<li>Input:
+
+<ul>
+
+<li>grokValidation - Object containing Grok statement and sample message</li>
+ </ul></li>
+
+<li>Returns:
+
+<ul>
+
+<li>200 - JSON results</li>
+ </ul></li>
+</ul></div>
+<div class="section">
+<h3><a name="POST_apiv1hdfs"></a><tt>POST /api/v1/hdfs</tt></h3>
+
+<ul>
+
+<li>Description: Writes contents to an HDFS file. Warning: this will overwrite the contents of a file if it already exists.</li>
+
+<li>Input:
+
+<ul>
+
+<li>path - Path to HDFS file</li>
+
+<li>contents - File contents</li>
+ </ul></li>
+
+<li>Returns:
+
+<ul>
+
+<li>200 - Contents were written</li>
+ </ul></li>
+</ul></div>
+<div class="section">
+<h3><a name="GET_apiv1hdfs"></a><tt>GET /api/v1/hdfs</tt></h3>
+
+<ul>
+
+<li>Description: Reads a file from HDFS and returns the contents</li>
+
+<li>Input:
+
+<ul>
+
+<li>path - Path to HDFS file</li>
+ </ul></li>
+
+<li>Returns:
+
+<ul>
+
+<li>200 - Returns file contents</li>
+ </ul></li>
+</ul></div>
+<div class="section">
+<h3><a name="DELETE_apiv1hdfs"></a><tt>DELETE /api/v1/hdfs</tt></h3>
+
+<ul>
+
+<li>Description: Deletes a file from HDFS</li>
+
+<li>Input:
+
+<ul>
+
+<li>path - Path to HDFS file</li>
+
+<li>recursive - Delete files recursively</li>
+ </ul></li>
+
+<li>Returns:
+
+<ul>
+
+<li>200 - File was deleted</li>
+
+<li>404 - File was not found in HDFS</li>
+ </ul></li>
+</ul></div>
+<div class="section">
+<h3><a name="GET_apiv1hdfslist"></a><tt>GET /api/v1/hdfs/list</tt></h3>
+
+<ul>
+
+<li>Description: Lists an HDFS directory</li>
+
+<li>Input:
+
+<ul>
+
+<li>path - Path to HDFS directory</li>
+ </ul></li>
+
+<li>Returns:
+
+<ul>
+
+<li>200 - HDFS directory list</li>
+ </ul></li>
+</ul></div>
+<div class="section">
+<h3><a name="GET_apiv1kafkatopic"></a><tt>GET /api/v1/kafka/topic</tt></h3>
+
+<ul>
+
+<li>Description: Retrieves all Kafka topics</li>
+
+<li>Returns:
+
+<ul>
+
+<li>200 - Returns a list of all Kafka topics</li>
+ </ul></li>
+</ul></div>
+<div class="section">
+<h3><a name="POST_apiv1kafkatopic"></a><tt>POST /api/v1/kafka/topic</tt></h3>
+
+<ul>
+
+<li>Description: Creates a new Kafka topic</li>
+
+<li>Input:
+
+<ul>
+
+<li>topic - Kafka topic</li>
+ </ul></li>
+
+<li>Returns:
+
+<ul>
+
+<li>200 - Returns saved Kafka topic</li>
+ </ul></li>
+</ul></div>
+<div class="section">
+<h3><a name="GET_apiv1kafkatopicname"></a><tt>GET /api/v1/kafka/topic/{name}</tt></h3>
+
+<ul>
+
+<li>Description: Retrieves a Kafka topic</li>
+
+<li>Input:
+
+<ul>
+
+<li>name - Kafka topic name</li>
+ </ul></li>
+
+<li>Returns:
+
+<ul>
+
+<li>200 - Returns Kafka topic</li>
+
+<li>404 - Kafka topic is missing</li>
+ </ul></li>
+</ul></div>
+<div class="section">
+<h3><a name="DELETE_apiv1kafkatopicname"></a><tt>DELETE /api/v1/kafka/topic/{name}</tt></h3>
+
+<ul>
+
+<li>Description: Deletes a Kafka topic</li>
+
+<li>Input:
+
+<ul>
+
+<li>name - Kafka topic name</li>
+ </ul></li>
+
+<li>Returns:
+
+<ul>
+
+<li>200 - Kafka topic was deleted</li>
+
+<li>404 - Kafka topic is missing</li>
+ </ul></li>
+</ul></div>
+<div class="section">
+<h3><a name="GET_apiv1kafkatopicnamesample"></a><tt>GET /api/v1/kafka/topic/{name}/sample</tt></h3>
+
+<ul>
+
+<li>Description: Retrieves a sample message from a Kafka topic using the most recent offset</li>
+
+<li>Input:
+
+<ul>
+
+<li>name - Kafka topic name</li>
+ </ul></li>
+
+<li>Returns:
+
+<ul>
+
+<li>200 - Returns sample message</li>
+
+<li>404 - Either Kafka topic is missing or contains no messages</li>
+ </ul></li>
+</ul></div>
+<div class="section">
+<h3><a name="GET_apiv1sensorenrichmentconfig"></a><tt>GET /api/v1/sensor/enrichment/config</tt></h3>
+
+<ul>
+
+<li>Description: Retrieves all SensorEnrichmentConfigs from Zookeeper</li>
+
+<li>Returns:
+
+<ul>
+
+<li>200 - Returns all SensorEnrichmentConfigs</li>
+ </ul></li>
+</ul></div>
+<div class="section">
+<h3><a name="GET_apiv1sensorenrichmentconfiglistavailableenrichments"></a><tt>GET /api/v1/sensor/enrichment/config/list/available/enrichments</tt></h3>
+
+<ul>
+
+<li>Description: Lists the available enrichments</li>
+
+<li>Returns:
+
+<ul>
+
+<li>200 - Returns a list of available enrichments</li>
+ </ul></li>
+</ul></div>
+<div class="section">
+<h3><a name="GET_apiv1sensorenrichmentconfiglistavailablethreattriageaggregators"></a><tt>GET /api/v1/sensor/enrichment/config/list/available/threat/triage/aggregators</tt></h3>
+
+<ul>
+
+<li>Description: Lists the available threat triage aggregators</li>
+
+<li>Returns:
+
+<ul>
+
+<li>200 - Returns a list of available threat triage aggregators</li>
+ </ul></li>
+</ul></div>
+<div class="section">
+<h3><a name="DELETE_apiv1sensorenrichmentconfigname"></a><tt>DELETE /api/v1/sensor/enrichment/config/{name}</tt></h3>
+
+<ul>
+
+<li>Description: Deletes a SensorEnrichmentConfig from Zookeeper</li>
+
+<li>Input:
+
+<ul>
+
+<li>name - SensorEnrichmentConfig name</li>
+ </ul></li>
+
+<li>Returns:
+
+<ul>
+
+<li>200 - SensorEnrichmentConfig was deleted</li>
+
+<li>404 - SensorEnrichmentConfig is missing</li>
+ </ul></li>
+</ul></div>
+<div class="section">
+<h3><a name="POST_apiv1sensorenrichmentconfigname"></a><tt>POST /api/v1/sensor/enrichment/config/{name}</tt></h3>
+
+<ul>
+
+<li>Description: Updates or creates a SensorEnrichmentConfig in Zookeeper</li>
+
+<li>Input:
+
+<ul>
+
+<li>sensorEnrichmentConfig - SensorEnrichmentConfig</li>
+
+<li>name - SensorEnrichmentConfig name</li>
+ </ul></li>
+
+<li>Returns:
+
+<ul>
+
+<li>200 - SensorEnrichmentConfig updated. Returns saved SensorEnrichmentConfig</li>
+
+<li>201 - SensorEnrichmentConfig created. Returns saved SensorEnrichmentConfig</li>
+ </ul></li>
+</ul></div>
+<div class="section">
+<h3><a name="GET_apiv1sensorenrichmentconfigname"></a><tt>GET /api/v1/sensor/enrichment/config/{name}</tt></h3>
+
+<ul>
+
+<li>Description: Retrieves a SensorEnrichmentConfig from Zookeeper</li>
+
+<li>Input:
+
+<ul>
+
+<li>name - SensorEnrichmentConfig name</li>
+ </ul></li>
+
+<li>Returns:
+
+<ul>
+
+<li>200 - Returns SensorEnrichmentConfig</li>
+
+<li>404 - SensorEnrichmentConfig is missing</li>
+ </ul></li>
+</ul></div>
+<div class="section">
+<h3><a name="GET_apiv1sensorindexingconfig"></a><tt>GET /api/v1/sensor/indexing/config</tt></h3>
+
+<ul>
+
+<li>Description: Retrieves all SensorIndexingConfigs from Zookeeper</li>
+
+<li>Returns:
+
+<ul>
+
+<li>200 - Returns all SensorIndexingConfigs</li>
+ </ul></li>
+</ul></div>
+<div class="section">
+<h3><a name="DELETE_apiv1sensorindexingconfigname"></a><tt>DELETE /api/v1/sensor/indexing/config/{name}</tt></h3>
+
+<ul>
+
+<li>Description: Deletes a SensorIndexingConfig from Zookeeper</li>
+
+<li>Input:
+
+<ul>
+
+<li>name - SensorIndexingConfig name</li>
+ </ul></li>
+
+<li>Returns:
+
+<ul>
+
+<li>200 - SensorIndexingConfig was deleted</li>
+
+<li>404 - SensorIndexingConfig is missing</li>
+ </ul></li>
+</ul></div>
+<div class="section">
+<h3><a name="POST_apiv1sensorindexingconfigname"></a><tt>POST /api/v1/sensor/indexing/config/{name}</tt></h3>
+
+<ul>
+
+<li>Description: Updates or creates a SensorIndexingConfig in Zookeeper</li>
+
+<li>Input:
+
+<ul>
+
+<li>sensorIndexingConfig - SensorIndexingConfig</li>
+
+<li>name - SensorIndexingConfig name</li>
+ </ul></li>
+
+<li>Returns:
+
+<ul>
+
+<li>200 - SensorIndexingConfig updated. Returns saved SensorIndexingConfig</li>
+
+<li>201 - SensorIndexingConfig created. Returns saved SensorIndexingConfig</li>
+ </ul></li>
+</ul></div>
+<div class="section">
+<h3><a name="GET_apiv1sensorindexingconfigname"></a><tt>GET /api/v1/sensor/indexing/config/{name}</tt></h3>
+
+<ul>
+
+<li>Description: Retrieves a SensorIndexingConfig from Zookeeper</li>
+
+<li>Input:
+
+<ul>
+
+<li>name - SensorIndexingConfig name</li>
+ </ul></li>
+
+<li>Returns:
+
+<ul>
+
+<li>200 - Returns SensorIndexingConfig</li>
+
+<li>404 - SensorIndexingConfig is missing</li>
+ </ul></li>
+</ul></div>
+<div class="section">
+<h3><a name="POST_apiv1sensorparserconfig"></a><tt>POST /api/v1/sensor/parser/config</tt></h3>
+
+<ul>
+
+<li>Description: Updates or creates a SensorParserConfig in Zookeeper</li>
+
+<li>Input:
+
+<ul>
+
+<li>sensorParserConfig - SensorParserConfig</li>
+ </ul></li>
+
+<li>Returns:
+
+<ul>
+
+<li>200 - SensorParserConfig updated. Returns saved SensorParserConfig</li>
+
+<li>201 - SensorParserConfig created. Returns saved SensorParserConfig</li>
+ </ul></li>
+</ul></div>
+<div class="section">
+<h3><a name="GET_apiv1sensorparserconfig"></a><tt>GET /api/v1/sensor/parser/config</tt></h3>
+
+<ul>
+
+<li>Description: Retrieves all SensorParserConfigs from Zookeeper</li>
+
+<li>Returns:
+
+<ul>
+
+<li>200 - Returns all SensorParserConfigs</li>
+ </ul></li>
+</ul></div>
+<div class="section">
+<h3><a name="GET_apiv1sensorparserconfiglistavailable"></a><tt>GET /api/v1/sensor/parser/config/list/available</tt></h3>
+
+<ul>
+
+<li>Description: Lists the available parser classes that can be found on the classpath</li>
+
+<li>Returns:
+
+<ul>
+
+<li>200 - Returns a list of available parser classes</li>
+ </ul></li>
+</ul></div>
+<div class="section">
+<h3><a name="POST_apiv1sensorparserconfigparseMessage"></a><tt>POST /api/v1/sensor/parser/config/parseMessage</tt></h3>
+
+<ul>
+
+<li>Description: Parses a sample message given a SensorParserConfig</li>
+
+<li>Input:
+
+<ul>
+
+<li>parseMessageRequest - Object containing a sample message and SensorParserConfig</li>
+ </ul></li>
+
+<li>Returns:
+
+<ul>
+
+<li>200 - Returns parsed message</li>
+ </ul></li>
+</ul></div>
+<div class="section">
+<h3><a name="GET_apiv1sensorparserconfigreloadavailable"></a><tt>GET /api/v1/sensor/parser/config/reload/available</tt></h3>
+
+<ul>
+
+<li>Description: Scans the classpath for available parser classes and reloads the cached parser class list</li>
+
+<li>Returns:
+
+<ul>
+
+<li>200 - Returns a list of available parser classes</li>
+ </ul></li>
+</ul></div>
+<div class="section">
+<h3><a name="DELETE_apiv1sensorparserconfigname"></a><tt>DELETE /api/v1/sensor/parser/config/{name}</tt></h3>
+
+<ul>
+
+<li>Description: Deletes a SensorParserConfig from Zookeeper</li>
+
+<li>Input:
+
+<ul>
+
+<li>name - SensorParserConfig name</li>
+ </ul></li>
+
+<li>Returns:
+
+<ul>
+
+<li>200 - SensorParserConfig was deleted</li>
+
+<li>404 - SensorParserConfig is missing</li>
+ </ul></li>
+</ul></div>
+<div class="section">
+<h3><a name="GET_apiv1sensorparserconfigname"></a><tt>GET /api/v1/sensor/parser/config/{name}</tt></h3>
+
+<ul>
+
+<li>Description: Retrieves a SensorParserConfig from Zookeeper</li>
+
+<li>Input:
+
+<ul>
+
+<li>name - SensorParserConfig name</li>
+ </ul></li>
+
+<li>Returns:
+
+<ul>
+
+<li>200 - Returns SensorParserConfig</li>
+
+<li>404 - SensorParserConfig is missing</li>
+ </ul></li>
+</ul></div>
+<div class="section">
+<h3><a name="POST_apiv1stellarapplytransformations"></a><tt>POST /api/v1/stellar/apply/transformations</tt></h3>
+
+<ul>
+
+<li>Description: Executes transformations against a sample message</li>
+
+<li>Input:
+
+<ul>
+
+<li>transformationValidation - Object containing SensorParserConfig and sample message</li>
+ </ul></li>
+
+<li>Returns:
+
+<ul>
+
+<li>200 - Returns transformation results</li>
+ </ul></li>
+</ul></div>
+<div class="section">
+<h3><a name="GET_apiv1stellarlist"></a><tt>GET /api/v1/stellar/list</tt></h3>
+
+<ul>
+
+<li>Description: Retrieves field transformations</li>
+
+<li>Returns:
+
+<ul>
+
+<li>200 - Returns a list field transformations</li>
+ </ul></li>
+</ul></div>
+<div class="section">
+<h3><a name="GET_apiv1stellarlistfunctions"></a><tt>GET /api/v1/stellar/list/functions</tt></h3>
+
+<ul>
+
+<li>Description: Lists the Stellar functions that can be found on the classpath</li>
+
+<li>Returns:
+
+<ul>
+
+<li>200 - Returns a list of Stellar functions</li>
+ </ul></li>
+</ul></div>
+<div class="section">
+<h3><a name="GET_apiv1stellarlistsimplefunctions"></a><tt>GET /api/v1/stellar/list/simple/functions</tt></h3>
+
+<ul>
+
+<li>Description: Lists the simple Stellar functions (functions with only 1 input) that can be found on the classpath</li>
+
+<li>Returns:
+
+<ul>
+
+<li>200 - Returns a list of simple Stellar functions</li>
+ </ul></li>
+</ul></div>
+<div class="section">
+<h3><a name="POST_apiv1stellarvalidaterules"></a><tt>POST /api/v1/stellar/validate/rules</tt></h3>
+
+<ul>
+
+<li>Description: Tests Stellar statements to ensure they are well-formed</li>
+
+<li>Input:
+
+<ul>
+
+<li>statements - List of statements to validate</li>
+ </ul></li>
+
+<li>Returns:
+
+<ul>
+
+<li>200 - Returns validation results</li>
+ </ul></li>
+</ul></div>
+<div class="section">
+<h3><a name="GET_apiv1storm"></a><tt>GET /api/v1/storm</tt></h3>
+
+<ul>
+
+<li>Description: Retrieves the status of all Storm topologies</li>
+
+<li>Returns:
+
+<ul>
+
+<li>200 - Returns a list of topologies with status information</li>
+ </ul></li>
+</ul></div>
+<div class="section">
+<h3><a name="GET_apiv1stormclientstatus"></a><tt>GET /api/v1/storm/client/status</tt></h3>
+
+<ul>
+
+<li>Description: Retrieves information about the Storm command line client</li>
+
+<li>Returns:
+
+<ul>
+
+<li>200 - Returns storm command line client information</li>
+ </ul></li>
+</ul></div>
+<div class="section">
+<h3><a name="GET_apiv1stormenrichment"></a><tt>GET /api/v1/storm/enrichment</tt></h3>
+
+<ul>
+
+<li>Description: Retrieves the status of the Storm enrichment topology</li>
+
+<li>Returns:
+
+<ul>
+
+<li>200 - Returns topology status information</li>
+
+<li>404 - Topology is missing</li>
+ </ul></li>
+</ul></div>
+<div class="section">
+<h3><a name="GET_apiv1stormenrichmentactivate"></a><tt>GET /api/v1/storm/enrichment/activate</tt></h3>
+
+<ul>
+
+<li>Description: Activates a Storm enrichment topology</li>
+
+<li>Returns:
+
+<ul>
+
+<li>200 - Returns activate response message</li>
+ </ul></li>
+</ul></div>
+<div class="section">
+<h3><a name="GET_apiv1stormenrichmentdeactivate"></a><tt>GET /api/v1/storm/enrichment/deactivate</tt></h3>
+
+<ul>
+
+<li>Description: Deactivates a Storm enrichment topology</li>
+
+<li>Returns:
+
+<ul>
+
+<li>200 - Returns deactivate response message</li>
+ </ul></li>
+</ul></div>
+<div class="section">
+<h3><a name="GET_apiv1stormenrichmentstart"></a><tt>GET /api/v1/storm/enrichment/start</tt></h3>
+
+<ul>
+
+<li>Description: Starts a Storm enrichment topology</li>
+
+<li>Returns:
+
+<ul>
+
+<li>200 - Returns start response message</li>
+ </ul></li>
+</ul></div>
+<div class="section">
+<h3><a name="GET_apiv1stormenrichmentstop"></a><tt>GET /api/v1/storm/enrichment/stop</tt></h3>
+
+<ul>
+
+<li>Description: Stops a Storm enrichment topology</li>
+
+<li>Input:
+
+<ul>
+
+<li>stopNow - Stop the topology immediately</li>
+ </ul></li>
+
+<li>Returns:
+
+<ul>
+
+<li>200 - Returns stop response message</li>
+ </ul></li>
+</ul></div>
+<div class="section">
+<h3><a name="GET_apiv1stormindexing"></a><tt>GET /api/v1/storm/indexing</tt></h3>
+
+<ul>
+
+<li>Description: Retrieves the status of the Storm indexing topology</li>
+
+<li>Returns:
+
+<ul>
+
+<li>200 - Returns topology status information</li>
+
+<li>404 - Topology is missing</li>
+ </ul></li>
+</ul></div>
+<div class="section">
+<h3><a name="GET_apiv1stormindexingactivate"></a><tt>GET /api/v1/storm/indexing/activate</tt></h3>
+
+<ul>
+
+<li>Description: Activates a Storm indexing topology</li>
+
+<li>Returns:
+
+<ul>
+
+<li>200 - Returns activate response message</li>
+ </ul></li>
+</ul></div>
+<div class="section">
+<h3><a name="GET_apiv1stormindexingdeactivate"></a><tt>GET /api/v1/storm/indexing/deactivate</tt></h3>
+
+<ul>
+
+<li>Description: Deactivates a Storm indexing topology</li>
+
+<li>Returns:
+
+<ul>
+
+<li>200 - Returns deactivate response message</li>
+ </ul></li>
+</ul></div>
+<div class="section">
+<h3><a name="GET_apiv1stormindexingstart"></a><tt>GET /api/v1/storm/indexing/start</tt></h3>
+
+<ul>
+
+<li>Description: Starts a Storm indexing topology</li>
+
+<li>Returns:
+
+<ul>
+
+<li>200 - Returns start response message</li>
+ </ul></li>
+</ul></div>
+<div class="section">
+<h3><a name="GET_apiv1stormindexingstop"></a><tt>GET /api/v1/storm/indexing/stop</tt></h3>
+
+<ul>
+
+<li>Description: Stops a Storm enrichment topology</li>
+
+<li>Input:
+
+<ul>
+
+<li>stopNow - Stop the topology immediately</li>
+ </ul></li>
+
+<li>Returns:
+
+<ul>
+
+<li>200 - Returns stop response message</li>
+ </ul></li>
+</ul></div>
+<div class="section">
+<h3><a name="GET_apiv1stormparseractivatename"></a><tt>GET /api/v1/storm/parser/activate/{name}</tt></h3>
+
+<ul>
+
+<li>Description: Activates a Storm parser topology</li>
+
+<li>Input:
+
+<ul>
+
+<li>name - Parser name</li>
+ </ul></li>
+
+<li>Returns:
+
+<ul>
+
+<li>200 - Returns activate response message</li>
+ </ul></li>
+</ul></div>
+<div class="section">
+<h3><a name="GET_apiv1stormparserdeactivatename"></a><tt>GET /api/v1/storm/parser/deactivate/{name}</tt></h3>
+
+<ul>
+
+<li>Description: Deactivates a Storm parser topology</li>
+
+<li>Input:
+
+<ul>
+
+<li>name - Parser name</li>
+ </ul></li>
+
+<li>Returns:
+
+<ul>
+
+<li>200 - Returns deactivate response message</li>
+ </ul></li>
+</ul></div>
+<div class="section">
+<h3><a name="GET_apiv1stormparserstartname"></a><tt>GET /api/v1/storm/parser/start/{name}</tt></h3>
+
+<ul>
+
+<li>Description: Starts a Storm parser topology</li>
+
+<li>Input:
+
+<ul>
+
+<li>name - Parser name</li>
+ </ul></li>
+
+<li>Returns:
+
+<ul>
+
+<li>200 - Returns start response message</li>
+ </ul></li>
+</ul></div>
+<div class="section">
+<h3><a name="GET_apiv1stormparserstopname"></a><tt>GET /api/v1/storm/parser/stop/{name}</tt></h3>
+
+<ul>
+
+<li>Description: Stops a Storm parser topology</li>
+
+<li>Input:
+
+<ul>
+
+<li>name - Parser name</li>
+
+<li>stopNow - Stop the topology immediately</li>
+ </ul></li>
+
+<li>Returns:
+
+<ul>
+
+<li>200 - Returns stop response message</li>
+ </ul></li>
+</ul></div>
+<div class="section">
+<h3><a name="GET_apiv1stormname"></a><tt>GET /api/v1/storm/{name}</tt></h3>
+
+<ul>
+
+<li>Description: Retrieves the status of a Storm topology</li>
+
+<li>Input:
+
+<ul>
+
+<li>name - Topology name</li>
+ </ul></li>
+
+<li>Returns:
+
+<ul>
+
+<li>200 - Returns topology status information</li>
+
+<li>404 - Topology is missing</li>
+ </ul></li>
+</ul></div>
+<div class="section">
+<h3><a name="GET_apiv1user"></a><tt>GET /api/v1/user</tt></h3>
+
+<ul>
+
+<li>Description: Retrieves the current user</li>
+
+<li>Returns:
+
+<ul>
+
+<li>200 - Current user</li>
+ </ul></li>
+</ul></div></div>
+<div class="section">
+<h2><a name="Testing"></a>Testing</h2>
+<p>Profiles are includes for both the metron-docker and Quick Dev environments.</p>
+<div class="section">
+<h3><a name="metron-docker"></a>metron-docker</h3>
+<p>Start the <a href="../../metron-docker/index.html">metron-docker</a> environment. Build the metron-rest module and start it with the Spring Boot Maven plugin:</p>
+
+<div class="source">
+<div class="source">
+<pre>mvn clean package
+mvn spring-boot:run -Drun.profiles=docker,dev
+</pre></div></div>
+<p>The metron-rest application will be available at <a class="externalLink" href="http://localhost:8080/swagger-ui.html#/">http://localhost:8080/swagger-ui.html#/</a>.</p></div>
+<div class="section">
+<h3><a name="Quick_Dev"></a>Quick Dev</h3>
+<p>Start the <a href="../../metron-deployment/vagrant/quick-dev-platform/index.html">Quick Dev</a> environment. Build the metron-rest module and start it with the Spring Boot Maven plugin:</p>
+
+<div class="source">
+<div class="source">
+<pre>mvn clean package
+mvn spring-boot:run -Drun.profiles=vagrant,dev
+</pre></div></div>
+<p>The metron-rest application will be available at <a class="externalLink" href="http://localhost:8080/swagger-ui.html#/">http://localhost:8080/swagger-ui.html#/</a>.</p>
+<p>To run the application locally on the Quick Dev host (node1), follow the <a href="#Installation">Installation</a> instructions above. Then set the METRON_SPRING_PROFILES_ACTIVE variable in <tt>/etc/sysconfig/metron</tt>:</p>
+
+<div class="source">
+<div class="source">
+<pre>METRON_SPRING_PROFILES_ACTIVE="vagrant,dev"
+</pre></div></div>
+<p>and start the application:</p>
+
+<div class="source">
+<div class="source">
+<pre>service metron-rest start
+</pre></div></div>
+<p>In a cluster with Kerberos enabled, update the security settings in <tt>/etc/sysconfig/metron</tt>. Security is disabled by default in the <tt>vagrant</tt> Spring profile so that setting must be overriden with the METRON_SPRING_OPTIONS variable:</p>
+
+<div class="source">
+<div class="source">
+<pre>METRON_SPRING_PROFILES_ACTIVE="vagrant,dev"
+METRON_JVMFLAGS="-Djava.security.auth.login.config=$METRON_HOME/client_jaas.conf"
+METRON_SPRING_OPTIONS="--kerberos.enabled=true"
+</pre></div></div>
+<p>The metron-rest application will be available at <a class="externalLink" href="http://node1:8082/swagger-ui.html#/">http://node1:8082/swagger-ui.html#/</a>.</p></div></div>
+<div class="section">
+<h2><a name="License"></a>License</h2>
+<p>This project depends on the Java Transaction API. See <a class="externalLink" href="https://java.net/projects/jta-spec/">https://java.net/projects/jta-spec/</a> for more details.</p></div>
+ </div>
+ </div>
+ </div>
+
+ <hr/>
+
+ <footer>
+ <div class="container-fluid">
+ <div class="row span12">Copyright © 2017
+ <a href="https://www.apache.org">The Apache Software Foundation</a>.
+ All Rights Reserved.
+
+ </div>
+
+
+
+ </div>
+ </footer>
+ </body>
+</html>
http://git-wip-us.apache.org/repos/asf/metron/blob/cf4c2ecd/current-book/metron-platform/index.html
----------------------------------------------------------------------
diff --git a/current-book/metron-platform/index.html b/current-book/metron-platform/index.html
index 0546ef1..44c1804 100644
--- a/current-book/metron-platform/index.html
+++ b/current-book/metron-platform/index.html
@@ -1,13 +1,13 @@
<!DOCTYPE html>
<!--
- | Generated by Apache Maven Doxia at 2017-02-23
+ | Generated by Apache Maven Doxia at 2017-06-27
| Rendered using Apache Maven Fluido Skin 1.3.0
-->
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
- <meta name="Date-Revision-yyyymmdd" content="20170223" />
+ <meta name="Date-Revision-yyyymmdd" content="20170627" />
<meta http-equiv="Content-Language" content="en" />
<title>Metron – Current Build</title>
<link rel="stylesheet" href="../css/apache-maven-fluido-1.3.0.min.css" />
@@ -30,14 +30,11 @@
<div class="container-fluid">
<div id="banner">
<div class="pull-left">
- <a href="http://metron.incubator.apache.org/" id="bannerLeft">
- <img src="../images/metron-logo.png" alt="Apache Metron - Incubating" width="148px" height="48px"/>
+ <a href="http://metron.apache.org/" id="bannerLeft">
+ <img src="../images/metron-logo.png" alt="Apache Metron" width="148px" height="48px"/>
</a>
</div>
- <div class="pull-right"> <a href="http://incubator.apache.org/" id="bannerRight">
- <img src="../images/ApacheIncubating_Logo.png" alt="Apache Incubating" width="192px" height="48px"/>
- </a>
- </div>
+ <div class="pull-right"> </div>
<div class="clear"><hr/></div>
</div>
@@ -51,8 +48,8 @@
</li>
<li class="divider ">/</li>
<li class="">
- <a href="http://metron.incubator.apache.org/" class="externalLink" title="Metron-Incubating">
- Metron-Incubating</a>
+ <a href="http://metron.apache.org/" class="externalLink" title="Metron">
+ Metron</a>
</li>
<li class="divider ">/</li>
<li class="">
@@ -64,8 +61,8 @@
- <li id="publishDate" class="pull-right">Last Published: 2017-02-23</li> <li class="divider pull-right">|</li>
- <li id="projectVersion" class="pull-right">Version: 0.3.1</li>
+ <li id="publishDate" class="pull-right">Last Published: 2017-06-27</li> <li class="divider pull-right">|</li>
+ <li id="projectVersion" class="pull-right">Version: 0.4.0</li>
</ul>
</div>
@@ -78,7 +75,7 @@
<ul class="nav nav-list">
<li class="nav-header">User Documentation</li>
-
+
<li>
<a href="../index.html" title="Metron">
@@ -99,7 +96,7 @@
<i class="icon-chevron-right"></i>
Analytics</a>
</li>
-
+
<li>
<a href="../metron-deployment/index.html" title="Deployment">
@@ -113,7 +110,21 @@
<i class="none"></i>
Docker</a>
</li>
-
+
+ <li>
+
+ <a href="../metron-interface/metron-config/index.html" title="Config">
+ <i class="none"></i>
+ Config</a>
+ </li>
+
+ <li>
+
+ <a href="../metron-interface/metron-rest/index.html" title="Rest">
+ <i class="none"></i>
+ Rest</a>
+ </li>
+
<li class="active">
<a href="#"><i class="icon-chevron-down"></i>Platform</a>
@@ -125,13 +136,13 @@
<i class="none"></i>
Api</a>
</li>
-
+
<li>
<a href="../metron-platform/metron-common/index.html" title="Common">
- <i class="none"></i>
+ <i class="icon-chevron-right"></i>
Common</a>
- </li>
+ </li>
<li>
@@ -174,9 +185,16 @@
<i class="none"></i>
Pcap-backend</a>
</li>
+
+ <li>
+
+ <a href="../metron-platform/metron-writer/index.html" title="Writer">
+ <i class="none"></i>
+ Writer</a>
+ </li>
</ul>
</li>
-
+
<li>
<a href="../metron-sensors/index.html" title="Sensors">
@@ -220,11 +238,11 @@ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License. --><h1>Current Build</h1>
<p><a name="Current_Build"></a></p>
-<p>The latest build of metron-platform is 0.3.1.</p>
+<p>The latest build of metron-platform is 0.4.0.</p>
<p>We are still in the process of merging/porting additional features from our production code base into this open source release. This release will be followed by a number of additional beta releases until the port is complete. We will also work on getting additional documentation and user/developer guides to the community as soon as we can. At this time we offer no support for the beta software, but will try to respond to requests as promptly as we can.</p>
<p><a name="metron-platform"></a></p>
<h1>metron-platform</h1>
-<p>Extensible set of Storm topologies and topology attributes for streaming, enriching, indexing, and storing telemetry in Hadoop. General information on Metron is available at <a class="externalLink" href="https://metron.incubator.apache.org/">https://metron.incubator.apache.org/</a></p>
+<p>Extensible set of Storm topologies and topology attributes for streaming, enriching, indexing, and storing telemetry in Hadoop. General information on Metron is available at <a class="externalLink" href="https://metron.apache.org/">https://metron.apache.org/</a></p>
<p><a name="Documentation"></a></p>
<h1>Documentation</h1>
<p>Please see documentation within each individual module for description and usage instructions. Sample topologies are provided under Metron_Topologies to get you started with the framework. We pre-assume knowledge of Hadoop, Storm, Kafka, and HBase.</p>
@@ -236,8 +254,9 @@ limitations under the License. --><h1>Current Build</h1>
<footer>
<div class="container-fluid">
- <div class="row span12">Copyright © 2017.
- All Rights Reserved.
+ <div class="row span12">Copyright © 2017
+ <a href="https://www.apache.org">The Apache Software Foundation</a>.
+ All Rights Reserved.
</div>
http://git-wip-us.apache.org/repos/asf/metron/blob/cf4c2ecd/current-book/metron-platform/metron-api/index.html
----------------------------------------------------------------------
diff --git a/current-book/metron-platform/metron-api/index.html b/current-book/metron-platform/metron-api/index.html
index fe2b9a6..607e4aa 100644
--- a/current-book/metron-platform/metron-api/index.html
+++ b/current-book/metron-platform/metron-api/index.html
@@ -1,13 +1,13 @@
<!DOCTYPE html>
<!--
- | Generated by Apache Maven Doxia at 2017-02-23
+ | Generated by Apache Maven Doxia at 2017-06-27
| Rendered using Apache Maven Fluido Skin 1.3.0
-->
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
- <meta name="Date-Revision-yyyymmdd" content="20170223" />
+ <meta name="Date-Revision-yyyymmdd" content="20170627" />
<meta http-equiv="Content-Language" content="en" />
<title>Metron – Metron PCAP Service</title>
<link rel="stylesheet" href="../../css/apache-maven-fluido-1.3.0.min.css" />
@@ -30,14 +30,11 @@
<div class="container-fluid">
<div id="banner">
<div class="pull-left">
- <a href="http://metron.incubator.apache.org/" id="bannerLeft">
- <img src="../../images/metron-logo.png" alt="Apache Metron - Incubating" width="148px" height="48px"/>
+ <a href="http://metron.apache.org/" id="bannerLeft">
+ <img src="../../images/metron-logo.png" alt="Apache Metron" width="148px" height="48px"/>
</a>
</div>
- <div class="pull-right"> <a href="http://incubator.apache.org/" id="bannerRight">
- <img src="../../images/ApacheIncubating_Logo.png" alt="Apache Incubating" width="192px" height="48px"/>
- </a>
- </div>
+ <div class="pull-right"> </div>
<div class="clear"><hr/></div>
</div>
@@ -51,8 +48,8 @@
</li>
<li class="divider ">/</li>
<li class="">
- <a href="http://metron.incubator.apache.org/" class="externalLink" title="Metron-Incubating">
- Metron-Incubating</a>
+ <a href="http://metron.apache.org/" class="externalLink" title="Metron">
+ Metron</a>
</li>
<li class="divider ">/</li>
<li class="">
@@ -64,8 +61,8 @@
- <li id="publishDate" class="pull-right">Last Published: 2017-02-23</li> <li class="divider pull-right">|</li>
- <li id="projectVersion" class="pull-right">Version: 0.3.1</li>
+ <li id="publishDate" class="pull-right">Last Published: 2017-06-27</li> <li class="divider pull-right">|</li>
+ <li id="projectVersion" class="pull-right">Version: 0.4.0</li>
</ul>
</div>
@@ -78,7 +75,7 @@
<ul class="nav nav-list">
<li class="nav-header">User Documentation</li>
-
+
<li>
<a href="../../index.html" title="Metron">
@@ -99,7 +96,7 @@
<i class="icon-chevron-right"></i>
Analytics</a>
</li>
-
+
<li>
<a href="../../metron-deployment/index.html" title="Deployment">
@@ -113,7 +110,21 @@
<i class="none"></i>
Docker</a>
</li>
-
+
+ <li>
+
+ <a href="../../metron-interface/metron-config/index.html" title="Config">
+ <i class="none"></i>
+ Config</a>
+ </li>
+
+ <li>
+
+ <a href="../../metron-interface/metron-rest/index.html" title="Rest">
+ <i class="none"></i>
+ Rest</a>
+ </li>
+
<li>
<a href="../../metron-platform/index.html" title="Platform">
@@ -125,13 +136,13 @@
<a href="#"><i class="none"></i>Api</a>
</li>
-
+
<li>
<a href="../../metron-platform/metron-common/index.html" title="Common">
- <i class="none"></i>
+ <i class="icon-chevron-right"></i>
Common</a>
- </li>
+ </li>
<li>
@@ -174,9 +185,16 @@
<i class="none"></i>
Pcap-backend</a>
</li>
+
+ <li>
+
+ <a href="../../metron-platform/metron-writer/index.html" title="Writer">
+ <i class="none"></i>
+ Writer</a>
+ </li>
</ul>
</li>
-
+
<li>
<a href="../../metron-sensors/index.html" title="Sensors">
@@ -272,8 +290,9 @@
<footer>
<div class="container-fluid">
- <div class="row span12">Copyright © 2017.
- All Rights Reserved.
+ <div class="row span12">Copyright © 2017
+ <a href="https://www.apache.org">The Apache Software Foundation</a>.
+ All Rights Reserved.
</div>
[03/10] metron git commit: METRON-1012 Update Metron public web site
for 0.4.0 release - push to asf-site from the generated code on master
Posted by ma...@apache.org.
http://git-wip-us.apache.org/repos/asf/metron/blob/cf4c2ecd/current-book/metron-platform/metron-enrichment/index.html
----------------------------------------------------------------------
diff --git a/current-book/metron-platform/metron-enrichment/index.html b/current-book/metron-platform/metron-enrichment/index.html
index 33615dd..771f646 100644
--- a/current-book/metron-platform/metron-enrichment/index.html
+++ b/current-book/metron-platform/metron-enrichment/index.html
@@ -1,13 +1,13 @@
<!DOCTYPE html>
<!--
- | Generated by Apache Maven Doxia at 2017-02-23
+ | Generated by Apache Maven Doxia at 2017-06-27
| Rendered using Apache Maven Fluido Skin 1.3.0
-->
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
- <meta name="Date-Revision-yyyymmdd" content="20170223" />
+ <meta name="Date-Revision-yyyymmdd" content="20170627" />
<meta http-equiv="Content-Language" content="en" />
<title>Metron – Enrichment</title>
<link rel="stylesheet" href="../../css/apache-maven-fluido-1.3.0.min.css" />
@@ -30,14 +30,11 @@
<div class="container-fluid">
<div id="banner">
<div class="pull-left">
- <a href="http://metron.incubator.apache.org/" id="bannerLeft">
- <img src="../../images/metron-logo.png" alt="Apache Metron - Incubating" width="148px" height="48px"/>
+ <a href="http://metron.apache.org/" id="bannerLeft">
+ <img src="../../images/metron-logo.png" alt="Apache Metron" width="148px" height="48px"/>
</a>
</div>
- <div class="pull-right"> <a href="http://incubator.apache.org/" id="bannerRight">
- <img src="../../images/ApacheIncubating_Logo.png" alt="Apache Incubating" width="192px" height="48px"/>
- </a>
- </div>
+ <div class="pull-right"> </div>
<div class="clear"><hr/></div>
</div>
@@ -51,8 +48,8 @@
</li>
<li class="divider ">/</li>
<li class="">
- <a href="http://metron.incubator.apache.org/" class="externalLink" title="Metron-Incubating">
- Metron-Incubating</a>
+ <a href="http://metron.apache.org/" class="externalLink" title="Metron">
+ Metron</a>
</li>
<li class="divider ">/</li>
<li class="">
@@ -64,8 +61,8 @@
- <li id="publishDate" class="pull-right">Last Published: 2017-02-23</li> <li class="divider pull-right">|</li>
- <li id="projectVersion" class="pull-right">Version: 0.3.1</li>
+ <li id="publishDate" class="pull-right">Last Published: 2017-06-27</li> <li class="divider pull-right">|</li>
+ <li id="projectVersion" class="pull-right">Version: 0.4.0</li>
</ul>
</div>
@@ -78,7 +75,7 @@
<ul class="nav nav-list">
<li class="nav-header">User Documentation</li>
-
+
<li>
<a href="../../index.html" title="Metron">
@@ -99,7 +96,7 @@
<i class="icon-chevron-right"></i>
Analytics</a>
</li>
-
+
<li>
<a href="../../metron-deployment/index.html" title="Deployment">
@@ -113,7 +110,21 @@
<i class="none"></i>
Docker</a>
</li>
-
+
+ <li>
+
+ <a href="../../metron-interface/metron-config/index.html" title="Config">
+ <i class="none"></i>
+ Config</a>
+ </li>
+
+ <li>
+
+ <a href="../../metron-interface/metron-rest/index.html" title="Rest">
+ <i class="none"></i>
+ Rest</a>
+ </li>
+
<li>
<a href="../../metron-platform/index.html" title="Platform">
@@ -127,13 +138,13 @@
<i class="none"></i>
Api</a>
</li>
-
+
<li>
<a href="../../metron-platform/metron-common/index.html" title="Common">
- <i class="none"></i>
+ <i class="icon-chevron-right"></i>
Common</a>
- </li>
+ </li>
<li>
@@ -174,9 +185,16 @@
<i class="none"></i>
Pcap-backend</a>
</li>
+
+ <li>
+
+ <a href="../../metron-platform/metron-writer/index.html" title="Writer">
+ <i class="none"></i>
+ Writer</a>
+ </li>
</ul>
</li>
-
+
<li>
<a href="../../metron-sensors/index.html" title="Sensors">
@@ -286,7 +304,7 @@
</tbody>
</table>
<p>The <tt>config</tt> map is intended to house enrichment specific configuration. For instance, for the <tt>hbaseEnrichment</tt>, the mappings between the enrichment types to the column families is specified.</p>
-<p>The <tt>fieldMap</tt>contents are of interest because they contain the routing and configuration information for the enrichments. When we say ‘routing’, we mean how the messages get split up and sent to the enrichment adapter bolts. The simplest, by far, is just providing a simple list as in</p>
+<p>The <tt>fieldMap</tt>contents are of interest because they contain the routing and configuration information for the enrichments.<br />When we say ‘routing’, we mean how the messages get split up and sent to the enrichment adapter bolts.<br />The simplest, by far, is just providing a simple list as in</p>
<div class="source">
<div class="source">
@@ -305,7 +323,11 @@
]
}
</pre></div></div>
-<p>Based on this sample config, both ip_src_addr and ip_dst_addr will go to the <tt>geo</tt>, <tt>host</tt>, and <tt>hbaseEnrichment</tt> adapter bolts. For the <tt>geo</tt>, <tt>host</tt> and <tt>hbaseEnrichment</tt>, this is sufficient. However, more complex enrichments may contain their own configuration. Currently, the <tt>stellar</tt> enrichment requires a more complex configuration, such as:</p>
+<p>Based on this sample config, both <tt>ip_src_addr</tt> and <tt>ip_dst_addr</tt> will go to the <tt>geo</tt>, <tt>host</tt>, and <tt>hbaseEnrichment</tt> adapter bolts. </p>
+<div class="section">
+<h4><a name="Stellar_Enrichment_Configuration"></a>Stellar Enrichment Configuration</h4>
+<p>For the <tt>geo</tt>, <tt>host</tt> and <tt>hbaseEnrichment</tt>, this is sufficient. However, more complex enrichments may contain their own configuration. Currently, the <tt>stellar</tt> enrichment is more adaptable and thus requires a more nuanced configuration.</p>
+<p>At its most basic, we want to take a message and apply a couple of enrichments, such as converting the <tt>hostname</tt> field to lowercase. We do this by specifying the transformation inside of the <tt>config</tt> for the <tt>stellar</tt> fieldMap. There are two syntaxes that are supported, specifying the transformations as a map with the key as the field and the value the stellar expression:</p>
<div class="source">
<div class="source">
@@ -313,34 +335,78 @@
...
"stellar" : {
"config" : {
- "numeric" : {
- "foo": "1 + 1"
- }
- ,"ALL_CAPS" : "TO_UPPER(source.type)"
+ "hostname" : "TO_LOWER(hostname)"
}
}
}
</pre></div></div>
-<p>Whereas the simpler enrichments just need a set of fields explicitly stated so they can be separated from the message and sent to the enrichment adapter bolt for enrichment and ultimately joined back in the join bolt, the stellar enrichment has its set of required fields implicitly stated through usage. For instance, if your stellar statement references a field, it should be included and if not, then it should not be included. We did not want to require users to make explicit the implicit.</p>
-<p>The other way in which the stellar enrichment is somewhat more complex is in how the statements are executed. In the general purpose case for a list of fields, those fields are used to create a message to send to the enrichment adapter bolt and that bolt’s worker will handle the fields one by one in serial for a given message. For stellar enrichment, we wanted to have a more complex design so that users could specify the groups of stellar statements sent to the same worker in the same message (and thus executed sequentially). Consider the following configuration:</p>
+<p>Another approach is to make the transformations as a list with the same <tt>var := expr</tt> syntax as is used in the Stellar REPL, such as:</p>
+
+<div class="source">
+<div class="source">
+<pre> "fieldMap": {
+ ...
+ "stellar" : {
+ "config" : [
+ "hostname := TO_LOWER(hostname)"
+ ]
+ }
+ }
+</pre></div></div>
+<p>Sometimes arbitrary stellar enrichments may take enough time that you would prefer to split some of them into groups and execute the groups of stellar enrichments in parallel. Take, for instance, if you wanted to do an HBase enrichment and a profiler call which were independent of one another. This usecase is supported by splitting the enrichments up as groups.</p>
+<p>Consider the following example:</p>
<div class="source">
<div class="source">
<pre> "fieldMap": {
+ ...
"stellar" : {
"config" : {
- "numeric" : {
- "foo": "1 + 1"
- "bar" : TO_LOWER(source.type)"
- }
- ,"text" : {
- "ALL_CAPS" : "TO_UPPER(source.type)"
- }
+ "malicious_domain_enrichment" : {
+ "is_bad_domain" : "ENRICHMENT_EXISTS('malicious_domains', ip_dst_addr, 'enrichments', 'cf')"
+ },
+ "login_profile" : [
+ "profile_window := PROFILE_WINDOW('from 6 months ago')",
+ "global_login_profile := PROFILE_GET('distinct_login_attempts', 'global', profile_window)",
+ "stats := STATS_MERGE(global_login_profile)",
+ "auth_attempts_median := STATS_PERCENTILE(stats, 0.5)",
+ "auth_attempts_sd := STATS_SD(stats)",
+ "profile_window := null",
+ "global_login_profile := null",
+ "stats := null"
+ ]
}
}
}
</pre></div></div>
-<p>We have a group called <tt>numeric</tt> whose stellar statements will be executed sequentially. In parallel to that, we have the group of stellar statements under the group <tt>text</tt> executing. The intent here is to allow you to not force higher latency operations to be done sequentially. You can use any name for your groupings you like. Be aware that the configuration is a map and duplicate configuration keys’ values are not combined, so the duplicate configuration value will be overwritten.</p></div>
+<p>Here we want to perform two enrichments that hit HBase and we would rather not run in sequence. These enrichments are entirely independent of one another (i.e. neither relies on the output of the other). In this case, we’ve created a group called <tt>malicious_domain_enrichment</tt> to inquire about whether the destination address exists in the HBase enrichment table in the <tt>malicious_domains</tt> enrichment type. This is a simple enrichment, so we can express the enrichment group as a map with the new field <tt>is_bad_domain</tt> being a key and the stellar expression associated with that operation being the associated value.</p>
+<p>In contrast, the stellar enrichment group <tt>login_profile</tt> is interacting with the profiler, has multiple temporary expressions (i.e. <tt>profile_window</tt>, <tt>global_login_profile</tt>, and <tt>stats</tt>) that are useful only within the context of this group of stellar expressions. In this case, we would need to ensure that we use the list construct when specifying the group and remember to set the temporary variables to <tt>null</tt> so they are not passed along.</p>
+<p>In general, things to note from this section are as follows:</p>
+
+<ul>
+
+<li>The stellar enrichments for the <tt>stellar</tt> enrichment adapter are specified in the <tt>config</tt> for the <tt>stellar</tt> enrichment adapter in the <tt>fieldMap</tt></li>
+
+<li>Groups of independent (i.e. no expression in any group depend on the output of an expression from an other group) may be executed in parallel</li>
+
+<li>If you have the need to use temporary variables, you may use the list construct. Ensure that you assign the variables to <tt>null</tt> before the end of the group.</li>
+
+<li><b>Ensure that you do not assign a field to a stellar expression which returns an object which JSON cannot represent.</b></li>
+
+<li>Fields assigned to Maps as part of stellar enrichments have the maps unfolded, similar to the HBase Enrichment
+
+<ul>
+
+<li>For example the stellar enrichment for field <tt>foo</tt> which assigns a map such as <tt>foo := { 'grok' : 1, 'bar' : 'baz'}</tt> would yield the following fields:
+
+<ul>
+
+<li><tt>foo.grok</tt> == <tt>1</tt></li>
+
+<li><tt>foo.bar</tt> == <tt>'baz'</tt></li>
+ </ul></li>
+ </ul></li>
+</ul></div></div>
<div class="section">
<h3><a name="The_threatIntel_Configuration"></a>The <tt>threatIntel</tt> Configuration</h3>
@@ -395,7 +461,7 @@
</tr>
</tbody>
</table>
-<p>The <tt>config</tt> map is intended to house threat intel specific configuration. For instance, for the <tt>hbaseThreatIntel</tt> threat intel adapter, the mappings between the enrichment types to the column families is specified.</p>
+<p>The <tt>config</tt> map is intended to house threat intel specific configuration. For instance, for the <tt>hbaseThreatIntel</tt> threat intel adapter, the mappings between the enrichment types to the column families is specified. The <tt>fieldMap</tt> configuration is similar to the <tt>enrichment</tt> configuration in that the adapters available are the same.</p>
<p>The <tt>triageConfig</tt> field is also a complex field and it bears some description:</p>
<table border="0" class="table table-striped">
@@ -442,6 +508,8 @@
<li><tt>rule</tt> : The rule, represented as a Stellar statement</li>
<li><tt>score</tt> : Associated threat triage score for the rule</li>
+
+<li><tt>reason</tt> : Reason the rule tripped. Can be represented as a Stellar statement</li>
</ul>
<p>An example of a rule is as follows:</p>
@@ -451,8 +519,9 @@
{
"name" : "is internal"
, "comment" : "determines if the destination is internal."
- , rule" : "IN_SUBNET(ip_dst_addr, '192.168.0.0/24')"
- , "score" : 10
+ , "rule" : "IN_SUBNET(ip_dst_addr, '192.168.0.0/24')"
+ , "score" : 10
+ , "reason" : "FORMAT('%s is internal', ip_dst_addr)"
}
]
</pre></div></div>
@@ -657,8 +726,9 @@
<footer>
<div class="container-fluid">
- <div class="row span12">Copyright © 2017.
- All Rights Reserved.
+ <div class="row span12">Copyright © 2017
+ <a href="https://www.apache.org">The Apache Software Foundation</a>.
+ All Rights Reserved.
</div>
http://git-wip-us.apache.org/repos/asf/metron/blob/cf4c2ecd/current-book/metron-platform/metron-indexing/index.html
----------------------------------------------------------------------
diff --git a/current-book/metron-platform/metron-indexing/index.html b/current-book/metron-platform/metron-indexing/index.html
index 1f5d0cf..febd70e 100644
--- a/current-book/metron-platform/metron-indexing/index.html
+++ b/current-book/metron-platform/metron-indexing/index.html
@@ -1,13 +1,13 @@
<!DOCTYPE html>
<!--
- | Generated by Apache Maven Doxia at 2017-02-23
+ | Generated by Apache Maven Doxia at 2017-06-27
| Rendered using Apache Maven Fluido Skin 1.3.0
-->
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
- <meta name="Date-Revision-yyyymmdd" content="20170223" />
+ <meta name="Date-Revision-yyyymmdd" content="20170627" />
<meta http-equiv="Content-Language" content="en" />
<title>Metron – Indexing</title>
<link rel="stylesheet" href="../../css/apache-maven-fluido-1.3.0.min.css" />
@@ -30,14 +30,11 @@
<div class="container-fluid">
<div id="banner">
<div class="pull-left">
- <a href="http://metron.incubator.apache.org/" id="bannerLeft">
- <img src="../../images/metron-logo.png" alt="Apache Metron - Incubating" width="148px" height="48px"/>
+ <a href="http://metron.apache.org/" id="bannerLeft">
+ <img src="../../images/metron-logo.png" alt="Apache Metron" width="148px" height="48px"/>
</a>
</div>
- <div class="pull-right"> <a href="http://incubator.apache.org/" id="bannerRight">
- <img src="../../images/ApacheIncubating_Logo.png" alt="Apache Incubating" width="192px" height="48px"/>
- </a>
- </div>
+ <div class="pull-right"> </div>
<div class="clear"><hr/></div>
</div>
@@ -51,8 +48,8 @@
</li>
<li class="divider ">/</li>
<li class="">
- <a href="http://metron.incubator.apache.org/" class="externalLink" title="Metron-Incubating">
- Metron-Incubating</a>
+ <a href="http://metron.apache.org/" class="externalLink" title="Metron">
+ Metron</a>
</li>
<li class="divider ">/</li>
<li class="">
@@ -64,8 +61,8 @@
- <li id="publishDate" class="pull-right">Last Published: 2017-02-23</li> <li class="divider pull-right">|</li>
- <li id="projectVersion" class="pull-right">Version: 0.3.1</li>
+ <li id="publishDate" class="pull-right">Last Published: 2017-06-27</li> <li class="divider pull-right">|</li>
+ <li id="projectVersion" class="pull-right">Version: 0.4.0</li>
</ul>
</div>
@@ -78,7 +75,7 @@
<ul class="nav nav-list">
<li class="nav-header">User Documentation</li>
-
+
<li>
<a href="../../index.html" title="Metron">
@@ -99,7 +96,7 @@
<i class="icon-chevron-right"></i>
Analytics</a>
</li>
-
+
<li>
<a href="../../metron-deployment/index.html" title="Deployment">
@@ -113,7 +110,21 @@
<i class="none"></i>
Docker</a>
</li>
-
+
+ <li>
+
+ <a href="../../metron-interface/metron-config/index.html" title="Config">
+ <i class="none"></i>
+ Config</a>
+ </li>
+
+ <li>
+
+ <a href="../../metron-interface/metron-rest/index.html" title="Rest">
+ <i class="none"></i>
+ Rest</a>
+ </li>
+
<li>
<a href="../../metron-platform/index.html" title="Platform">
@@ -127,13 +138,13 @@
<i class="none"></i>
Api</a>
</li>
-
+
<li>
<a href="../../metron-platform/metron-common/index.html" title="Common">
- <i class="none"></i>
+ <i class="icon-chevron-right"></i>
Common</a>
- </li>
+ </li>
<li>
@@ -174,9 +185,16 @@
<i class="none"></i>
Pcap-backend</a>
</li>
+
+ <li>
+
+ <a href="../../metron-platform/metron-writer/index.html" title="Writer">
+ <i class="none"></i>
+ Writer</a>
+ </li>
</ul>
</li>
-
+
<li>
<a href="../../metron-sensors/index.html" title="Sensors">
@@ -232,7 +250,7 @@
<li>An indexing bolt configured to write to HDFS under <tt>/apps/metron/enrichment/indexed</tt></li>
</ul>
-<p>Errors during indexing are sent to a kafka queue called <tt>index_errors</tt></p></div>
+<p>By default, errors during indexing are sent back into the <tt>indexing</tt> kafka queue so that they can be indexed and archived.</p></div>
<div class="section">
<h2><a name="Sensor_Indexing_Configuration"></a>Sensor Indexing Configuration</h2>
<p>The sensor specific configuration is intended to configure the indexing used for a given sensor type (e.g. <tt>snort</tt>). </p>
@@ -384,7 +402,7 @@
<p>The <tt>indexing</tt> kafka queue is a collection point from the enrichment topology. As such, make sure that the number of partitions in the kafka topic is sufficient to handle the throughput that you expect.</p></div>
<div class="section">
<h2><a name="Indexing_Topology"></a>Indexing Topology</h2>
-<p>The enrichment topology as started by the <tt>$METRON_HOME/bin/start_elasticsearch_topology.sh</tt> or <tt>$METRON_HOME/bin/start_solr_topology.sh</tt> script uses a default of one executor per bolt. In a real production system, this should be customized by modifying the flux file in <tt>$METRON_HOME/flux/indexing/remote.yaml</tt>. </p>
+<p>The <tt>indexing</tt> topology as started by the <tt>$METRON_HOME/bin/start_elasticsearch_topology.sh</tt> or <tt>$METRON_HOME/bin/start_solr_topology.sh</tt> script uses a default of one executor per bolt. In a real production system, this should be customized by modifying the flux file in <tt>$METRON_HOME/flux/indexing/remote.yaml</tt>. </p>
<ul>
@@ -415,8 +433,9 @@
<footer>
<div class="container-fluid">
- <div class="row span12">Copyright © 2017.
- All Rights Reserved.
+ <div class="row span12">Copyright © 2017
+ <a href="https://www.apache.org">The Apache Software Foundation</a>.
+ All Rights Reserved.
</div>
http://git-wip-us.apache.org/repos/asf/metron/blob/cf4c2ecd/current-book/metron-platform/metron-management/index.html
----------------------------------------------------------------------
diff --git a/current-book/metron-platform/metron-management/index.html b/current-book/metron-platform/metron-management/index.html
index c8d1d13..6efed91 100644
--- a/current-book/metron-platform/metron-management/index.html
+++ b/current-book/metron-platform/metron-management/index.html
@@ -1,13 +1,13 @@
<!DOCTYPE html>
<!--
- | Generated by Apache Maven Doxia at 2017-02-23
+ | Generated by Apache Maven Doxia at 2017-06-27
| Rendered using Apache Maven Fluido Skin 1.3.0
-->
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
- <meta name="Date-Revision-yyyymmdd" content="20170223" />
+ <meta name="Date-Revision-yyyymmdd" content="20170627" />
<meta http-equiv="Content-Language" content="en" />
<title>Metron – Stellar REPL Management Utilities</title>
<link rel="stylesheet" href="../../css/apache-maven-fluido-1.3.0.min.css" />
@@ -30,14 +30,11 @@
<div class="container-fluid">
<div id="banner">
<div class="pull-left">
- <a href="http://metron.incubator.apache.org/" id="bannerLeft">
- <img src="../../images/metron-logo.png" alt="Apache Metron - Incubating" width="148px" height="48px"/>
+ <a href="http://metron.apache.org/" id="bannerLeft">
+ <img src="../../images/metron-logo.png" alt="Apache Metron" width="148px" height="48px"/>
</a>
</div>
- <div class="pull-right"> <a href="http://incubator.apache.org/" id="bannerRight">
- <img src="../../images/ApacheIncubating_Logo.png" alt="Apache Incubating" width="192px" height="48px"/>
- </a>
- </div>
+ <div class="pull-right"> </div>
<div class="clear"><hr/></div>
</div>
@@ -51,8 +48,8 @@
</li>
<li class="divider ">/</li>
<li class="">
- <a href="http://metron.incubator.apache.org/" class="externalLink" title="Metron-Incubating">
- Metron-Incubating</a>
+ <a href="http://metron.apache.org/" class="externalLink" title="Metron">
+ Metron</a>
</li>
<li class="divider ">/</li>
<li class="">
@@ -64,8 +61,8 @@
- <li id="publishDate" class="pull-right">Last Published: 2017-02-23</li> <li class="divider pull-right">|</li>
- <li id="projectVersion" class="pull-right">Version: 0.3.1</li>
+ <li id="publishDate" class="pull-right">Last Published: 2017-06-27</li> <li class="divider pull-right">|</li>
+ <li id="projectVersion" class="pull-right">Version: 0.4.0</li>
</ul>
</div>
@@ -78,7 +75,7 @@
<ul class="nav nav-list">
<li class="nav-header">User Documentation</li>
-
+
<li>
<a href="../../index.html" title="Metron">
@@ -99,7 +96,7 @@
<i class="icon-chevron-right"></i>
Analytics</a>
</li>
-
+
<li>
<a href="../../metron-deployment/index.html" title="Deployment">
@@ -113,7 +110,21 @@
<i class="none"></i>
Docker</a>
</li>
-
+
+ <li>
+
+ <a href="../../metron-interface/metron-config/index.html" title="Config">
+ <i class="none"></i>
+ Config</a>
+ </li>
+
+ <li>
+
+ <a href="../../metron-interface/metron-rest/index.html" title="Rest">
+ <i class="none"></i>
+ Rest</a>
+ </li>
+
<li>
<a href="../../metron-platform/index.html" title="Platform">
@@ -127,13 +138,13 @@
<i class="none"></i>
Api</a>
</li>
-
+
<li>
<a href="../../metron-platform/metron-common/index.html" title="Common">
- <i class="none"></i>
+ <i class="icon-chevron-right"></i>
Common</a>
- </li>
+ </li>
<li>
@@ -174,9 +185,16 @@
<i class="none"></i>
Pcap-backend</a>
</li>
+
+ <li>
+
+ <a href="../../metron-platform/metron-writer/index.html" title="Writer">
+ <i class="none"></i>
+ Writer</a>
+ </li>
</ul>
</li>
-
+
<li>
<a href="../../metron-sensors/index.html" title="Sensors">
@@ -674,7 +692,7 @@
<li>writer - The writer to update (e.g. elasticsearch, solr or hdfs)</li>
-<li>size - batch size (integer)</li>
+<li>size - batch size (integer), defaults to 1, meaning batching disabled</li>
</ul></li>
<li>Returns: The String representation of the config in zookeeper</li>
@@ -1589,8 +1607,9 @@ SION('is_both') ] )
<footer>
<div class="container-fluid">
- <div class="row span12">Copyright © 2017.
- All Rights Reserved.
+ <div class="row span12">Copyright © 2017
+ <a href="https://www.apache.org">The Apache Software Foundation</a>.
+ All Rights Reserved.
</div>
http://git-wip-us.apache.org/repos/asf/metron/blob/cf4c2ecd/current-book/metron-platform/metron-parsers/index.html
----------------------------------------------------------------------
diff --git a/current-book/metron-platform/metron-parsers/index.html b/current-book/metron-platform/metron-parsers/index.html
index 10bbd15..f7d13a6 100644
--- a/current-book/metron-platform/metron-parsers/index.html
+++ b/current-book/metron-platform/metron-parsers/index.html
@@ -1,13 +1,13 @@
<!DOCTYPE html>
<!--
- | Generated by Apache Maven Doxia at 2017-02-23
+ | Generated by Apache Maven Doxia at 2017-06-27
| Rendered using Apache Maven Fluido Skin 1.3.0
-->
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
- <meta name="Date-Revision-yyyymmdd" content="20170223" />
+ <meta name="Date-Revision-yyyymmdd" content="20170627" />
<meta http-equiv="Content-Language" content="en" />
<title>Metron – Parsers</title>
<link rel="stylesheet" href="../../css/apache-maven-fluido-1.3.0.min.css" />
@@ -30,14 +30,11 @@
<div class="container-fluid">
<div id="banner">
<div class="pull-left">
- <a href="http://metron.incubator.apache.org/" id="bannerLeft">
- <img src="../../images/metron-logo.png" alt="Apache Metron - Incubating" width="148px" height="48px"/>
+ <a href="http://metron.apache.org/" id="bannerLeft">
+ <img src="../../images/metron-logo.png" alt="Apache Metron" width="148px" height="48px"/>
</a>
</div>
- <div class="pull-right"> <a href="http://incubator.apache.org/" id="bannerRight">
- <img src="../../images/ApacheIncubating_Logo.png" alt="Apache Incubating" width="192px" height="48px"/>
- </a>
- </div>
+ <div class="pull-right"> </div>
<div class="clear"><hr/></div>
</div>
@@ -51,8 +48,8 @@
</li>
<li class="divider ">/</li>
<li class="">
- <a href="http://metron.incubator.apache.org/" class="externalLink" title="Metron-Incubating">
- Metron-Incubating</a>
+ <a href="http://metron.apache.org/" class="externalLink" title="Metron">
+ Metron</a>
</li>
<li class="divider ">/</li>
<li class="">
@@ -64,8 +61,8 @@
- <li id="publishDate" class="pull-right">Last Published: 2017-02-23</li> <li class="divider pull-right">|</li>
- <li id="projectVersion" class="pull-right">Version: 0.3.1</li>
+ <li id="publishDate" class="pull-right">Last Published: 2017-06-27</li> <li class="divider pull-right">|</li>
+ <li id="projectVersion" class="pull-right">Version: 0.4.0</li>
</ul>
</div>
@@ -78,7 +75,7 @@
<ul class="nav nav-list">
<li class="nav-header">User Documentation</li>
-
+
<li>
<a href="../../index.html" title="Metron">
@@ -99,7 +96,7 @@
<i class="icon-chevron-right"></i>
Analytics</a>
</li>
-
+
<li>
<a href="../../metron-deployment/index.html" title="Deployment">
@@ -113,7 +110,21 @@
<i class="none"></i>
Docker</a>
</li>
-
+
+ <li>
+
+ <a href="../../metron-interface/metron-config/index.html" title="Config">
+ <i class="none"></i>
+ Config</a>
+ </li>
+
+ <li>
+
+ <a href="../../metron-interface/metron-rest/index.html" title="Rest">
+ <i class="none"></i>
+ Rest</a>
+ </li>
+
<li>
<a href="../../metron-platform/index.html" title="Platform">
@@ -127,13 +138,13 @@
<i class="none"></i>
Api</a>
</li>
-
+
<li>
<a href="../../metron-platform/metron-common/index.html" title="Common">
- <i class="none"></i>
+ <i class="icon-chevron-right"></i>
Common</a>
- </li>
+ </li>
<li>
@@ -174,9 +185,16 @@
<i class="none"></i>
Pcap-backend</a>
</li>
+
+ <li>
+
+ <a href="../../metron-platform/metron-writer/index.html" title="Writer">
+ <i class="none"></i>
+ Writer</a>
+ </li>
</ul>
</li>
-
+
<li>
<a href="../../metron-sensors/index.html" title="Sensors">
@@ -252,7 +270,7 @@
<div class="section">
<h2><a name="Parser_Architecture"></a>Parser Architecture</h2>
<p><img src="../../images/parser_arch.png" alt="Architecture" /></p>
-<p>Data flows through the parser bolt via kafka and into the <tt>enrichments</tt> topology in kafka. Errors are collected with the context of the error (e.g. stacktrace) and original message causing the error and sent to an <tt>error</tt> queue. Invalid messages as determined by global validation functions are sent to an <tt>invalid</tt> queue. </p></div>
+<p>Data flows through the parser bolt via kafka and into the <tt>enrichments</tt> topology in kafka. Errors are collected with the context of the error (e.g. stacktrace) and original message causing the error and sent to an <tt>error</tt> queue. Invalid messages as determined by global validation functions are also treated as errors and sent to an <tt>error</tt> queue. </p></div>
<div class="section">
<h2><a name="Message_Format"></a>Message Format</h2>
<p>All Metron messages follow a specific format in order to ingest a message. If a message does not conform to this format it will be dropped and put onto an error queue for further examination. The message must be of a JSON format and must have a JSON tag message like so:</p>
@@ -260,7 +278,6 @@
<div class="source">
<div class="source">
<pre>{"message" : message content}
-
</pre></div></div>
<p>Where appropriate there is also a standardization around the 5-tuple JSON fields. This is done so the topology correlation engine further down stream can correlate messages from different topologies by these fields. We are currently working on expanding the message standardization beyond these fields, but this feature is not yet availabe. The standard field names are as follows:</p>
@@ -295,7 +312,6 @@
"original_string": xxx,
"additional-field 1": xxx,
}
-
}
</pre></div></div></div>
<div class="section">
@@ -537,9 +553,6 @@ HH:mm:ss', MAP_GET(dc, dc2tz, 'UTC') )"
-ewp,--error_writer_p <PARALLELISM_HINT> Error Writer Parallelism
Hint
-h,--help This screen
- -iwnt,--invalid_writer_num_tasks <NUM_TASKS> Invalid Writer Num Tasks
- -iwp,--invalid_writer_p <PARALLELISM_HINT> Invalid Message Writer
- Parallelism Hint
-k,--kafka <BROKER_URL> Kafka Broker URL
-mt,--message_timeout <TIMEOUT_IN_SECS> Message Timeout in Seconds
-mtp,--max_task_parallelism <MAX_TASK> Max task parallelism
@@ -560,34 +573,35 @@ HH:mm:ss', MAP_GET(dc, dc2tz, 'UTC') )"
<ul>
-<li>retryDelayMaxMs</li>
+<li><tt>spout.pollTimeoutMs</tt> - Specifies the time, in milliseconds, spent waiting in poll if data is not available. Default is 2s</li>
-<li>retryDelayMultiplier</li>
+<li><tt>spout.firstPollOffsetStrategy</tt> - Sets the offset used by the Kafka spout in the first poll to Kafka broker upon process start. One of
-<li>retryInitialDelayMs</li>
-
-<li>stateUpdateIntervalMs</li>
-
-<li>bufferSizeBytes</li>
-
-<li>fetchMaxWait</li>
+<ul>
+
+<li><tt>EARLIEST</tt></li>
+
+<li><tt>LATEST</tt></li>
+
+<li><tt>UNCOMMITTED_EARLIEST</tt> - Last uncommitted and if offsets aren’t found, defaults to earliest. NOTE: This is the default.</li>
+
+<li><tt>UNCOMMITTED_LATEST</tt> - Last uncommitted and if offsets aren’t found, defaults to latest.</li>
+ </ul></li>
-<li>fetchSizeBytes</li>
+<li><tt>spout.offsetCommitPeriodMs</tt> - Specifies the period, in milliseconds, the offset commit task is periodically called. Default is 15s.</li>
-<li>maxOffsetBehind</li>
+<li><tt>spout.maxUncommittedOffsets</tt> - Defines the max number of polled offsets (records) that can be pending commit, before another poll can take place. Once this limit is reached, no more offsets (records) can be polled until the next successful commit(s) sets the number of pending offsets bellow the threshold. The default is 10,000,000.</li>
-<li>metricsTimeBucketSizeInSecs</li>
+<li><tt>spout.maxRetries</tt> - Defines the max number of retrials in case of tuple failure. The default is to retry forever, which means that no new records are committed until the previous polled records have been acked. This guarantees at once delivery of all the previously polled records. By specifying a finite value for maxRetries, the user decides to sacrifice guarantee of delivery for the previous polled records in favor of processing more records.</li>
-<li>socketTimeoutMs</li>
+<li>Any of the configs in the Consumer API for <a class="externalLink" href="http://kafka.apache.org/0100/documentation.html#newconsumerconfigs">Kafka 0.10.x</a></li>
</ul>
-<p>These are described in some detail <a class="externalLink" href="https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.4/bk_storm-user-guide/content/storm-kafka-api-ref.html">here</a>.</p>
-<p>For instance, creating a JSON file which will set the <tt>bufferSizeBytes</tt> to 2MB and <tt>retryDelayMaxMs</tt> to 2000 would look like</p>
+<p>For instance, creating a JSON file which will set the offsets to <tt>UNCOMMITTED_EARLIEST</tt></p>
<div class="source">
<div class="source">
<pre>{
- "bufferSizeBytes" : 2000000,
- "retryDelayMaxMs" : 2000
+ "spout.firstPollOffsetStrategy" : "UNCOMMITTED_EARLIEST"
}
</pre></div></div>
<p>This would be loaded by passing the file as argument to <tt>--extra_kafka_spout_config</tt></p></div>
@@ -654,15 +668,6 @@ HH:mm:ss', MAP_GET(dc, dc2tz, 'UTC') )"
<li><tt>--error_writer_p</tt> : The parallelism hint for the error writer bolt</li>
</ul></li>
-
-<li>The Invalid Message Writer Bolt
-
-<ul>
-
-<li><tt>--invalid_writer_num_tasks</tt> : The number of tasks for the error writer bolt</li>
-
-<li><tt>--invalid_writer_p</tt> : The parallelism hint for the error writer bolt</li>
- </ul></li>
</ul>
<p>Finally, if workers and executors are new to you, the following might be of use to you:</p>
@@ -678,8 +683,9 @@ HH:mm:ss', MAP_GET(dc, dc2tz, 'UTC') )"
<footer>
<div class="container-fluid">
- <div class="row span12">Copyright © 2017.
- All Rights Reserved.
+ <div class="row span12">Copyright © 2017
+ <a href="https://www.apache.org">The Apache Software Foundation</a>.
+ All Rights Reserved.
</div>
http://git-wip-us.apache.org/repos/asf/metron/blob/cf4c2ecd/current-book/metron-platform/metron-pcap-backend/index.html
----------------------------------------------------------------------
diff --git a/current-book/metron-platform/metron-pcap-backend/index.html b/current-book/metron-platform/metron-pcap-backend/index.html
index a36dc50..af673d5 100644
--- a/current-book/metron-platform/metron-pcap-backend/index.html
+++ b/current-book/metron-platform/metron-pcap-backend/index.html
@@ -1,13 +1,13 @@
<!DOCTYPE html>
<!--
- | Generated by Apache Maven Doxia at 2017-02-23
+ | Generated by Apache Maven Doxia at 2017-06-27
| Rendered using Apache Maven Fluido Skin 1.3.0
-->
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
- <meta name="Date-Revision-yyyymmdd" content="20170223" />
+ <meta name="Date-Revision-yyyymmdd" content="20170627" />
<meta http-equiv="Content-Language" content="en" />
<title>Metron – Metron PCAP Backend</title>
<link rel="stylesheet" href="../../css/apache-maven-fluido-1.3.0.min.css" />
@@ -30,14 +30,11 @@
<div class="container-fluid">
<div id="banner">
<div class="pull-left">
- <a href="http://metron.incubator.apache.org/" id="bannerLeft">
- <img src="../../images/metron-logo.png" alt="Apache Metron - Incubating" width="148px" height="48px"/>
+ <a href="http://metron.apache.org/" id="bannerLeft">
+ <img src="../../images/metron-logo.png" alt="Apache Metron" width="148px" height="48px"/>
</a>
</div>
- <div class="pull-right"> <a href="http://incubator.apache.org/" id="bannerRight">
- <img src="../../images/ApacheIncubating_Logo.png" alt="Apache Incubating" width="192px" height="48px"/>
- </a>
- </div>
+ <div class="pull-right"> </div>
<div class="clear"><hr/></div>
</div>
@@ -51,8 +48,8 @@
</li>
<li class="divider ">/</li>
<li class="">
- <a href="http://metron.incubator.apache.org/" class="externalLink" title="Metron-Incubating">
- Metron-Incubating</a>
+ <a href="http://metron.apache.org/" class="externalLink" title="Metron">
+ Metron</a>
</li>
<li class="divider ">/</li>
<li class="">
@@ -64,8 +61,8 @@
- <li id="publishDate" class="pull-right">Last Published: 2017-02-23</li> <li class="divider pull-right">|</li>
- <li id="projectVersion" class="pull-right">Version: 0.3.1</li>
+ <li id="publishDate" class="pull-right">Last Published: 2017-06-27</li> <li class="divider pull-right">|</li>
+ <li id="projectVersion" class="pull-right">Version: 0.4.0</li>
</ul>
</div>
@@ -78,7 +75,7 @@
<ul class="nav nav-list">
<li class="nav-header">User Documentation</li>
-
+
<li>
<a href="../../index.html" title="Metron">
@@ -99,7 +96,7 @@
<i class="icon-chevron-right"></i>
Analytics</a>
</li>
-
+
<li>
<a href="../../metron-deployment/index.html" title="Deployment">
@@ -113,7 +110,21 @@
<i class="none"></i>
Docker</a>
</li>
-
+
+ <li>
+
+ <a href="../../metron-interface/metron-config/index.html" title="Config">
+ <i class="none"></i>
+ Config</a>
+ </li>
+
+ <li>
+
+ <a href="../../metron-interface/metron-rest/index.html" title="Rest">
+ <i class="none"></i>
+ Rest</a>
+ </li>
+
<li>
<a href="../../metron-platform/index.html" title="Platform">
@@ -127,13 +138,13 @@
<i class="none"></i>
Api</a>
</li>
-
+
<li>
<a href="../../metron-platform/metron-common/index.html" title="Common">
- <i class="none"></i>
+ <i class="icon-chevron-right"></i>
Common</a>
- </li>
+ </li>
<li>
@@ -174,9 +185,16 @@
<a href="#"><i class="none"></i>Pcap-backend</a>
</li>
+
+ <li>
+
+ <a href="../../metron-platform/metron-writer/index.html" title="Writer">
+ <i class="none"></i>
+ Writer</a>
+ </li>
</ul>
</li>
-
+
<li>
<a href="../../metron-sensors/index.html" title="Sensors">
@@ -207,7 +225,31 @@
<h1>Metron PCAP Backend</h1>
<p><a name="Metron_PCAP_Backend"></a></p>
-<p>The purpose of the Metron PCAP backend is to create a storm topology capable of ingesting rapidly raw packet capture data directly into HDFS from Kafka.</p>
+<p>The purpose of the Metron PCAP backend is to create a storm topology capable of rapidly ingesting raw packet capture data directly into HDFS from Kafka.</p>
+
+<ul>
+
+<li><a href="#the-sensors-feeding-kafka">Sensors</a></li>
+
+<li><a href="#the-pcap-topology">PCAP Topology</a></li>
+
+<li><a href="#the-files-on-hdfs">HDFS Files</a></li>
+
+<li><a href="#Configuration">Configuration</a></li>
+
+<li><a href="#Starting_the_Topology">Starting the Topology</a></li>
+
+<li><a href="#Utilities">Utilities</a>
+
+<ul>
+
+<li><a href="#Inspector_Utility">Inspector Utility</a></li>
+
+<li><a href="#Query_Filter_Utility">Query Filter Utility</a></li>
+ </ul></li>
+
+<li><a href="#Performance_Tuning">Performance Tuning</a></li>
+</ul>
<div class="section">
<h2><a name="The_Sensors_Feeding_Kafka"></a>The Sensors Feeding Kafka</h2>
<p>This component must be fed by fast packet capture components upstream via Kafka. The two supported components shipped with Metron are as follows:</p>
@@ -257,15 +299,19 @@
<p>These files contain a set of packet data with headers on them in sequence files.</p></div>
<div class="section">
<h2><a name="Configuration"></a>Configuration</h2>
-<p>The configuration file for the Flux topology is located at <tt>$METRON_HOME/config/etc/env/pcap.properties</tt> and the possible options are as follows:</p>
+<p>The configuration file for the Flux topology is located at <tt>$METRON_HOME/config/pcap.properties</tt> and the possible options are as follows:</p>
<ul>
<li><tt>spout.kafka.topic.pcap</tt> : The kafka topic to listen to</li>
+<li><tt>storm.auto.credentials</tt> : The kerberos ticket renewal. If running on a kerberized cluster, this should be <tt>['org.apache.storm.security.auth.kerberos.AutoTGT']</tt></li>
+
+<li><tt>kafka.security.protocol</tt> : The security protocol to use for kafka. This should be <tt>PLAINTEXT</tt> for a non-kerberized cluster and probably <tt>SASL_PLAINTEXT</tt> for a kerberized cluster.</li>
+
<li><tt>kafka.zk</tt> : The comma separated zookeeper quorum (i.e. host:2181,host2:2181)</li>
-<li><tt>kafka.pcap.start</tt> : One of <tt>START</tt>, <tt>END</tt>, <tt>WHERE_I_LEFT_OFF</tt> representing where to start listening on the queue.</li>
+<li><tt>kafka.pcap.start</tt> : One of <tt>EARLIEST</tt>, <tt>LATEST</tt>, <tt>UNCOMMITTED_EARLIEST</tt>, <tt>UNCOMMITTED_LATEST</tt> representing where to start listening on the queue.</li>
<li><tt>kafka.pcap.numPackets</tt> : The number of packets to keep in one file.</li>
@@ -301,7 +347,7 @@
<li>fixed</li>
-<li>query (Metron Stellar)</li>
+<li>query (via Stellar)</li>
</ul>
<p>The tool is executed via </p>
@@ -324,6 +370,7 @@
and end_time. Default is to use time in
millis since the epoch.
-dp,--ip_dst_port <arg> Destination port
+ -pf,--packet_filter <arg> Packet filter regex
-et,--end_time <arg> Packet end time range. Default is current
system time.
-nr,--num_reducers <arg> The number of reducers to use. Default
@@ -354,7 +401,217 @@
-h,--help Display help
-q,--query <arg> Query string to use as a filter
-st,--start_time <arg> (required) Packet start time range.
-</pre></div></div></div></div></div>
+</pre></div></div>
+<p>The Query filter’s <tt>--query</tt> argument specifies the Stellar expression to execute on each packet. To interact with the packet, a few variables are exposed:</p>
+
+<ul>
+
+<li><tt>packet</tt> : The packet data (a <tt>byte[]</tt>)</li>
+
+<li><tt>ip_src_addr</tt> : The source address for the packet (a <tt>String</tt>)</li>
+
+<li><tt>ip_src_port</tt> : The source port for the packet (an <tt>Integer</tt>)</li>
+
+<li><tt>ip_dst_addr</tt> : The destination address for the packet (a <tt>String</tt>)</li>
+
+<li><tt>ip_dst_port</tt> : The destination port for the packet (an <tt>Integer</tt>)</li>
+</ul></div>
+<div class="section">
+<h4><a name="Binary_Regex"></a>Binary Regex</h4>
+<p>Filtering can be done both by the packet header as well as via a binary regular expression which can be run on the packet payload itself. This filter can be specified via:</p>
+
+<ul>
+
+<li>The <tt>-pf</tt> or <tt>--packet_filter</tt> options for the fixed query filter</li>
+
+<li>The <tt>BYTEARRAY_MATCHER(pattern, data)</tt> Stellar function. The first argument is the regex pattern and the second argument is the data. The packet data will be exposed via the<tt>packet</tt> variable in Stellar.</li>
+</ul>
+<p>The format of this regular expression is described <a class="externalLink" href="https://github.com/nishihatapalmer/byteseek/blob/master/sequencesyntax.md">here</a>.</p></div></div></div>
+<div class="section">
+<h2><a name="Performance_Tuning"></a>Performance Tuning</h2>
+<p>The PCAP topology is extremely lightweight and functions as a Spout-only topology. In order to tune the topology, users currently must specify a combination of properties in pcap.properties as well as configuration in the pcap remote.yaml flux file itself. Tuning the number of partitions in your Kafka topic will have a dramatic impact on performance as well. We ran data into Kafka at 1.1 Gbps and our tests resulted in configuring 128 partitions for our kakfa topic along with the following settings in pcap.properties and remote.yaml (unrelated properties for performance have been removed):</p>
+<div class="section">
+<h3><a name="pcap.properties_file"></a>pcap.properties file</h3>
+
+<div class="source">
+<div class="source">
+<pre>spout.kafka.topic.pcap=pcap
+storm.topology.workers=16
+kafka.spout.parallelism=128
+kafka.pcap.numPackets=1000000000
+kafka.pcap.maxTimeMS=0
+hdfs.replication=1
+hdfs.sync.every=10000
+</pre></div></div>
+<p>You’ll notice that the number of kakfa partitions equals the spout parallelism, and this is no coincidence. The ordering guarantees for a partition in Kafka enforces that you may have no more consumers than 1 per topic. Any additional parallelism will leave you with dormant threads consuming resources but performing no additional work. For our cluster with 4 Storm Supervisors, we found 16 workers to provide optimal throughput as well. We were largely IO bound rather than CPU bound with the incoming PCAP data.</p></div>
+<div class="section">
+<h3><a name="remote.yaml"></a>remote.yaml</h3>
+<p>In the flux file, we introduced the following configuration:</p>
+
+<div class="source">
+<div class="source">
+<pre>name: "pcap"
+config:
+ topology.workers: ${storm.topology.workers}
+ topology.worker.childopts: ${topology.worker.childopts}
+ topology.auto-credentials: ${storm.auto.credentials}
+ topology.ackers.executors: 0
+components:
+
+ # Any kafka props for the producer go here.
+ - id: "kafkaProps"
+ className: "java.util.HashMap"
+ configMethods:
+ - name: "put"
+ args:
+ - "value.deserializer"
+ - "org.apache.kafka.common.serialization.ByteArrayDeserializer"
+ - name: "put"
+ args:
+ - "key.deserializer"
+ - "org.apache.kafka.common.serialization.ByteArrayDeserializer"
+ - name: "put"
+ args:
+ - "group.id"
+ - "pcap"
+ - name: "put"
+ args:
+ - "security.protocol"
+ - "${kafka.security.protocol}"
+ - name: "put"
+ args:
+ - "poll.timeout.ms"
+ - 100
+ - name: "put"
+ args:
+ - "offset.commit.period.ms"
+ - 30000
+ - name: "put"
+ args:
+ - "session.timeout.ms"
+ - 30000
+ - name: "put"
+ args:
+ - "max.uncommitted.offsets"
+ - 200000000
+ - name: "put"
+ args:
+ - "max.poll.interval.ms"
+ - 10
+ - name: "put"
+ args:
+ - "max.poll.records"
+ - 200000
+ - name: "put"
+ args:
+ - "receive.buffer.bytes"
+ - 431072
+ - name: "put"
+ args:
+ - "max.partition.fetch.bytes"
+ - 8097152
+
+ - id: "hdfsProps"
+ className: "java.util.HashMap"
+ configMethods:
+ - name: "put"
+ args:
+ - "io.file.buffer.size"
+ - 1000000
+ - name: "put"
+ args:
+ - "dfs.blocksize"
+ - 1073741824
+
+ - id: "kafkaConfig"
+ className: "org.apache.metron.storm.kafka.flux.SimpleStormKafkaBuilder"
+ constructorArgs:
+ - ref: "kafkaProps"
+ # topic name
+ - "${spout.kafka.topic.pcap}"
+ - "${kafka.zk}"
+ configMethods:
+ - name: "setFirstPollOffsetStrategy"
+ args:
+ # One of EARLIEST, LATEST, UNCOMMITTED_EARLIEST, UNCOMMITTED_LATEST
+ - ${kafka.pcap.start}
+
+ - id: "writerConfig"
+ className: "org.apache.metron.spout.pcap.HDFSWriterConfig"
+ configMethods:
+ - name: "withOutputPath"
+ args:
+ - "${kafka.pcap.out}"
+ - name: "withNumPackets"
+ args:
+ - ${kafka.pcap.numPackets}
+ - name: "withMaxTimeMS"
+ args:
+ - ${kafka.pcap.maxTimeMS}
+ - name: "withZookeeperQuorum"
+ args:
+ - "${kafka.zk}"
+ - name: "withSyncEvery"
+ args:
+ - ${hdfs.sync.every}
+ - name: "withReplicationFactor"
+ args:
+ - ${hdfs.replication}
+ - name: "withHDFSConfig"
+ args:
+ - ref: "hdfsProps"
+ - name: "withDeserializer"
+ args:
+ - "${kafka.pcap.ts_scheme}"
+ - "${kafka.pcap.ts_granularity}"
+spouts:
+ - id: "kafkaSpout"
+ className: "org.apache.metron.spout.pcap.KafkaToHDFSSpout"
+ parallelism: ${kafka.spout.parallelism}
+ constructorArgs:
+ - ref: "kafkaConfig"
+ - ref: "writerConfig"
+
+</pre></div></div>
+<div class="section">
+<h4><a name="Flux_Changes_Introduced"></a>Flux Changes Introduced</h4>
+<div class="section">
+<h5><a name="Topology_Configuration"></a>Topology Configuration</h5>
+<p>The only change here is <tt>topology.ackers.executors: 0</tt>, which disables Storm tuple acking for maximum throughput.</p></div>
+<div class="section">
+<h5><a name="Kafka_configuration"></a>Kafka configuration</h5>
+
+<div class="source">
+<div class="source">
+<pre>poll.timeout.ms
+offset.commit.period.ms
+session.timeout.ms
+max.uncommitted.offsets
+max.poll.interval.ms
+max.poll.records
+receive.buffer.bytes
+max.partition.fetch.bytes
+</pre></div></div></div>
+<div class="section">
+<h5><a name="Writer_Configuration"></a>Writer Configuration</h5>
+<p>This is a combination of settings for the HDFSWriter (see pcap.properties values above) as well as HDFS.</p>
+<p><b>HDFS config</b></p>
+<p>Component config HashMap with the following properties:</p>
+
+<div class="source">
+<div class="source">
+<pre>io.file.buffer.size
+dfs.blocksize
+</pre></div></div>
+<p><b>Writer config</b></p>
+<p>References the HDFS props component specified above.</p>
+
+<div class="source">
+<div class="source">
+<pre> - name: "withHDFSConfig"
+ args:
+ - ref: "hdfsProps"
+</pre></div></div></div></div></div></div>
</div>
</div>
</div>
@@ -363,8 +620,9 @@
<footer>
<div class="container-fluid">
- <div class="row span12">Copyright © 2017.
- All Rights Reserved.
+ <div class="row span12">Copyright © 2017
+ <a href="https://www.apache.org">The Apache Software Foundation</a>.
+ All Rights Reserved.
</div>
[07/10] metron git commit: METRON-1012 Update Metron public web site
for 0.4.0 release - push to asf-site from the generated code on master
Posted by ma...@apache.org.
http://git-wip-us.apache.org/repos/asf/metron/blob/cf4c2ecd/current-book/metron-deployment/packaging/docker/ansible-docker/index.html
----------------------------------------------------------------------
diff --git a/current-book/metron-deployment/packaging/docker/ansible-docker/index.html b/current-book/metron-deployment/packaging/docker/ansible-docker/index.html
index e01ec59..f1e4572 100644
--- a/current-book/metron-deployment/packaging/docker/ansible-docker/index.html
+++ b/current-book/metron-deployment/packaging/docker/ansible-docker/index.html
@@ -1,13 +1,13 @@
<!DOCTYPE html>
<!--
- | Generated by Apache Maven Doxia at 2017-02-23
+ | Generated by Apache Maven Doxia at 2017-06-27
| Rendered using Apache Maven Fluido Skin 1.3.0
-->
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
- <meta name="Date-Revision-yyyymmdd" content="20170223" />
+ <meta name="Date-Revision-yyyymmdd" content="20170627" />
<meta http-equiv="Content-Language" content="en" />
<title>Metron – Overview</title>
<link rel="stylesheet" href="../../../../css/apache-maven-fluido-1.3.0.min.css" />
@@ -30,14 +30,11 @@
<div class="container-fluid">
<div id="banner">
<div class="pull-left">
- <a href="http://metron.incubator.apache.org/" id="bannerLeft">
- <img src="../../../../images/metron-logo.png" alt="Apache Metron - Incubating" width="148px" height="48px"/>
+ <a href="http://metron.apache.org/" id="bannerLeft">
+ <img src="../../../../images/metron-logo.png" alt="Apache Metron" width="148px" height="48px"/>
</a>
</div>
- <div class="pull-right"> <a href="http://incubator.apache.org/" id="bannerRight">
- <img src="../../../../images/ApacheIncubating_Logo.png" alt="Apache Incubating" width="192px" height="48px"/>
- </a>
- </div>
+ <div class="pull-right"> </div>
<div class="clear"><hr/></div>
</div>
@@ -51,8 +48,8 @@
</li>
<li class="divider ">/</li>
<li class="">
- <a href="http://metron.incubator.apache.org/" class="externalLink" title="Metron-Incubating">
- Metron-Incubating</a>
+ <a href="http://metron.apache.org/" class="externalLink" title="Metron">
+ Metron</a>
</li>
<li class="divider ">/</li>
<li class="">
@@ -64,8 +61,8 @@
- <li id="publishDate" class="pull-right">Last Published: 2017-02-23</li> <li class="divider pull-right">|</li>
- <li id="projectVersion" class="pull-right">Version: 0.3.1</li>
+ <li id="publishDate" class="pull-right">Last Published: 2017-06-27</li> <li class="divider pull-right">|</li>
+ <li id="projectVersion" class="pull-right">Version: 0.4.0</li>
</ul>
</div>
@@ -78,7 +75,7 @@
<ul class="nav nav-list">
<li class="nav-header">User Documentation</li>
-
+
<li>
<a href="../../../../index.html" title="Metron">
@@ -99,7 +96,7 @@
<i class="icon-chevron-right"></i>
Analytics</a>
</li>
-
+
<li>
<a href="../../../../metron-deployment/index.html" title="Deployment">
@@ -109,6 +106,20 @@
<li>
+ <a href="../../../../metron-deployment/Kerberos-ambari-setup.html" title="Kerberos-ambari-setup">
+ <i class="none"></i>
+ Kerberos-ambari-setup</a>
+ </li>
+
+ <li>
+
+ <a href="../../../../metron-deployment/Kerberos-manual-setup.html" title="Kerberos-manual-setup">
+ <i class="none"></i>
+ Kerberos-manual-setup</a>
+ </li>
+
+ <li>
+
<a href="../../../../metron-deployment/amazon-ec2/index.html" title="Amazon-ec2">
<i class="none"></i>
Amazon-ec2</a>
@@ -128,11 +139,11 @@
<li>
- <a href="../../../../metron-deployment/packer-build/index.html" title="Packer-build">
+ <a href="../../../../metron-deployment/packaging/packer-build/index.html" title="Packer-build">
<i class="none"></i>
Packer-build</a>
</li>
-
+
<li>
<a href="../../../../metron-deployment/roles/index.html" title="Roles">
@@ -155,14 +166,28 @@
<i class="none"></i>
Docker</a>
</li>
-
+
+ <li>
+
+ <a href="../../../../metron-interface/metron-config/index.html" title="Config">
+ <i class="none"></i>
+ Config</a>
+ </li>
+
+ <li>
+
+ <a href="../../../../metron-interface/metron-rest/index.html" title="Rest">
+ <i class="none"></i>
+ Rest</a>
+ </li>
+
<li>
<a href="../../../../metron-platform/index.html" title="Platform">
<i class="icon-chevron-right"></i>
Platform</a>
</li>
-
+
<li>
<a href="../../../../metron-sensors/index.html" title="Sensors">
@@ -199,7 +224,7 @@
<ol style="list-style-type: decimal">
-<li>Install Docker [https://www.docker.com/products/overview]</li>
+<li>Install Docker ( <a class="externalLink" href="https://www.docker.com/products/overview">https://www.docker.com/products/overview</a> )</li>
<li>Navigate to <project-directory>/metron-deployment/packaging/docker/ansible-docker</li>
@@ -207,14 +232,14 @@
</ol></div>
<div class="section">
<h2><a name="Using_the_Container"></a>Using the Container</h2>
-<p>Full instructions are found on the wiki [https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=65144361].</p>
-<p>tl;dr</p>
+<p>Full instructions are found on the wiki at <a class="externalLink" href="https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=65144361">https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=65144361</a></p>
+<p>tl;dr:</p>
<ol style="list-style-type: decimal">
-<li>docker run -it -v <project-directory>:/root/incubator-metron ansible-docker:2.0.0.2 bash</li>
+<li>docker run -it -v <project-directory>:/root/metron ansible-docker:2.0.0.2 bash</li>
-<li>cd /root/incubator-metron</li>
+<li>cd /root/metron</li>
<li>mvn clean package -DskipTests</li>
</ol></div>
@@ -226,8 +251,9 @@
<footer>
<div class="container-fluid">
- <div class="row span12">Copyright © 2017.
- All Rights Reserved.
+ <div class="row span12">Copyright © 2017
+ <a href="https://www.apache.org">The Apache Software Foundation</a>.
+ All Rights Reserved.
</div>
http://git-wip-us.apache.org/repos/asf/metron/blob/cf4c2ecd/current-book/metron-deployment/packaging/docker/rpm-docker/index.html
----------------------------------------------------------------------
diff --git a/current-book/metron-deployment/packaging/docker/rpm-docker/index.html b/current-book/metron-deployment/packaging/docker/rpm-docker/index.html
index ae6948e..7efd459 100644
--- a/current-book/metron-deployment/packaging/docker/rpm-docker/index.html
+++ b/current-book/metron-deployment/packaging/docker/rpm-docker/index.html
@@ -1,13 +1,13 @@
<!DOCTYPE html>
<!--
- | Generated by Apache Maven Doxia at 2017-02-23
+ | Generated by Apache Maven Doxia at 2017-06-27
| Rendered using Apache Maven Fluido Skin 1.3.0
-->
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
- <meta name="Date-Revision-yyyymmdd" content="20170223" />
+ <meta name="Date-Revision-yyyymmdd" content="20170627" />
<meta http-equiv="Content-Language" content="en" />
<title>Metron – Overview</title>
<link rel="stylesheet" href="../../../../css/apache-maven-fluido-1.3.0.min.css" />
@@ -30,14 +30,11 @@
<div class="container-fluid">
<div id="banner">
<div class="pull-left">
- <a href="http://metron.incubator.apache.org/" id="bannerLeft">
- <img src="../../../../images/metron-logo.png" alt="Apache Metron - Incubating" width="148px" height="48px"/>
+ <a href="http://metron.apache.org/" id="bannerLeft">
+ <img src="../../../../images/metron-logo.png" alt="Apache Metron" width="148px" height="48px"/>
</a>
</div>
- <div class="pull-right"> <a href="http://incubator.apache.org/" id="bannerRight">
- <img src="../../../../images/ApacheIncubating_Logo.png" alt="Apache Incubating" width="192px" height="48px"/>
- </a>
- </div>
+ <div class="pull-right"> </div>
<div class="clear"><hr/></div>
</div>
@@ -51,8 +48,8 @@
</li>
<li class="divider ">/</li>
<li class="">
- <a href="http://metron.incubator.apache.org/" class="externalLink" title="Metron-Incubating">
- Metron-Incubating</a>
+ <a href="http://metron.apache.org/" class="externalLink" title="Metron">
+ Metron</a>
</li>
<li class="divider ">/</li>
<li class="">
@@ -64,8 +61,8 @@
- <li id="publishDate" class="pull-right">Last Published: 2017-02-23</li> <li class="divider pull-right">|</li>
- <li id="projectVersion" class="pull-right">Version: 0.3.1</li>
+ <li id="publishDate" class="pull-right">Last Published: 2017-06-27</li> <li class="divider pull-right">|</li>
+ <li id="projectVersion" class="pull-right">Version: 0.4.0</li>
</ul>
</div>
@@ -78,7 +75,7 @@
<ul class="nav nav-list">
<li class="nav-header">User Documentation</li>
-
+
<li>
<a href="../../../../index.html" title="Metron">
@@ -99,7 +96,7 @@
<i class="icon-chevron-right"></i>
Analytics</a>
</li>
-
+
<li>
<a href="../../../../metron-deployment/index.html" title="Deployment">
@@ -109,6 +106,20 @@
<li>
+ <a href="../../../../metron-deployment/Kerberos-ambari-setup.html" title="Kerberos-ambari-setup">
+ <i class="none"></i>
+ Kerberos-ambari-setup</a>
+ </li>
+
+ <li>
+
+ <a href="../../../../metron-deployment/Kerberos-manual-setup.html" title="Kerberos-manual-setup">
+ <i class="none"></i>
+ Kerberos-manual-setup</a>
+ </li>
+
+ <li>
+
<a href="../../../../metron-deployment/amazon-ec2/index.html" title="Amazon-ec2">
<i class="none"></i>
Amazon-ec2</a>
@@ -128,11 +139,11 @@
<li>
- <a href="../../../../metron-deployment/packer-build/index.html" title="Packer-build">
+ <a href="../../../../metron-deployment/packaging/packer-build/index.html" title="Packer-build">
<i class="none"></i>
Packer-build</a>
</li>
-
+
<li>
<a href="../../../../metron-deployment/roles/index.html" title="Roles">
@@ -155,14 +166,28 @@
<i class="none"></i>
Docker</a>
</li>
-
+
+ <li>
+
+ <a href="../../../../metron-interface/metron-config/index.html" title="Config">
+ <i class="none"></i>
+ Config</a>
+ </li>
+
+ <li>
+
+ <a href="../../../../metron-interface/metron-rest/index.html" title="Rest">
+ <i class="none"></i>
+ Rest</a>
+ </li>
+
<li>
<a href="../../../../metron-platform/index.html" title="Platform">
<i class="icon-chevron-right"></i>
Platform</a>
</li>
-
+
<li>
<a href="../../../../metron-sensors/index.html" title="Sensors">
@@ -199,7 +224,7 @@
<ol style="list-style-type: decimal">
-<li>Install Docker [https://www.docker.com/products/overview]</li>
+<li>Install Docker ( <a class="externalLink" href="https://www.docker.com/products/overview">https://www.docker.com/products/overview</a> )</li>
<li>Navigate to <project-directory>/metron-deployment/packaging/rpm-docker</li>
@@ -213,8 +238,9 @@
<footer>
<div class="container-fluid">
- <div class="row span12">Copyright © 2017.
- All Rights Reserved.
+ <div class="row span12">Copyright © 2017
+ <a href="https://www.apache.org">The Apache Software Foundation</a>.
+ All Rights Reserved.
</div>
http://git-wip-us.apache.org/repos/asf/metron/blob/cf4c2ecd/current-book/metron-deployment/packaging/packer-build/index.html
----------------------------------------------------------------------
diff --git a/current-book/metron-deployment/packaging/packer-build/index.html b/current-book/metron-deployment/packaging/packer-build/index.html
new file mode 100644
index 0000000..e5a9e53
--- /dev/null
+++ b/current-book/metron-deployment/packaging/packer-build/index.html
@@ -0,0 +1,298 @@
+<!DOCTYPE html>
+<!--
+ | Generated by Apache Maven Doxia at 2017-06-27
+ | Rendered using Apache Maven Fluido Skin 1.3.0
+-->
+<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
+ <head>
+ <meta charset="UTF-8" />
+ <meta name="viewport" content="width=device-width, initial-scale=1.0" />
+ <meta name="Date-Revision-yyyymmdd" content="20170627" />
+ <meta http-equiv="Content-Language" content="en" />
+ <title>Metron – Build Metron Images</title>
+ <link rel="stylesheet" href="../../../css/apache-maven-fluido-1.3.0.min.css" />
+ <link rel="stylesheet" href="../../../css/site.css" />
+ <link rel="stylesheet" href="../../../css/print.css" media="print" />
+
+
+ <script type="text/javascript" src="../../../js/apache-maven-fluido-1.3.0.min.js"></script>
+
+
+
+<script type="text/javascript">$( document ).ready( function() { $( '.carousel' ).carousel( { interval: 3500 } ) } );</script>
+
+ </head>
+ <body class="topBarDisabled">
+
+
+
+
+ <div class="container-fluid">
+ <div id="banner">
+ <div class="pull-left">
+ <a href="http://metron.apache.org/" id="bannerLeft">
+ <img src="../../../images/metron-logo.png" alt="Apache Metron" width="148px" height="48px"/>
+ </a>
+ </div>
+ <div class="pull-right"> </div>
+ <div class="clear"><hr/></div>
+ </div>
+
+ <div id="breadcrumbs">
+ <ul class="breadcrumb">
+
+
+ <li class="">
+ <a href="http://www.apache.org" class="externalLink" title="Apache">
+ Apache</a>
+ </li>
+ <li class="divider ">/</li>
+ <li class="">
+ <a href="http://metron.apache.org/" class="externalLink" title="Metron">
+ Metron</a>
+ </li>
+ <li class="divider ">/</li>
+ <li class="">
+ <a href="../../../index.html" title="Documentation">
+ Documentation</a>
+ </li>
+ <li class="divider ">/</li>
+ <li class="">Build Metron Images</li>
+
+
+
+ <li id="publishDate" class="pull-right">Last Published: 2017-06-27</li> <li class="divider pull-right">|</li>
+ <li id="projectVersion" class="pull-right">Version: 0.4.0</li>
+
+ </ul>
+ </div>
+
+
+ <div class="row-fluid">
+ <div id="leftColumn" class="span3">
+ <div class="well sidebar-nav">
+
+
+ <ul class="nav nav-list">
+ <li class="nav-header">User Documentation</li>
+
+ <li>
+
+ <a href="../../../index.html" title="Metron">
+ <i class="icon-chevron-down"></i>
+ Metron</a>
+ <ul class="nav nav-list">
+
+ <li>
+
+ <a href="../../../Upgrading.html" title="Upgrading">
+ <i class="none"></i>
+ Upgrading</a>
+ </li>
+
+ <li>
+
+ <a href="../../../metron-analytics/index.html" title="Analytics">
+ <i class="icon-chevron-right"></i>
+ Analytics</a>
+ </li>
+
+ <li>
+
+ <a href="../../../metron-deployment/index.html" title="Deployment">
+ <i class="icon-chevron-down"></i>
+ Deployment</a>
+ <ul class="nav nav-list">
+
+ <li>
+
+ <a href="../../../metron-deployment/Kerberos-ambari-setup.html" title="Kerberos-ambari-setup">
+ <i class="none"></i>
+ Kerberos-ambari-setup</a>
+ </li>
+
+ <li>
+
+ <a href="../../../metron-deployment/Kerberos-manual-setup.html" title="Kerberos-manual-setup">
+ <i class="none"></i>
+ Kerberos-manual-setup</a>
+ </li>
+
+ <li>
+
+ <a href="../../../metron-deployment/amazon-ec2/index.html" title="Amazon-ec2">
+ <i class="none"></i>
+ Amazon-ec2</a>
+ </li>
+
+ <li>
+
+ <a href="../../../metron-deployment/packaging/docker/ansible-docker/index.html" title="Ansible-docker">
+ <i class="none"></i>
+ Ansible-docker</a>
+ </li>
+
+ <li>
+
+ <a href="../../../metron-deployment/packaging/docker/rpm-docker/index.html" title="Rpm-docker">
+ <i class="none"></i>
+ Rpm-docker</a>
+ </li>
+
+ <li class="active">
+
+ <a href="#"><i class="none"></i>Packer-build</a>
+ </li>
+
+ <li>
+
+ <a href="../../../metron-deployment/roles/index.html" title="Roles">
+ <i class="icon-chevron-right"></i>
+ Roles</a>
+ </li>
+
+ <li>
+
+ <a href="../../../metron-deployment/vagrant/index.html" title="Vagrant">
+ <i class="icon-chevron-right"></i>
+ Vagrant</a>
+ </li>
+ </ul>
+ </li>
+
+ <li>
+
+ <a href="../../../metron-docker/index.html" title="Docker">
+ <i class="none"></i>
+ Docker</a>
+ </li>
+
+ <li>
+
+ <a href="../../../metron-interface/metron-config/index.html" title="Config">
+ <i class="none"></i>
+ Config</a>
+ </li>
+
+ <li>
+
+ <a href="../../../metron-interface/metron-rest/index.html" title="Rest">
+ <i class="none"></i>
+ Rest</a>
+ </li>
+
+ <li>
+
+ <a href="../../../metron-platform/index.html" title="Platform">
+ <i class="icon-chevron-right"></i>
+ Platform</a>
+ </li>
+
+ <li>
+
+ <a href="../../../metron-sensors/index.html" title="Sensors">
+ <i class="icon-chevron-right"></i>
+ Sensors</a>
+ </li>
+ </ul>
+ </li>
+ </ul>
+
+
+
+ <hr class="divider" />
+
+ <div id="poweredBy">
+ <div class="clear"></div>
+ <div class="clear"></div>
+ <div class="clear"></div>
+ <a href="http://maven.apache.org/" title="Built by Maven" class="poweredBy">
+ <img class="builtBy" alt="Built by Maven" src="../../../images/logos/maven-feather.png" />
+ </a>
+ </div>
+ </div>
+ </div>
+
+
+ <div id="bodyColumn" class="span9" >
+
+ <h1>Build Metron Images</h1>
+<p>Based on the fantastic <a class="externalLink" href="https://github.com/chef/bento">Bento</a> project developed by Chef.</p>
+<div class="section">
+<h2><a name="Images_Provided"></a>Images Provided</h2>
+
+<ul>
+
+<li>base-centos-6.7: Centos 6.7 + HDP. Used in the full-dev-platform Vagrant image</li>
+
+<li>quick-dev-centos-6.7: Centos 6.7 + HDP + Metron. Used for the quick-dev-platform Vagrant image.</li>
+</ul></div>
+<div class="section">
+<h2><a name="Prerequisites"></a>Prerequisites</h2>
+
+<ul>
+
+<li><a class="externalLink" href="https://www.packer.io/">Packer</a> 0.12.2</li>
+
+<li><a class="externalLink" href="https://www.virtualbox.org/">Virtualbox</a> 5.0.16+ (Tested with 5.0.20)</li>
+</ul>
+<p>Build Both Images ———————- Navigate to <your-project-directory>/metron-deployment/packer-build Execute bin/bento build</p>
+<p>Packer will build both images and export .box files to the ./builds directory.</p>
+<p>Build Single Images ———————- Navigate to <i>your-project-directory</i>/metron-deployment/packer-build</p>
+
+<ul>
+
+<li>Base Centos (full-dev)</li>
+</ul>
+
+<div class="source">
+<div class="source">
+<pre>bin/bento build base-centos-6.7.json
+</pre></div></div>
+
+<ul>
+
+<li>Quick Dev</li>
+</ul>
+
+<div class="source">
+<div class="source">
+<pre>bin/bento build quick-dev-centos-6.7.json
+</pre></div></div>
+<p>Using Your New Box File ———————- Modify the relevant Vagrantfile (full-dev-platform or quick-dev-platform) replacing the lines:</p>
+
+<div class="source">
+<div class="source">
+<pre><pre><code>config.vm.box = "<i>box_name</i>"
+config.ssh.insert_key = true</code></pre>
+</pre></div></div>
+<p>with</p>
+
+<div class="source">
+<div class="source">
+<pre><pre></code>config.vm.box = "<i>test_box_name</i>"
+config.vm.box = "<i>PathToBoxfile/Boxfilename</i>"
+config.ssh.insert_key = true</code></pre>
+</pre></div></div>
+<p>Launch the image as usual.</p>
+<p>Node: Vagrant will cache boxes, you can force Vagrant to reload your box by running <tt>vagrant box remove <i>test_box_name</i></tt> before launching your new image.</p></div>
+ </div>
+ </div>
+ </div>
+
+ <hr/>
+
+ <footer>
+ <div class="container-fluid">
+ <div class="row span12">Copyright © 2017
+ <a href="https://www.apache.org">The Apache Software Foundation</a>.
+ All Rights Reserved.
+
+ </div>
+
+
+
+ </div>
+ </footer>
+ </body>
+</html>
http://git-wip-us.apache.org/repos/asf/metron/blob/cf4c2ecd/current-book/metron-deployment/roles/index.html
----------------------------------------------------------------------
diff --git a/current-book/metron-deployment/roles/index.html b/current-book/metron-deployment/roles/index.html
index cdcf082..107f1ad 100644
--- a/current-book/metron-deployment/roles/index.html
+++ b/current-book/metron-deployment/roles/index.html
@@ -1,13 +1,13 @@
<!DOCTYPE html>
<!--
- | Generated by Apache Maven Doxia at 2017-02-23
+ | Generated by Apache Maven Doxia at 2017-06-27
| Rendered using Apache Maven Fluido Skin 1.3.0
-->
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
- <meta name="Date-Revision-yyyymmdd" content="20170223" />
+ <meta name="Date-Revision-yyyymmdd" content="20170627" />
<meta http-equiv="Content-Language" content="en" />
<title>Metron – Ansible Roles</title>
<link rel="stylesheet" href="../../css/apache-maven-fluido-1.3.0.min.css" />
@@ -30,14 +30,11 @@
<div class="container-fluid">
<div id="banner">
<div class="pull-left">
- <a href="http://metron.incubator.apache.org/" id="bannerLeft">
- <img src="../../images/metron-logo.png" alt="Apache Metron - Incubating" width="148px" height="48px"/>
+ <a href="http://metron.apache.org/" id="bannerLeft">
+ <img src="../../images/metron-logo.png" alt="Apache Metron" width="148px" height="48px"/>
</a>
</div>
- <div class="pull-right"> <a href="http://incubator.apache.org/" id="bannerRight">
- <img src="../../images/ApacheIncubating_Logo.png" alt="Apache Incubating" width="192px" height="48px"/>
- </a>
- </div>
+ <div class="pull-right"> </div>
<div class="clear"><hr/></div>
</div>
@@ -51,8 +48,8 @@
</li>
<li class="divider ">/</li>
<li class="">
- <a href="http://metron.incubator.apache.org/" class="externalLink" title="Metron-Incubating">
- Metron-Incubating</a>
+ <a href="http://metron.apache.org/" class="externalLink" title="Metron">
+ Metron</a>
</li>
<li class="divider ">/</li>
<li class="">
@@ -64,8 +61,8 @@
- <li id="publishDate" class="pull-right">Last Published: 2017-02-23</li> <li class="divider pull-right">|</li>
- <li id="projectVersion" class="pull-right">Version: 0.3.1</li>
+ <li id="publishDate" class="pull-right">Last Published: 2017-06-27</li> <li class="divider pull-right">|</li>
+ <li id="projectVersion" class="pull-right">Version: 0.4.0</li>
</ul>
</div>
@@ -78,7 +75,7 @@
<ul class="nav nav-list">
<li class="nav-header">User Documentation</li>
-
+
<li>
<a href="../../index.html" title="Metron">
@@ -99,7 +96,7 @@
<i class="icon-chevron-right"></i>
Analytics</a>
</li>
-
+
<li>
<a href="../../metron-deployment/index.html" title="Deployment">
@@ -109,6 +106,20 @@
<li>
+ <a href="../../metron-deployment/Kerberos-ambari-setup.html" title="Kerberos-ambari-setup">
+ <i class="none"></i>
+ Kerberos-ambari-setup</a>
+ </li>
+
+ <li>
+
+ <a href="../../metron-deployment/Kerberos-manual-setup.html" title="Kerberos-manual-setup">
+ <i class="none"></i>
+ Kerberos-manual-setup</a>
+ </li>
+
+ <li>
+
<a href="../../metron-deployment/amazon-ec2/index.html" title="Amazon-ec2">
<i class="none"></i>
Amazon-ec2</a>
@@ -130,11 +141,11 @@
<li>
- <a href="../../metron-deployment/packer-build/index.html" title="Packer-build">
+ <a href="../../metron-deployment/packaging/packer-build/index.html" title="Packer-build">
<i class="none"></i>
Packer-build</a>
</li>
-
+
<li class="active">
<a href="#"><i class="icon-chevron-down"></i>Roles</a>
@@ -142,13 +153,6 @@
<li>
- <a href="../../metron-deployment/roles/kibana/index.html" title="Kibana">
- <i class="none"></i>
- Kibana</a>
- </li>
-
- <li>
-
<a href="../../metron-deployment/roles/monit/index.html" title="Monit">
<i class="none"></i>
Monit</a>
@@ -199,14 +203,28 @@
<i class="none"></i>
Docker</a>
</li>
-
+
+ <li>
+
+ <a href="../../metron-interface/metron-config/index.html" title="Config">
+ <i class="none"></i>
+ Config</a>
+ </li>
+
+ <li>
+
+ <a href="../../metron-interface/metron-rest/index.html" title="Rest">
+ <i class="none"></i>
+ Rest</a>
+ </li>
+
<li>
<a href="../../metron-platform/index.html" title="Platform">
<i class="icon-chevron-right"></i>
Platform</a>
</li>
-
+
<li>
<a href="../../metron-sensors/index.html" title="Sensors">
@@ -240,13 +258,13 @@
<ul>
-<li>Kibana</li>
-
<li>Monit</li>
<li>OpenTaxii</li>
-<li>Pcap Relay</li>
+<li>Pcap Replay</li>
+
+<li>Sensor Stubs</li>
<li>Sensor Test Mode</li>
</ul>
@@ -258,8 +276,9 @@
<footer>
<div class="container-fluid">
- <div class="row span12">Copyright © 2017.
- All Rights Reserved.
+ <div class="row span12">Copyright © 2017
+ <a href="https://www.apache.org">The Apache Software Foundation</a>.
+ All Rights Reserved.
</div>
http://git-wip-us.apache.org/repos/asf/metron/blob/cf4c2ecd/current-book/metron-deployment/roles/monit/index.html
----------------------------------------------------------------------
diff --git a/current-book/metron-deployment/roles/monit/index.html b/current-book/metron-deployment/roles/monit/index.html
index 07f3bcd..74db63d 100644
--- a/current-book/metron-deployment/roles/monit/index.html
+++ b/current-book/metron-deployment/roles/monit/index.html
@@ -1,13 +1,13 @@
<!DOCTYPE html>
<!--
- | Generated by Apache Maven Doxia at 2017-02-23
+ | Generated by Apache Maven Doxia at 2017-06-27
| Rendered using Apache Maven Fluido Skin 1.3.0
-->
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
- <meta name="Date-Revision-yyyymmdd" content="20170223" />
+ <meta name="Date-Revision-yyyymmdd" content="20170627" />
<meta http-equiv="Content-Language" content="en" />
<title>Metron – Monit Integration</title>
<link rel="stylesheet" href="../../../css/apache-maven-fluido-1.3.0.min.css" />
@@ -30,14 +30,11 @@
<div class="container-fluid">
<div id="banner">
<div class="pull-left">
- <a href="http://metron.incubator.apache.org/" id="bannerLeft">
- <img src="../../../images/metron-logo.png" alt="Apache Metron - Incubating" width="148px" height="48px"/>
+ <a href="http://metron.apache.org/" id="bannerLeft">
+ <img src="../../../images/metron-logo.png" alt="Apache Metron" width="148px" height="48px"/>
</a>
</div>
- <div class="pull-right"> <a href="http://incubator.apache.org/" id="bannerRight">
- <img src="../../../images/ApacheIncubating_Logo.png" alt="Apache Incubating" width="192px" height="48px"/>
- </a>
- </div>
+ <div class="pull-right"> </div>
<div class="clear"><hr/></div>
</div>
@@ -51,8 +48,8 @@
</li>
<li class="divider ">/</li>
<li class="">
- <a href="http://metron.incubator.apache.org/" class="externalLink" title="Metron-Incubating">
- Metron-Incubating</a>
+ <a href="http://metron.apache.org/" class="externalLink" title="Metron">
+ Metron</a>
</li>
<li class="divider ">/</li>
<li class="">
@@ -64,8 +61,8 @@
- <li id="publishDate" class="pull-right">Last Published: 2017-02-23</li> <li class="divider pull-right">|</li>
- <li id="projectVersion" class="pull-right">Version: 0.3.1</li>
+ <li id="publishDate" class="pull-right">Last Published: 2017-06-27</li> <li class="divider pull-right">|</li>
+ <li id="projectVersion" class="pull-right">Version: 0.4.0</li>
</ul>
</div>
@@ -78,7 +75,7 @@
<ul class="nav nav-list">
<li class="nav-header">User Documentation</li>
-
+
<li>
<a href="../../../index.html" title="Metron">
@@ -99,7 +96,7 @@
<i class="icon-chevron-right"></i>
Analytics</a>
</li>
-
+
<li>
<a href="../../../metron-deployment/index.html" title="Deployment">
@@ -109,6 +106,20 @@
<li>
+ <a href="../../../metron-deployment/Kerberos-ambari-setup.html" title="Kerberos-ambari-setup">
+ <i class="none"></i>
+ Kerberos-ambari-setup</a>
+ </li>
+
+ <li>
+
+ <a href="../../../metron-deployment/Kerberos-manual-setup.html" title="Kerberos-manual-setup">
+ <i class="none"></i>
+ Kerberos-manual-setup</a>
+ </li>
+
+ <li>
+
<a href="../../../metron-deployment/amazon-ec2/index.html" title="Amazon-ec2">
<i class="none"></i>
Amazon-ec2</a>
@@ -130,11 +141,11 @@
<li>
- <a href="../../../metron-deployment/packer-build/index.html" title="Packer-build">
+ <a href="../../../metron-deployment/packaging/packer-build/index.html" title="Packer-build">
<i class="none"></i>
Packer-build</a>
</li>
-
+
<li>
<a href="../../../metron-deployment/roles/index.html" title="Roles">
@@ -142,13 +153,6 @@
Roles</a>
<ul class="nav nav-list">
- <li>
-
- <a href="../../../metron-deployment/roles/kibana/index.html" title="Kibana">
- <i class="none"></i>
- Kibana</a>
- </li>
-
<li class="active">
<a href="#"><i class="none"></i>Monit</a>
@@ -199,14 +203,28 @@
<i class="none"></i>
Docker</a>
</li>
-
+
+ <li>
+
+ <a href="../../../metron-interface/metron-config/index.html" title="Config">
+ <i class="none"></i>
+ Config</a>
+ </li>
+
+ <li>
+
+ <a href="../../../metron-interface/metron-rest/index.html" title="Rest">
+ <i class="none"></i>
+ Rest</a>
+ </li>
+
<li>
<a href="../../../metron-platform/index.html" title="Platform">
<i class="icon-chevron-right"></i>
Platform</a>
</li>
-
+
<li>
<a href="../../../metron-sensors/index.html" title="Sensors">
@@ -263,7 +281,7 @@
" Ambari @ http://ec2-52-39-4-93.us-west-2.compute.amazonaws.com:8080",
" Sensor Status @ http://ec2-52-39-4-93.us-west-2.compute.amazonaws.com:2812",
" Topology Status @ http://ec2-52-39-130-62.us-west-2.compute.amazonaws.com:2812",
- "For additional information, see https://metron.incubator.apache.org/'"
+ "For additional information, see https://metron.apache.org/'"
]
}
</pre></div></div></li>
@@ -314,8 +332,9 @@
<footer>
<div class="container-fluid">
- <div class="row span12">Copyright © 2017.
- All Rights Reserved.
+ <div class="row span12">Copyright © 2017
+ <a href="https://www.apache.org">The Apache Software Foundation</a>.
+ All Rights Reserved.
</div>
http://git-wip-us.apache.org/repos/asf/metron/blob/cf4c2ecd/current-book/metron-deployment/roles/opentaxii/index.html
----------------------------------------------------------------------
diff --git a/current-book/metron-deployment/roles/opentaxii/index.html b/current-book/metron-deployment/roles/opentaxii/index.html
index 5965ce9..8d8cae4 100644
--- a/current-book/metron-deployment/roles/opentaxii/index.html
+++ b/current-book/metron-deployment/roles/opentaxii/index.html
@@ -1,13 +1,13 @@
<!DOCTYPE html>
<!--
- | Generated by Apache Maven Doxia at 2017-02-23
+ | Generated by Apache Maven Doxia at 2017-06-27
| Rendered using Apache Maven Fluido Skin 1.3.0
-->
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
- <meta name="Date-Revision-yyyymmdd" content="20170223" />
+ <meta name="Date-Revision-yyyymmdd" content="20170627" />
<meta http-equiv="Content-Language" content="en" />
<title>Metron – OpenTAXII</title>
<link rel="stylesheet" href="../../../css/apache-maven-fluido-1.3.0.min.css" />
@@ -30,14 +30,11 @@
<div class="container-fluid">
<div id="banner">
<div class="pull-left">
- <a href="http://metron.incubator.apache.org/" id="bannerLeft">
- <img src="../../../images/metron-logo.png" alt="Apache Metron - Incubating" width="148px" height="48px"/>
+ <a href="http://metron.apache.org/" id="bannerLeft">
+ <img src="../../../images/metron-logo.png" alt="Apache Metron" width="148px" height="48px"/>
</a>
</div>
- <div class="pull-right"> <a href="http://incubator.apache.org/" id="bannerRight">
- <img src="../../../images/ApacheIncubating_Logo.png" alt="Apache Incubating" width="192px" height="48px"/>
- </a>
- </div>
+ <div class="pull-right"> </div>
<div class="clear"><hr/></div>
</div>
@@ -51,8 +48,8 @@
</li>
<li class="divider ">/</li>
<li class="">
- <a href="http://metron.incubator.apache.org/" class="externalLink" title="Metron-Incubating">
- Metron-Incubating</a>
+ <a href="http://metron.apache.org/" class="externalLink" title="Metron">
+ Metron</a>
</li>
<li class="divider ">/</li>
<li class="">
@@ -64,8 +61,8 @@
- <li id="publishDate" class="pull-right">Last Published: 2017-02-23</li> <li class="divider pull-right">|</li>
- <li id="projectVersion" class="pull-right">Version: 0.3.1</li>
+ <li id="publishDate" class="pull-right">Last Published: 2017-06-27</li> <li class="divider pull-right">|</li>
+ <li id="projectVersion" class="pull-right">Version: 0.4.0</li>
</ul>
</div>
@@ -78,7 +75,7 @@
<ul class="nav nav-list">
<li class="nav-header">User Documentation</li>
-
+
<li>
<a href="../../../index.html" title="Metron">
@@ -99,7 +96,7 @@
<i class="icon-chevron-right"></i>
Analytics</a>
</li>
-
+
<li>
<a href="../../../metron-deployment/index.html" title="Deployment">
@@ -109,6 +106,20 @@
<li>
+ <a href="../../../metron-deployment/Kerberos-ambari-setup.html" title="Kerberos-ambari-setup">
+ <i class="none"></i>
+ Kerberos-ambari-setup</a>
+ </li>
+
+ <li>
+
+ <a href="../../../metron-deployment/Kerberos-manual-setup.html" title="Kerberos-manual-setup">
+ <i class="none"></i>
+ Kerberos-manual-setup</a>
+ </li>
+
+ <li>
+
<a href="../../../metron-deployment/amazon-ec2/index.html" title="Amazon-ec2">
<i class="none"></i>
Amazon-ec2</a>
@@ -130,11 +141,11 @@
<li>
- <a href="../../../metron-deployment/packer-build/index.html" title="Packer-build">
+ <a href="../../../metron-deployment/packaging/packer-build/index.html" title="Packer-build">
<i class="none"></i>
Packer-build</a>
</li>
-
+
<li>
<a href="../../../metron-deployment/roles/index.html" title="Roles">
@@ -144,13 +155,6 @@
<li>
- <a href="../../../metron-deployment/roles/kibana/index.html" title="Kibana">
- <i class="none"></i>
- Kibana</a>
- </li>
-
- <li>
-
<a href="../../../metron-deployment/roles/monit/index.html" title="Monit">
<i class="none"></i>
Monit</a>
@@ -199,14 +203,28 @@
<i class="none"></i>
Docker</a>
</li>
-
+
+ <li>
+
+ <a href="../../../metron-interface/metron-config/index.html" title="Config">
+ <i class="none"></i>
+ Config</a>
+ </li>
+
+ <li>
+
+ <a href="../../../metron-interface/metron-rest/index.html" title="Rest">
+ <i class="none"></i>
+ Rest</a>
+ </li>
+
<li>
<a href="../../../metron-platform/index.html" title="Platform">
<i class="icon-chevron-right"></i>
Platform</a>
</li>
-
+
<li>
<a href="../../../metron-sensors/index.html" title="Sensors">
@@ -406,8 +424,9 @@ taxii-poll --discovery http://hailataxii.com/taxii-data -c guest.phishtank_com -
<footer>
<div class="container-fluid">
- <div class="row span12">Copyright © 2017.
- All Rights Reserved.
+ <div class="row span12">Copyright © 2017
+ <a href="https://www.apache.org">The Apache Software Foundation</a>.
+ All Rights Reserved.
</div>
http://git-wip-us.apache.org/repos/asf/metron/blob/cf4c2ecd/current-book/metron-deployment/roles/pcap_replay/index.html
----------------------------------------------------------------------
diff --git a/current-book/metron-deployment/roles/pcap_replay/index.html b/current-book/metron-deployment/roles/pcap_replay/index.html
index cd95448..7d072dc 100644
--- a/current-book/metron-deployment/roles/pcap_replay/index.html
+++ b/current-book/metron-deployment/roles/pcap_replay/index.html
@@ -1,13 +1,13 @@
<!DOCTYPE html>
<!--
- | Generated by Apache Maven Doxia at 2017-02-23
+ | Generated by Apache Maven Doxia at 2017-06-27
| Rendered using Apache Maven Fluido Skin 1.3.0
-->
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
- <meta name="Date-Revision-yyyymmdd" content="20170223" />
+ <meta name="Date-Revision-yyyymmdd" content="20170627" />
<meta http-equiv="Content-Language" content="en" />
<title>Metron – Pcap Replay</title>
<link rel="stylesheet" href="../../../css/apache-maven-fluido-1.3.0.min.css" />
@@ -30,14 +30,11 @@
<div class="container-fluid">
<div id="banner">
<div class="pull-left">
- <a href="http://metron.incubator.apache.org/" id="bannerLeft">
- <img src="../../../images/metron-logo.png" alt="Apache Metron - Incubating" width="148px" height="48px"/>
+ <a href="http://metron.apache.org/" id="bannerLeft">
+ <img src="../../../images/metron-logo.png" alt="Apache Metron" width="148px" height="48px"/>
</a>
</div>
- <div class="pull-right"> <a href="http://incubator.apache.org/" id="bannerRight">
- <img src="../../../images/ApacheIncubating_Logo.png" alt="Apache Incubating" width="192px" height="48px"/>
- </a>
- </div>
+ <div class="pull-right"> </div>
<div class="clear"><hr/></div>
</div>
@@ -51,8 +48,8 @@
</li>
<li class="divider ">/</li>
<li class="">
- <a href="http://metron.incubator.apache.org/" class="externalLink" title="Metron-Incubating">
- Metron-Incubating</a>
+ <a href="http://metron.apache.org/" class="externalLink" title="Metron">
+ Metron</a>
</li>
<li class="divider ">/</li>
<li class="">
@@ -64,8 +61,8 @@
- <li id="publishDate" class="pull-right">Last Published: 2017-02-23</li> <li class="divider pull-right">|</li>
- <li id="projectVersion" class="pull-right">Version: 0.3.1</li>
+ <li id="publishDate" class="pull-right">Last Published: 2017-06-27</li> <li class="divider pull-right">|</li>
+ <li id="projectVersion" class="pull-right">Version: 0.4.0</li>
</ul>
</div>
@@ -78,7 +75,7 @@
<ul class="nav nav-list">
<li class="nav-header">User Documentation</li>
-
+
<li>
<a href="../../../index.html" title="Metron">
@@ -99,7 +96,7 @@
<i class="icon-chevron-right"></i>
Analytics</a>
</li>
-
+
<li>
<a href="../../../metron-deployment/index.html" title="Deployment">
@@ -109,6 +106,20 @@
<li>
+ <a href="../../../metron-deployment/Kerberos-ambari-setup.html" title="Kerberos-ambari-setup">
+ <i class="none"></i>
+ Kerberos-ambari-setup</a>
+ </li>
+
+ <li>
+
+ <a href="../../../metron-deployment/Kerberos-manual-setup.html" title="Kerberos-manual-setup">
+ <i class="none"></i>
+ Kerberos-manual-setup</a>
+ </li>
+
+ <li>
+
<a href="../../../metron-deployment/amazon-ec2/index.html" title="Amazon-ec2">
<i class="none"></i>
Amazon-ec2</a>
@@ -130,11 +141,11 @@
<li>
- <a href="../../../metron-deployment/packer-build/index.html" title="Packer-build">
+ <a href="../../../metron-deployment/packaging/packer-build/index.html" title="Packer-build">
<i class="none"></i>
Packer-build</a>
</li>
-
+
<li>
<a href="../../../metron-deployment/roles/index.html" title="Roles">
@@ -144,13 +155,6 @@
<li>
- <a href="../../../metron-deployment/roles/kibana/index.html" title="Kibana">
- <i class="none"></i>
- Kibana</a>
- </li>
-
- <li>
-
<a href="../../../metron-deployment/roles/monit/index.html" title="Monit">
<i class="none"></i>
Monit</a>
@@ -199,14 +203,28 @@
<i class="none"></i>
Docker</a>
</li>
-
+
+ <li>
+
+ <a href="../../../metron-interface/metron-config/index.html" title="Config">
+ <i class="none"></i>
+ Config</a>
+ </li>
+
+ <li>
+
+ <a href="../../../metron-interface/metron-rest/index.html" title="Rest">
+ <i class="none"></i>
+ Rest</a>
+ </li>
+
<li>
<a href="../../../metron-platform/index.html" title="Platform">
<i class="icon-chevron-right"></i>
Platform</a>
</li>
-
+
<li>
<a href="../../../metron-sensors/index.html" title="Sensors">
@@ -281,8 +299,9 @@
<footer>
<div class="container-fluid">
- <div class="row span12">Copyright © 2017.
- All Rights Reserved.
+ <div class="row span12">Copyright © 2017
+ <a href="https://www.apache.org">The Apache Software Foundation</a>.
+ All Rights Reserved.
</div>
http://git-wip-us.apache.org/repos/asf/metron/blob/cf4c2ecd/current-book/metron-deployment/roles/sensor-stubs/index.html
----------------------------------------------------------------------
diff --git a/current-book/metron-deployment/roles/sensor-stubs/index.html b/current-book/metron-deployment/roles/sensor-stubs/index.html
index 5f16354..df96a6a 100644
--- a/current-book/metron-deployment/roles/sensor-stubs/index.html
+++ b/current-book/metron-deployment/roles/sensor-stubs/index.html
@@ -1,13 +1,13 @@
<!DOCTYPE html>
<!--
- | Generated by Apache Maven Doxia at 2017-02-23
+ | Generated by Apache Maven Doxia at 2017-06-27
| Rendered using Apache Maven Fluido Skin 1.3.0
-->
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
- <meta name="Date-Revision-yyyymmdd" content="20170223" />
+ <meta name="Date-Revision-yyyymmdd" content="20170627" />
<meta http-equiv="Content-Language" content="en" />
<title>Metron – Sensor Stubs</title>
<link rel="stylesheet" href="../../../css/apache-maven-fluido-1.3.0.min.css" />
@@ -30,14 +30,11 @@
<div class="container-fluid">
<div id="banner">
<div class="pull-left">
- <a href="http://metron.incubator.apache.org/" id="bannerLeft">
- <img src="../../../images/metron-logo.png" alt="Apache Metron - Incubating" width="148px" height="48px"/>
+ <a href="http://metron.apache.org/" id="bannerLeft">
+ <img src="../../../images/metron-logo.png" alt="Apache Metron" width="148px" height="48px"/>
</a>
</div>
- <div class="pull-right"> <a href="http://incubator.apache.org/" id="bannerRight">
- <img src="../../../images/ApacheIncubating_Logo.png" alt="Apache Incubating" width="192px" height="48px"/>
- </a>
- </div>
+ <div class="pull-right"> </div>
<div class="clear"><hr/></div>
</div>
@@ -51,8 +48,8 @@
</li>
<li class="divider ">/</li>
<li class="">
- <a href="http://metron.incubator.apache.org/" class="externalLink" title="Metron-Incubating">
- Metron-Incubating</a>
+ <a href="http://metron.apache.org/" class="externalLink" title="Metron">
+ Metron</a>
</li>
<li class="divider ">/</li>
<li class="">
@@ -64,8 +61,8 @@
- <li id="publishDate" class="pull-right">Last Published: 2017-02-23</li> <li class="divider pull-right">|</li>
- <li id="projectVersion" class="pull-right">Version: 0.3.1</li>
+ <li id="publishDate" class="pull-right">Last Published: 2017-06-27</li> <li class="divider pull-right">|</li>
+ <li id="projectVersion" class="pull-right">Version: 0.4.0</li>
</ul>
</div>
@@ -78,7 +75,7 @@
<ul class="nav nav-list">
<li class="nav-header">User Documentation</li>
-
+
<li>
<a href="../../../index.html" title="Metron">
@@ -99,7 +96,7 @@
<i class="icon-chevron-right"></i>
Analytics</a>
</li>
-
+
<li>
<a href="../../../metron-deployment/index.html" title="Deployment">
@@ -109,6 +106,20 @@
<li>
+ <a href="../../../metron-deployment/Kerberos-ambari-setup.html" title="Kerberos-ambari-setup">
+ <i class="none"></i>
+ Kerberos-ambari-setup</a>
+ </li>
+
+ <li>
+
+ <a href="../../../metron-deployment/Kerberos-manual-setup.html" title="Kerberos-manual-setup">
+ <i class="none"></i>
+ Kerberos-manual-setup</a>
+ </li>
+
+ <li>
+
<a href="../../../metron-deployment/amazon-ec2/index.html" title="Amazon-ec2">
<i class="none"></i>
Amazon-ec2</a>
@@ -130,11 +141,11 @@
<li>
- <a href="../../../metron-deployment/packer-build/index.html" title="Packer-build">
+ <a href="../../../metron-deployment/packaging/packer-build/index.html" title="Packer-build">
<i class="none"></i>
Packer-build</a>
</li>
-
+
<li>
<a href="../../../metron-deployment/roles/index.html" title="Roles">
@@ -144,13 +155,6 @@
<li>
- <a href="../../../metron-deployment/roles/kibana/index.html" title="Kibana">
- <i class="none"></i>
- Kibana</a>
- </li>
-
- <li>
-
<a href="../../../metron-deployment/roles/monit/index.html" title="Monit">
<i class="none"></i>
Monit</a>
@@ -199,14 +203,28 @@
<i class="none"></i>
Docker</a>
</li>
-
+
+ <li>
+
+ <a href="../../../metron-interface/metron-config/index.html" title="Config">
+ <i class="none"></i>
+ Config</a>
+ </li>
+
+ <li>
+
+ <a href="../../../metron-interface/metron-rest/index.html" title="Rest">
+ <i class="none"></i>
+ Rest</a>
+ </li>
+
<li>
<a href="../../../metron-platform/index.html" title="Platform">
<i class="icon-chevron-right"></i>
Platform</a>
</li>
-
+
<li>
<a href="../../../metron-sensors/index.html" title="Sensors">
@@ -327,8 +345,9 @@ vagrant --ansible-skip-tags="sensor-stubs,solr" up
<footer>
<div class="container-fluid">
- <div class="row span12">Copyright © 2017.
- All Rights Reserved.
+ <div class="row span12">Copyright © 2017
+ <a href="https://www.apache.org">The Apache Software Foundation</a>.
+ All Rights Reserved.
</div>
[09/10] metron git commit: METRON-1012 Update Metron public web site
for 0.4.0 release - push to asf-site from the generated code on master
Posted by ma...@apache.org.
http://git-wip-us.apache.org/repos/asf/metron/blob/cf4c2ecd/current-book/metron-analytics/metron-profiler-client/index.html
----------------------------------------------------------------------
diff --git a/current-book/metron-analytics/metron-profiler-client/index.html b/current-book/metron-analytics/metron-profiler-client/index.html
index 54dd5d6..47cef0a 100644
--- a/current-book/metron-analytics/metron-profiler-client/index.html
+++ b/current-book/metron-analytics/metron-profiler-client/index.html
@@ -1,13 +1,13 @@
<!DOCTYPE html>
<!--
- | Generated by Apache Maven Doxia at 2017-02-23
+ | Generated by Apache Maven Doxia at 2017-06-27
| Rendered using Apache Maven Fluido Skin 1.3.0
-->
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
- <meta name="Date-Revision-yyyymmdd" content="20170223" />
+ <meta name="Date-Revision-yyyymmdd" content="20170627" />
<meta http-equiv="Content-Language" content="en" />
<title>Metron – Metron Profiler Client</title>
<link rel="stylesheet" href="../../css/apache-maven-fluido-1.3.0.min.css" />
@@ -30,14 +30,11 @@
<div class="container-fluid">
<div id="banner">
<div class="pull-left">
- <a href="http://metron.incubator.apache.org/" id="bannerLeft">
- <img src="../../images/metron-logo.png" alt="Apache Metron - Incubating" width="148px" height="48px"/>
+ <a href="http://metron.apache.org/" id="bannerLeft">
+ <img src="../../images/metron-logo.png" alt="Apache Metron" width="148px" height="48px"/>
</a>
</div>
- <div class="pull-right"> <a href="http://incubator.apache.org/" id="bannerRight">
- <img src="../../images/ApacheIncubating_Logo.png" alt="Apache Incubating" width="192px" height="48px"/>
- </a>
- </div>
+ <div class="pull-right"> </div>
<div class="clear"><hr/></div>
</div>
@@ -51,8 +48,8 @@
</li>
<li class="divider ">/</li>
<li class="">
- <a href="http://metron.incubator.apache.org/" class="externalLink" title="Metron-Incubating">
- Metron-Incubating</a>
+ <a href="http://metron.apache.org/" class="externalLink" title="Metron">
+ Metron</a>
</li>
<li class="divider ">/</li>
<li class="">
@@ -64,8 +61,8 @@
- <li id="publishDate" class="pull-right">Last Published: 2017-02-23</li> <li class="divider pull-right">|</li>
- <li id="projectVersion" class="pull-right">Version: 0.3.1</li>
+ <li id="publishDate" class="pull-right">Last Published: 2017-06-27</li> <li class="divider pull-right">|</li>
+ <li id="projectVersion" class="pull-right">Version: 0.4.0</li>
</ul>
</div>
@@ -78,7 +75,7 @@
<ul class="nav nav-list">
<li class="nav-header">User Documentation</li>
-
+
<li>
<a href="../../index.html" title="Metron">
@@ -127,7 +124,7 @@
</li>
</ul>
</li>
-
+
<li>
<a href="../../metron-deployment/index.html" title="Deployment">
@@ -141,14 +138,28 @@
<i class="none"></i>
Docker</a>
</li>
-
+
+ <li>
+
+ <a href="../../metron-interface/metron-config/index.html" title="Config">
+ <i class="none"></i>
+ Config</a>
+ </li>
+
+ <li>
+
+ <a href="../../metron-interface/metron-rest/index.html" title="Rest">
+ <i class="none"></i>
+ Rest</a>
+ </li>
+
<li>
<a href="../../metron-platform/index.html" title="Platform">
<i class="icon-chevron-right"></i>
Platform</a>
</li>
-
+
<li>
<a href="../../metron-sensors/index.html" title="Sensors">
@@ -182,25 +193,9 @@
<p>This project provides a client API for accessing the profiles generated by the <a href="../metron-profiler/index.html">Metron Profiler</a>. This includes both a Java API and Stellar API for accessing the profile data. The primary use case is to extract profile data for use during model scoring.</p>
<div class="section">
<h2><a name="Stellar_Client_API"></a>Stellar Client API</h2>
-<p>The following are usage examples that show how the Stellar API can be used to read profiles generated by the <a href="../metron-profiler/index.html">Metron Profiler</a>. This API would be used in conjunction with other Stellar functions like <a href="../../metron-platform/metron-common/index.html#MAAS_MODEL_APPLY"><tt>MAAS_MODEL_APPLY</tt></a> to perform model scoring on streaming data.</p>
-<p>These examples assume a profile has been defined called ‘snort-alerts’ that tracks the number of Snort alerts associated with an IP address over time. The profile definition might look similar to the following.</p>
-
-<div class="source">
-<div class="source">
-<pre>{
- "profiles": [
- {
- "profile": "snort-alerts",
- "foreach": "ip_src_addr",
- "onlyif": "source.type == 'snort'",
- "update": { "s": "STATS_ADD(s, 1)" },
- "result": "STATS_MEAN(s)"
- }
- ]
-}
-</pre></div></div>
-<p>During model scoring the entity being scored, in this case a particular IP address, will be known. The following examples shows how this profile data might be retrieved.</p>
-<p>The Stellar client consists of the <tt>PROFILE_GET</tt> command, which takes the following arguments:</p>
+<div class="section">
+<h3><a name="PROFILE_GET"></a><tt>PROFILE_GET</tt></h3>
+<p>The <tt>PROFILE_GET</tt> command allows you to select all of the profile measurements written. This command takes the following arguments:</p>
<div class="source">
<div class="source">
@@ -217,21 +212,8 @@ OPTIONAL:
</pre></div></div>
<p>There is an older calling format where <tt>groups_list</tt> is specified as a sequence of group names, “varargs” style, instead of a List object. This format is still supported for backward compatibility, but it is deprecated, and it is disallowed if the optional <tt>config_overrides</tt> argument is used.</p>
<p>The <tt>periods</tt> field is (likely) the output of another Stellar function which defines the times to include.</p>
-<p><tt>PROFILE_FIXED</tt>: The profiler periods associated with a fixed lookback starting from now. These are ProfilePeriod objects.</p>
-
-<div class="source">
-<div class="source">
-<pre>REQUIRED:
- durationAgo - How long ago should values be retrieved from?
- units - The units of 'durationAgo'.
-OPTIONAL:
- config_overrides - Optional - Map (in curly braces) of name:value pairs, each overriding the global config parameter
- of the same name. Default is the empty Map, meaning no overrides.
-
-e.g. To retrieve all the profiles for the last 5 hours. PROFILE_GET('profile', 'entity', PROFILE_FIXED(5, 'HOURS'))
-</pre></div></div>
<div class="section">
-<h3><a name="Groups_list_argument"></a>Groups_list argument</h3>
+<h4><a name="Groups_list_argument"></a>Groups_list argument</h4>
<p>The <tt>groups_list</tt> argument in the client must exactly correspond to the <a href="../metron-profiler/index.html#groupBy"><tt>groupBy</tt></a> configuration in the profile definition. If <tt>groupBy</tt> was not used in the profile, <tt>groups_list</tt> must be empty in the client. If <tt>groupBy</tt> was used in the profile, then the client <tt>groups_list</tt> is <b>not</b> optional; it must be the same length as the <tt>groupBy</tt> list, and specify exactly one selected group value for each <tt>groupBy</tt> criterion, in the same order. For example:</p>
<div class="source">
@@ -241,7 +223,7 @@ Then in PROFILE_GET, an allowed groups value would be: [ “3”, &#x
which will select only records from Tuesdays with port number 8080.
</pre></div></div></div>
<div class="section">
-<h3><a name="Configuration_and_the_config_overrides_argument"></a>Configuration and the config_overrides argument</h3>
+<h4><a name="Configuration_and_the_config_overrides_argument"></a>Configuration and the config_overrides argument</h4>
<p>By default, the Profiler creates profiles with a period duration of 15 minutes. This means that data is accumulated, summarized and flushed every 15 minutes. The Client API must also have knowledge of this duration to correctly retrieve the profile data. If the Client is expecting 15 minute periods, it will not be able to read data generated by a Profiler that was configured for 1 hour periods, and will return zero results. </p>
<p>Similarly, all six Client configuration parameters listed in the table below must match the Profiler configuration parameter settings from the time the profile was created. The period duration and other configuration parameters from the Profiler topology are stored in local filesystem at <tt>$METRON_HOME/config/profiler.properties</tt>. The Stellar Client API can be configured correspondingly by setting the following properties in Metron’s global configuration, on local filesystem at <tt>$METRON_HOME/config/zookeeper/global.json</tt>, then uploaded to Zookeeper (at <tt>/metron/topology/global</tt>) by using <tt>zk_load_configs.sh</tt>: </p>
@@ -334,43 +316,433 @@ which will select only records from Tuesdays with port number 8080.
<td> </td>
</tr>
</tbody>
-</table></div>
+</table></div></div>
+<div class="section">
+<h3><a name="Profile_Selectors"></a>Profile Selectors</h3>
+<p>You will notice that the third argument for <tt>PROFILE_GET</tt> is a list of <tt>ProfilePeriod</tt> objects. This list is expected to be produced by another Stellar function. There are a couple options available.</p>
+<div class="section">
+<h4><a name="PROFILE_FIXED"></a><tt>PROFILE_FIXED</tt></h4>
+<p>The profiler periods associated with a fixed lookback starting from now. These are ProfilePeriod objects.</p>
+
+<div class="source">
+<div class="source">
+<pre>REQUIRED:
+ durationAgo - How long ago should values be retrieved from?
+ units - The units of 'durationAgo'.
+OPTIONAL:
+ config_overrides - Optional - Map (in curly braces) of name:value pairs, each overriding the global config parameter
+ of the same name. Default is the empty Map, meaning no overrides.
+
+e.g. To retrieve all the profiles for the last 5 hours. PROFILE_GET('profile', 'entity', PROFILE_FIXED(5, 'HOURS'))
+</pre></div></div>
+<p>Note that the <tt>config_overrides</tt> parameter operates exactly as the <tt>config_overrides</tt> argument in <tt>PROFILE_GET</tt>. The only available parameters for override are:</p>
+
+<ul>
+
+<li><tt>profiler.client.period.duration</tt></li>
+
+<li><tt>profiler.client.period.duration.units</tt></li>
+</ul></div>
+<div class="section">
+<h4><a name="PROFILE_WINDOW"></a><tt>PROFILE_WINDOW</tt></h4>
+<p><tt>PROFILE_WINDOW</tt> is intended to provide a finer-level of control over selecting windows for profiles:</p>
+
+<ul>
+
+<li>Specify windows relative to the data timestamp (see the optional <tt>now</tt> parameter below)</li>
+
+<li>Specify non-contiguous windows to better handle seasonal data (e.g. the last hour for every day for the last month)</li>
+
+<li>Specify profile output excluding holidays</li>
+
+<li>Specify only profile output on a specific day of the week</li>
+</ul>
+<p>It does this by a domain specific language mimicking natural language that defines the windows excluded.</p>
+
+<div class="source">
+<div class="source">
+<pre>REQUIRED:
+ windowSelector - The statement specifying the window to select.
+ now - Optional - The timestamp to use for now.
+OPTIONAL:
+ config_overrides - Optional - Map (in curly braces) of name:value pairs, each overriding the global config parameter
+ of the same name. Default is the empty Map, meaning no overrides.
+
+e.g. To retrieve all the measurements written for 'profile' and 'entity' for the last hour
+on the same weekday excluding weekends and US holidays across the last 14 days:
+PROFILE_GET('profile', 'entity', PROFILE_WINDOW('1 hour window every 24 hours starting from 14 days ago including the current day of the week excluding weekends, holidays:us'))
+</pre></div></div>
+<p>Note that the <tt>config_overrides</tt> parameter operates exactly as the <tt>config_overrides</tt> argument in <tt>PROFILE_GET</tt>. The only available parameters for override are:</p>
+
+<ul>
+
+<li><tt>profiler.client.period.duration</tt></li>
+
+<li><tt>profiler.client.period.duration.units</tt></li>
+</ul>
+<div class="section">
+<h5><a name="The_Profile_Selector_Language"></a>The Profile Selector Language</h5>
+<p>The domain specific language can be broken into a series of clauses, some optional</p>
+
+<ul>
+
+<li><a href="#Temporal_Window_Width"><span style="color:blue">Total Temporal Duration</span></a> - The total range of time in which windows may be specified</li>
+
+<li><a href="#InclusionExclusion_specifiers"><span style="color:red">Temporal Window Width</span></a> - How large each temporal window</li>
+
+<li><a href="#Skip_distance"><span style="color:green">Skip distance</span></a> (optional)- How far to skip between when one window starts and when the next begins</li>
+
+<li><a href="#InclusionExclusion_specifiers"><span style="color:purple">Inclusion/Exclusion specifiers</span></a> (optional) - The set of specifiers to further filter the window</li>
+</ul>
+<p>One <i>must</i> specify either a total temporal duration or a temporal window width. The remaining clauses are optional. During the course of the following discussion, we will color code the clauses in the examples and link them to the relevant section for more detail.</p>
+<p>From a high level, the language fits the following three forms, which are composed of the clauses above:</p>
+
+<ul>
+
+<li><a href="#Temporal_Window_Width"><span style="color:red">time_interval WINDOW?</span></a> <a href="#InclusionExclusion_specifiers"><span style="color:purple">(INCLUDING specifier_list)? (EXCLUDING specifier_list)?</span></a></li>
+
+<li><a href="#Temporal_Window_Width"><span style="color:red">time_interval WINDOW?</span></a> <a href="#Skip_distance"><span style="color:green">EVERY time_interval</span></a> <a href="#Total_Temporal_Duration"><span style="color:blue">FROM time_interval (TO time_interval)?</span></a> <a href="#InclusionExclusion_specifiers"><span style="color:purple">(INCLUDING specifier_list)? (EXCLUDING specifier_list)?</span></a></li>
+
+<li><a href="#Total_Temporal_Duration"><span style="color:blue">FROM time_interval (TO time_interval)?</span></a></li>
+</ul>
+<div class="section">
+<h6><a name="Total_Temporal_Duration"></a><span style="color:blue">Total Temporal Duration</span></h6>
+<p>Total temporal duration is specified by a phrase: <tt>FROM time_interval AGO TO time_interval AGO</tt> This indicates the beginning and ending of a time interval. This is an inclusive duration.</p>
+
+<ul>
+
+<li><tt>FROM</tt> - Can be the words “from” or “starting from”</li>
+
+<li><tt>time_interval</tt> - A time amount followed by a unit (e.g. 1 hour). Fractional amounts are not supported. The unit may be “minute”, “day”, “hour” with any pluralization.</li>
+
+<li><tt>TO</tt> - Can be the words “until” or “to”</li>
+
+<li><tt>AGO</tt> - Optionally the word “ago”</li>
+</ul>
+<p>The <tt>TO time_interval AGO</tt> portion is optional. If unspecified then it is expected that the time interval ends now.</p>
+<p>Due to the vagaries of the english language, the from and the to portions, if both specified, are interchangeable with regard to which one specifies the start and which specifies the end. </p>
+<p>In other words “<a href="#Total_Temporal_Duration"><span style="color:blue">starting from 1 hour ago to 30 minutes ago</span></a>” and “<a href="#Total_Temporal_Duration"><span style="color:blue">starting from 30 minutes ago to 1 hour ago</span></a>” specify the same temporal duration.</p>
+<p><b>Examples</b></p>
+
+<ul>
+
+<li>A duration starting 1 hour ago and ending now
+
+<ul>
+
+<li><a href="#Total_Temporal_Duration"><span style="color:blue">from 1 hour ago</span></a></li>
+
+<li><a href="#Total_Temporal_Duration"><span style="color:blue">from 1 hour</span></a></li>
+
+<li><a href="#Total_Temporal_Duration"><span style="color:blue">starting from 1 hour ago</span></a></li>
+
+<li><a href="#Total_Temporal_Duration"><span style="color:blue">starting from 1 hour</span></a></li>
+ </ul></li>
+
+<li>A duration starting 1 hour ago and ending 30 minutes ago:
+
+<ul>
+
+<li><a href="#Total_Temporal_Duration"><span style="color:blue">from 1 hour ago until 30 minutes ago</span></a></li>
+
+<li><a href="#Total_Temporal_Duration"><span style="color:blue">from 30 minutes ago until 1 hour ago</span></a></li>
+
+<li><a href="#Total_Temporal_Duration"><span style="color:blue">starting from 1 hour ago to 30 minutes ago</span></a></li>
+
+<li><a href="#Total_Temporal_Duration"><span style="color:blue">starting from 1 hour to 30 minutes</span></a></li>
+ </ul></li>
+</ul></div>
+<div class="section">
+<h6><a name="Temporal_Window_Width"></a><span style="color:red">Temporal Window Width</span></h6>
+<p>Temporal window width is the specification of a window. A window is may either repeat within total temporal duration or may fill the total temporal duration. This is an inclusive window. A window is specified by the phrase: <tt>time_interval WINDOW</tt></p>
+
+<ul>
+
+<li><tt>time_interval</tt> - A time amount followed by a unit (e.g. 1 hour). Fractional amounts are not supported. The unit may be “minute”, “day”, “hour” with any pluralization.</li>
+
+<li><tt>WINDOW</tt> - Optionally the word “window”</li>
+</ul>
+<p><b>Examples</b></p>
+
+<ul>
+
+<li>A fixed window starting 2 hours ago and going until now
+
+<ul>
+
+<li><a href="#Temporal_Window_Width"><span style="color:red">2 hour</span></a></li>
+
+<li><a href="#Temporal_Window_Width"><span style="color:red">2 hours</span></a></li>
+
+<li><a href="#Temporal_Window_Width"><span style="color:red">2 hours window</span></a></li>
+ </ul></li>
+
+<li>A repeating 30 minute window starting 2 hours ago and repeating every hour until now. This would result in 2 30-minute wide windows: 2 hours ago and 1 hour ago
+
+<ul>
+
+<li><a href="#Temporal_Window_Width"><span style="color:red">30 minute window</span></a> <a href="#Skip_distance"><span style="color:green">every 1 hour</span></a> <a href="#Total_Temporal_Duration"><span style="color:blue">starting from 2 hours ago</span></a></li>
+
+<li><a href="#Temporal_Window_Width"><span style="color:red">30 minute windows</span></a> <a href="#Skip_distance"><span style="color:green">every 1 hour</span></a> <a href="#Total_Temporal_Duration"><span style="color:blue">from 2 hours ago</span></a></li>
+ </ul></li>
+
+<li>A repeating 30 minute window starting 2 hours ago and repeating every hour until 30 minutes ago. This would result in 2 30-minute wide windows: 2 hours ago and 1 hour ago
+
+<ul>
+
+<li><a href="#Temporal_Window_Width"><span style="color:red">30 minute window</span></a> <a href="#Skip_distance"><span style="color:green">every 1 hour</span></a> <a href="#Total_Temporal_Duration"><span style="color:blue">starting from 2 hours ago until 30 minutes ago</span></a></li>
+
+<li><a href="#Temporal_Window_Width"><span style="color:red">30 minutes window</span></a> <a href="#Skip_distance"><span style="color:green">every 1 hour</span></a> <a href="#Total_Temporal_Duration"><span style="color:blue">from 2 hours ago to 30 minutes ago</span></a></li>
+
+<li><a href="#Temporal_Window_Width"><span style="color:red">30 minutes window</span></a> <a href="#Skip_distance"><span style="color:green">for every 1 hour</span></a> <a href="#Total_Temporal_Duration"><span style="color:blue">from 30 minutes ago to 2 hours ago</span></a></li>
+ </ul></li>
+</ul></div>
+<div class="section">
+<h6><a name="Skip_distance"></a><span style="color:green">Skip distance</span></h6>
+<p>Skip distance is the amount of time between temporal window beginnings that the next window starts. It is, in effect, the window period. </p>
+<p>It is specified by the phrase <tt>EVERY time_interval</tt></p>
+
+<ul>
+
+<li><tt>time_interval</tt> - A time amount followed by a unit (e.g. 1 hour). Fractional amounts are not supported. The unit may be “minute”, “day”, “hour” with any pluralization.</li>
+
+<li><tt>EVERY</tt> - The word/phrase “every” or “for every”</li>
+</ul>
+<p><b>Examples</b></p>
+
+<ul>
+
+<li>A repeating 30 minute window starting 2 hours ago and repeating every hour until now. This would result in 2 30-minute wide windows: 2 hours ago and 1 hour ago
+
+<ul>
+
+<li><a href="#Temporal_Window_Width"><span style="color:red">30 minute window</span></a> <a href="#Skip_distance"><span style="color:green">every 1 hour</span></a> <a href="#Total_Temporal_Duration"><span style="color:blue">starting from 2 hours ago </span></a></li>
+
+<li><a href="#Temporal_Window_Width"><span style="color:red">30 minutes window</span></a> <a href="#Skip_distance"><span style="color:green">every 1 hour</span></a> <a href="#Total_Temporal_Duration"><span style="color:blue">from 2 hours ago </span></a></li>
+ </ul></li>
+
+<li>A repeating 30 minute window starting 2 hours ago and repeating every hour until 30 minutes ago. This would result in 2 30-minute wide windows: 2 hours ago and 1 hour ago
+
+<ul>
+
+<li><a href="#Temporal_Window_Width"><span style="color:red">30 minute window</span></a> <a href="#Skip_distance"><span style="color:green">every 1 hour</span></a> <a href="#Total_Temporal_Duration"><span style="color:blue">starting from 2 hours ago until 30 minutes ago</span></a></li>
+
+<li><a href="#Temporal_Window_Width"><span style="color:red">30 minutes window</span></a> <a href="#Skip_distance"><span style="color:green">every 1 hour</span></a> <a href="#Total_Temporal_Duration"><span style="color:blue">from 2 hours ago to 30 minutes ago</span></a></li>
+
+<li><a href="#Temporal_Window_Width"><span style="color:red">30 minutes window</span></a> <a href="#Skip_distance"><span style="color:green">for every 1 hour</span></a> <a href="#Total_Temporal_Duration"><span style="color:blue">from 30 minutes ago to 2 hours ago</span></a></li>
+ </ul></li>
+</ul></div>
+<div class="section">
+<h6><a name="InclusionExclusion_specifiers"></a><span style="color:purple">Inclusion/Exclusion specifiers</span></h6>
+<p>Inclusion and Exclusion specifiers operate as filters on the set of windows. They operate on the window beginning timestamp.</p>
+<p>For inclusion specifiers, windows who are passed by <i>any</i> of the set of inclusion specifiers are included.<br />inclusion specifiers. Similarly, windows who are passed by <i>any</i> of the set of exclusion specifiers are excluded. Exclusion specifiers trump inclusion specifiers.</p>
+<p>Specifiers follow one of the following formats depending on if it is an inclusion or exclusion specifier:</p>
+
+<ul>
+
+<li><tt>INCLUSION specifier, specifier, ...</tt>
+
+<ul>
+
+<li><tt>INCLUSION</tt> can be “include”, “includes” or “including”</li>
+ </ul></li>
+
+<li><tt>EXCLUSION specifier, specifier, ...</tt>
+
+<ul>
+
+<li><tt>EXCLUSION</tt> can be “exclude”, “excludes” or “excluding”</li>
+ </ul></li>
+</ul>
+<p>The specifiers are a set of fixed specifiers available as part of the language:</p>
+
+<ul>
+
+<li>Fixed day of week-based specifiers - includes or excludes if the window is on the specified day of the week
+
+<ul>
+
+<li>“monday” or “mondays”</li>
+
+<li>“tuesday” or “tuesdays”</li>
+
+<li>“wednesday” or “wednesdays”</li>
+
+<li>“thursday” or “thursdays”</li>
+
+<li>“friday” or “fridays”</li>
+
+<li>“saturday” or “saturdays”</li>
+
+<li>“sunday” or “sundays”</li>
+
+<li>“weekday” or “weekdays”</li>
+
+<li>“weekend” or "“weekends”</li>
+ </ul></li>
+
+<li>Relative day of week-based specifiers - includes or excludes based on the day of week relative to now
+
+<ul>
+
+<li>“current day of the week”</li>
+
+<li>“current day of week”</li>
+
+<li>“this day of the week”</li>
+
+<li>“this day of week”</li>
+ </ul></li>
+
+<li>Specified date - includes or excludes based on the specified date
+
+<ul>
+
+<li>“date” - Takes up to 2 arguments
+
+<ul>
+
+<li>The day in <tt>yyyy/MM/dd</tt> format if no second argument is provided</li>
+
+<li>Optionally the format to specify the first argument in</li>
+
+<li>Example: <tt>date:2017/12/25</tt> would include or exclude December 25, 2017</li>
+
+<li>Example: <tt>date:20171225:yyyyMMdd</tt> would include or exclude December 25, 2017</li>
+ </ul></li>
+ </ul></li>
+
+<li>Holidays - includes or excludes based on if the window starts during a holiday
+
+<ul>
+
+<li>“holiday” or “holidays”
+
+<ul>
+
+<li>Arguments form the jollyday hierarchy of holidays. e.g. “us:nyc” would be holidays for New York City, USA</li>
+
+<li>If none is specified, it will choose based on locale.</li>
+
+<li>Countries supported are those supported in <a class="externalLink" href="https://github.com/svendiedrichsen/jollyday/tree/master/src/main/resources/holidays">jollyday</a></li>
+
+<li>Example: <tt>holiday:us:nyc</tt> would be the holidays of New York City, USA</li>
+
+<li>Example: <tt>holiday:hu</tt> would be the holidays of Hungary</li>
+ </ul></li>
+ </ul></li>
+</ul>
+<p><b>WARNING: Daylight Savings Time effects</b></p>
+<p>While Universal Time (UTC) is nice and constant, many servers are set to local timezones that enable Daylight Savings Time (DST). This means that twice a year, on DST transition weekends, “Sunday” is either 23 or 25 hours long. However, durations specified as “7 days ago” are always interpreted as “7*24 hours ago”. This can lead to some surprising effects when using days of the week as inclusion or exclusion specifiers.</p>
+<p>For example, the profile window specified by the phrase “30 minute window every 24 hours from 7 days ago” will always have 7 thirty-minute intervals, and these will normally occur on 5 weekdays and 2 weekend days. However, if you invoke this window at 12:15am any day during the week following the start of DST, you will get these intervals (supposing you start early on a Wednesday morning):</p>
+
+<div class="source">
+<div class="source">
+<pre>Tuesday 12:15am-12:45am (yesterday)
+Monday 12:15am-12:45am
+Saturday 11:15pm-11:45pm (skipped Sunday!)
+Friday 11:15pm-11:45pm
+Thursday 11:15pm-11:45pm
+Wednesday 11:15pm-11:45pm
+Tuesday 11:15pm-11:45pm
+</pre></div></div>
+<p>Sunday got skipped over because it was only 23 hours long; that is, there were 24 hours between Saturday 11:15pm and Monday 12:15am. So if you specified “excluding weekends”, you would get 6 days’ intervals instead of the expected 5. There are multiple variations on this theme.</p>
+<p>Remember that the underlying time is kept in UTC, so the data is always correct. It is only when attempting to interpret UTC as local time, date, and day, that these confusions may occur. They may be eliminated by setting your server timezone to UTC, or otherwise disabling DST.</p>
+<p><b>Examples</b></p>
+<p>Assume these are executed at noon.</p>
+
+<ul>
+
+<li>A 1 hour window for the past 8 ‘current day of the week’
+
+<ul>
+
+<li><a href="#Temporal_Window_Width"><span style="color:red">1 hour window</span></a> <a href="#Skip_distance"><span style="color:green">every 24 hours</span></a> <a href="#Total_Temporal_Duration"><span style="color:blue">from 56 days ago</span></a> <a href="#InclusionExclusion_specifiers"><span style="color:purple">including this day of the week</span></a></li>
+ </ul></li>
+
+<li>A 1 hour window for the past 8 tuesdays
+
+<ul>
+
+<li><a href="#Temporal_Window_Width"><span style="color:red">1 hour window</span></a> <a href="#Skip_distance"><span style="color:green">every 24 hours</span></a> <a href="#Total_Temporal_Duration"><span style="color:blue">from 56 days ago</span></a> <a href="#InclusionExclusion_specifiers"><span style="color:purple">including tuesdays</span></a></li>
+ </ul></li>
+
+<li>A 30 minute window every tuesday at noon starting 14 days ago until now
+
+<ul>
+
+<li><a href="#Temporal_Window_Width"><span style="color:red">30 minute window</span></a> <a href="#Skip_distance"><span style="color:green">every 24 hours</span></a> <a href="#Total_Temporal_Duration"><span style="color:blue">from 14 days ago</span></a> <a href="#InclusionExclusion_specifiers"><span style="color:purple">including tuesdays</span></a></li>
+ </ul></li>
+
+<li>A 30 minute window every day except holidays and weekends at noon starting 14 days ago until now
+
+<ul>
+
+<li><a href="#Temporal_Window_Width"><span style="color:red">30 minute window</span></a> <a href="#Skip_distance"><span style="color:green">every 24 hours</span></a> <a href="#Total_Temporal_Duration"><span style="color:blue">from 14 days ago</span></a> <a href="#InclusionExclusion_specifiers"><span style="color:purple">excluding holidays:us, weekends</span></a></li>
+
+<li><a href="#Temporal_Window_Width"><span style="color:red">30 minute window</span></a> <a href="#Skip_distance"><span style="color:green">every 24 hours</span></a> <a href="#Total_Temporal_Duration"><span style="color:blue">from 14 days ago</span></a> <a href="#InclusionExclusion_specifiers"><span style="color:purple">including weekdays excluding holidays:us, weekends</span></a></li>
+ </ul></li>
+
+<li>A 30 minute window at noon every day from 7 days ago including saturdays and excluding weekends. Because exclusions trump inclusions, the following will never yield any windows
+
+<ul>
+
+<li><a href="#Temporal_Window_Width"><span style="color:red">30 minute window</span></a> <a href="#Skip_distance"><span style="color:green">every 24 hours</span></a> <a href="#Total_Temporal_Duration"><span style="color:blue">from 7 days ago</span></a> <a href="#InclusionExclusion_specifiers"><span style="color:purple">including saturdays excluding weekends</span></a></li>
+ </ul></li>
+</ul></div></div></div></div>
<div class="section">
<h3><a name="Errors"></a>Errors</h3>
-<p>The most common result of incorrect PROFILE_GET arguments or Client configuration parameters is an empty result set, rather than an error. The Client cannot effectively validate the arguments, because the Profiler configuration parameters may be changed and the profile itself does not store them. The person doing the querying must carry forward the knowledge of the Profiler configuration parameters from the time of profile creation, and use corresponding PROFILE_GET arguments and Client configuration parameters when querying the data.</p></div>
+<p>The most common result of incorrect <tt>PROFILE_GET</tt> arguments or Client configuration parameters is an empty result set, rather than an error. The Client cannot effectively validate the arguments, because the Profiler configuration parameters may be changed and the profile itself does not store them. The person doing the querying must carry forward the knowledge of the Profiler configuration parameters from the time of profile creation, and use corresponding <tt>PROFILE_GET</tt> arguments and Client configuration parameters when querying the data.</p></div>
<div class="section">
<h3><a name="Examples"></a>Examples</h3>
-<p>Retrieve all values of ‘snort-alerts’ from ‘10.0.0.1’ over the past 4 hours.</p>
+<p>The following are usage examples that show how the Stellar API can be used to read profiles generated by the <a href="../metron-profiler/index.html">Metron Profiler</a>. This API would be used in conjunction with other Stellar functions like <a href="../../metron-platform/metron-common/index.html#MAAS_MODEL_APPLY"><tt>MAAS_MODEL_APPLY</tt></a> to perform model scoring on streaming data.</p>
+<p>These examples assume a profile has been defined called ‘snort-alerts’ that tracks the number of Snort alerts associated with an IP address over time. The profile definition might look similar to the following.</p>
+
+<div class="source">
+<div class="source">
+<pre>{
+ "profiles": [
+ {
+ "profile": "snort-alerts",
+ "foreach": "ip_src_addr",
+ "onlyif": "source.type == 'snort'",
+ "update": { "s": "STATS_ADD(s, 1)" },
+ "result": "STATS_MEAN(s)"
+ }
+ ]
+}
+</pre></div></div>
+<p>During model scoring the entity being scored, in this case a particular IP address, will be known. The following examples shows how this profile data might be retrieved. Retrieve all values of ‘snort-alerts’ from ‘10.0.0.1’ over the past 4 hours.</p>
<div class="source">
<div class="source">
-<pre>PROFILE_GET('snort-alerts', '10.0.0.1', 4, 'HOURS')
+<pre>PROFILE_GET('snort-alerts', '10.0.0.1', PROFILE_FIXED(4, 'HOURS'))
</pre></div></div>
<p>Retrieve all values of ‘snort-alerts’ from ‘10.0.0.1’ over the past 2 days.</p>
<div class="source">
<div class="source">
-<pre>PROFILE_GET('snort-alerts', '10.0.0.1', 2, 'DAYS')
+<pre>PROFILE_GET('snort-alerts', '10.0.0.1', PROFILE_FIXED(2, 'DAYS'))
</pre></div></div>
<p>If the profile had been defined to group the data by weekday versus weekend, then the following example would apply:</p>
-<p>Retrieve all values of ‘snort-alerts’ from ‘10.0.0.1’ that occurred on ‘weekdays’ over the past month.</p>
+<p>Retrieve all values of ‘snort-alerts’ from ‘10.0.0.1’ that occurred on ‘weekdays’ over the past 30 days.</p>
<div class="source">
<div class="source">
-<pre>PROFILE_GET('snort-alerts', '10.0.0.1', 1, 'MONTHS', ['weekdays'] )
+<pre>PROFILE_GET('snort-alerts', '10.0.0.1', PROFILE_FIXED(30, 'DAYS'), ['weekdays'] )
</pre></div></div>
<p>The client may need to use a configuration different from the current Client configuration settings. For example, perhaps you are on a cluster shared with other analysts, and need to access a profile that was constructed 2 months ago using different period duration, while they are accessing more recent profiles constructed with the currently configured period duration. For this situation, you may use the <tt>config_overrides</tt> argument:</p>
<p>Retrieve all values of ‘snort-alerts’ from ‘10.0.0.1’ over the past 2 days, with no <tt>groupBy</tt>, and overriding the usual global client configuration parameters for window duration.</p>
<div class="source">
<div class="source">
-<pre>PROFILE_GET('profile1', 'entity1', 2, 'DAYS', [], {'profiler.client.period.duration' : '2', 'profiler.client.period.duration.units' : 'MINUTES'})
+<pre>PROFILE_GET('profile1', 'entity1', PROFILE_FIXED(2, 'DAYS', {'profiler.client.period.duration' : '2', 'profiler.client.period.duration.units' : 'MINUTES'}), [])
</pre></div></div>
-<p>Retrieve all values of ‘snort-alerts’ from ‘10.0.0.1’ that occurred on ‘weekdays’ over the past month, overriding the usual global client configuration parameters for window duration.</p>
+<p>Retrieve all values of ‘snort-alerts’ from ‘10.0.0.1’ that occurred on ‘weekdays’ over the past 30 days, overriding the usual global client configuration parameters for window duration.</p>
<div class="source">
<div class="source">
-<pre>PROFILE_GET('profile1', 'entity1', 1, 'MONTHS', ['weekdays'], {'profiler.client.period.duration' : '2', 'profiler.client.period.duration.units' : 'MINUTES'})
+<pre>PROFILE_GET('profile1', 'entity1', PROFILE_FIXED(30, 'DAYS', {'profiler.client.period.duration' : '2', 'profiler.client.period.duration.units' : 'MINUTES'}), ['weekdays'] )
</pre></div></div></div></div>
<div class="section">
<h2><a name="Getting_Started"></a>Getting Started</h2>
@@ -414,8 +786,9 @@ Returns: The selected profile measurements.
<footer>
<div class="container-fluid">
- <div class="row span12">Copyright © 2017.
- All Rights Reserved.
+ <div class="row span12">Copyright © 2017
+ <a href="https://www.apache.org">The Apache Software Foundation</a>.
+ All Rights Reserved.
</div>
http://git-wip-us.apache.org/repos/asf/metron/blob/cf4c2ecd/current-book/metron-analytics/metron-profiler/index.html
----------------------------------------------------------------------
diff --git a/current-book/metron-analytics/metron-profiler/index.html b/current-book/metron-analytics/metron-profiler/index.html
index adb7ffb..2f727dc 100644
--- a/current-book/metron-analytics/metron-profiler/index.html
+++ b/current-book/metron-analytics/metron-profiler/index.html
@@ -1,13 +1,13 @@
<!DOCTYPE html>
<!--
- | Generated by Apache Maven Doxia at 2017-02-23
+ | Generated by Apache Maven Doxia at 2017-06-27
| Rendered using Apache Maven Fluido Skin 1.3.0
-->
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
- <meta name="Date-Revision-yyyymmdd" content="20170223" />
+ <meta name="Date-Revision-yyyymmdd" content="20170627" />
<meta http-equiv="Content-Language" content="en" />
<title>Metron – Metron Profiler</title>
<link rel="stylesheet" href="../../css/apache-maven-fluido-1.3.0.min.css" />
@@ -30,14 +30,11 @@
<div class="container-fluid">
<div id="banner">
<div class="pull-left">
- <a href="http://metron.incubator.apache.org/" id="bannerLeft">
- <img src="../../images/metron-logo.png" alt="Apache Metron - Incubating" width="148px" height="48px"/>
+ <a href="http://metron.apache.org/" id="bannerLeft">
+ <img src="../../images/metron-logo.png" alt="Apache Metron" width="148px" height="48px"/>
</a>
</div>
- <div class="pull-right"> <a href="http://incubator.apache.org/" id="bannerRight">
- <img src="../../images/ApacheIncubating_Logo.png" alt="Apache Incubating" width="192px" height="48px"/>
- </a>
- </div>
+ <div class="pull-right"> </div>
<div class="clear"><hr/></div>
</div>
@@ -51,8 +48,8 @@
</li>
<li class="divider ">/</li>
<li class="">
- <a href="http://metron.incubator.apache.org/" class="externalLink" title="Metron-Incubating">
- Metron-Incubating</a>
+ <a href="http://metron.apache.org/" class="externalLink" title="Metron">
+ Metron</a>
</li>
<li class="divider ">/</li>
<li class="">
@@ -64,8 +61,8 @@
- <li id="publishDate" class="pull-right">Last Published: 2017-02-23</li> <li class="divider pull-right">|</li>
- <li id="projectVersion" class="pull-right">Version: 0.3.1</li>
+ <li id="publishDate" class="pull-right">Last Published: 2017-06-27</li> <li class="divider pull-right">|</li>
+ <li id="projectVersion" class="pull-right">Version: 0.4.0</li>
</ul>
</div>
@@ -78,7 +75,7 @@
<ul class="nav nav-list">
<li class="nav-header">User Documentation</li>
-
+
<li>
<a href="../../index.html" title="Metron">
@@ -127,7 +124,7 @@
</li>
</ul>
</li>
-
+
<li>
<a href="../../metron-deployment/index.html" title="Deployment">
@@ -141,14 +138,28 @@
<i class="none"></i>
Docker</a>
</li>
-
+
+ <li>
+
+ <a href="../../metron-interface/metron-config/index.html" title="Config">
+ <i class="none"></i>
+ Config</a>
+ </li>
+
+ <li>
+
+ <a href="../../metron-interface/metron-rest/index.html" title="Rest">
+ <i class="none"></i>
+ Rest</a>
+ </li>
+
<li>
<a href="../../metron-platform/index.html" title="Platform">
<i class="icon-chevron-right"></i>
Platform</a>
</li>
-
+
<li>
<a href="../../metron-sensors/index.html" title="Sensors">
@@ -271,8 +282,13 @@ hbase(main):001:0> count 'profiler'
<li>
<p>Use the Profiler Client to read the profile data. The below example <tt>PROFILE_GET</tt> command will read data written by the sample profile given above, if 10.0.0.1 is one of the input values for <tt>ip_src_addr</tt>. More information on configuring and using the client can be found <a href="../metron-profiler-client/index.html">here</a>. It is assumed that the <tt>PROFILE_GET</tt> client is correctly configured before using it.</p>
-<p>``` $ bin/stellar -z node1:2181</p>
-<p>[Stellar]>>> PROFILE_GET( “test”, “10.0.0.1”, PROFILE_FIXED(30, “MINUTES”)) [451, 448] ```</p></li>
+
+<div class="source">
+<div class="source">
+<pre>$ bin/stellar -z node1:2181
+[Stellar]>>> PROFILE_GET( "test", "10.0.0.1", PROFILE_FIXED(30, "MINUTES"))
+[451, 448]
+</pre></div></div></li>
</ol></div>
<div class="section">
<h2><a name="Creating_Profiles"></a>Creating Profiles</h2>
@@ -285,9 +301,9 @@ hbase(main):001:0> count 'profiler'
<th>Name </th>
-<th>Required </th>
+<th> </th>
-<th>Description </th>
+<th>Description</th>
</tr>
</thead>
<tbody>
@@ -298,7 +314,7 @@ hbase(main):001:0> count 'profiler'
<td>Required </td>
-<td>Unique name identifying the profile. </td>
+<td>Unique name identifying the profile.</td>
</tr>
<tr class="a">
@@ -307,7 +323,7 @@ hbase(main):001:0> count 'profiler'
<td>Required </td>
-<td>A separate profile is maintained “for each” of these. </td>
+<td>A separate profile is maintained “for each” of these.</td>
</tr>
<tr class="b">
@@ -316,7 +332,7 @@ hbase(main):001:0> count 'profiler'
<td>Optional </td>
-<td>Boolean expression that determines if a message should be applied to the profile. </td>
+<td>Boolean expression that determines if a message should be applied to the profile.</td>
</tr>
<tr class="a">
@@ -325,7 +341,7 @@ hbase(main):001:0> count 'profiler'
<td>Optional </td>
-<td>One or more Stellar expressions used to group the profile measurements when persisted. </td>
+<td>One or more Stellar expressions used to group the profile measurements when persisted.</td>
</tr>
<tr class="b">
@@ -334,7 +350,7 @@ hbase(main):001:0> count 'profiler'
<td>Optional </td>
-<td>One or more expressions executed at the start of a window period. </td>
+<td>One or more expressions executed at the start of a window period.</td>
</tr>
<tr class="a">
@@ -343,7 +359,7 @@ hbase(main):001:0> count 'profiler'
<td>Required </td>
-<td>One or more expressions executed when a message is applied to the profile. </td>
+<td>One or more expressions executed when a message is applied to the profile.</td>
</tr>
<tr class="b">
@@ -352,7 +368,7 @@ hbase(main):001:0> count 'profiler'
<td>Required </td>
-<td>A Stellar expression that is executed when the window period expires. </td>
+<td>Stellar expressions that are executed when the window period expires.</td>
</tr>
<tr class="a">
@@ -361,7 +377,7 @@ hbase(main):001:0> count 'profiler'
<td>Optional </td>
-<td>Profile data is purged after this period of time, specified in milliseconds. </td>
+<td>Profile data is purged after this period of time, specified in milliseconds.</td>
</tr>
</tbody>
</table>
@@ -415,7 +431,49 @@ hbase(main):001:0> count 'profiler'
<div class="section">
<h3><a name="result"></a><tt>result</tt></h3>
<p><i>Required</i></p>
-<p>A Stellar expression that is executed when the window period expires. The expression is expected to summarize the messages that were applied to the profile over the window period, using the state accumulated by the updates. The result will typically be a single numeric value, but it may be any serializable object, as shown in Example 4 below. </p></div>
+<p>Stellar expressions that are executed when the window period expires. The expressions are expected to summarize the messages that were applied to the profile over the window period. In the most basic form a single result is persisted for later retrieval.</p>
+
+<div class="source">
+<div class="source">
+<pre>"result": "var1 + var2"
+</pre></div></div>
+<p>For more advanced use cases, a profile can generate two types of results. A profile can define one or both of these result types at the same time. </p>
+
+<ul>
+
+<li><tt>profile</tt>: A required expression that defines a value that is persisted for later retrieval.</li>
+
+<li><tt>triage</tt>: An optional expression that defines values that are accessible within the Threat Triage process.</li>
+</ul>
+<p><b>profile</b></p>
+<p>A required Stellar expression that results in a value that is persisted in the profile store for later retrieval. The expression can result in any object that is Kryo serializable. These values can be retrieved for later use with the <a href="../metron-profiler-client/index.html">Profiler Client</a>. </p>
+
+<div class="source">
+<div class="source">
+<pre>"result": {
+ "profile": "2 + 2"
+}
+</pre></div></div>
+<p>An alternative, simplified form is also acceptable.</p>
+
+<div class="source">
+<div class="source">
+<pre>"result": "2 + 2"
+</pre></div></div>
+<p><b>triage</b></p>
+<p>An optional map of one or more Stellar expressions. The value of each expression is made available to the Threat Triage process under the given name. Each expression must result in a either a primitive type, like an integer, long, or short, or a String. All other types will result in an error.</p>
+<p>In the following example, three values, the minimum, the maximum and the mean are appended to a message. This message is consumed by Metron, like other sources of telemetry, and each of these values are accessible from within the Threat Triage process using the given field names; <tt>min</tt>, <tt>max</tt>, and <tt>mean</tt>.</p>
+
+<div class="source">
+<div class="source">
+<pre>"result": {
+ "triage": {
+ "min": "STATS_MIN(stats)",
+ "max": "STATS_MAX(stats)",
+ "mean": "STATS_MEAN(stats)"
+ }
+}
+</pre></div></div></div>
<div class="section">
<h3><a name="expires"></a><tt>expires</tt></h3>
<p><i>Optional</i></p>
@@ -431,7 +489,7 @@ hbase(main):001:0> count 'profiler'
<th>Setting </th>
-<th>Description </th>
+<th>Description</th>
</tr>
</thead>
<tbody>
@@ -440,84 +498,91 @@ hbase(main):001:0> count 'profiler'
<td>profiler.workers </td>
-<td>The number of worker processes to create for the topology. </td>
+<td>The number of worker processes to create for the topology.</td>
</tr>
<tr class="a">
<td>profiler.executors </td>
-<td>The number of executors to spawn per component. </td>
+<td>The number of executors to spawn per component.</td>
</tr>
<tr class="b">
<td>profiler.input.topic </td>
-<td>The name of the Kafka topic from which to consume data. </td>
+<td>The name of the Kafka topic from which to consume data.</td>
</tr>
<tr class="a">
-<td>profiler.period.duration </td>
+<td>profiler.output.topic </td>
-<td>The duration of each profile period. This value should be defined along with <tt>profiler.period.duration.units</tt>. </td>
+<td>The name of the Kafka topic to which profile data is written. Only used with profiles that use the <a href="#result"><tt>triage</tt> result field</a>.</td>
</tr>
<tr class="b">
-<td>profiler.period.duration.units </td>
+<td>profiler.period.duration </td>
-<td>The units used to specify the <tt>profiler.period.duration</tt>. </td>
+<td>The duration of each profile period. This value should be defined along with <tt>profiler.period.duration.units</tt>.</td>
</tr>
<tr class="a">
-<td>profiler.ttl </td>
+<td>profiler.period.duration.units </td>
-<td>If a message has not been applied to a Profile in this period of time, the Profile will be forgotten and its resources will be cleaned up. This value should be defined along with <tt>profiler.ttl.units</tt>. </td>
+<td>The units used to specify the <tt>profiler.period.duration</tt>.</td>
</tr>
<tr class="b">
-<td>profiler.ttl.units </td>
+<td>profiler.ttl </td>
-<td>The units used to specify the <tt>profiler.ttl</tt>. </td>
+<td>If a message has not been applied to a Profile in this period of time, the Profile will be forgotten and its resources will be cleaned up. This value should be defined along with <tt>profiler.ttl.units</tt>.</td>
</tr>
<tr class="a">
-<td>profiler.hbase.salt.divisor </td>
+<td>profiler.ttl.units </td>
-<td>A salt is prepended to the row key to help prevent hotspotting. This constant is used to generate the salt. Ideally, this constant should be roughly equal to the number of nodes in the Hbase cluster. </td>
+<td>The units used to specify the <tt>profiler.ttl</tt>.</td>
</tr>
<tr class="b">
-<td>profiler.hbase.table </td>
+<td>profiler.hbase.salt.divisor </td>
-<td>The name of the HBase table that profiles are written to. </td>
+<td>A salt is prepended to the row key to help prevent hotspotting. This constant is used to generate the salt. Ideally, this constant should be roughly equal to the number of nodes in the Hbase cluster.</td>
</tr>
<tr class="a">
-<td>profiler.hbase.column.family </td>
+<td>profiler.hbase.table </td>
-<td>The column family used to store profiles. </td>
+<td>The name of the HBase table that profiles are written to.</td>
</tr>
<tr class="b">
-<td>profiler.hbase.batch </td>
+<td>profiler.hbase.column.family </td>
-<td>The number of puts that are written in a single batch. </td>
+<td>The column family used to store profiles.</td>
</tr>
<tr class="a">
+<td>profiler.hbase.batch </td>
+
+<td>The number of puts that are written in a single batch.</td>
+ </tr>
+
+<tr class="b">
+
<td>profiler.hbase.flush.interval.seconds </td>
-<td>The maximum number of seconds between batch writes to HBase. </td>
+<td>The maximum number of seconds between batch writes to HBase.</td>
</tr>
</tbody>
</table>
@@ -766,8 +831,9 @@ hbase(main):001:0> count 'profiler'
<footer>
<div class="container-fluid">
- <div class="row span12">Copyright © 2017.
- All Rights Reserved.
+ <div class="row span12">Copyright © 2017
+ <a href="https://www.apache.org">The Apache Software Foundation</a>.
+ All Rights Reserved.
</div>
http://git-wip-us.apache.org/repos/asf/metron/blob/cf4c2ecd/current-book/metron-analytics/metron-statistics/HLLP.html
----------------------------------------------------------------------
diff --git a/current-book/metron-analytics/metron-statistics/HLLP.html b/current-book/metron-analytics/metron-statistics/HLLP.html
index 7d389ab..0017621 100644
--- a/current-book/metron-analytics/metron-statistics/HLLP.html
+++ b/current-book/metron-analytics/metron-statistics/HLLP.html
@@ -1,13 +1,13 @@
<!DOCTYPE html>
<!--
- | Generated by Apache Maven Doxia at 2017-02-23
+ | Generated by Apache Maven Doxia at 2017-06-27
| Rendered using Apache Maven Fluido Skin 1.3.0
-->
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
- <meta name="Date-Revision-yyyymmdd" content="20170223" />
+ <meta name="Date-Revision-yyyymmdd" content="20170627" />
<meta http-equiv="Content-Language" content="en" />
<title>Metron – HyperLogLogPlus</title>
<link rel="stylesheet" href="../../css/apache-maven-fluido-1.3.0.min.css" />
@@ -30,14 +30,11 @@
<div class="container-fluid">
<div id="banner">
<div class="pull-left">
- <a href="http://metron.incubator.apache.org/" id="bannerLeft">
- <img src="../../images/metron-logo.png" alt="Apache Metron - Incubating" width="148px" height="48px"/>
+ <a href="http://metron.apache.org/" id="bannerLeft">
+ <img src="../../images/metron-logo.png" alt="Apache Metron" width="148px" height="48px"/>
</a>
</div>
- <div class="pull-right"> <a href="http://incubator.apache.org/" id="bannerRight">
- <img src="../../images/ApacheIncubating_Logo.png" alt="Apache Incubating" width="192px" height="48px"/>
- </a>
- </div>
+ <div class="pull-right"> </div>
<div class="clear"><hr/></div>
</div>
@@ -51,8 +48,8 @@
</li>
<li class="divider ">/</li>
<li class="">
- <a href="http://metron.incubator.apache.org/" class="externalLink" title="Metron-Incubating">
- Metron-Incubating</a>
+ <a href="http://metron.apache.org/" class="externalLink" title="Metron">
+ Metron</a>
</li>
<li class="divider ">/</li>
<li class="">
@@ -64,8 +61,8 @@
- <li id="publishDate" class="pull-right">Last Published: 2017-02-23</li> <li class="divider pull-right">|</li>
- <li id="projectVersion" class="pull-right">Version: 0.3.1</li>
+ <li id="publishDate" class="pull-right">Last Published: 2017-06-27</li> <li class="divider pull-right">|</li>
+ <li id="projectVersion" class="pull-right">Version: 0.4.0</li>
</ul>
</div>
@@ -78,7 +75,7 @@
<ul class="nav nav-list">
<li class="nav-header">User Documentation</li>
-
+
<li>
<a href="../../index.html" title="Metron">
@@ -136,7 +133,7 @@
</li>
</ul>
</li>
-
+
<li>
<a href="../../metron-deployment/index.html" title="Deployment">
@@ -150,14 +147,28 @@
<i class="none"></i>
Docker</a>
</li>
-
+
+ <li>
+
+ <a href="../../metron-interface/metron-config/index.html" title="Config">
+ <i class="none"></i>
+ Config</a>
+ </li>
+
+ <li>
+
+ <a href="../../metron-interface/metron-rest/index.html" title="Rest">
+ <i class="none"></i>
+ Rest</a>
+ </li>
+
<li>
<a href="../../metron-platform/index.html" title="Platform">
<i class="icon-chevron-right"></i>
Platform</a>
</li>
-
+
<li>
<a href="../../metron-sensors/index.html" title="Sensors">
@@ -201,7 +212,7 @@
<p>Below is a table detailing the performance characteristics when running HLLP over multiple data set sizes. There are three primary variables manipulated during the test: cardinality, sparse set precision, and normal set precision. Each of these values will have an impact on error rates, execution time, and total memory consumption.</p></div>
<div class="section">
<h3><a name="Choosing_p_sp"></a>Choosing p, sp</h3>
-<p>There is a tradeoff when choosing values for p and sp. A higher value for sp means a lower error rate for instances when the sparse set is being used. However, a higher value for sp also requires more memory, which means the algorithm will cut over to the dense representation more quickly. The Google paper (listed above) recommends settings of p=25, sp=14 for optimal memory use and algorithm accuracy, and these are the defaults provided by the Apache Metron (Incubating) implementation in Stellar. You’ll notice in the tests below, memory consumption tops out around 12KB for this setting, regardless of cardinality size.</p></div>
+<p>There is a tradeoff when choosing values for p and sp. A higher value for sp means a lower error rate for instances when the sparse set is being used. However, a higher value for sp also requires more memory, which means the algorithm will cut over to the dense representation more quickly. The Google paper (listed above) recommends settings of p=25, sp=14 for optimal memory use and algorithm accuracy, and these are the defaults provided by the Apache Metron implementation in Stellar. You’ll notice in the tests below, memory consumption tops out around 12KB for this setting, regardless of cardinality size.</p></div>
<div class="section">
<h3><a name="Key"></a>Key</h3>
<hr />
@@ -7574,8 +7585,9 @@ format err as %: true
<footer>
<div class="container-fluid">
- <div class="row span12">Copyright © 2017.
- All Rights Reserved.
+ <div class="row span12">Copyright © 2017
+ <a href="https://www.apache.org">The Apache Software Foundation</a>.
+ All Rights Reserved.
</div>
http://git-wip-us.apache.org/repos/asf/metron/blob/cf4c2ecd/current-book/metron-analytics/metron-statistics/index.html
----------------------------------------------------------------------
diff --git a/current-book/metron-analytics/metron-statistics/index.html b/current-book/metron-analytics/metron-statistics/index.html
index c167eaf..c663271 100644
--- a/current-book/metron-analytics/metron-statistics/index.html
+++ b/current-book/metron-analytics/metron-statistics/index.html
@@ -1,13 +1,13 @@
<!DOCTYPE html>
<!--
- | Generated by Apache Maven Doxia at 2017-02-23
+ | Generated by Apache Maven Doxia at 2017-06-27
| Rendered using Apache Maven Fluido Skin 1.3.0
-->
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
- <meta name="Date-Revision-yyyymmdd" content="20170223" />
+ <meta name="Date-Revision-yyyymmdd" content="20170627" />
<meta http-equiv="Content-Language" content="en" />
<title>Metron – Statistics and Mathematical Functions</title>
<link rel="stylesheet" href="../../css/apache-maven-fluido-1.3.0.min.css" />
@@ -30,14 +30,11 @@
<div class="container-fluid">
<div id="banner">
<div class="pull-left">
- <a href="http://metron.incubator.apache.org/" id="bannerLeft">
- <img src="../../images/metron-logo.png" alt="Apache Metron - Incubating" width="148px" height="48px"/>
+ <a href="http://metron.apache.org/" id="bannerLeft">
+ <img src="../../images/metron-logo.png" alt="Apache Metron" width="148px" height="48px"/>
</a>
</div>
- <div class="pull-right"> <a href="http://incubator.apache.org/" id="bannerRight">
- <img src="../../images/ApacheIncubating_Logo.png" alt="Apache Incubating" width="192px" height="48px"/>
- </a>
- </div>
+ <div class="pull-right"> </div>
<div class="clear"><hr/></div>
</div>
@@ -51,8 +48,8 @@
</li>
<li class="divider ">/</li>
<li class="">
- <a href="http://metron.incubator.apache.org/" class="externalLink" title="Metron-Incubating">
- Metron-Incubating</a>
+ <a href="http://metron.apache.org/" class="externalLink" title="Metron">
+ Metron</a>
</li>
<li class="divider ">/</li>
<li class="">
@@ -64,8 +61,8 @@
- <li id="publishDate" class="pull-right">Last Published: 2017-02-23</li> <li class="divider pull-right">|</li>
- <li id="projectVersion" class="pull-right">Version: 0.3.1</li>
+ <li id="publishDate" class="pull-right">Last Published: 2017-06-27</li> <li class="divider pull-right">|</li>
+ <li id="projectVersion" class="pull-right">Version: 0.4.0</li>
</ul>
</div>
@@ -78,7 +75,7 @@
<ul class="nav nav-list">
<li class="nav-header">User Documentation</li>
-
+
<li>
<a href="../../index.html" title="Metron">
@@ -136,7 +133,7 @@
</li>
</ul>
</li>
-
+
<li>
<a href="../../metron-deployment/index.html" title="Deployment">
@@ -150,14 +147,28 @@
<i class="none"></i>
Docker</a>
</li>
-
+
+ <li>
+
+ <a href="../../metron-interface/metron-config/index.html" title="Config">
+ <i class="none"></i>
+ Config</a>
+ </li>
+
+ <li>
+
+ <a href="../../metron-interface/metron-rest/index.html" title="Rest">
+ <i class="none"></i>
+ Rest</a>
+ </li>
+
<li>
<a href="../../metron-platform/index.html" title="Platform">
<i class="icon-chevron-right"></i>
Platform</a>
</li>
-
+
<li>
<a href="../../metron-sensors/index.html" title="Sensors">
@@ -225,7 +236,7 @@
<li>hyperLogLogPlus - the hllp set</li>
</ul></li>
-<li>Returns: Long value representing the cardinality for this set</li>
+<li>Returns: Long value representing the cardinality for this set. Cardinality of a null set is 0.</li>
</ul></div>
<div class="section">
<h4><a name="HLLP_INIT"></a><tt>HLLP_INIT</tt></h4>
@@ -259,7 +270,7 @@
<li>hllp - List of hllp estimators to merge. Takes a single hllp set or a list.</li>
</ul></li>
-<li>Returns: A new merged HyperLogLogPlus estimator set</li>
+<li>Returns: A new merged HyperLogLogPlus estimator set. Passing an empty list returns null.</li>
</ul></div></div>
<div class="section">
<h3><a name="Mathematical_Functions"></a>Mathematical Functions</h3>
@@ -800,8 +811,6 @@ if __name__ == '__main__':
<div class="source">
<div class="source">
<pre>{
- "index": "mad",
- "batchSize": 1,
"enrichment": {
"fieldMap": {
"stellar" : {
@@ -828,6 +837,23 @@ PROFILE_GET( 'sketchy_mad', 'global', PROFILE_FIXED(10, 'MINUTES')) ), value)&qu
}
}
}
+</pre></div></div>
+<p>We also need an indexing configuration. Create the following in <tt>$METRON_HOME/config/zookeeper/indexing/mad.json</tt>:</p>
+
+<div class="source">
+<div class="source">
+<pre>{
+ "hdfs" : {
+ "index": "mad",
+ "batchSize": 1,
+ "enabled" : true
+ },
+ "elasticsearch" : {
+ "index": "mad",
+ "batchSize": 1,
+ "enabled" : true
+ }
+}
</pre></div></div></div>
<div class="section">
<h4><a name="The_Profiler"></a>The Profiler</h4>
@@ -903,8 +929,9 @@ PROFILE_GET( 'sketchy_mad', 'global', PROFILE_FIXED(10, 'MINUTES')) ), value)&qu
<footer>
<div class="container-fluid">
- <div class="row span12">Copyright © 2017.
- All Rights Reserved.
+ <div class="row span12">Copyright © 2017
+ <a href="https://www.apache.org">The Apache Software Foundation</a>.
+ All Rights Reserved.
</div>