You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@metron.apache.org by ce...@apache.org on 2017/02/09 14:12:18 UTC
svn commit: r18254 [8/13] - in /dev/incubator/metron/0.3.1-RC3-incubating:
./ book-site/ book-site/css/ book-site/image-archive/ book-site/images/
book-site/images/logos/ book-site/images/profiles/ book-site/img/
book-site/js/ book-site/metron-analytic...
Added: dev/incubator/metron/0.3.1-RC3-incubating/book-site/metron-deployment/roles/opentaxii/index.html
==============================================================================
--- dev/incubator/metron/0.3.1-RC3-incubating/book-site/metron-deployment/roles/opentaxii/index.html (added)
+++ dev/incubator/metron/0.3.1-RC3-incubating/book-site/metron-deployment/roles/opentaxii/index.html Thu Feb 9 14:12:17 2017
@@ -0,0 +1,419 @@
+<!DOCTYPE html>
+<!--
+ | Generated by Apache Maven Doxia at 2017-02-09
+ | Rendered using Apache Maven Fluido Skin 1.3.0
+-->
+<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
+ <head>
+ <meta charset="UTF-8" />
+ <meta name="viewport" content="width=device-width, initial-scale=1.0" />
+ <meta name="Date-Revision-yyyymmdd" content="20170209" />
+ <meta http-equiv="Content-Language" content="en" />
+ <title>Metron – OpenTAXII</title>
+ <link rel="stylesheet" href="../../../css/apache-maven-fluido-1.3.0.min.css" />
+ <link rel="stylesheet" href="../../../css/site.css" />
+ <link rel="stylesheet" href="../../../css/print.css" media="print" />
+
+
+ <script type="text/javascript" src="../../../js/apache-maven-fluido-1.3.0.min.js"></script>
+
+
+
+<script type="text/javascript">$( document ).ready( function() { $( '.carousel' ).carousel( { interval: 3500 } ) } );</script>
+
+ </head>
+ <body class="topBarDisabled">
+
+
+
+
+ <div class="container-fluid">
+ <div id="banner">
+ <div class="pull-left">
+ <a href="http://metron.incubator.apache.org/" id="bannerLeft">
+ <img src="../../../images/metron-logo.png" alt="Apache Metron - Incubating" width="148px" height="48px"/>
+ </a>
+ </div>
+ <div class="pull-right"> <a href="http://incubator.apache.org/" id="bannerRight">
+ <img src="../../../images/ApacheIncubating_Logo.png" alt="Apache Incubating" width="192px" height="48px"/>
+ </a>
+ </div>
+ <div class="clear"><hr/></div>
+ </div>
+
+ <div id="breadcrumbs">
+ <ul class="breadcrumb">
+
+
+ <li class="">
+ <a href="http://www.apache.org" class="externalLink" title="Apache">
+ Apache</a>
+ </li>
+ <li class="divider ">/</li>
+ <li class="">
+ <a href="http://metron.incubator.apache.org/" class="externalLink" title="Metron-Incubating">
+ Metron-Incubating</a>
+ </li>
+ <li class="divider ">/</li>
+ <li class="">
+ <a href="../../../index.html" title="Documentation">
+ Documentation</a>
+ </li>
+ <li class="divider ">/</li>
+ <li class="">OpenTAXII</li>
+
+
+
+ <li id="publishDate" class="pull-right">Last Published: 2017-02-09</li> <li class="divider pull-right">|</li>
+ <li id="projectVersion" class="pull-right">Version: 0.3.1</li>
+
+ </ul>
+ </div>
+
+
+ <div class="row-fluid">
+ <div id="leftColumn" class="span3">
+ <div class="well sidebar-nav">
+
+
+ <ul class="nav nav-list">
+ <li class="nav-header">User Documentation</li>
+
+ <li>
+
+ <a href="../../../index.html" title="Metron">
+ <i class="icon-chevron-down"></i>
+ Metron</a>
+ <ul class="nav nav-list">
+
+ <li>
+
+ <a href="../../../Upgrading.html" title="Upgrading">
+ <i class="none"></i>
+ Upgrading</a>
+ </li>
+
+ <li>
+
+ <a href="../../../metron-analytics/index.html" title="Analytics">
+ <i class="icon-chevron-right"></i>
+ Analytics</a>
+ </li>
+
+ <li>
+
+ <a href="../../../metron-deployment/index.html" title="Deployment">
+ <i class="icon-chevron-down"></i>
+ Deployment</a>
+ <ul class="nav nav-list">
+
+ <li>
+
+ <a href="../../../metron-deployment/amazon-ec2/index.html" title="Amazon-ec2">
+ <i class="none"></i>
+ Amazon-ec2</a>
+ </li>
+
+ <li>
+
+ <a href="../../../metron-deployment/packaging/docker/ansible-docker/index.html" title="Ansible-docker">
+ <i class="none"></i>
+ Ansible-docker</a>
+ </li>
+
+ <li>
+
+ <a href="../../../metron-deployment/packaging/docker/rpm-docker/index.html" title="Rpm-docker">
+ <i class="none"></i>
+ Rpm-docker</a>
+ </li>
+
+ <li>
+
+ <a href="../../../metron-deployment/packer-build/index.html" title="Packer-build">
+ <i class="none"></i>
+ Packer-build</a>
+ </li>
+
+ <li>
+
+ <a href="../../../metron-deployment/roles/index.html" title="Roles">
+ <i class="icon-chevron-down"></i>
+ Roles</a>
+ <ul class="nav nav-list">
+
+ <li>
+
+ <a href="../../../metron-deployment/roles/kibana/index.html" title="Kibana">
+ <i class="none"></i>
+ Kibana</a>
+ </li>
+
+ <li>
+
+ <a href="../../../metron-deployment/roles/monit/index.html" title="Monit">
+ <i class="none"></i>
+ Monit</a>
+ </li>
+
+ <li class="active">
+
+ <a href="#"><i class="none"></i>Opentaxii</a>
+ </li>
+
+ <li>
+
+ <a href="../../../metron-deployment/roles/pcap_replay/index.html" title="Pcap_replay">
+ <i class="none"></i>
+ Pcap_replay</a>
+ </li>
+
+ <li>
+
+ <a href="../../../metron-deployment/roles/sensor-stubs/index.html" title="Sensor-stubs">
+ <i class="none"></i>
+ Sensor-stubs</a>
+ </li>
+
+ <li>
+
+ <a href="../../../metron-deployment/roles/sensor-test-mode/index.html" title="Sensor-test-mode">
+ <i class="none"></i>
+ Sensor-test-mode</a>
+ </li>
+ </ul>
+ </li>
+
+ <li>
+
+ <a href="../../../metron-deployment/vagrant/index.html" title="Vagrant">
+ <i class="icon-chevron-right"></i>
+ Vagrant</a>
+ </li>
+ </ul>
+ </li>
+
+ <li>
+
+ <a href="../../../metron-docker/index.html" title="Docker">
+ <i class="none"></i>
+ Docker</a>
+ </li>
+
+ <li>
+
+ <a href="../../../metron-platform/index.html" title="Platform">
+ <i class="icon-chevron-right"></i>
+ Platform</a>
+ </li>
+
+ <li>
+
+ <a href="../../../metron-sensors/index.html" title="Sensors">
+ <i class="icon-chevron-right"></i>
+ Sensors</a>
+ </li>
+ </ul>
+ </li>
+ </ul>
+
+
+
+ <hr class="divider" />
+
+ <div id="poweredBy">
+ <div class="clear"></div>
+ <div class="clear"></div>
+ <div class="clear"></div>
+ <a href="http://maven.apache.org/" title="Built by Maven" class="poweredBy">
+ <img class="builtBy" alt="Built by Maven" src="../../../images/logos/maven-feather.png" />
+ </a>
+ </div>
+ </div>
+ </div>
+
+
+ <div id="bodyColumn" class="span9" >
+
+ <h1>OpenTAXII</h1>
+<p><a name="OpenTAXII"></a></p>
+<p>Installs <a class="externalLink" href="https://github.com/EclecticIQ/OpenTAXII">OpenTAXII</a> as a deamon that can be launched via a SysV service script. The complementary client implementation, <a class="externalLink" href="https://github.com/EclecticIQ/cabby">Cabby</a> is also installed.</p>
+<p>OpenTAXII is a robust Python implementation of TAXII Services that delivers a rich feature set and friendly pythonic API. <a class="externalLink" href="https://stixproject.github.io/">TAXII</a> (Trusted Automated eXchange of Indicator Information) is a collection of specifications defining a set of services and message exchanges used for sharing cyber threat intelligence information between parties.</p>
+<div class="section">
+<h2><a name="Getting_Started"></a>Getting Started</h2>
+<p>After deployment completes the OpenTAXII service is installed and running. A set of <a class="externalLink" href="http://hailataxii.com/">Hail a TAXII</a> threat intel collections have been defined and configured. Use the <tt>status</tt> option to view the collections that have been defined.</p>
+
+<div class="source">
+<div class="source">
+<pre>$ service opentaxii status
+Checking opentaxii... Running
+guest.phishtank_com 0
+guest.Abuse_ch 0
+guest.CyberCrime_Tracker 0
+guest.EmergingThreats_rules 0
+guest.Lehigh_edu 0
+guest.MalwareDomainList_Hostlist 0
+guest.blutmagie_de_torExits 0
+guest.dataForLast_7daysOnly 0
+guest.dshield_BlockList 0
+</pre></div></div>
+<p>Notice that each collections contain zero records. None of the data is automatically synced during deployment. To sync the data manually use the <tt>sync</tt> option as defined below. The following example does not provide a begin and end time so the data will be fetched for the current day only.</p>
+
+<div class="source">
+<div class="source">
+<pre># service opentaxii sync guest.blutmagie_de_torExits
+2016-04-21 20:34:42,511 INFO: Starting new HTTP connection (1): localhost
+2016-04-21 20:34:42,540 INFO: Response received for Inbox_Message from http://localhost:9000/services/inbox
+2016-04-21 20:34:42,542 INFO: Sending Inbox_Message to http://localhost:9000/services/inbox
+...
+2016-04-21 20:34:42,719 INFO: Response received for Poll_Request from http://localhost:9000/services/poll
+2016-04-21 20:34:42,719 INFO: Content blocks count: 1618, is partial: False
+</pre></div></div>
+<p>The OpenTAXII service now contains 1,618 threat intel records indicating Tor Exit nodes.</p>
+
+<div class="source">
+<div class="source">
+<pre>[root@source ~]# service opentaxii status
+Checking opentaxii... Running
+guest.phishtank_com 0
+guest.Abuse_ch 0
+guest.CyberCrime_Tracker 0
+guest.EmergingThreats_rules 0
+guest.Lehigh_edu 0
+guest.MalwareDomainList_Hostlist 0
+guest.blutmagie_de_torExits 1618
+guest.dataForLast_7daysOnly 0
+guest.dshield_BlockList 0
+</pre></div></div></div>
+<div class="section">
+<h2><a name="Usage"></a>Usage</h2>
+<p>A standard SysV script has been installed to manage OpenTAXII. The following functions are available.</p>
+<p><tt>start</tt> <tt>stop</tt> <tt>restart</tt> the OpenTAXII service</p>
+<p><tt>status</tt> of the OpenTAXII service. The command displays the collections that have been defined and the number of records in each.</p>
+
+<div class="source">
+<div class="source">
+<pre>$ service opentaxii status
+Checking opentaxii... Running
+guest.phishtank_com 984
+guest.Abuse_ch 45
+guest.CyberCrime_Tracker 482
+guest.EmergingThreats_rules 0
+guest.Lehigh_edu 1030
+guest.MalwareDomainList_Hostlist 84
+guest.blutmagie_de_torExits 3236
+guest.dataForLast_7daysOnly 3377
+guest.dshield_BlockList 0
+</pre></div></div>
+<p><tt>setup</tt> Initializes the services and collections required to operate the OpenTAXII service. This will destroy all existing data. The user is prompted to continue before any data is destroyed.</p>
+
+<div class="source">
+<div class="source">
+<pre># service opentaxii setup
+WARNING: force reset and destroy all opentaxii data? [Ny]: y
+Stopping opentaxii ..Ok
+2016-04-21T19:56:01.886157Z [opentaxii.server] info: api.persistence.loaded {timestamp=2016-04-21T19:56:01.886157Z, logger=opentaxii.server, api_class=SQLDatabaseAPI, event=api.persistence.loaded, level=info}
+2016-04-21T19:56:01.896503Z [opentaxii.server] info: api.auth.loaded {timestamp=2016-04-21T19:56:01.896503Z, logger=opentaxii.server, api_class=SQLDatabaseAPI, event=api.auth.loaded, level=info}
+2016-04-21T19:56:01.896655Z [opentaxii.server] info: taxiiserver.configured {timestamp=2016-04-21T19:56:01.896655Z, logger=opentaxii.server, event=taxiiserver.configured, level=info}
+...
+Ok
+</pre></div></div>
+<p><tt>sync [collection] [begin-at] [end-at]</tt> Syncs the threat intel data available at <a class="externalLink" href="http://hailataxii.com/">Hail a TAXII</a>. If no begin and end date is provided then data is synced over the current day only.</p>
+
+<ul>
+
+<li><tt>collection</tt> Name of the collection to sync.</li>
+
+<li><tt>begin-at</tt> Exclusive begin of time window; ISO8601</li>
+
+<li><tt>end-at</tt> Inclusive end of time window; ISO8601</li>
+</ul>
+
+<div class="source">
+<div class="source">
+<pre>$ service opentaxii sync guest.phishtank_com
++ /usr/local/opentaxii/opentaxii-venv/bin/taxii-proxy --poll-path http://hailataxii.com/taxii-data --poll-collection guest.phishtank_com --inbox-path http://localhost:9000/services/guest.phishtank_com-inbox --inbox-collection guest.phishtank_com --binding urn:stix.mitre.org:xml:1.1.1 --begin 2016-04-21 --end 2016-04-22
+2016-04-21 17:36:23,778 INFO: Sending Poll_Request to http://hailataxii.com/taxii-data
+2016-04-21 17:36:23,784 INFO: Starting new HTTP connection (1): hailataxii.com
+2016-04-21 17:36:24,175 INFO: Response received for Poll_Request from http://hailataxii.com/taxii-data
+2016-04-21 17:36:24,274 INFO: Sending Inbox_Message to http://localhost:9000/services/guest.phishtank_com-inbox
+...
+2016-04-21 17:36:34,867 INFO: Response received for Poll_Request from http://localhost:9000/services/guest.phishtank_com-poll
+2016-04-21 17:36:34,868 INFO: Content blocks count: 6993, is partial: False
+</pre></div></div>
+<div class="section">
+<h3><a name="Troubleshooting"></a>Troubleshooting</h3>
+<p>Should you need to explore the installation, here are instructions on doing so.</p>
+<p>OpenTAXII is installed in a virtual environment. Before exploring the environment run the following commands to perform the necessary setup. The specific paths may change depending on your Ansible settings.</p>
+
+<div class="source">
+<div class="source">
+<pre>export LD_LIBRARY_PATH=/opt/rh/python27/root/usr/lib64
+export OPENTAXII_CONFIG=/usr/local/opentaxii/etc/opentaxii-conf.yml
+cd /usr/local/opentaxii
+. opentaxii-venv/bin/activate
+</pre></div></div>
+<p>Discover available services.</p>
+
+<div class="source">
+<div class="source">
+<pre>taxii-discovery --discovery http://localhost:9000/services/discovery
+taxii-discovery --discovery http://hailataxii.com/taxii-data
+</pre></div></div>
+<p>Explore available collections.</p>
+
+<div class="source">
+<div class="source">
+<pre>taxii-collections --discovery http://localhost:9000/services/discovery
+taxii-collections --discovery http://hailataxii.com/taxii-data
+</pre></div></div>
+<p>Read data from a collection.</p>
+
+<div class="source">
+<div class="source">
+<pre>taxii-poll --discovery http://localhost:9000/services/discovery -c guest.phishtank_com
+taxii-poll --discovery http://hailataxii.com/taxii-data -c guest.phishtank_com --begin 2016-04-20
+</pre></div></div>
+<p>Manually load data into a collection.</p>
+
+<div class="source">
+<div class="source">
+<pre>taxii-push \
+ --discovery http://localhost:9000/services/discovery \
+ --dest phishtank \
+ --content-file data.xml \
+ --username guest \
+ --password guest
+</pre></div></div>
+<p>Fetch data from a remote service and mirror it locally.</p>
+
+<div class="source">
+<div class="source">
+<pre>taxii-proxy --poll-path http://hailataxii.com/taxii-data \
+ --poll-collection guest.phishtank_com \
+ --inbox-path http://localhost:9000/services/guest.phishtank_com-inbox \
+ --inbox-collection guest.phishtank_com \
+ --binding urn:stix.mitre.org:xml:1.1.1 \
+ --inbox-username guest \
+ --inbox-password guest \
+ --begin 2016-04-20
+</pre></div></div></div></div>
+ </div>
+ </div>
+ </div>
+
+ <hr/>
+
+ <footer>
+ <div class="container-fluid">
+ <div class="row span12">Copyright © 2017.
+ All Rights Reserved.
+
+ </div>
+
+
+
+ </div>
+ </footer>
+ </body>
+</html>
Added: dev/incubator/metron/0.3.1-RC3-incubating/book-site/metron-deployment/roles/pcap_replay/index.html
==============================================================================
--- dev/incubator/metron/0.3.1-RC3-incubating/book-site/metron-deployment/roles/pcap_replay/index.html (added)
+++ dev/incubator/metron/0.3.1-RC3-incubating/book-site/metron-deployment/roles/pcap_replay/index.html Thu Feb 9 14:12:17 2017
@@ -0,0 +1,294 @@
+<!DOCTYPE html>
+<!--
+ | Generated by Apache Maven Doxia at 2017-02-09
+ | Rendered using Apache Maven Fluido Skin 1.3.0
+-->
+<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
+ <head>
+ <meta charset="UTF-8" />
+ <meta name="viewport" content="width=device-width, initial-scale=1.0" />
+ <meta name="Date-Revision-yyyymmdd" content="20170209" />
+ <meta http-equiv="Content-Language" content="en" />
+ <title>Metron – Pcap Replay</title>
+ <link rel="stylesheet" href="../../../css/apache-maven-fluido-1.3.0.min.css" />
+ <link rel="stylesheet" href="../../../css/site.css" />
+ <link rel="stylesheet" href="../../../css/print.css" media="print" />
+
+
+ <script type="text/javascript" src="../../../js/apache-maven-fluido-1.3.0.min.js"></script>
+
+
+
+<script type="text/javascript">$( document ).ready( function() { $( '.carousel' ).carousel( { interval: 3500 } ) } );</script>
+
+ </head>
+ <body class="topBarDisabled">
+
+
+
+
+ <div class="container-fluid">
+ <div id="banner">
+ <div class="pull-left">
+ <a href="http://metron.incubator.apache.org/" id="bannerLeft">
+ <img src="../../../images/metron-logo.png" alt="Apache Metron - Incubating" width="148px" height="48px"/>
+ </a>
+ </div>
+ <div class="pull-right"> <a href="http://incubator.apache.org/" id="bannerRight">
+ <img src="../../../images/ApacheIncubating_Logo.png" alt="Apache Incubating" width="192px" height="48px"/>
+ </a>
+ </div>
+ <div class="clear"><hr/></div>
+ </div>
+
+ <div id="breadcrumbs">
+ <ul class="breadcrumb">
+
+
+ <li class="">
+ <a href="http://www.apache.org" class="externalLink" title="Apache">
+ Apache</a>
+ </li>
+ <li class="divider ">/</li>
+ <li class="">
+ <a href="http://metron.incubator.apache.org/" class="externalLink" title="Metron-Incubating">
+ Metron-Incubating</a>
+ </li>
+ <li class="divider ">/</li>
+ <li class="">
+ <a href="../../../index.html" title="Documentation">
+ Documentation</a>
+ </li>
+ <li class="divider ">/</li>
+ <li class="">Pcap Replay</li>
+
+
+
+ <li id="publishDate" class="pull-right">Last Published: 2017-02-09</li> <li class="divider pull-right">|</li>
+ <li id="projectVersion" class="pull-right">Version: 0.3.1</li>
+
+ </ul>
+ </div>
+
+
+ <div class="row-fluid">
+ <div id="leftColumn" class="span3">
+ <div class="well sidebar-nav">
+
+
+ <ul class="nav nav-list">
+ <li class="nav-header">User Documentation</li>
+
+ <li>
+
+ <a href="../../../index.html" title="Metron">
+ <i class="icon-chevron-down"></i>
+ Metron</a>
+ <ul class="nav nav-list">
+
+ <li>
+
+ <a href="../../../Upgrading.html" title="Upgrading">
+ <i class="none"></i>
+ Upgrading</a>
+ </li>
+
+ <li>
+
+ <a href="../../../metron-analytics/index.html" title="Analytics">
+ <i class="icon-chevron-right"></i>
+ Analytics</a>
+ </li>
+
+ <li>
+
+ <a href="../../../metron-deployment/index.html" title="Deployment">
+ <i class="icon-chevron-down"></i>
+ Deployment</a>
+ <ul class="nav nav-list">
+
+ <li>
+
+ <a href="../../../metron-deployment/amazon-ec2/index.html" title="Amazon-ec2">
+ <i class="none"></i>
+ Amazon-ec2</a>
+ </li>
+
+ <li>
+
+ <a href="../../../metron-deployment/packaging/docker/ansible-docker/index.html" title="Ansible-docker">
+ <i class="none"></i>
+ Ansible-docker</a>
+ </li>
+
+ <li>
+
+ <a href="../../../metron-deployment/packaging/docker/rpm-docker/index.html" title="Rpm-docker">
+ <i class="none"></i>
+ Rpm-docker</a>
+ </li>
+
+ <li>
+
+ <a href="../../../metron-deployment/packer-build/index.html" title="Packer-build">
+ <i class="none"></i>
+ Packer-build</a>
+ </li>
+
+ <li>
+
+ <a href="../../../metron-deployment/roles/index.html" title="Roles">
+ <i class="icon-chevron-down"></i>
+ Roles</a>
+ <ul class="nav nav-list">
+
+ <li>
+
+ <a href="../../../metron-deployment/roles/kibana/index.html" title="Kibana">
+ <i class="none"></i>
+ Kibana</a>
+ </li>
+
+ <li>
+
+ <a href="../../../metron-deployment/roles/monit/index.html" title="Monit">
+ <i class="none"></i>
+ Monit</a>
+ </li>
+
+ <li>
+
+ <a href="../../../metron-deployment/roles/opentaxii/index.html" title="Opentaxii">
+ <i class="none"></i>
+ Opentaxii</a>
+ </li>
+
+ <li class="active">
+
+ <a href="#"><i class="none"></i>Pcap_replay</a>
+ </li>
+
+ <li>
+
+ <a href="../../../metron-deployment/roles/sensor-stubs/index.html" title="Sensor-stubs">
+ <i class="none"></i>
+ Sensor-stubs</a>
+ </li>
+
+ <li>
+
+ <a href="../../../metron-deployment/roles/sensor-test-mode/index.html" title="Sensor-test-mode">
+ <i class="none"></i>
+ Sensor-test-mode</a>
+ </li>
+ </ul>
+ </li>
+
+ <li>
+
+ <a href="../../../metron-deployment/vagrant/index.html" title="Vagrant">
+ <i class="icon-chevron-right"></i>
+ Vagrant</a>
+ </li>
+ </ul>
+ </li>
+
+ <li>
+
+ <a href="../../../metron-docker/index.html" title="Docker">
+ <i class="none"></i>
+ Docker</a>
+ </li>
+
+ <li>
+
+ <a href="../../../metron-platform/index.html" title="Platform">
+ <i class="icon-chevron-right"></i>
+ Platform</a>
+ </li>
+
+ <li>
+
+ <a href="../../../metron-sensors/index.html" title="Sensors">
+ <i class="icon-chevron-right"></i>
+ Sensors</a>
+ </li>
+ </ul>
+ </li>
+ </ul>
+
+
+
+ <hr class="divider" />
+
+ <div id="poweredBy">
+ <div class="clear"></div>
+ <div class="clear"></div>
+ <div class="clear"></div>
+ <a href="http://maven.apache.org/" title="Built by Maven" class="poweredBy">
+ <img class="builtBy" alt="Built by Maven" src="../../../images/logos/maven-feather.png" />
+ </a>
+ </div>
+ </div>
+ </div>
+
+
+ <div id="bodyColumn" class="span9" >
+
+ <h1>Pcap Replay</h1>
+<p>This project enables packet capture data to be replayed through a network interface to simulate live network traffic. This can be used to support functional, performance, and load testing of Apache Metron.</p>
+<div class="section">
+<h2><a name="Getting_Started"></a>Getting Started</h2>
+<p>To replay packet capture data, simply start the <tt>pcap-replay</tt> SysV service. To do this run the following command.</p>
+
+<div class="source">
+<div class="source">
+<pre>service pcap-replay start
+</pre></div></div>
+<p>All additional options accepted by <tt>tcpreplay</tt> can be passed to the service script to modify how the network data is replayed. For example, this makes it simple to control the amount and rate of data replayed during functional, performance and load testing.</p>
+<p>Example: Replay data at a rate of 10 mbps.</p>
+
+<div class="source">
+<div class="source">
+<pre>service pcap-replay start --mbps 10
+</pre></div></div>
+<p>Example: Replay data at a rate of 10 packets per second.</p>
+
+<div class="source">
+<div class="source">
+<pre>service pcap-replay start --pps 10
+</pre></div></div>
+<p>All nodes on the same subnet with their network interface set to promiscuous mode will then be able to capture the network traffic being replayed. To validate, simply run something like the following.</p>
+
+<div class="source">
+<div class="source">
+<pre>tcpdump -i eth1
+</pre></div></div></div>
+<div class="section">
+<h2><a name="Data"></a>Data</h2>
+<p>An example packet capture file has been installed at <tt>/opt/pcap-replay/example.pcap</tt>. By default, the network traffic contained within this file is continually replayed. </p>
+<p>To replay your own packet capture data, simply add any number of files containing <tt>libpcap</tt> formatted packet capture data to <tt>/opt/pcap-replay</tt>. The files must end with the <tt>.pcap</tt> extension. To pick up newly installed files, simply restart the service.</p>
+
+<div class="source">
+<div class="source">
+<pre>service pcap-replay restart
+</pre></div></div></div>
+ </div>
+ </div>
+ </div>
+
+ <hr/>
+
+ <footer>
+ <div class="container-fluid">
+ <div class="row span12">Copyright © 2017.
+ All Rights Reserved.
+
+ </div>
+
+
+
+ </div>
+ </footer>
+ </body>
+</html>
Added: dev/incubator/metron/0.3.1-RC3-incubating/book-site/metron-deployment/roles/sensor-stubs/index.html
==============================================================================
--- dev/incubator/metron/0.3.1-RC3-incubating/book-site/metron-deployment/roles/sensor-stubs/index.html (added)
+++ dev/incubator/metron/0.3.1-RC3-incubating/book-site/metron-deployment/roles/sensor-stubs/index.html Thu Feb 9 14:12:17 2017
@@ -0,0 +1,340 @@
+<!DOCTYPE html>
+<!--
+ | Generated by Apache Maven Doxia at 2017-02-09
+ | Rendered using Apache Maven Fluido Skin 1.3.0
+-->
+<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
+ <head>
+ <meta charset="UTF-8" />
+ <meta name="viewport" content="width=device-width, initial-scale=1.0" />
+ <meta name="Date-Revision-yyyymmdd" content="20170209" />
+ <meta http-equiv="Content-Language" content="en" />
+ <title>Metron – Sensor Stubs</title>
+ <link rel="stylesheet" href="../../../css/apache-maven-fluido-1.3.0.min.css" />
+ <link rel="stylesheet" href="../../../css/site.css" />
+ <link rel="stylesheet" href="../../../css/print.css" media="print" />
+
+
+ <script type="text/javascript" src="../../../js/apache-maven-fluido-1.3.0.min.js"></script>
+
+
+
+<script type="text/javascript">$( document ).ready( function() { $( '.carousel' ).carousel( { interval: 3500 } ) } );</script>
+
+ </head>
+ <body class="topBarDisabled">
+
+
+
+
+ <div class="container-fluid">
+ <div id="banner">
+ <div class="pull-left">
+ <a href="http://metron.incubator.apache.org/" id="bannerLeft">
+ <img src="../../../images/metron-logo.png" alt="Apache Metron - Incubating" width="148px" height="48px"/>
+ </a>
+ </div>
+ <div class="pull-right"> <a href="http://incubator.apache.org/" id="bannerRight">
+ <img src="../../../images/ApacheIncubating_Logo.png" alt="Apache Incubating" width="192px" height="48px"/>
+ </a>
+ </div>
+ <div class="clear"><hr/></div>
+ </div>
+
+ <div id="breadcrumbs">
+ <ul class="breadcrumb">
+
+
+ <li class="">
+ <a href="http://www.apache.org" class="externalLink" title="Apache">
+ Apache</a>
+ </li>
+ <li class="divider ">/</li>
+ <li class="">
+ <a href="http://metron.incubator.apache.org/" class="externalLink" title="Metron-Incubating">
+ Metron-Incubating</a>
+ </li>
+ <li class="divider ">/</li>
+ <li class="">
+ <a href="../../../index.html" title="Documentation">
+ Documentation</a>
+ </li>
+ <li class="divider ">/</li>
+ <li class="">Sensor Stubs</li>
+
+
+
+ <li id="publishDate" class="pull-right">Last Published: 2017-02-09</li> <li class="divider pull-right">|</li>
+ <li id="projectVersion" class="pull-right">Version: 0.3.1</li>
+
+ </ul>
+ </div>
+
+
+ <div class="row-fluid">
+ <div id="leftColumn" class="span3">
+ <div class="well sidebar-nav">
+
+
+ <ul class="nav nav-list">
+ <li class="nav-header">User Documentation</li>
+
+ <li>
+
+ <a href="../../../index.html" title="Metron">
+ <i class="icon-chevron-down"></i>
+ Metron</a>
+ <ul class="nav nav-list">
+
+ <li>
+
+ <a href="../../../Upgrading.html" title="Upgrading">
+ <i class="none"></i>
+ Upgrading</a>
+ </li>
+
+ <li>
+
+ <a href="../../../metron-analytics/index.html" title="Analytics">
+ <i class="icon-chevron-right"></i>
+ Analytics</a>
+ </li>
+
+ <li>
+
+ <a href="../../../metron-deployment/index.html" title="Deployment">
+ <i class="icon-chevron-down"></i>
+ Deployment</a>
+ <ul class="nav nav-list">
+
+ <li>
+
+ <a href="../../../metron-deployment/amazon-ec2/index.html" title="Amazon-ec2">
+ <i class="none"></i>
+ Amazon-ec2</a>
+ </li>
+
+ <li>
+
+ <a href="../../../metron-deployment/packaging/docker/ansible-docker/index.html" title="Ansible-docker">
+ <i class="none"></i>
+ Ansible-docker</a>
+ </li>
+
+ <li>
+
+ <a href="../../../metron-deployment/packaging/docker/rpm-docker/index.html" title="Rpm-docker">
+ <i class="none"></i>
+ Rpm-docker</a>
+ </li>
+
+ <li>
+
+ <a href="../../../metron-deployment/packer-build/index.html" title="Packer-build">
+ <i class="none"></i>
+ Packer-build</a>
+ </li>
+
+ <li>
+
+ <a href="../../../metron-deployment/roles/index.html" title="Roles">
+ <i class="icon-chevron-down"></i>
+ Roles</a>
+ <ul class="nav nav-list">
+
+ <li>
+
+ <a href="../../../metron-deployment/roles/kibana/index.html" title="Kibana">
+ <i class="none"></i>
+ Kibana</a>
+ </li>
+
+ <li>
+
+ <a href="../../../metron-deployment/roles/monit/index.html" title="Monit">
+ <i class="none"></i>
+ Monit</a>
+ </li>
+
+ <li>
+
+ <a href="../../../metron-deployment/roles/opentaxii/index.html" title="Opentaxii">
+ <i class="none"></i>
+ Opentaxii</a>
+ </li>
+
+ <li>
+
+ <a href="../../../metron-deployment/roles/pcap_replay/index.html" title="Pcap_replay">
+ <i class="none"></i>
+ Pcap_replay</a>
+ </li>
+
+ <li class="active">
+
+ <a href="#"><i class="none"></i>Sensor-stubs</a>
+ </li>
+
+ <li>
+
+ <a href="../../../metron-deployment/roles/sensor-test-mode/index.html" title="Sensor-test-mode">
+ <i class="none"></i>
+ Sensor-test-mode</a>
+ </li>
+ </ul>
+ </li>
+
+ <li>
+
+ <a href="../../../metron-deployment/vagrant/index.html" title="Vagrant">
+ <i class="icon-chevron-right"></i>
+ Vagrant</a>
+ </li>
+ </ul>
+ </li>
+
+ <li>
+
+ <a href="../../../metron-docker/index.html" title="Docker">
+ <i class="none"></i>
+ Docker</a>
+ </li>
+
+ <li>
+
+ <a href="../../../metron-platform/index.html" title="Platform">
+ <i class="icon-chevron-right"></i>
+ Platform</a>
+ </li>
+
+ <li>
+
+ <a href="../../../metron-sensors/index.html" title="Sensors">
+ <i class="icon-chevron-right"></i>
+ Sensors</a>
+ </li>
+ </ul>
+ </li>
+ </ul>
+
+
+
+ <hr class="divider" />
+
+ <div id="poweredBy">
+ <div class="clear"></div>
+ <div class="clear"></div>
+ <div class="clear"></div>
+ <a href="http://maven.apache.org/" title="Built by Maven" class="poweredBy">
+ <img class="builtBy" alt="Built by Maven" src="../../../images/logos/maven-feather.png" />
+ </a>
+ </div>
+ </div>
+ </div>
+
+
+ <div id="bodyColumn" class="span9" >
+
+ <h1>Sensor Stubs</h1>
+<p><a name="Sensor_Stubs"></a></p>
+<p>A service has been created to simulate the behavior of a sensor by sending canned telemetry data to a Kafka topic. These “Sensor Stubs” consume fewer resources than the actual sensor that they replace.</p>
+<div class="section">
+<div class="section">
+<h3><a name="aQ_How_do_the_sensor_stubs_work"></a>(Q) How do the sensor stubs work?</h3>
+<p>The stubs are installed with a set of canned data for each sensor type; Bro, Snort and YAF. A subset of this canned data is randomly selected and sent to the Kafka topic in batches. The timestamp of each message is updated to match current system time. </p></div>
+<div class="section">
+<h3><a name="aQ_How_do_I_configure_the_message_rate"></a>(Q) How do I configure the message rate?</h3>
+<p>The number of telemetry messages sent in each batch, along with the time delay between batches is configurable. Before installation, these values can be configured by redefining <tt>sensor_stubs_delay</tt> and <tt>sensor_stubs_count</tt>. The values can also be configured by altering the deployed system service script at <tt>/etc/init.d/sensor-stubs</tt>.</p></div>
+<div class="section">
+<h3><a name="aQ_How_do_I_install_the_sensor_stubs"></a>(Q) How do I install the sensor stubs?</h3>
+<p>Using the default playbooks, this role can be installed by using the Ansible tag <tt>sensor-stubs</tt>. This service is installed on the same hosts where the sensors would be; defined by the <tt>sensors</tt> host group.</p>
+<p>The defaults for the “Quick Dev” and “Full Dev” environments have been changed so that the Sensor Stubs are installed by default, rather than the sensors themselves. The Amazon EC2 environment continues to install the original sensors by default.</p></div>
+<div class="section">
+<h3><a name="aQ_How_do_I_use_the_sensor_stubs"></a>(Q) How do I use the sensor stubs?</h3>
+<p>Start all sensor stubs. The output includes the PID for each running sensor stub.</p>
+
+<div class="source">
+<div class="source">
+<pre>$ service sensor-stubs start
+Starting sensor-stubs...
+ bro: Ok [26505]
+ yaf: Ok [26507]
+ snort: Ok [26509]
+</pre></div></div>
+<p>Check the status of each sensor stub.</p>
+
+<div class="source">
+<div class="source">
+<pre>$ service sensor-stubs status
+Checking sensor-stubs...
+ bro: Running [26505]
+ yaf: Running [26507]
+ snort: Running [26509]
+</pre></div></div>
+<p>Stop all sensor stubs.</p>
+
+<div class="source">
+<div class="source">
+<pre>$ service sensor-stubs stop
+Stopping sensor-stubs...
+.. bro: Ok [26505]
+.. yaf: Ok [26507]
+.. snort: Ok [26509]
+</pre></div></div>
+<p>Check the status. All sensor stubs should be stopped.</p>
+
+<div class="source">
+<div class="source">
+<pre>$ service sensor-stubs status
+Checking sensor-stubs...
+ bro: Not running
+ yaf: Not running
+ snort: Not running
+</pre></div></div>
+<p>Start only the Bro sensor stub.</p>
+
+<div class="source">
+<div class="source">
+<pre>$ service sensor-stubs start bro
+Starting sensor-stubs...
+ bro: OK [11616]
+</pre></div></div>
+<p>Stop the Bro sensor stub.</p>
+
+<div class="source">
+<div class="source">
+<pre>$ service sensor-stubs stop bro
+Stopping sensor-stubs...
+.. bro: Ok [11616]
+</pre></div></div></div>
+<div class="section">
+<h3><a name="aQ_How_do_I_install_the_original_sensors"></a>(Q) How do I install the original sensors?</h3>
+<p>The default behavior can be changed by skipping the <tt>sensor-stubs</tt> flag and including the <tt>sensors</tt> flag. For example, to deploy “Quick Dev” with the original sensors run the following command.</p>
+
+<div class="source">
+<div class="source">
+<pre>cd metron-deployment/vagrant/quick-dev-platform
+vagrant --ansible-skip-tags="sensor-stubs,solr" up
+</pre></div></div></div>
+<div class="section">
+<h3><a name="aQ_Where_does_the_mock_data_come_from"></a>(Q) Where does the mock data come from?</h3>
+<p>The data produced by the sensor stubs was generated by running the sensors against the example pcap file that is distributed with Metron. This ensures that the data produced by the sensor stubs is similar to the data produced when using the actual sensors.</p></div></div>
+ </div>
+ </div>
+ </div>
+
+ <hr/>
+
+ <footer>
+ <div class="container-fluid">
+ <div class="row span12">Copyright © 2017.
+ All Rights Reserved.
+
+ </div>
+
+
+
+ </div>
+ </footer>
+ </body>
+</html>
Added: dev/incubator/metron/0.3.1-RC3-incubating/book-site/metron-deployment/roles/sensor-test-mode/index.html
==============================================================================
--- dev/incubator/metron/0.3.1-RC3-incubating/book-site/metron-deployment/roles/sensor-test-mode/index.html (added)
+++ dev/incubator/metron/0.3.1-RC3-incubating/book-site/metron-deployment/roles/sensor-test-mode/index.html Thu Feb 9 14:12:17 2017
@@ -0,0 +1,284 @@
+<!DOCTYPE html>
+<!--
+ | Generated by Apache Maven Doxia at 2017-02-09
+ | Rendered using Apache Maven Fluido Skin 1.3.0
+-->
+<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
+ <head>
+ <meta charset="UTF-8" />
+ <meta name="viewport" content="width=device-width, initial-scale=1.0" />
+ <meta name="Date-Revision-yyyymmdd" content="20170209" />
+ <meta http-equiv="Content-Language" content="en" />
+ <title>Metron – Sensor Test Mode</title>
+ <link rel="stylesheet" href="../../../css/apache-maven-fluido-1.3.0.min.css" />
+ <link rel="stylesheet" href="../../../css/site.css" />
+ <link rel="stylesheet" href="../../../css/print.css" media="print" />
+
+
+ <script type="text/javascript" src="../../../js/apache-maven-fluido-1.3.0.min.js"></script>
+
+
+
+<script type="text/javascript">$( document ).ready( function() { $( '.carousel' ).carousel( { interval: 3500 } ) } );</script>
+
+ </head>
+ <body class="topBarDisabled">
+
+
+
+
+ <div class="container-fluid">
+ <div id="banner">
+ <div class="pull-left">
+ <a href="http://metron.incubator.apache.org/" id="bannerLeft">
+ <img src="../../../images/metron-logo.png" alt="Apache Metron - Incubating" width="148px" height="48px"/>
+ </a>
+ </div>
+ <div class="pull-right"> <a href="http://incubator.apache.org/" id="bannerRight">
+ <img src="../../../images/ApacheIncubating_Logo.png" alt="Apache Incubating" width="192px" height="48px"/>
+ </a>
+ </div>
+ <div class="clear"><hr/></div>
+ </div>
+
+ <div id="breadcrumbs">
+ <ul class="breadcrumb">
+
+
+ <li class="">
+ <a href="http://www.apache.org" class="externalLink" title="Apache">
+ Apache</a>
+ </li>
+ <li class="divider ">/</li>
+ <li class="">
+ <a href="http://metron.incubator.apache.org/" class="externalLink" title="Metron-Incubating">
+ Metron-Incubating</a>
+ </li>
+ <li class="divider ">/</li>
+ <li class="">
+ <a href="../../../index.html" title="Documentation">
+ Documentation</a>
+ </li>
+ <li class="divider ">/</li>
+ <li class="">Sensor Test Mode</li>
+
+
+
+ <li id="publishDate" class="pull-right">Last Published: 2017-02-09</li> <li class="divider pull-right">|</li>
+ <li id="projectVersion" class="pull-right">Version: 0.3.1</li>
+
+ </ul>
+ </div>
+
+
+ <div class="row-fluid">
+ <div id="leftColumn" class="span3">
+ <div class="well sidebar-nav">
+
+
+ <ul class="nav nav-list">
+ <li class="nav-header">User Documentation</li>
+
+ <li>
+
+ <a href="../../../index.html" title="Metron">
+ <i class="icon-chevron-down"></i>
+ Metron</a>
+ <ul class="nav nav-list">
+
+ <li>
+
+ <a href="../../../Upgrading.html" title="Upgrading">
+ <i class="none"></i>
+ Upgrading</a>
+ </li>
+
+ <li>
+
+ <a href="../../../metron-analytics/index.html" title="Analytics">
+ <i class="icon-chevron-right"></i>
+ Analytics</a>
+ </li>
+
+ <li>
+
+ <a href="../../../metron-deployment/index.html" title="Deployment">
+ <i class="icon-chevron-down"></i>
+ Deployment</a>
+ <ul class="nav nav-list">
+
+ <li>
+
+ <a href="../../../metron-deployment/amazon-ec2/index.html" title="Amazon-ec2">
+ <i class="none"></i>
+ Amazon-ec2</a>
+ </li>
+
+ <li>
+
+ <a href="../../../metron-deployment/packaging/docker/ansible-docker/index.html" title="Ansible-docker">
+ <i class="none"></i>
+ Ansible-docker</a>
+ </li>
+
+ <li>
+
+ <a href="../../../metron-deployment/packaging/docker/rpm-docker/index.html" title="Rpm-docker">
+ <i class="none"></i>
+ Rpm-docker</a>
+ </li>
+
+ <li>
+
+ <a href="../../../metron-deployment/packer-build/index.html" title="Packer-build">
+ <i class="none"></i>
+ Packer-build</a>
+ </li>
+
+ <li>
+
+ <a href="../../../metron-deployment/roles/index.html" title="Roles">
+ <i class="icon-chevron-down"></i>
+ Roles</a>
+ <ul class="nav nav-list">
+
+ <li>
+
+ <a href="../../../metron-deployment/roles/kibana/index.html" title="Kibana">
+ <i class="none"></i>
+ Kibana</a>
+ </li>
+
+ <li>
+
+ <a href="../../../metron-deployment/roles/monit/index.html" title="Monit">
+ <i class="none"></i>
+ Monit</a>
+ </li>
+
+ <li>
+
+ <a href="../../../metron-deployment/roles/opentaxii/index.html" title="Opentaxii">
+ <i class="none"></i>
+ Opentaxii</a>
+ </li>
+
+ <li>
+
+ <a href="../../../metron-deployment/roles/pcap_replay/index.html" title="Pcap_replay">
+ <i class="none"></i>
+ Pcap_replay</a>
+ </li>
+
+ <li>
+
+ <a href="../../../metron-deployment/roles/sensor-stubs/index.html" title="Sensor-stubs">
+ <i class="none"></i>
+ Sensor-stubs</a>
+ </li>
+
+ <li class="active">
+
+ <a href="#"><i class="none"></i>Sensor-test-mode</a>
+ </li>
+ </ul>
+ </li>
+
+ <li>
+
+ <a href="../../../metron-deployment/vagrant/index.html" title="Vagrant">
+ <i class="icon-chevron-right"></i>
+ Vagrant</a>
+ </li>
+ </ul>
+ </li>
+
+ <li>
+
+ <a href="../../../metron-docker/index.html" title="Docker">
+ <i class="none"></i>
+ Docker</a>
+ </li>
+
+ <li>
+
+ <a href="../../../metron-platform/index.html" title="Platform">
+ <i class="icon-chevron-right"></i>
+ Platform</a>
+ </li>
+
+ <li>
+
+ <a href="../../../metron-sensors/index.html" title="Sensors">
+ <i class="icon-chevron-right"></i>
+ Sensors</a>
+ </li>
+ </ul>
+ </li>
+ </ul>
+
+
+
+ <hr class="divider" />
+
+ <div id="poweredBy">
+ <div class="clear"></div>
+ <div class="clear"></div>
+ <div class="clear"></div>
+ <a href="http://maven.apache.org/" title="Built by Maven" class="poweredBy">
+ <img class="builtBy" alt="Built by Maven" src="../../../images/logos/maven-feather.png" />
+ </a>
+ </div>
+ </div>
+ </div>
+
+
+ <div id="bodyColumn" class="span9" >
+
+ <h1>Sensor Test Mode</h1>
+<p>A role that configures each of the sensors to produce the maximum amount of telemetry data. This role is useful only for testing. It can be useful to support functional, performance, and load testing of Apache Metron.</p>
+<p>The role does the following to maximize the amount of telemetry data produced by each Metron sensor.</p>
+
+<ul>
+
+<li>Plays a packet capture file through a network interface to simulate live network traffic.</li>
+
+<li>Configures <a class="externalLink" href="https://tools.netsa.cert.org/yaf/yaf.html">YAF</a> with <tt>idle-timeout=0</tt>. This causes a flow record to be produced for every network packet received.</li>
+
+<li>Configures <a class="externalLink" href="https://www.snort.org/">Snort</a> to produce an alert for every network packet received.</li>
+</ul>
+<div class="section">
+<h2><a name="Getting_Started"></a>Getting Started</h2>
+<p>To enable the <tt>sensor-test-mode</tt> role apply the role to the <tt>sensors</tt> host group in your Ansible playbook.</p>
+
+<div class="source">
+<div class="source">
+<pre>- hosts: sensors
+ roles:
+ - role: sensor-test-mode
+</pre></div></div>
+<p>The role has also been added to the default <tt>metron_install.yml</tt> playbook so that it can be turned on/off with a property in both the local Virtualbox and the remote EC2 deployments.</p>
+
+<div class="source">
+<div class="source">
+<pre>sensor_test_mode: True
+</pre></div></div></div>
+ </div>
+ </div>
+ </div>
+
+ <hr/>
+
+ <footer>
+ <div class="container-fluid">
+ <div class="row span12">Copyright © 2017.
+ All Rights Reserved.
+
+ </div>
+
+
+
+ </div>
+ </footer>
+ </body>
+</html>
Added: dev/incubator/metron/0.3.1-RC3-incubating/book-site/metron-deployment/vagrant/codelab-platform/index.html
==============================================================================
--- dev/incubator/metron/0.3.1-RC3-incubating/book-site/metron-deployment/vagrant/codelab-platform/index.html (added)
+++ dev/incubator/metron/0.3.1-RC3-incubating/book-site/metron-deployment/vagrant/codelab-platform/index.html Thu Feb 9 14:12:17 2017
@@ -0,0 +1,315 @@
+<!DOCTYPE html>
+<!--
+ | Generated by Apache Maven Doxia at 2017-02-09
+ | Rendered using Apache Maven Fluido Skin 1.3.0
+-->
+<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
+ <head>
+ <meta charset="UTF-8" />
+ <meta name="viewport" content="width=device-width, initial-scale=1.0" />
+ <meta name="Date-Revision-yyyymmdd" content="20170209" />
+ <meta http-equiv="Content-Language" content="en" />
+ <title>Metron – Developer Image for Apache Metron on Virtualbox</title>
+ <link rel="stylesheet" href="../../../css/apache-maven-fluido-1.3.0.min.css" />
+ <link rel="stylesheet" href="../../../css/site.css" />
+ <link rel="stylesheet" href="../../../css/print.css" media="print" />
+
+
+ <script type="text/javascript" src="../../../js/apache-maven-fluido-1.3.0.min.js"></script>
+
+
+
+<script type="text/javascript">$( document ).ready( function() { $( '.carousel' ).carousel( { interval: 3500 } ) } );</script>
+
+ </head>
+ <body class="topBarDisabled">
+
+
+
+
+ <div class="container-fluid">
+ <div id="banner">
+ <div class="pull-left">
+ <a href="http://metron.incubator.apache.org/" id="bannerLeft">
+ <img src="../../../images/metron-logo.png" alt="Apache Metron - Incubating" width="148px" height="48px"/>
+ </a>
+ </div>
+ <div class="pull-right"> <a href="http://incubator.apache.org/" id="bannerRight">
+ <img src="../../../images/ApacheIncubating_Logo.png" alt="Apache Incubating" width="192px" height="48px"/>
+ </a>
+ </div>
+ <div class="clear"><hr/></div>
+ </div>
+
+ <div id="breadcrumbs">
+ <ul class="breadcrumb">
+
+
+ <li class="">
+ <a href="http://www.apache.org" class="externalLink" title="Apache">
+ Apache</a>
+ </li>
+ <li class="divider ">/</li>
+ <li class="">
+ <a href="http://metron.incubator.apache.org/" class="externalLink" title="Metron-Incubating">
+ Metron-Incubating</a>
+ </li>
+ <li class="divider ">/</li>
+ <li class="">
+ <a href="../../../index.html" title="Documentation">
+ Documentation</a>
+ </li>
+ <li class="divider ">/</li>
+ <li class="">Developer Image for Apache Metron on Virtualbox</li>
+
+
+
+ <li id="publishDate" class="pull-right">Last Published: 2017-02-09</li> <li class="divider pull-right">|</li>
+ <li id="projectVersion" class="pull-right">Version: 0.3.1</li>
+
+ </ul>
+ </div>
+
+
+ <div class="row-fluid">
+ <div id="leftColumn" class="span3">
+ <div class="well sidebar-nav">
+
+
+ <ul class="nav nav-list">
+ <li class="nav-header">User Documentation</li>
+
+ <li>
+
+ <a href="../../../index.html" title="Metron">
+ <i class="icon-chevron-down"></i>
+ Metron</a>
+ <ul class="nav nav-list">
+
+ <li>
+
+ <a href="../../../Upgrading.html" title="Upgrading">
+ <i class="none"></i>
+ Upgrading</a>
+ </li>
+
+ <li>
+
+ <a href="../../../metron-analytics/index.html" title="Analytics">
+ <i class="icon-chevron-right"></i>
+ Analytics</a>
+ </li>
+
+ <li>
+
+ <a href="../../../metron-deployment/index.html" title="Deployment">
+ <i class="icon-chevron-down"></i>
+ Deployment</a>
+ <ul class="nav nav-list">
+
+ <li>
+
+ <a href="../../../metron-deployment/amazon-ec2/index.html" title="Amazon-ec2">
+ <i class="none"></i>
+ Amazon-ec2</a>
+ </li>
+
+ <li>
+
+ <a href="../../../metron-deployment/packaging/docker/ansible-docker/index.html" title="Ansible-docker">
+ <i class="none"></i>
+ Ansible-docker</a>
+ </li>
+
+ <li>
+
+ <a href="../../../metron-deployment/packaging/docker/rpm-docker/index.html" title="Rpm-docker">
+ <i class="none"></i>
+ Rpm-docker</a>
+ </li>
+
+ <li>
+
+ <a href="../../../metron-deployment/packer-build/index.html" title="Packer-build">
+ <i class="none"></i>
+ Packer-build</a>
+ </li>
+
+ <li>
+
+ <a href="../../../metron-deployment/roles/index.html" title="Roles">
+ <i class="icon-chevron-right"></i>
+ Roles</a>
+ </li>
+
+ <li>
+
+ <a href="../../../metron-deployment/vagrant/index.html" title="Vagrant">
+ <i class="icon-chevron-down"></i>
+ Vagrant</a>
+ <ul class="nav nav-list">
+
+ <li class="active">
+
+ <a href="#"><i class="none"></i>Codelab-platform</a>
+ </li>
+
+ <li>
+
+ <a href="../../../metron-deployment/vagrant/fastcapa-test-platform/index.html" title="Fastcapa-test-platform">
+ <i class="none"></i>
+ Fastcapa-test-platform</a>
+ </li>
+
+ <li>
+
+ <a href="../../../metron-deployment/vagrant/full-dev-platform/index.html" title="Full-dev-platform">
+ <i class="none"></i>
+ Full-dev-platform</a>
+ </li>
+
+ <li>
+
+ <a href="../../../metron-deployment/vagrant/quick-dev-platform/index.html" title="Quick-dev-platform">
+ <i class="none"></i>
+ Quick-dev-platform</a>
+ </li>
+ </ul>
+ </li>
+ </ul>
+ </li>
+
+ <li>
+
+ <a href="../../../metron-docker/index.html" title="Docker">
+ <i class="none"></i>
+ Docker</a>
+ </li>
+
+ <li>
+
+ <a href="../../../metron-platform/index.html" title="Platform">
+ <i class="icon-chevron-right"></i>
+ Platform</a>
+ </li>
+
+ <li>
+
+ <a href="../../../metron-sensors/index.html" title="Sensors">
+ <i class="icon-chevron-right"></i>
+ Sensors</a>
+ </li>
+ </ul>
+ </li>
+ </ul>
+
+
+
+ <hr class="divider" />
+
+ <div id="poweredBy">
+ <div class="clear"></div>
+ <div class="clear"></div>
+ <div class="clear"></div>
+ <a href="http://maven.apache.org/" title="Built by Maven" class="poweredBy">
+ <img class="builtBy" alt="Built by Maven" src="../../../images/logos/maven-feather.png" />
+ </a>
+ </div>
+ </div>
+ </div>
+
+
+ <div id="bodyColumn" class="span9" >
+
+ <h1>Developer Image for Apache Metron on Virtualbox</h1>
+<p>This image is a fully functional Metron installation that has been pre-loaded with Ambari, HDP and Metron.</p>
+<div class="section">
+<h2><a name="Getting_Started"></a>Getting Started</h2>
+<div class="section">
+<h3><a name="Prerequisites"></a>Prerequisites</h3>
+<p>As with the Singlenode Full Image, the computer used to deploy Apache Metron will need to have the following components installed.</p>
+
+<ul>
+
+<li><a class="externalLink" href="https://github.com/ansible/ansible">Ansible</a> 2.0.0.2</li>
+
+<li><a class="externalLink" href="https://www.vagrantup.com">Vagrant</a> 1.8.1</li>
+
+<li><a href="virtualbox.org">Virtualbox</a> 5.0.16</li>
+
+<li>Python 2.7.11</li>
+
+<li>Maven 3.3.9</li>
+</ul></div>
+<div class="section">
+<h3><a name="Ensure_vagrant_hostmanager_is_installed"></a>Ensure vagrant hostmanager is installed</h3>
+<p>To check and make sure you have the plugin installed execute the following:</p>
+
+<div class="source">
+<div class="source">
+<pre> vagrant plugin list
+</pre></div></div>
+<p>If you have it installed you should see the following listed in the output:</p>
+
+<div class="source">
+<div class="source">
+<pre> vagrant-hostmanager (1.8.1)
+</pre></div></div>
+<p>If it is not installed, you can install it with the following command:</p>
+
+<div class="source">
+<div class="source">
+<pre> vagrant plugin install vagrant-hostmanager
+</pre></div></div></div>
+<div class="section">
+<h3><a name="Launch_the_Metron_Development_Image"></a>Launch the Metron Development Image</h3>
+<p>Start the image with the following commands:</p>
+
+<div class="source">
+<div class="source">
+<pre> cd metron-deployment/vagrant/codelab-platform
+ ./run.sh
+</pre></div></div></div>
+<div class="section">
+<h3><a name="Work_with_Metron"></a>Work with Metron</h3>
+<p>As you build out new capabilities for Metron, you will need to re-deploy the Storm topologies. To do so, first HALT the running Storm topologies and then run:</p>
+
+<div class="source">
+<div class="source">
+<pre>./run_enrichment_role.sh
+</pre></div></div>
+<p>Remember Navigate to the following resources to explore your newly minted Apache Metron environment.</p>
+
+<ul>
+
+<li><a class="externalLink" href="http://node1:5000">Metron</a></li>
+
+<li><a class="externalLink" href="http://node1:8080">Ambari</a></li>
+
+<li><a class="externalLink" href="http://node1:2812">Services</a></li>
+</ul>
+<p>Connecting to the host through SSH is as simple as running the following command.</p>
+
+<div class="source">
+<div class="source">
+<pre> vagrant ssh
+</pre></div></div></div></div>
+ </div>
+ </div>
+ </div>
+
+ <hr/>
+
+ <footer>
+ <div class="container-fluid">
+ <div class="row span12">Copyright © 2017.
+ All Rights Reserved.
+
+ </div>
+
+
+
+ </div>
+ </footer>
+ </body>
+</html>
Added: dev/incubator/metron/0.3.1-RC3-incubating/book-site/metron-deployment/vagrant/fastcapa-test-platform/index.html
==============================================================================
--- dev/incubator/metron/0.3.1-RC3-incubating/book-site/metron-deployment/vagrant/fastcapa-test-platform/index.html (added)
+++ dev/incubator/metron/0.3.1-RC3-incubating/book-site/metron-deployment/vagrant/fastcapa-test-platform/index.html Thu Feb 9 14:12:17 2017
@@ -0,0 +1,313 @@
+<!DOCTYPE html>
+<!--
+ | Generated by Apache Maven Doxia at 2017-02-09
+ | Rendered using Apache Maven Fluido Skin 1.3.0
+-->
+<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
+ <head>
+ <meta charset="UTF-8" />
+ <meta name="viewport" content="width=device-width, initial-scale=1.0" />
+ <meta name="Date-Revision-yyyymmdd" content="20170209" />
+ <meta http-equiv="Content-Language" content="en" />
+ <title>Metron – Fastcapa Test Environment</title>
+ <link rel="stylesheet" href="../../../css/apache-maven-fluido-1.3.0.min.css" />
+ <link rel="stylesheet" href="../../../css/site.css" />
+ <link rel="stylesheet" href="../../../css/print.css" media="print" />
+
+
+ <script type="text/javascript" src="../../../js/apache-maven-fluido-1.3.0.min.js"></script>
+
+
+
+<script type="text/javascript">$( document ).ready( function() { $( '.carousel' ).carousel( { interval: 3500 } ) } );</script>
+
+ </head>
+ <body class="topBarDisabled">
+
+
+
+
+ <div class="container-fluid">
+ <div id="banner">
+ <div class="pull-left">
+ <a href="http://metron.incubator.apache.org/" id="bannerLeft">
+ <img src="../../../images/metron-logo.png" alt="Apache Metron - Incubating" width="148px" height="48px"/>
+ </a>
+ </div>
+ <div class="pull-right"> <a href="http://incubator.apache.org/" id="bannerRight">
+ <img src="../../../images/ApacheIncubating_Logo.png" alt="Apache Incubating" width="192px" height="48px"/>
+ </a>
+ </div>
+ <div class="clear"><hr/></div>
+ </div>
+
+ <div id="breadcrumbs">
+ <ul class="breadcrumb">
+
+
+ <li class="">
+ <a href="http://www.apache.org" class="externalLink" title="Apache">
+ Apache</a>
+ </li>
+ <li class="divider ">/</li>
+ <li class="">
+ <a href="http://metron.incubator.apache.org/" class="externalLink" title="Metron-Incubating">
+ Metron-Incubating</a>
+ </li>
+ <li class="divider ">/</li>
+ <li class="">
+ <a href="../../../index.html" title="Documentation">
+ Documentation</a>
+ </li>
+ <li class="divider ">/</li>
+ <li class="">Fastcapa Test Environment</li>
+
+
+
+ <li id="publishDate" class="pull-right">Last Published: 2017-02-09</li> <li class="divider pull-right">|</li>
+ <li id="projectVersion" class="pull-right">Version: 0.3.1</li>
+
+ </ul>
+ </div>
+
+
+ <div class="row-fluid">
+ <div id="leftColumn" class="span3">
+ <div class="well sidebar-nav">
+
+
+ <ul class="nav nav-list">
+ <li class="nav-header">User Documentation</li>
+
+ <li>
+
+ <a href="../../../index.html" title="Metron">
+ <i class="icon-chevron-down"></i>
+ Metron</a>
+ <ul class="nav nav-list">
+
+ <li>
+
+ <a href="../../../Upgrading.html" title="Upgrading">
+ <i class="none"></i>
+ Upgrading</a>
+ </li>
+
+ <li>
+
+ <a href="../../../metron-analytics/index.html" title="Analytics">
+ <i class="icon-chevron-right"></i>
+ Analytics</a>
+ </li>
+
+ <li>
+
+ <a href="../../../metron-deployment/index.html" title="Deployment">
+ <i class="icon-chevron-down"></i>
+ Deployment</a>
+ <ul class="nav nav-list">
+
+ <li>
+
+ <a href="../../../metron-deployment/amazon-ec2/index.html" title="Amazon-ec2">
+ <i class="none"></i>
+ Amazon-ec2</a>
+ </li>
+
+ <li>
+
+ <a href="../../../metron-deployment/packaging/docker/ansible-docker/index.html" title="Ansible-docker">
+ <i class="none"></i>
+ Ansible-docker</a>
+ </li>
+
+ <li>
+
+ <a href="../../../metron-deployment/packaging/docker/rpm-docker/index.html" title="Rpm-docker">
+ <i class="none"></i>
+ Rpm-docker</a>
+ </li>
+
+ <li>
+
+ <a href="../../../metron-deployment/packer-build/index.html" title="Packer-build">
+ <i class="none"></i>
+ Packer-build</a>
+ </li>
+
+ <li>
+
+ <a href="../../../metron-deployment/roles/index.html" title="Roles">
+ <i class="icon-chevron-right"></i>
+ Roles</a>
+ </li>
+
+ <li>
+
+ <a href="../../../metron-deployment/vagrant/index.html" title="Vagrant">
+ <i class="icon-chevron-down"></i>
+ Vagrant</a>
+ <ul class="nav nav-list">
+
+ <li>
+
+ <a href="../../../metron-deployment/vagrant/codelab-platform/index.html" title="Codelab-platform">
+ <i class="none"></i>
+ Codelab-platform</a>
+ </li>
+
+ <li class="active">
+
+ <a href="#"><i class="none"></i>Fastcapa-test-platform</a>
+ </li>
+
+ <li>
+
+ <a href="../../../metron-deployment/vagrant/full-dev-platform/index.html" title="Full-dev-platform">
+ <i class="none"></i>
+ Full-dev-platform</a>
+ </li>
+
+ <li>
+
+ <a href="../../../metron-deployment/vagrant/quick-dev-platform/index.html" title="Quick-dev-platform">
+ <i class="none"></i>
+ Quick-dev-platform</a>
+ </li>
+ </ul>
+ </li>
+ </ul>
+ </li>
+
+ <li>
+
+ <a href="../../../metron-docker/index.html" title="Docker">
+ <i class="none"></i>
+ Docker</a>
+ </li>
+
+ <li>
+
+ <a href="../../../metron-platform/index.html" title="Platform">
+ <i class="icon-chevron-right"></i>
+ Platform</a>
+ </li>
+
+ <li>
+
+ <a href="../../../metron-sensors/index.html" title="Sensors">
+ <i class="icon-chevron-right"></i>
+ Sensors</a>
+ </li>
+ </ul>
+ </li>
+ </ul>
+
+
+
+ <hr class="divider" />
+
+ <div id="poweredBy">
+ <div class="clear"></div>
+ <div class="clear"></div>
+ <div class="clear"></div>
+ <a href="http://maven.apache.org/" title="Built by Maven" class="poweredBy">
+ <img class="builtBy" alt="Built by Maven" src="../../../images/logos/maven-feather.png" />
+ </a>
+ </div>
+ </div>
+ </div>
+
+
+ <div id="bodyColumn" class="span9" >
+
+ <h1>Fastcapa Test Environment</h1>
+<p>Provides a test environment for the development and testing of Fastcapa. The environment is automatically validated after it is created to ensure that Fastcapa is behaving correctly.</p>
+<p>Two virtualized nodes are launched with Vagrant that can communicate with one another over a private network. </p>
+
+<ul>
+
+<li>The <tt>source</tt> node uses Metron’s <tt>pcap_replay</tt> functionality to transmit raw network packet data over a private network.</li>
+
+<li>The <tt>sink</tt> node is running <tt>fastcapa</tt> and is capturing these network packets.</li>
+
+<li>Fastcapa then transforms and bundles the packets into a message.</li>
+
+<li>The message is sent to a Kafka broker running on the <tt>source</tt> node.</li>
+</ul>
+<div class="section">
+<h2><a name="Getting_Started"></a>Getting Started</h2>
+<p>Simply run <tt>vagrant up</tt> to launch the environment. Automated tests are executed after provisioning completes to ensure that Fastcapa and the rest of the environment is functioning properly.</p>
+
+<div class="source">
+<div class="source">
+<pre>$ vagrant up
+==> source: Running provisioner: ansible...
+ source: Running ansible-playbook...
+...
+TASK [debug] *******************************************************************
+ok: [source] => {
+ "msg": "Successfully received packets sent from pcap-replay!"
+}
+...
+TASK [debug] *******************************************************************
+ok: [source] => {
+ "msg": "Successfully received a Kafka message from fastcapa!"
+}
+</pre></div></div></div>
+<div class="section">
+<h2><a name="Going_Deeper"></a>Going Deeper</h2>
+<p>This section will outline in more detail the environment and how to interact with it.</p>
+<div class="section">
+<h3><a name="source"></a><tt>source</tt></h3>
+<p>To validate that the <tt>source</tt> node is functioning properly, run the following commands.</p>
+<p>First, ensure that the <tt>pcap-replay</tt> service is running.</p>
+
+<div class="source">
+<div class="source">
+<pre>vagrant ssh source
+sudo service pcap-replay status
+</pre></div></div>
+<p>Use <tt>tcpdump</tt> to ensure that the raw packet data is being sent over the private network. Enter ‘CTRL-C’ to kill the <tt>tcpdump</tt> process once you are able to see that packets are being sent.</p>
+
+<div class="source">
+<div class="source">
+<pre>sudo yum -y install tcpdump
+sudo tcpdump -i enp0s8
+</pre></div></div></div>
+<div class="section">
+<h3><a name="sink"></a><tt>sink</tt></h3>
+<p>Next validate that the <tt>sink</tt> is functioning properly. Run the following commands starting from the host operating system. </p>
+<p>First, ensure that the <tt>fastcapa</tt> service is running.</p>
+
+<div class="source">
+<div class="source">
+<pre>vagrant ssh sink
+service fastcapa status
+</pre></div></div>
+<p>Ensure that the raw network packet data is being received by Kafka.</p>
+
+<div class="source">
+<div class="source">
+<pre>/usr/hdp/current/kafka-broker/bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic pcap
+</pre></div></div>
+<p>Enter ‘CTRL-C’ to kill the <tt>kafka-console-consumer</tt> process once you are able to see that packets are being sent. These packets will appear to be gibberish in the console. This is the raw binary network packet data after all.</p></div></div>
+ </div>
+ </div>
+ </div>
+
+ <hr/>
+
+ <footer>
+ <div class="container-fluid">
+ <div class="row span12">Copyright © 2017.
+ All Rights Reserved.
+
+ </div>
+
+
+
+ </div>
+ </footer>
+ </body>
+</html>