You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@metron.apache.org by ma...@apache.org on 2017/12/08 09:27:19 UTC
svn commit: r23442 [8/24] - in /dev/metron/0.4.2-RC1: ./ site-book/
site-book/css/ site-book/images/ site-book/images/logos/
site-book/images/profiles/ site-book/img/ site-book/js/
site-book/metron-analytics/ site-book/metron-analytics/metron-maas-serv...
Added: dev/metron/0.4.2-RC1/site-book/metron-deployment/Kerberos-manual-setup.html
==============================================================================
--- dev/metron/0.4.2-RC1/site-book/metron-deployment/Kerberos-manual-setup.html (added)
+++ dev/metron/0.4.2-RC1/site-book/metron-deployment/Kerberos-manual-setup.html Fri Dec 8 09:27:19 2017
@@ -0,0 +1,906 @@
+<!DOCTYPE html>
+<!--
+ | Generated by Apache Maven Doxia at 2017-12-08
+ | Rendered using Apache Maven Fluido Skin 1.3.0
+-->
+<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
+ <head>
+ <meta charset="UTF-8" />
+ <meta name="viewport" content="width=device-width, initial-scale=1.0" />
+ <meta name="Date-Revision-yyyymmdd" content="20171208" />
+ <meta http-equiv="Content-Language" content="en" />
+ <title>Metron – Kerberos Setup</title>
+ <link rel="stylesheet" href="../css/apache-maven-fluido-1.3.0.min.css" />
+ <link rel="stylesheet" href="../css/site.css" />
+ <link rel="stylesheet" href="../css/print.css" media="print" />
+
+
+ <script type="text/javascript" src="../js/apache-maven-fluido-1.3.0.min.js"></script>
+
+
+
+<script type="text/javascript">$( document ).ready( function() { $( '.carousel' ).carousel( { interval: 3500 } ) } );</script>
+
+ </head>
+ <body class="topBarDisabled">
+
+
+
+
+ <div class="container-fluid">
+ <div id="banner">
+ <div class="pull-left">
+ <a href="http://metron.apache.org/" id="bannerLeft">
+ <img src="../images/metron-logo.png" alt="Apache Metron" width="148px" height="48px"/>
+ </a>
+ </div>
+ <div class="pull-right"> </div>
+ <div class="clear"><hr/></div>
+ </div>
+
+ <div id="breadcrumbs">
+ <ul class="breadcrumb">
+
+
+ <li class="">
+ <a href="http://www.apache.org" class="externalLink" title="Apache">
+ Apache</a>
+ </li>
+ <li class="divider ">/</li>
+ <li class="">
+ <a href="http://metron.apache.org/" class="externalLink" title="Metron">
+ Metron</a>
+ </li>
+ <li class="divider ">/</li>
+ <li class="">
+ <a href="../index.html" title="Documentation">
+ Documentation</a>
+ </li>
+ <li class="divider ">/</li>
+ <li class="">Kerberos Setup</li>
+
+
+
+ <li id="publishDate" class="pull-right">Last Published: 2017-12-08</li> <li class="divider pull-right">|</li>
+ <li id="projectVersion" class="pull-right">Version: 0.4.2</li>
+
+ </ul>
+ </div>
+
+
+ <div class="row-fluid">
+ <div id="leftColumn" class="span3">
+ <div class="well sidebar-nav">
+
+
+ <ul class="nav nav-list">
+ <li class="nav-header">User Documentation</li>
+
+ <li>
+
+ <a href="../index.html" title="Metron">
+ <i class="icon-chevron-down"></i>
+ Metron</a>
+ <ul class="nav nav-list">
+
+ <li>
+
+ <a href="../Upgrading.html" title="Upgrading">
+ <i class="none"></i>
+ Upgrading</a>
+ </li>
+
+ <li>
+
+ <a href="../metron-analytics/index.html" title="Analytics">
+ <i class="icon-chevron-right"></i>
+ Analytics</a>
+ </li>
+
+ <li>
+
+ <a href="../metron-contrib/metron-docker/index.html" title="Docker">
+ <i class="none"></i>
+ Docker</a>
+ </li>
+
+ <li>
+
+ <a href="../metron-deployment/index.html" title="Deployment">
+ <i class="icon-chevron-down"></i>
+ Deployment</a>
+ <ul class="nav nav-list">
+
+ <li>
+
+ <a href="../metron-deployment/Kerberos-ambari-setup.html" title="Kerberos-ambari-setup">
+ <i class="none"></i>
+ Kerberos-ambari-setup</a>
+ </li>
+
+ <li class="active">
+
+ <a href="#"><i class="none"></i>Kerberos-manual-setup</a>
+ </li>
+
+ <li>
+
+ <a href="../metron-deployment/amazon-ec2/index.html" title="Amazon-ec2">
+ <i class="none"></i>
+ Amazon-ec2</a>
+ </li>
+
+ <li>
+
+ <a href="../metron-deployment/other-examples/index.html" title="Other-examples">
+ <i class="icon-chevron-right"></i>
+ Other-examples</a>
+ </li>
+
+ <li>
+
+ <a href="../metron-deployment/packaging/ambari/index.html" title="Ambari">
+ <i class="none"></i>
+ Ambari</a>
+ </li>
+
+ <li>
+
+ <a href="../metron-deployment/packaging/docker/ansible-docker/index.html" title="Ansible-docker">
+ <i class="none"></i>
+ Ansible-docker</a>
+ </li>
+
+ <li>
+
+ <a href="../metron-deployment/packaging/docker/rpm-docker/index.html" title="Rpm-docker">
+ <i class="none"></i>
+ Rpm-docker</a>
+ </li>
+
+ <li>
+
+ <a href="../metron-deployment/packaging/packer-build/index.html" title="Packer-build">
+ <i class="none"></i>
+ Packer-build</a>
+ </li>
+
+ <li>
+
+ <a href="../metron-deployment/roles/index.html" title="Roles">
+ <i class="icon-chevron-right"></i>
+ Roles</a>
+ </li>
+
+ <li>
+
+ <a href="../metron-deployment/vagrant/index.html" title="Vagrant">
+ <i class="icon-chevron-right"></i>
+ Vagrant</a>
+ </li>
+ </ul>
+ </li>
+
+ <li>
+
+ <a href="../metron-interface/metron-alerts/index.html" title="Alerts">
+ <i class="none"></i>
+ Alerts</a>
+ </li>
+
+ <li>
+
+ <a href="../metron-interface/metron-config/index.html" title="Config">
+ <i class="none"></i>
+ Config</a>
+ </li>
+
+ <li>
+
+ <a href="../metron-interface/metron-rest/index.html" title="Rest">
+ <i class="none"></i>
+ Rest</a>
+ </li>
+
+ <li>
+
+ <a href="../metron-platform/index.html" title="Platform">
+ <i class="icon-chevron-right"></i>
+ Platform</a>
+ </li>
+
+ <li>
+
+ <a href="../metron-sensors/index.html" title="Sensors">
+ <i class="icon-chevron-right"></i>
+ Sensors</a>
+ </li>
+
+ <li>
+
+ <a href="../metron-stellar/stellar-3rd-party-example/index.html" title="Stellar-3rd-party-example">
+ <i class="none"></i>
+ Stellar-3rd-party-example</a>
+ </li>
+
+ <li>
+
+ <a href="../metron-stellar/stellar-common/index.html" title="Stellar-common">
+ <i class="icon-chevron-right"></i>
+ Stellar-common</a>
+ </li>
+
+ <li>
+
+ <a href="../use-cases/index.html" title="Use-cases">
+ <i class="icon-chevron-right"></i>
+ Use-cases</a>
+ </li>
+ </ul>
+ </li>
+ </ul>
+
+
+
+ <hr class="divider" />
+
+ <div id="poweredBy">
+ <div class="clear"></div>
+ <div class="clear"></div>
+ <div class="clear"></div>
+ <a href="http://maven.apache.org/" title="Built by Maven" class="poweredBy">
+ <img class="builtBy" alt="Built by Maven" src="../images/logos/maven-feather.png" />
+ </a>
+ </div>
+ </div>
+ </div>
+
+
+ <div id="bodyColumn" class="span9" >
+
+ <h1>Kerberos Setup</h1>
+<p>This document provides instructions for kerberizing Metron’s Vagrant-based development environments. These instructions do not cover the Ambari MPack or sensors. General Kerberization notes can be found in the metron-deployment <a href="../index.html">README.md</a>.</p>
+
+<ul>
+
+<li><a href="#Setup">Setup</a></li>
+
+<li><a href="#Setup_a_KDC">Setup a KDC</a></li>
+
+<li><a href="#Verify_KDC">Verify KDC</a></li>
+
+<li><a href="#Enable_Kerberos">Enable Kerberos</a></li>
+
+<li><a href="#Kafka_Authorization">Kafka Authorization</a></li>
+
+<li><a href="#HBase_Authorization">HBase Authorization</a></li>
+
+<li><a href="#Storm_Authorization">Storm Authorization</a></li>
+
+<li><a href="#Start_Metron">Start Metron</a></li>
+
+<li><a href="#Push_Data">Push Data</a></li>
+
+<li><a href="#More_Information">More Information</a></li>
+</ul>
+<div class="section">
+<h2><a name="Setup"></a>Setup</h2>
+
+<ol style="list-style-type: decimal">
+
+<li>
+<p>Deploy the <a href="vagrant/full-dev-platform/index.html">development environment.</a>.</p></li>
+
+<li>
+<p>Export the following environment variables. These need to be set for the remainder of the instructions. Replace <tt>node1</tt> with the appropriate hosts, if you are running Metron anywhere other than Vagrant.</p>
+
+<div class="source">
+<div class="source">
+<pre># execute as root
+sudo su -
+export KAFKA_HOME="/usr/hdp/current/kafka-broker"
+export ZOOKEEPER=node1:2181
+export ELASTICSEARCH=node1:9200
+export BROKERLIST=node1:6667
+export HDP_HOME="/usr/hdp/current"
+export KAFKA_HOME="${HDP_HOME}/kafka-broker"
+export METRON_VERSION="0.4.2"
+export METRON_HOME="/usr/metron/${METRON_VERSION}"
+</pre></div></div></li>
+
+<li>
+<p>Execute the following commands as root.</p>
+
+<div class="source">
+<div class="source">
+<pre>sudo su -
+</pre></div></div></li>
+
+<li>
+<p>Stop all Metron topologies. They will be restarted again once Kerberos has been enabled.</p>
+
+<div class="source">
+<div class="source">
+<pre>for topology in bro snort enrichment indexing; do
+ storm kill $topology;
+done
+</pre></div></div></li>
+
+<li>
+<p>Create the <tt>metron</tt> user’s home directory in HDFS.</p>
+
+<div class="source">
+<div class="source">
+<pre>sudo -u hdfs hdfs dfs -mkdir /user/metron
+sudo -u hdfs hdfs dfs -chown metron:hdfs /user/metron
+sudo -u hdfs hdfs dfs -chmod 770 /user/metron
+</pre></div></div></li>
+</ol></div>
+<div class="section">
+<h2><a name="Setup_a_KDC"></a>Setup a KDC</h2>
+
+<ol style="list-style-type: decimal">
+
+<li>
+<p>Install dependencies.</p>
+
+<div class="source">
+<div class="source">
+<pre>yum -y install krb5-server krb5-libs krb5-workstation
+</pre></div></div></li>
+
+<li>
+<p>Define the current host as the KDC.</p>
+
+<div class="source">
+<div class="source">
+<pre>KDC=`hostname`
+sed -i.orig 's/kerberos.example.com/'"$KDC"'/g' /etc/krb5.conf
+cp -f /etc/krb5.conf /var/lib/ambari-server/resources/scripts
+</pre></div></div></li>
+
+<li>
+<p>Ensure that the KDC can issue renewable tickets. This may be necessary on a real cluster, but should not be on <a href="vagrant/full-dev-platform/index.html">Full Dev</a>.</p>
+<p>Edit <tt>/var/kerberos/krb5kdc/kdc.conf</tt> and ensure the following is added to the <tt>realm</tt> section</p>
+
+<div class="source">
+<div class="source">
+<pre>max_renewable_life = 7d
+</pre></div></div></li>
+
+<li>
+<p>Create the KDC principal database. You will be prompted for a password. This step takes a moment.</p>
+
+<div class="source">
+<div class="source">
+<pre>kdb5_util create -s
+</pre></div></div></li>
+
+<li>
+<p>Start the KDC and ensure that it starts on boot.</p>
+
+<div class="source">
+<div class="source">
+<pre>/etc/rc.d/init.d/krb5kdc start
+chkconfig krb5kdc on
+</pre></div></div></li>
+
+<li>
+<p>Start the Kerberos Admin service and ensure that it starts on boot. </p>
+
+<div class="source">
+<div class="source">
+<pre>/etc/rc.d/init.d/kadmin start
+chkconfig kadmin on
+</pre></div></div></li>
+
+<li>
+<p>Setup the <tt>admin</tt> principal. You will be prompted for a password; do not forget it.</p>
+
+<div class="source">
+<div class="source">
+<pre>kadmin.local -q "addprinc admin/admin"
+</pre></div></div></li>
+
+<li>
+<p>Setup the <tt>metron</tt> principal. You will <tt>kinit</tt> as the <tt>metron</tt> principal when running topologies. You will be prompted for a password; do not forget it.</p>
+
+<div class="source">
+<div class="source">
+<pre>kadmin.local -q "addprinc metron"
+</pre></div></div></li>
+</ol></div>
+<div class="section">
+<h2><a name="Verify_KDC"></a>Verify KDC</h2>
+
+<ol style="list-style-type: decimal">
+
+<li>
+<p>Ticket renewal is disallowed by default in many Linux distributions. If the KDC cannot issue renewable tickets, an error will be thrown when starting Metron’s Storm topologies:</p>
+
+<div class="source">
+<div class="source">
+<pre>Exception in thread "main" java.lang.RuntimeException:
+java.lang.RuntimeException: The TGT found is not renewable
+</pre></div></div></li>
+
+<li>
+<p>Ensure the Metron keytab is renewable. Look for the ‘R’ flag in the output of the following command.</p>
+
+<div class="source">
+<div class="source">
+<pre>klist -f
+</pre></div></div>
+
+<ul>
+
+<li>If the ‘R’ flags are present, you may skip to next section.</li>
+
+<li>If the ‘R’ flags are absent, you will need to follow the below steps:</li>
+ </ul></li>
+
+<li>
+<p>If the KDC is already setup, then editing <tt>max_life</tt> and <tt>max_renewable_life</tt> in <tt>/var/kerberos/krb5kdc/kdc.conf</tt>, then restarting <tt>kadmin</tt> and <tt>krb5kdc</tt> services will not change the policies for existing users.</p>
+<p>You need to set the renew lifetime for existing users and the <tt>krbtgt</tt> realm. Modify the appropriate principals to allow renewable tickets using the following commands. Adjust the parameters to match your desired KDC parameters:</p>
+
+<div class="source">
+<div class="source">
+<pre>kadmin.local -q "modprinc -maxlife 1days -maxrenewlife 7days +allow_renewable krbtgt/EXAMPLE.COM@EXAMPLE.COM"
+kadmin.local -q "modprinc -maxlife 1days -maxrenewlife 7days +allow_renewable metron@EXAMPLE.COM"
+</pre></div></div></li>
+</ol></div>
+<div class="section">
+<h2><a name="Enable_Kerberos"></a>Enable Kerberos</h2>
+
+<ol style="list-style-type: decimal">
+
+<li>
+<p>In <a class="externalLink" href="http://node1:8080">Ambari</a>, setup Storm to use Kerberos and run worker jobs as the submitting user.</p>
+<p>a. Add the following properties to the custom storm-site:</p>
+
+<div class="source">
+<div class="source">
+<pre>topology.auto-credentials=['org.apache.storm.security.auth.kerberos.AutoTGT']
+nimbus.credential.renewers.classes=['org.apache.storm.security.auth.kerberos.AutoTGT']
+supervisor.run.worker.as.user=true
+</pre></div></div>
+<p>b. In the Storm config section in Ambari, choose “Add Property” under custom storm-site:</p>
+<p><img src="../images/ambari-storm-site.png" alt="custom storm-site" /></p>
+<p>c. In the dialog window, choose the “bulk property add mode” toggle button and add the below values:</p>
+<p><img src="../images/ambari-storm-site-properties.png" alt="custom storm-site properties" /></p></li>
+
+<li>
+<p>Kerberize the cluster via Ambari. More detailed documentation can be found <a class="externalLink" href="http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.5.3/bk_security/content/_enabling_kerberos_security_in_ambari.html">here</a>.</p>
+<p>a. For this exercise, choose existing MIT KDC (this is what we setup and installed in the previous steps.)</p>
+<p><img src="../images/enable-kerberos.png" alt="enable keberos" /></p>
+<p><img src="../images/enable-kerberos-started.png" alt="enable keberos get started" /></p>
+<p>b. Setup Kerberos configuration. Realm is EXAMPLE.COM. The admin principal will end up as admin/admin@EXAMPLE.COM when testing the KDC. Use the password you entered during the step for adding the admin principal.</p>
+<p><img src="../images/enable-kerberos-configure-kerberos.png" alt="enable keberos configure" /></p>
+<p>c. Click through to “Start and Test Services.” Let the cluster spin up, but don’t worry about starting up Metron via Ambari - we’re going to run the parsers manually against the rest of the Hadoop cluster Kerberized. The wizard will fail at starting Metron, but this is OK. Click “continue.” When you’re finished, the custom storm-site should look similar to the following:</p>
+<p><img src="../images/custom-storm-site-final.png" alt="enable keberos configure" /></p></li>
+
+<li>
+<p>Create a Metron keytab</p>
+
+<div class="source">
+<div class="source">
+<pre>kadmin.local -q "ktadd -k metron.headless.keytab metron@EXAMPLE.COM"
+cp metron.headless.keytab /etc/security/keytabs
+chown metron:hadoop /etc/security/keytabs/metron.headless.keytab
+chmod 440 /etc/security/keytabs/metron.headless.keytab
+</pre></div></div></li>
+</ol></div>
+<div class="section">
+<h2><a name="Kafka_Authorization"></a>Kafka Authorization</h2>
+
+<ol style="list-style-type: decimal">
+
+<li>
+<p>Acquire a Kerberos ticket using the <tt>metron</tt> principal.</p>
+
+<div class="source">
+<div class="source">
+<pre>kinit -kt /etc/security/keytabs/metron.headless.keytab metron@EXAMPLE.COM
+</pre></div></div></li>
+
+<li>
+<p>Create any additional Kafka topics that you will need. We need to create the topics before adding the required ACLs. The current full dev installation will deploy bro, snort, enrichments, and indexing only. For example, you may want to add a topic for ‘yaf’ telemetry.</p>
+
+<div class="source">
+<div class="source">
+<pre>${KAFKA_HOME}/bin/kafka-topics.sh \
+ --zookeeper ${ZOOKEEPER} \
+ --create \
+ --topic yaf \
+ --partitions 1 \
+ --replication-factor 1
+</pre></div></div></li>
+
+<li>
+<p>Setup Kafka ACLs for the <tt>bro</tt>, <tt>snort</tt>, <tt>enrichments</tt>, and <tt>indexing</tt> topics. Run the same command against any additional topics that you might be using; for example <tt>yaf</tt>.</p>
+
+<div class="source">
+<div class="source">
+<pre>export KERB_USER=metron
+for topic in bro snort enrichments indexing; do
+ ${KAFKA_HOME}/bin/kafka-acls.sh \
+ --authorizer kafka.security.auth.SimpleAclAuthorizer \
+ --authorizer-properties zookeeper.connect=${ZOOKEEPER} \
+ --add \
+ --allow-principal User:${KERB_USER} \
+ --topic ${topic}
+done
+</pre></div></div></li>
+
+<li>
+<p>Setup Kafka ACLs for the consumer groups. This command sets the ACLs for Bro, Snort, YAF, Enrichments, Indexing, and the Profiler. Execute the same command for any additional Parsers that you may be running.</p>
+
+<div class="source">
+<div class="source">
+<pre>export KERB_USER=metron
+for group in bro_parser snort_parser yaf_parser enrichments indexing profiler; do
+ ${KAFKA_HOME}/bin/kafka-acls.sh \
+ --authorizer kafka.security.auth.SimpleAclAuthorizer \
+ --authorizer-properties zookeeper.connect=${ZOOKEEPER} \
+ --add \
+ --allow-principal User:${KERB_USER} \
+ --group ${group}
+done
+</pre></div></div></li>
+
+<li>
+<p>Add the <tt>metron</tt> principal to the <tt>kafka-cluster</tt> ACL.</p>
+
+<div class="source">
+<div class="source">
+<pre>${KAFKA_HOME}/bin/kafka-acls.sh \
+ --authorizer kafka.security.auth.SimpleAclAuthorizer \
+ --authorizer-properties zookeeper.connect=${ZOOKEEPER} \
+ --add \
+ --allow-principal User:${KERB_USER} \
+ --cluster kafka-cluster
+</pre></div></div></li>
+</ol></div>
+<div class="section">
+<h2><a name="HBase_Authorization"></a>HBase Authorization</h2>
+
+<ol style="list-style-type: decimal">
+
+<li>
+<p>Acquire a Kerberos ticket using the <tt>hbase</tt> principal</p>
+
+<div class="source">
+<div class="source">
+<pre>kinit -kt /etc/security/keytabs/hbase.headless.keytab hbase-metron_cluster@EXAMPLE.COM
+</pre></div></div></li>
+
+<li>
+<p>Grant permissions for the HBase tables used in Metron.</p>
+
+<div class="source">
+<div class="source">
+<pre>echo "grant 'metron', 'RW', 'threatintel'" | hbase shell
+echo "grant 'metron', 'RW', 'enrichment'" | hbase shell
+</pre></div></div></li>
+
+<li>
+<p>If you are using the Profiler, do the same for its HBase table.</p>
+
+<div class="source">
+<div class="source">
+<pre>echo "create 'profiler', 'P'" | hbase shell
+echo "grant 'metron', 'RW', 'profiler', 'P'" | hbase shell
+</pre></div></div></li>
+</ol></div>
+<div class="section">
+<h2><a name="Storm_Authorization"></a>Storm Authorization</h2>
+
+<ol style="list-style-type: decimal">
+
+<li>
+<p>Switch to the <tt>metron</tt> user and acquire a Kerberos ticket for the <tt>metron</tt> principal.</p>
+
+<div class="source">
+<div class="source">
+<pre>su metron
+kinit -kt /etc/security/keytabs/metron.headless.keytab metron@EXAMPLE.COM
+</pre></div></div></li>
+
+<li>
+<p>Create the directory <tt>/home/metron/.storm</tt> and switch to that directory.</p>
+
+<div class="source">
+<div class="source">
+<pre>mkdir /home/metron/.storm
+cd /home/metron/.storm
+</pre></div></div></li>
+
+<li>
+<p>Ensure the Metron keytab is renewable. See <a href="#Verify_KDC">Verify KDC</a> above.</p></li>
+
+<li>
+<p>Create a client JAAS file at <tt>/home/metron/.storm/client_jaas.conf</tt>. This should look identical to the Storm client JAAS file located at <tt>/etc/storm/conf/client_jaas.conf</tt> except for the addition of a <tt>Client</tt> stanza. The <tt>Client</tt> stanza is used for Zookeeper. All quotes and semicolons are necessary.</p>
+
+<div class="source">
+<div class="source">
+<pre>cat << EOF > client_jaas.conf
+StormClient {
+ com.sun.security.auth.module.Krb5LoginModule required
+ useTicketCache=true
+ renewTicket=true
+ serviceName="nimbus";
+};
+Client {
+ com.sun.security.auth.module.Krb5LoginModule required
+ useKeyTab=true
+ keyTab="/etc/security/keytabs/metron.headless.keytab"
+ storeKey=true
+ useTicketCache=false
+ serviceName="zookeeper"
+ principal="metron@EXAMPLE.COM";
+};
+KafkaClient {
+ com.sun.security.auth.module.Krb5LoginModule required
+ useKeyTab=true
+ keyTab="/etc/security/keytabs/metron.headless.keytab"
+ storeKey=true
+ useTicketCache=false
+ serviceName="kafka"
+ principal="metron@EXAMPLE.COM";
+};
+EOF
+</pre></div></div></li>
+
+<li>
+<p>Create a YAML file at <tt>/home/metron/.storm/storm.yaml</tt>. This should point to the client JAAS file. Set the array of nimbus hosts accordingly.</p>
+
+<div class="source">
+<div class="source">
+<pre>cat << EOF > /home/metron/.storm/storm.yaml
+nimbus.seeds : ['node1']
+java.security.auth.login.config : '/home/metron/.storm/client_jaas.conf'
+storm.thrift.transport : 'org.apache.storm.security.auth.kerberos.KerberosSaslTransportPlugin'
+EOF
+</pre></div></div></li>
+
+<li>
+<p>Create an auxiliary storm configuration file at <tt>/home/metron/storm-config.json</tt>. Note the login config option in the file points to the client JAAS file.</p>
+
+<div class="source">
+<div class="source">
+<pre>cat << EOF > /home/metron/storm-config.json
+{
+ "topology.worker.childopts" : "-Djava.security.auth.login.config=/home/metron/.storm/client_jaas.conf"
+}
+EOF
+</pre></div></div></li>
+
+<li>
+<p>Configure the Enrichment, Indexing and Profiler topologies to use the client JAAS file. To do this, the following key-value pairs:</p>
+
+<ul>
+
+<li><tt>kafka.security.protocol=PLAINTEXTSASL</tt></li>
+
+<li><tt>topology.worker.childopts=-Djava.security.auth.login.config=/home/metron/.storm/client_jaas.conf</tt></li>
+ </ul>
+<p>must be added to each of the topology properties files:</p>
+
+<ul>
+
+<li><tt>${METRON_HOME}/config/enrichment.properties</tt></li>
+
+<li><tt>${METRON_HOME}/config/elasticsearch.properties</tt></li>
+
+<li><tt>${METRON_HOME}/config/profiler.properties</tt></li>
+ </ul>
+<p>You may use the following command to automate this step:</p>
+
+<div class="source">
+<div class="source">
+<pre>for file in enrichment.properties elasticsearch.properties profiler.properties; do
+ echo ${file}
+ sed -i "s/^kafka.security.protocol=.*/kafka.security.protocol=PLAINTEXTSASL/" "${METRON_HOME}/config/${file}"
+ sed -i "s/^topology.worker.childopts=.*/topology.worker.childopts=-Djava.security.auth.login.config=\/home\/metron\/.storm\/client_jaas.conf/" "${METRON_HOME}/config/${file}"
+done
+</pre></div></div></li>
+</ol></div>
+<div class="section">
+<h2><a name="Start_Metron"></a>Start Metron</h2>
+
+<ol style="list-style-type: decimal">
+
+<li>
+<p>Switch to the <tt>metron</tt> user and acquire a Kerberos ticket for the <tt>metron</tt> principal.</p>
+
+<div class="source">
+<div class="source">
+<pre>su metron
+kinit -kt /etc/security/keytabs/metron.headless.keytab metron@EXAMPLE.COM
+</pre></div></div></li>
+
+<li>
+<p>Restart the parser topologies. Be sure to pass in the new parameter, <tt>-ksp</tt> or <tt>--kafka_security_protocol</tt>. The following command will start only the Bro and Snort topologies. Execute the same command for any other Parsers that you may need, for example <tt>yaf</tt>.</p>
+
+<div class="source">
+<div class="source">
+<pre>for parser in bro snort; do
+ ${METRON_HOME}/bin/start_parser_topology.sh \
+ -z ${ZOOKEEPER} \
+ -s ${parser} \
+ -ksp SASL_PLAINTEXT \
+ -e /home/metron/storm-config.json;
+done
+</pre></div></div></li>
+
+<li>
+<p>Restart the Enrichment and Indexing topologies.</p>
+
+<div class="source">
+<div class="source">
+<pre>${METRON_HOME}/bin/start_enrichment_topology.sh
+${METRON_HOME}/bin/start_elasticsearch_topology.sh
+</pre></div></div></li>
+</ol>
+<p>Metron should be ready to receive data.</p></div>
+<div class="section">
+<h2><a name="Push_Data"></a>Push Data</h2>
+
+<ol style="list-style-type: decimal">
+
+<li>
+<p>Push some sample data to one of the parser topics. E.g for Bro we took raw data from <a href="../metron-platform/metron-integration-test/src/main/sample/data/bro/raw/BroExampleOutput/index.html">metron/metron-platform/metron-integration-test/src/main/sample/data/bro/raw/BroExampleOutput</a></p>
+
+<div class="source">
+<div class="source">
+<pre>cat sample-bro.txt | ${KAFKA_HOME}/kafka-broker/bin/kafka-console-producer.sh \
+ --broker-list ${BROKERLIST} \
+ --security-protocol SASL_PLAINTEXT \
+ --topic bro
+</pre></div></div></li>
+
+<li>
+<p>Wait a few moments for data to flow through the system and then check for data in the Elasticsearch indices. Replace yaf with whichever parser type you’ve chosen.</p>
+
+<div class="source">
+<div class="source">
+<pre>curl -XGET "${ELASTICSEARCH}/bro*/_search"
+curl -XGET "${ELASTICSEARCH}/bro*/_count"
+</pre></div></div></li>
+
+<li>
+<p>You should have data flowing from the parsers all the way through to the indexes. This completes the Kerberization instructions</p></li>
+</ol></div>
+<div class="section">
+<h2><a name="More_Information"></a>More Information</h2>
+<div class="section">
+<h3><a name="Kerberos"></a>Kerberos</h3>
+<p>Unsure of your Kerberos principal associated with a keytab? There are a couple ways to get this. One is via the list of principals that Ambari provides via downloadable csv. If you didn’t download this list, you can also check the principal manually by running the following against the keytab.</p>
+
+<div class="source">
+<div class="source">
+<pre>klist -kt /etc/security/keytabs/<keytab-file-name>
+</pre></div></div>
+<p>E.g.</p>
+
+<div class="source">
+<div class="source">
+<pre>klist -kt /etc/security/keytabs/hbase.headless.keytab
+Keytab name: FILE:/etc/security/keytabs/hbase.headless.keytab
+KVNO Timestamp Principal
+---- ----------------- --------------------------------------------------------
+ 1 03/28/17 19:29:36 hbase-metron_cluster@EXAMPLE.COM
+ 1 03/28/17 19:29:36 hbase-metron_cluster@EXAMPLE.COM
+ 1 03/28/17 19:29:36 hbase-metron_cluster@EXAMPLE.COM
+ 1 03/28/17 19:29:36 hbase-metron_cluster@EXAMPLE.COM
+ 1 03/28/17 19:29:36 hbase-metron_cluster@EXAMPLE.COM
+</pre></div></div></div>
+<div class="section">
+<h3><a name="Kafka_with_Kerberos_enabled"></a>Kafka with Kerberos enabled</h3>
+<div class="section">
+<h4><a name="Running_Sensors"></a>Running Sensors</h4>
+<p>A couple steps are required to produce data to a Kerberized Kafka topic. On the host you’ll be setting up your sensor(s), switch to the metron user and create a client_jaas.conf file in the metron home directory if one doesn’t already exist. It should be owned by metron:metron and contain at least the following stanza that tells the Kafka client how to interact with Kerberos:</p>
+
+<div class="source">
+<div class="source">
+<pre>su - metron
+cat ${METRON_HOME}/client_jaas.conf
+...
+KafkaClient {
+ com.sun.security.auth.module.Krb5LoginModule required
+ useKeyTab=true
+ keyTab="/etc/security/keytabs/metron.headless.keytab"
+ storeKey=true
+ useTicketCache=false
+ serviceName="kafka"
+ principal="metron@EXAMPLE.COM";
+};
+</pre></div></div>
+<p>You’ll also need to set KAFKA_OPTS to tell the Kafka client how to interact with Kerberos.</p>
+
+<div class="source">
+<div class="source">
+<pre>export KAFKA_OPTS="-Djava.security.auth.login.config=${METRON_HOME}/client_jaas.conf"
+</pre></div></div>
+<p>For sensors that leverage the Kafka console producer to pipe data into Metron, e.g. Snort and Yaf, you will need to modify the corresponding sensor shell scripts or config to append the SASL security protocol property. <tt>--security-protocol SASL_PLAINTEXT</tt>. Be sure to kinit with the metron user’s keytab before executing the script that starts the sensor.</p>
+<p>More notes can be found in <a href="../metron-sensors/index.html">metron/metron-sensors/README.md</a></p></div>
+<div class="section">
+<h4><a name="Write_data_to_a_topic_with_SASL"></a>Write data to a topic with SASL</h4>
+
+<div class="source">
+<div class="source">
+<pre>cat sample-yaf.txt | ${KAFKA_HOME}/bin/kafka-console-producer.sh \
+ --broker-list ${BROKERLIST} \
+ --security-protocol PLAINTEXTSASL \
+ --topic yaf
+</pre></div></div></div>
+<div class="section">
+<h4><a name="View_topic_data_from_latest_offset_with_SASL"></a>View topic data from latest offset with SASL</h4>
+
+<div class="source">
+<div class="source">
+<pre>${KAFKA_HOME}/bin/kafka-console-consumer.sh \
+ --zookeeper ${ZOOKEEPER} \
+ --security-protocol PLAINTEXTSASL \
+ --topic yaf
+</pre></div></div></div>
+<div class="section">
+<h4><a name="Modify_the_sensor-stubs_to_send_logs_via_SASL"></a>Modify the sensor-stubs to send logs via SASL</h4>
+
+<div class="source">
+<div class="source">
+<pre>sed -i 's/node1:6667 --topic/node1:6667 --security-protocol PLAINTEXTSASL --topic/' /opt/sensor-stubs/bin/start-*-stub
+for sensorstub in bro snort; do
+ service sensor-stubs stop ${sensorstub};
+ service sensor-stubs start ${sensorstub};
+done
+</pre></div></div></div>
+<div class="section">
+<h4><a name="Model_as_a_Service_on_Kerberos"></a>Model as a Service on Kerberos</h4>
+<p>MaaS works with kerberos, you have to remember to kinit with the metron user. There is one small issue out of the box (particularly on vagrant), you get an error like so when running <tt>$METRON_HOME/bin/maas_service.sh</tt>:</p>
+
+<div class="source">
+<div class="source">
+<pre>Requested user metron is not whitelisted and has id 501,which is below the minimum allowed 1000.
+</pre></div></div>
+<p>In order to correct this, you should:</p>
+
+<ul>
+
+<li>Navigate to the Yarn configuration in Ambari</li>
+
+<li>Click on “Advanced”</li>
+
+<li>Scroll to “Advanced yarn-env”</li>
+
+<li>Adjust the “Minimum user ID for submitting job” config to 500 from 1000</li>
+
+<li>You should then restart Yarn to have the change take effect.</li>
+</ul></div></div>
+<div class="section">
+<h3><a name="References"></a>References</h3>
+
+<ul>
+
+<li><a class="externalLink" href="https://github.com/apache/storm/blob/master/SECURITY.md">https://github.com/apache/storm/blob/master/SECURITY.md</a></li>
+</ul></div></div>
+ </div>
+ </div>
+ </div>
+
+ <hr/>
+
+ <footer>
+ <div class="container-fluid">
+ <div class="row span12">Copyright © 2017
+ <a href="https://www.apache.org">The Apache Software Foundation</a>.
+ All Rights Reserved.
+
+ </div>
+
+
+
+ </div>
+ </footer>
+ </body>
+</html>
Added: dev/metron/0.4.2-RC1/site-book/metron-deployment/amazon-ec2/index.html
==============================================================================
--- dev/metron/0.4.2-RC1/site-book/metron-deployment/amazon-ec2/index.html (added)
+++ dev/metron/0.4.2-RC1/site-book/metron-deployment/amazon-ec2/index.html Fri Dec 8 09:27:19 2017
@@ -0,0 +1,603 @@
+<!DOCTYPE html>
+<!--
+ | Generated by Apache Maven Doxia at 2017-12-08
+ | Rendered using Apache Maven Fluido Skin 1.3.0
+-->
+<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
+ <head>
+ <meta charset="UTF-8" />
+ <meta name="viewport" content="width=device-width, initial-scale=1.0" />
+ <meta name="Date-Revision-yyyymmdd" content="20171208" />
+ <meta http-equiv="Content-Language" content="en" />
+ <title>Metron – Apache Metron on Amazon EC2</title>
+ <link rel="stylesheet" href="../../css/apache-maven-fluido-1.3.0.min.css" />
+ <link rel="stylesheet" href="../../css/site.css" />
+ <link rel="stylesheet" href="../../css/print.css" media="print" />
+
+
+ <script type="text/javascript" src="../../js/apache-maven-fluido-1.3.0.min.js"></script>
+
+
+
+<script type="text/javascript">$( document ).ready( function() { $( '.carousel' ).carousel( { interval: 3500 } ) } );</script>
+
+ </head>
+ <body class="topBarDisabled">
+
+
+
+
+ <div class="container-fluid">
+ <div id="banner">
+ <div class="pull-left">
+ <a href="http://metron.apache.org/" id="bannerLeft">
+ <img src="../../images/metron-logo.png" alt="Apache Metron" width="148px" height="48px"/>
+ </a>
+ </div>
+ <div class="pull-right"> </div>
+ <div class="clear"><hr/></div>
+ </div>
+
+ <div id="breadcrumbs">
+ <ul class="breadcrumb">
+
+
+ <li class="">
+ <a href="http://www.apache.org" class="externalLink" title="Apache">
+ Apache</a>
+ </li>
+ <li class="divider ">/</li>
+ <li class="">
+ <a href="http://metron.apache.org/" class="externalLink" title="Metron">
+ Metron</a>
+ </li>
+ <li class="divider ">/</li>
+ <li class="">
+ <a href="../../index.html" title="Documentation">
+ Documentation</a>
+ </li>
+ <li class="divider ">/</li>
+ <li class="">Apache Metron on Amazon EC2</li>
+
+
+
+ <li id="publishDate" class="pull-right">Last Published: 2017-12-08</li> <li class="divider pull-right">|</li>
+ <li id="projectVersion" class="pull-right">Version: 0.4.2</li>
+
+ </ul>
+ </div>
+
+
+ <div class="row-fluid">
+ <div id="leftColumn" class="span3">
+ <div class="well sidebar-nav">
+
+
+ <ul class="nav nav-list">
+ <li class="nav-header">User Documentation</li>
+
+ <li>
+
+ <a href="../../index.html" title="Metron">
+ <i class="icon-chevron-down"></i>
+ Metron</a>
+ <ul class="nav nav-list">
+
+ <li>
+
+ <a href="../../Upgrading.html" title="Upgrading">
+ <i class="none"></i>
+ Upgrading</a>
+ </li>
+
+ <li>
+
+ <a href="../../metron-analytics/index.html" title="Analytics">
+ <i class="icon-chevron-right"></i>
+ Analytics</a>
+ </li>
+
+ <li>
+
+ <a href="../../metron-contrib/metron-docker/index.html" title="Docker">
+ <i class="none"></i>
+ Docker</a>
+ </li>
+
+ <li>
+
+ <a href="../../metron-deployment/index.html" title="Deployment">
+ <i class="icon-chevron-down"></i>
+ Deployment</a>
+ <ul class="nav nav-list">
+
+ <li>
+
+ <a href="../../metron-deployment/Kerberos-ambari-setup.html" title="Kerberos-ambari-setup">
+ <i class="none"></i>
+ Kerberos-ambari-setup</a>
+ </li>
+
+ <li>
+
+ <a href="../../metron-deployment/Kerberos-manual-setup.html" title="Kerberos-manual-setup">
+ <i class="none"></i>
+ Kerberos-manual-setup</a>
+ </li>
+
+ <li class="active">
+
+ <a href="#"><i class="none"></i>Amazon-ec2</a>
+ </li>
+
+ <li>
+
+ <a href="../../metron-deployment/other-examples/index.html" title="Other-examples">
+ <i class="icon-chevron-right"></i>
+ Other-examples</a>
+ </li>
+
+ <li>
+
+ <a href="../../metron-deployment/packaging/ambari/index.html" title="Ambari">
+ <i class="none"></i>
+ Ambari</a>
+ </li>
+
+ <li>
+
+ <a href="../../metron-deployment/packaging/docker/ansible-docker/index.html" title="Ansible-docker">
+ <i class="none"></i>
+ Ansible-docker</a>
+ </li>
+
+ <li>
+
+ <a href="../../metron-deployment/packaging/docker/rpm-docker/index.html" title="Rpm-docker">
+ <i class="none"></i>
+ Rpm-docker</a>
+ </li>
+
+ <li>
+
+ <a href="../../metron-deployment/packaging/packer-build/index.html" title="Packer-build">
+ <i class="none"></i>
+ Packer-build</a>
+ </li>
+
+ <li>
+
+ <a href="../../metron-deployment/roles/index.html" title="Roles">
+ <i class="icon-chevron-right"></i>
+ Roles</a>
+ </li>
+
+ <li>
+
+ <a href="../../metron-deployment/vagrant/index.html" title="Vagrant">
+ <i class="icon-chevron-right"></i>
+ Vagrant</a>
+ </li>
+ </ul>
+ </li>
+
+ <li>
+
+ <a href="../../metron-interface/metron-alerts/index.html" title="Alerts">
+ <i class="none"></i>
+ Alerts</a>
+ </li>
+
+ <li>
+
+ <a href="../../metron-interface/metron-config/index.html" title="Config">
+ <i class="none"></i>
+ Config</a>
+ </li>
+
+ <li>
+
+ <a href="../../metron-interface/metron-rest/index.html" title="Rest">
+ <i class="none"></i>
+ Rest</a>
+ </li>
+
+ <li>
+
+ <a href="../../metron-platform/index.html" title="Platform">
+ <i class="icon-chevron-right"></i>
+ Platform</a>
+ </li>
+
+ <li>
+
+ <a href="../../metron-sensors/index.html" title="Sensors">
+ <i class="icon-chevron-right"></i>
+ Sensors</a>
+ </li>
+
+ <li>
+
+ <a href="../../metron-stellar/stellar-3rd-party-example/index.html" title="Stellar-3rd-party-example">
+ <i class="none"></i>
+ Stellar-3rd-party-example</a>
+ </li>
+
+ <li>
+
+ <a href="../../metron-stellar/stellar-common/index.html" title="Stellar-common">
+ <i class="icon-chevron-right"></i>
+ Stellar-common</a>
+ </li>
+
+ <li>
+
+ <a href="../../use-cases/index.html" title="Use-cases">
+ <i class="icon-chevron-right"></i>
+ Use-cases</a>
+ </li>
+ </ul>
+ </li>
+ </ul>
+
+
+
+ <hr class="divider" />
+
+ <div id="poweredBy">
+ <div class="clear"></div>
+ <div class="clear"></div>
+ <div class="clear"></div>
+ <a href="http://maven.apache.org/" title="Built by Maven" class="poweredBy">
+ <img class="builtBy" alt="Built by Maven" src="../../images/logos/maven-feather.png" />
+ </a>
+ </div>
+ </div>
+ </div>
+
+
+ <div id="bodyColumn" class="span9" >
+
+ <h1>Apache Metron on Amazon EC2</h1>
+<p>This project fully automates the provisioning of Apache Metron on Amazon EC2 infrastructure. Starting with only your Amazon EC2 credentials, this project will create a fully-functioning, end-to-end, multi-node cluster running Apache Metron.</p>
+<p>Warning: Amazon will charge for the use of their resources when running Apache Metron. The amount will vary based on the number and size of hosts, along with current Amazon pricing structure. Be sure to stop or terminate all of the hosts instantiated by Apache Metron when not in use to avoid unnecessary charges.</p>
+<div class="section">
+<h2><a name="Getting_Started"></a>Getting Started</h2>
+<div class="section">
+<h3><a name="Prerequisites"></a>Prerequisites</h3>
+<p>The host used to deploy Apache Metron will need the following software tools installed. The following versions are known to work as of the time of this writing, but by no means are these the only working versions.</p>
+
+<ul>
+
+<li>Ansible 2.0.0.2 or 2.2.2.0</li>
+
+<li>Python 2.7.11</li>
+
+<li>Maven 3.3.9</li>
+</ul>
+<p>Any platform that supports these tools is suitable, but the following instructions cover only macOS. The easiest means of installing these tools on a Mac is to use the excellent <a class="externalLink" href="http://brew.sh/">Homebrew</a> project.</p>
+
+<ol style="list-style-type: decimal">
+
+<li>
+<p>Install Homebrew by running the following command in a terminal. Refer to the <a class="externalLink" href="http://brew.sh/">Homebrew</a> home page for the latest installation instructions.</p>
+
+<div class="source">
+<div class="source">
+<pre> /usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"
+</pre></div></div></li>
+
+<li>
+<p>With Homebrew installed, run the following command in a terminal to install all of the required tools.</p>
+
+<div class="source">
+<div class="source">
+<pre> brew cask install java
+ brew install maven git
+</pre></div></div></li>
+
+<li>
+<p>Install Ansible by following the instructions <a class="externalLink" href="http://docs.ansible.com/ansible/intro_installation.html#latest-releases-via-pip">here</a>.</p></li>
+
+<li>
+<p>Ensure that a public SSH key is located at <tt>~/.ssh/id_rsa.pub</tt>.</p>
+
+<div class="source">
+<div class="source">
+<pre> $ cat ~/.ssh/id_rsa.pub
+ ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQChv5GJxPjR39UJV7VY17ivbLVlxFrH7UHwh1Jsjem4d1eYiAtde5N2y65/HRNxWbhYli9ED8k0/MRP92ejewucEbrPNq5mytPqdC4IvZ98Ln2GbqTDwvlP3T7xa/wYFOpFsOmXXql8216wSrnrS4f3XK7ze34S6/VmY+lsBYnr3dzyj8sG/mexpJgFS/w83mWJV0e/ryf4Hd7P6DZ5fO+nmTXfKNK22ga4ctcnbZ+toYcPL+ODCh8598XCKVo97XjwF5OxN3vl1p1HHguo3cHB4H1OIaqX5mUt59gFIZcAXUME89PO6NUiZDd3RTstpf125nQVkQAHu2fvW96/f037 nick@localhost
+</pre></div></div>
+<p>If this file does not exist, run the following command at a terminal and accept all defaults. Only the public key, not the private key, will be uploaded to Amazon and configured on each host to enable SSH connectivity. While it is possible to create and use an alternative key those details will not be covered. </p>
+
+<div class="source">
+<div class="source">
+<pre> ssh-keygen -t rsa
+</pre></div></div></li>
+</ol></div>
+<div class="section">
+<h3><a name="Amazon_Web_Services"></a>Amazon Web Services</h3>
+<p>If you already have an Amazon Web Services account that you have used to deploy EC2 hosts, then you should be able to skip the next few steps.</p>
+
+<ol style="list-style-type: decimal">
+
+<li>
+<p>Head over to <a class="externalLink" href="http://aws.amazon.com/">Amazon Web Services</a> and create an account. As part of the account creation process you will need to provide a credit card to cover any charges that may apply.</p></li>
+
+<li>
+<p>Create a set of user credentials through <a class="externalLink" href="https://console.aws.amazon.com/iam/">Amazon’s Identity and Access Management (IAM) </a> dashboard. On the IAM dashboard menu click “Users” and then “Create New User”. Provide a name and ensure that “Generate an access key for each user” remains checked. Download the credentials and keep them for later use.</p></li>
+
+<li>
+<p>While still in <a class="externalLink" href="https://console.aws.amazon.com/iam/">Amazon’s Identity and Access Management (IAM) </a> dashboard, click on the user that was previously created. Click the “Permissions” tab and then the “Attach Policy” button. Attach the following policies to the user.</p>
+
+<ul>
+
+<li>AmazonEC2FullAccess</li>
+
+<li>AmazonVPCFullAccess</li>
+ </ul></li>
+
+<li>
+<p>Apache Metron uses the <a class="externalLink" href="https://aws.amazon.com/marketplace/pp/B00NQAYLWO">official, open source CentOS 6</a> Amazon Machine Image (AMI). If you have never used this AMI before then you will need to accept Amazon’s terms and conditions. Navigate to the <a class="externalLink" href="https://aws.amazon.com/marketplace/pp/B00NQAYLWO">web page for this AMI</a> and click the “Continue” button. Choose the “Manual Launch” tab then click the “Accept Software Terms” button.</p></li>
+</ol>
+<p>Having successfully created your Amazon Web Services account, hopefully you will find that the most difficult tasks are behind us. </p></div>
+<div class="section">
+<h3><a name="Deploy_Metron"></a>Deploy Metron</h3>
+
+<ol style="list-style-type: decimal">
+
+<li>
+<p>Use the Amazon access key by exporting its values via the shell’s environment. This allows Ansible to authenticate with Amazon EC2. For example:</p>
+
+<div class="source">
+<div class="source">
+<pre> export AWS_ACCESS_KEY_ID="AKIAI6NRFEO27E5FFELQ"
+ export AWS_SECRET_ACCESS_KEY="vTDydWJQnAer7OWauUS150i+9Np7hfCXrrVVP6ed"
+</pre></div></div>
+<p>Notice: You must replace the access key values above with values from your own access key.</p></li>
+
+<li>
+<p>Start the Apache Metron deployment process. When prompted provide a unique name for your Metron environment or accept the default. </p>
+
+<div class="source">
+<div class="source">
+<pre> $ ./run.sh
+ Metron Environment [metron-test]: my-metron-env
+ ...
+</pre></div></div>
+<p>The process is likely to take between 70-90 minutes. Fortunately, everything is fully automated and you should feel free to grab a coffee.</p></li>
+</ol></div>
+<div class="section">
+<h3><a name="Explore_Metron"></a>Explore Metron</h3>
+
+<ol style="list-style-type: decimal">
+
+<li>
+<p>After the deployment has completed successfully, a message like the following will be displayed. Navigate to the specified resources to explore your newly minted Apache Metron environment.</p>
+
+<div class="source">
+<div class="source">
+<pre> TASK [debug] *******************************************************************
+ ok: [localhost] => {
+ "Success": [
+ "Apache Metron deployed successfully",
+ " Metron @ http://ec2-52-37-255-142.us-west-2.compute.amazonaws.com:5000",
+ " Ambari @ http://ec2-52-37-225-202.us-west-2.compute.amazonaws.com:8080",
+ " Sensors @ ec2-52-37-225-202.us-west-2.compute.amazonaws.com on tap0",
+ "For additional information, see https://metron.apache.org/'"
+ ]
+ }
+</pre></div></div></li>
+
+<li>
+<p>Each of the provisioned hosts will be accessible from the internet. Connecting to one over SSH as the user <tt>centos</tt> will not require a password as it will authenticate with the pre-defined SSH key. </p>
+
+<div class="source">
+<div class="source">
+<pre> ssh centos@ec2-52-91-215-174.compute-1.amazonaws.com
+</pre></div></div></li>
+</ol></div></div>
+<div class="section">
+<h2><a name="Advanced_Usage"></a>Advanced Usage</h2>
+<div class="section">
+<h3><a name="Multiple_Environments"></a>Multiple Environments</h3>
+<p>This process can support provisioning of multiple, isolated environments. Simply change the <tt>env</tt> settings in <tt>conf/defaults.yml</tt>. For example, you might provision separate development, test, and production environments.</p>
+
+<div class="source">
+<div class="source">
+<pre>env: metron-test
+</pre></div></div></div>
+<div class="section">
+<h3><a name="Selective_Provisioning"></a>Selective Provisioning</h3>
+<p>To provision only subsets of the entire Metron deployment, Ansible tags can be specified. For example, to only deploy the sensors on an Amazon EC2 environment, run the following command.</p>
+
+<div class="source">
+<div class="source">
+<pre>ansible-playbook -i ec2.py playbook.yml --tags "ec2,sensors"
+</pre></div></div></div>
+<div class="section">
+<h3><a name="Custom_SSH_Key"></a>Custom SSH Key</h3>
+<p>By default, the playbook will attempt to register your public SSH key <tt>~/.ssh/id_rsa.pub</tt> with each provisioned host. This enables Ansible to communicate with each host using an SSH connection. If would prefer to use another key simply add the path to the public key file to the <tt>key_file</tt> property in <tt>conf/defaults.yml</tt>.</p>
+<p>For example, generate a new SSH key for Metron that will be stored at <tt>~/.ssh/my-metron-key</tt>.</p>
+
+<div class="source">
+<div class="source">
+<pre>$ ssh-keygen -q -f ~/.ssh/my-metron-key
+Enter passphrase (empty for no passphrase):
+Enter same passphrase again:
+</pre></div></div>
+<p>Add the path to the newly created SSH public key to <tt>conf/defaults.yml</tt>.</p>
+
+<div class="source">
+<div class="source">
+<pre>key_file: ~/.ssh/metron-private-key.pub
+</pre></div></div></div></div>
+<div class="section">
+<h2><a name="Common_Errors"></a>Common Errors</h2>
+<div class="section">
+<h3><a name="Error:_unsupported_operation_exception_custom_format_isnt_supported"></a>Error: [unsupported_operation_exception] custom format isn’t supported</h3>
+<p>This error might be seen within Metron’s default dashboard in Kibana 4. This occurs when the index templates do not exist for the Snort, Bro or YAF indices in Elasticsearch. </p>
+<p>The dashboard expects fields to be of a certain type. If the index templates have not been loaded correctly, the data types for the fields in these indices will be incorrect and the dashboard will display this error.</p>
+<div class="section">
+<h4><a name="Solution"></a>Solution</h4>
+<p>If you see this error, please report your findings by creating a JIRA or dropping an email to the Metron Users mailing list. Follow these steps to work around the problem.</p>
+
+<ol style="list-style-type: decimal">
+
+<li>
+<p>Define which Elasticsearch host to interact with. Any Elasticsearch host should work.</p>
+
+<div class="source">
+<div class="source">
+<pre>export ES_HOST="http://ec2-52-25-237-20.us-west-2.compute.amazonaws.com:9200"
+</pre></div></div></li>
+
+<li>
+<p>Confirm the index templates are in fact missing. </p>
+
+<div class="source">
+<div class="source">
+<pre>curl -s -XGET $ES_HOST/_template
+</pre></div></div></li>
+
+<li>
+<p>Manually load the index templates.</p>
+
+<div class="source">
+<div class="source">
+<pre>cd metron-deployment
+curl -s -XPOST $ES_HOST/_template/bro_index -d @roles/metron_elasticsearch_templates/files/es_templates/bro_index.template
+curl -s -XPOST $ES_HOST/_template/snort_index -d @roles/metron_elasticsearch_templates/files/es_templates/snort_index.template
+curl -s -XPOST $ES_HOST/_template/yaf_index -d @roles/metron_elasticsearch_templates/files/es_templates/yaf_index.template
+</pre></div></div></li>
+
+<li>
+<p>Delete the existing indexes. Only a new index will use the templates defined in the previous step.</p>
+
+<div class="source">
+<div class="source">
+<pre>curl -s -XDELETE "$ES_HOST/yaf_index*"
+curl -s -XDELETE "$ES_HOST/bro_index*"
+curl -s -XDELETE "$ES_HOST/snort_index*"
+</pre></div></div></li>
+
+<li>
+<p>Open up Kibana and wait for the new indexes to be created. The dashboard should now work.</p></li>
+</ol></div></div>
+<div class="section">
+<h3><a name="Error:_No_handler_was_ready_to_authenticateCheck_your_credentials"></a>Error: ‘No handler was ready to authenticate…Check your credentials’</h3>
+
+<div class="source">
+<div class="source">
+<pre>TASK [Define keypair] **********************************************************
+failed: [localhost] => (item=ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDXbcb1AlWsEPP
+ r9jEFrn0yun3PYNidJ/...david@hasselhoff.com) => {"failed": true, "item": "ssh-r
+ sa AAAAB3NzaC1yc2EAAAADAQABAAABAQDXbcb1AlWsEPPr9jEFr... david@hasselhoff.com",
+ "msg": "No handler was ready to authenticate. 1 handlers were checked.
+ ['HmacAuthV4Handler'] Check your credentials"}
+</pre></div></div>
+<div class="section">
+<h4><a name="Solution_1"></a>Solution 1</h4>
+<p>This occurs when Ansible does not have the correct AWS access keys. The following commands must return a valid access key that is defined within Amazon’s <a class="externalLink" href="https://console.aws.amazon.com/iam/">Identity and Access Management</a> console. </p>
+
+<div class="source">
+<div class="source">
+<pre>$ echo $AWS_ACCESS_KEY_ID
+AKIAI6NRFEO27E5FFELQ
+
+$ echo $AWS_SECRET_ACCESS_KEY
+vTDydWJQnAer7OWauUS150i+9Np7hfCXrrVVP6ed
+</pre></div></div></div>
+<div class="section">
+<h4><a name="Solution_2"></a>Solution 2</h4>
+<p>This error can occur if you have exported the correct AWS access key, but you are using <tt>sudo</tt> to run the Ansible playbook. Do not use the <tt>sudo</tt> command when running the Ansible playbook.</p></div></div>
+<div class="section">
+<h3><a name="Error:_OptInRequired:__you_need_to_accept_terms_and_subscribe"></a>Error: ‘OptInRequired: … you need to accept terms and subscribe’</h3>
+
+<div class="source">
+<div class="source">
+<pre>TASK [metron-test: Instantiate 1 host(s) as sensors,ambari_master,metron,ec2] **
+fatal: [localhost]: FAILED! => {"changed": false, "failed": true, "msg":
+"Instance creation failed => OptInRequired: In order to use this AWS Marketplace
+product you need to accept terms and subscribe. To do so please visit
+http://aws.amazon.com/marketplace/pp?sku=6x5jmcajty9edm3f211pqjfn2"}
+to retry, use: --limit @playbook.retry
+</pre></div></div>
+<div class="section">
+<h4><a name="Solution"></a>Solution</h4>
+<p>Apache Metron uses the <a class="externalLink" href="https://aws.amazon.com/marketplace/pp?sku=6x5jmcajty9edm3f211pqjfn2">official CentOS 6 Amazon Machine Image</a> when provisioning hosts. Amazon requires that you accept certain terms and conditions when using any Amazon Machine Image (AMI). Follow the link provided in the error message to accept the terms and conditions then re-run the playbook. </p></div></div>
+<div class="section">
+<h3><a name="Error:_PendingVerification:_Your_account_is_currently_being_verified"></a>Error: ‘PendingVerification: Your account is currently being verified’</h3>
+
+<div class="source">
+<div class="source">
+<pre>TASK [metron-test: Instantiate 1 host(s) as sensors,ambari_master,metron,ec2] **
+fatal: [localhost]: FAILED! => {"changed": false, "failed": true, "msg":
+"Instance creation failed => PendingVerification: Your account is currently
+being verified. Verification normally takes less than 2 hours. Until your
+account is verified, you may not be able to launch additional instances or
+create additional volumes. If you are still receiving this message after more
+than 2 hours, please let us know by writing to aws-verification@amazon.com. We
+appreciate your patience."}
+to retry, use: --limit @playbook.retry
+</pre></div></div>
+<div class="section">
+<h4><a name="Solution"></a>Solution</h4>
+<p>This will occur if you are attempting to deploy Apache Metron using a newly created Amazon Web Services account. Follow the advice of the message and wait until Amazon’s verification process is complete. Amazon has some additional <a class="externalLink" href="http://docs.aws.amazon.com/AWSEC2/latest/APIReference/errors-overview.html">advice for dealing with this error and more</a>.</p>
+
+<blockquote>
+<p>Your account is pending verification. Until the verification process is complete, you may not be able to carry out requests with this account. If you have questions, contact <a class="externalLink" href="http://console.aws.amazon.com/support/home#/">AWS Support</a>.</p>
+</blockquote></div></div>
+<div class="section">
+<h3><a name="Error:_Instance_creation_failed__InstanceLimitExceeded"></a>Error: ‘Instance creation failed => InstanceLimitExceeded’</h3>
+
+<div class="source">
+<div class="source">
+<pre>TASK [metron-test: Instantiate 3 host(s) as search,metron,ec2] *****************
+fatal: [localhost]: FAILED! => {"changed": false, "failed": true, "msg":
+"Instance creation failed => InstanceLimitExceeded: You have requested more
+instances (11) than your current instance limit of 10 allows for the specified
+instance type. Please visit http://aws.amazon.com/contact-us/ec2-request to
+request an adjustment to this limit."}
+to retry, use: --limit @playbook.retry
+</pre></div></div>
+<div class="section">
+<h4><a name="Solution"></a>Solution</h4>
+<p>This will occur if Apache Metron attempts to deploy more host instances than allowed by your account. The total number of instances required for Apache Metron can be reduced by editing <tt>deployment/amazon-ec/playbook.yml</tt>. Perhaps a better alternative is to request of Amazon that this limit be increased. Amazon has some additional <a class="externalLink" href="http://docs.aws.amazon.com/AWSEC2/latest/APIReference/errors-overview.html">advice for dealing with this error and more</a>.</p>
+
+<blockquote>
+<p>You’ve reached the limit on the number of instances you can run concurrently. The limit depends on the instance type. For more information, see <a class="externalLink" href="http://aws.amazon.com/ec2/faqs/#How_many_instances_can_I_run_in_Amazon_EC2">How many instances can I run in Amazon EC2</a>. If you need additional instances, complete the <a class="externalLink" href="https://console.aws.amazon.com/support/home#/case/create?issueType=service-limit-increase&limitType=service-code-ec2-instances">Amazon EC2 Instance Request Form</a>.</p>
+</blockquote></div></div>
+<div class="section">
+<h3><a name="Error:_SSH_encountered_an_unknown_error_during_the_connection"></a>Error: ‘SSH encountered an unknown error during the connection’</h3>
+
+<div class="source">
+<div class="source">
+<pre>TASK [setup] *******************************************************************
+fatal: [ec2-52-26-113-221.us-west-2.compute.amazonaws.com]: UNREACHABLE! => {
+ "changed": false, "msg": "SSH encountered an unknown error during the
+ connection. We recommend you re-run the command using -vvvv, which will enable
+ SSH debugging output to help diagnose the issue", "unreachable": true}
+</pre></div></div>
+<div class="section">
+<h4><a name="Solution"></a>Solution</h4>
+<p>This most often indicates that Ansible cannot connect to the host with the SSH key that it has access to. This could occur if hosts are provisioned with one SSH key, but the playbook is executed subsequently with a different SSH key. The issue can be addressed by either altering the <tt>key_file</tt> variable to point to the key that was used to provision the hosts or by simply terminating all hosts and re-running the playbook.</p></div></div></div>
+ </div>
+ </div>
+ </div>
+
+ <hr/>
+
+ <footer>
+ <div class="container-fluid">
+ <div class="row span12">Copyright © 2017
+ <a href="https://www.apache.org">The Apache Software Foundation</a>.
+ All Rights Reserved.
+
+ </div>
+
+
+
+ </div>
+ </footer>
+ </body>
+</html>