You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@metron.apache.org by ce...@apache.org on 2017/02/08 13:49:54 UTC
svn commit: r18223 [3/12] - in
/dev/incubator/metron/0.3.1-RC2-incubating/book-site: ./ css/
image-archive/ images/ images/logos/ images/profiles/ img/ js/
metron-analytics/ metron-analytics/metron-maas-service/
metron-analytics/metron-profiler-client/...
Added: dev/incubator/metron/0.3.1-RC2-incubating/book-site/metron-analytics/metron-profiler-client/index.html
==============================================================================
--- dev/incubator/metron/0.3.1-RC2-incubating/book-site/metron-analytics/metron-profiler-client/index.html (added)
+++ dev/incubator/metron/0.3.1-RC2-incubating/book-site/metron-analytics/metron-profiler-client/index.html Wed Feb 8 13:49:53 2017
@@ -0,0 +1,427 @@
+<!DOCTYPE html>
+<!--
+ | Generated by Apache Maven Doxia at 2017-02-08
+ | Rendered using Apache Maven Fluido Skin 1.3.0
+-->
+<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
+ <head>
+ <meta charset="UTF-8" />
+ <meta name="viewport" content="width=device-width, initial-scale=1.0" />
+ <meta name="Date-Revision-yyyymmdd" content="20170208" />
+ <meta http-equiv="Content-Language" content="en" />
+ <title>Metron – Metron Profiler Client</title>
+ <link rel="stylesheet" href="../../css/apache-maven-fluido-1.3.0.min.css" />
+ <link rel="stylesheet" href="../../css/site.css" />
+ <link rel="stylesheet" href="../../css/print.css" media="print" />
+
+
+ <script type="text/javascript" src="../../js/apache-maven-fluido-1.3.0.min.js"></script>
+
+
+
+<script type="text/javascript">$( document ).ready( function() { $( '.carousel' ).carousel( { interval: 3500 } ) } );</script>
+
+ </head>
+ <body class="topBarDisabled">
+
+
+
+
+ <div class="container-fluid">
+ <div id="banner">
+ <div class="pull-left">
+ <a href="http://metron.incubator.apache.org/" id="bannerLeft">
+ <img src="../../images/metron-logo.png" alt="Apache Metron - Incubating" width="148px" height="48px"/>
+ </a>
+ </div>
+ <div class="pull-right"> <a href="http://incubator.apache.org/" id="bannerRight">
+ <img src="../../images/ApacheIncubating_Logo.png" alt="Apache Incubating" width="192px" height="48px"/>
+ </a>
+ </div>
+ <div class="clear"><hr/></div>
+ </div>
+
+ <div id="breadcrumbs">
+ <ul class="breadcrumb">
+
+
+ <li class="">
+ <a href="http://www.apache.org" class="externalLink" title="Apache">
+ Apache</a>
+ </li>
+ <li class="divider ">/</li>
+ <li class="">
+ <a href="http://metron.incubator.apache.org/" class="externalLink" title="Metron-Incubating">
+ Metron-Incubating</a>
+ </li>
+ <li class="divider ">/</li>
+ <li class="">
+ <a href="../../index.html" title="Documentation">
+ Documentation</a>
+ </li>
+ <li class="divider ">/</li>
+ <li class="">Metron Profiler Client</li>
+
+
+
+ <li id="publishDate" class="pull-right">Last Published: 2017-02-08</li> <li class="divider pull-right">|</li>
+ <li id="projectVersion" class="pull-right">Version: 0.3.1</li>
+
+ </ul>
+ </div>
+
+
+ <div class="row-fluid">
+ <div id="leftColumn" class="span3">
+ <div class="well sidebar-nav">
+
+
+ <ul class="nav nav-list">
+ <li class="nav-header">User Documentation</li>
+
+ <li>
+
+ <a href="../../index.html" title="Metron">
+ <i class="icon-chevron-down"></i>
+ Metron</a>
+ <ul class="nav nav-list">
+
+ <li>
+
+ <a href="../../Upgrading.html" title="Upgrading">
+ <i class="none"></i>
+ Upgrading</a>
+ </li>
+
+ <li>
+
+ <a href="../../metron-analytics/index.html" title="Analytics">
+ <i class="icon-chevron-down"></i>
+ Analytics</a>
+ <ul class="nav nav-list">
+
+ <li>
+
+ <a href="../../metron-analytics/metron-maas-service/index.html" title="Maas-service">
+ <i class="none"></i>
+ Maas-service</a>
+ </li>
+
+ <li>
+
+ <a href="../../metron-analytics/metron-profiler/index.html" title="Profiler">
+ <i class="none"></i>
+ Profiler</a>
+ </li>
+
+ <li class="active">
+
+ <a href="#"><i class="none"></i>Profiler-client</a>
+ </li>
+
+ <li>
+
+ <a href="../../metron-analytics/metron-statistics/index.html" title="Statistics">
+ <i class="icon-chevron-right"></i>
+ Statistics</a>
+ </li>
+ </ul>
+ </li>
+
+ <li>
+
+ <a href="../../metron-deployment/index.html" title="Deployment">
+ <i class="icon-chevron-right"></i>
+ Deployment</a>
+ </li>
+
+ <li>
+
+ <a href="../../metron-docker/index.html" title="Docker">
+ <i class="none"></i>
+ Docker</a>
+ </li>
+
+ <li>
+
+ <a href="../../metron-platform/index.html" title="Platform">
+ <i class="icon-chevron-right"></i>
+ Platform</a>
+ </li>
+
+ <li>
+
+ <a href="../../metron-sensors/index.html" title="Sensors">
+ <i class="icon-chevron-right"></i>
+ Sensors</a>
+ </li>
+ </ul>
+ </li>
+ </ul>
+
+
+
+ <hr class="divider" />
+
+ <div id="poweredBy">
+ <div class="clear"></div>
+ <div class="clear"></div>
+ <div class="clear"></div>
+ <a href="http://maven.apache.org/" title="Built by Maven" class="poweredBy">
+ <img class="builtBy" alt="Built by Maven" src="../../images/logos/maven-feather.png" />
+ </a>
+ </div>
+ </div>
+ </div>
+
+
+ <div id="bodyColumn" class="span9" >
+
+ <h1>Metron Profiler Client</h1>
+<p><a name="Metron_Profiler_Client"></a></p>
+<p>This project provides a client API for accessing the profiles generated by the <a href="../metron-profiler/index.html">Metron Profiler</a>. This includes both a Java API and Stellar API for accessing the profile data. The primary use case is to extract profile data for use during model scoring.</p>
+<div class="section">
+<h2><a name="Stellar_Client_API"></a>Stellar Client API</h2>
+<p>The following are usage examples that show how the Stellar API can be used to read profiles generated by the <a href="../metron-profiler/index.html">Metron Profiler</a>. This API would be used in conjunction with other Stellar functions like <a href="../../metron-platform/metron-common/index.html"><tt>MAAS_MODEL_APPLY</tt></a> to perform model scoring on streaming data.</p>
+<p>These examples assume a profile has been defined called ‘snort-alerts’ that tracks the number of Snort alerts associated with an IP address over time. The profile definition might look similar to the following.</p>
+
+<div class="source">
+<div class="source">
+<pre>{
+ "profiles": [
+ {
+ "profile": "snort-alerts",
+ "foreach": "ip_src_addr",
+ "onlyif": "source.type == 'snort'",
+ "update": { "s": "STATS_ADD(s, 1)" },
+ "result": "STATS_MEAN(s)"
+ }
+ ]
+}
+</pre></div></div>
+<p>During model scoring the entity being scored, in this case a particular IP address, will be known. The following examples shows how this profile data might be retrieved.</p>
+<p>The Stellar client consists of the <tt>PROFILE_GET</tt> command, which takes the following arguments:</p>
+
+<div class="source">
+<div class="source">
+<pre>REQUIRED:
+ profile - The name of the profile
+ entity - The name of the entity
+ periods - The list of profile periods to grab. These are ProfilePeriod objects.
+OPTIONAL:
+ groups_list - Optional, must correspond to the 'groupBy' list used in profile creation - List (in square brackets) of
+ groupBy values used to filter the profile. Default is the empty list, meaning groupBy was not used when
+ creating the profile.
+ config_overrides - Optional - Map (in curly braces) of name:value pairs, each overriding the global config parameter
+ of the same name. Default is the empty Map, meaning no overrides.
+</pre></div></div>
+<p>There is an older calling format where <tt>groups_list</tt> is specified as a sequence of group names, “varargs” style, instead of a List object. This format is still supported for backward compatibility, but it is deprecated, and it is disallowed if the optional <tt>config_overrides</tt> argument is used.</p>
+<p>The <tt>periods</tt> field is (likely) the output of another Stellar function which defines the times to include.</p>
+<p><tt>PROFILE_FIXED</tt>: The profiler periods associated with a fixed lookback starting from now. These are ProfilePeriod objects.</p>
+
+<div class="source">
+<div class="source">
+<pre>REQUIRED:
+ durationAgo - How long ago should values be retrieved from?
+ units - The units of 'durationAgo'.
+OPTIONAL:
+ config_overrides - Optional - Map (in curly braces) of name:value pairs, each overriding the global config parameter
+ of the same name. Default is the empty Map, meaning no overrides.
+
+e.g. To retrieve all the profiles for the last 5 hours. PROFILE_GET('profile', 'entity', PROFILE_FIXED(5, 'HOURS'))
+</pre></div></div>
+<div class="section">
+<h3><a name="Groups_list_argument"></a>Groups_list argument</h3>
+<p>The <tt>groups_list</tt> argument in the client must exactly correspond to the <a href="../metron-profiler/index.html#groupBy"><tt>groupBy</tt></a> configuration in the profile definition. If <tt>groupBy</tt> was not used in the profile, <tt>groups_list</tt> must be empty in the client. If <tt>groupBy</tt> was used in the profile, then the client <tt>groups_list</tt> is <b>not</b> optional; it must be the same length as the <tt>groupBy</tt> list, and specify exactly one selected group value for each <tt>groupBy</tt> criterion, in the same order. For example:</p>
+
+<div class="source">
+<div class="source">
+<pre>If in Profile, the groupBy criteria are: [ “DAY_OF_WEEK()”, “URL_TO_PORT()” ]
+Then in PROFILE_GET, an allowed groups value would be: [ “3”, “8080” ]
+which will select only records from Tuesdays with port number 8080.
+</pre></div></div></div>
+<div class="section">
+<h3><a name="Configuration_and_the_config_overrides_argument"></a>Configuration and the config_overrides argument</h3>
+<p>By default, the Profiler creates profiles with a period duration of 15 minutes. This means that data is accumulated, summarized and flushed every 15 minutes. The Client API must also have knowledge of this duration to correctly retrieve the profile data. If the Client is expecting 15 minute periods, it will not be able to read data generated by a Profiler that was configured for 1 hour periods, and will return zero results. </p>
+<p>Similarly, all six Client configuration parameters listed in the table below must match the Profiler configuration parameter settings from the time the profile was created. The period duration and other configuration parameters from the Profiler topology are stored in local filesystem at <tt>$METRON_HOME/config/profiler.properties</tt>. The Stellar Client API can be configured correspondingly by setting the following properties in Metron’s global configuration, on local filesystem at <tt>$METRON_HOME/config/zookeeper/global.json</tt>, then uploaded to Zookeeper (at <tt>/metron/topology/global</tt>) by using <tt>zk_load_configs.sh</tt>: </p>
+
+<div class="source">
+<div class="source">
+<pre> $ cd $METRON_HOME
+ $ bin/zk_load_configs.sh -m PUSH -i config/zookeeper/ -z node1:2181
+</pre></div></div>
+<p>Any of these six Client configuration parameters may be overridden at run time using the <tt>config_overrides</tt> Map argument in PROFILE_GET. The primary use case is when historical profiles have been created with a different Profiler configuration than is currently configured, and the analyst needing to access them does not want to change the global Client configuration so as not to disrupt the work of other analysts working with current profiles.</p>
+
+<table border="0" class="table table-striped">
+ <thead>
+
+<tr class="a">
+
+<th>Key </th>
+
+<th>Description </th>
+
+<th>Required </th>
+
+<th>Default </th>
+ </tr>
+ </thead>
+ <tbody>
+
+<tr class="b">
+
+<td>profiler.client.period.duration </td>
+
+<td>The duration of each profile period. This value should be defined along with <tt>profiler.client.period.duration.units</tt>. </td>
+
+<td>Optional </td>
+
+<td>15 </td>
+ </tr>
+
+<tr class="a">
+
+<td>profiler.client.period.duration.units </td>
+
+<td>The units used to specify the profile period duration. This value should be defined along with <tt>profiler.client.period.duration</tt>. </td>
+
+<td>Optional </td>
+
+<td>MINUTES </td>
+ </tr>
+
+<tr class="b">
+
+<td>profiler.client.hbase.table </td>
+
+<td>The name of the HBase table used to store profile data. </td>
+
+<td>Optional </td>
+
+<td>profiler </td>
+ </tr>
+
+<tr class="a">
+
+<td>profiler.client.hbase.column.family </td>
+
+<td>The name of the HBase column family used to store profile data. </td>
+
+<td>Optional </td>
+
+<td>P </td>
+ </tr>
+
+<tr class="b">
+
+<td>profiler.client.salt.divisor </td>
+
+<td>The salt divisor used to store profile data. </td>
+
+<td>Optional </td>
+
+<td>1000 </td>
+ </tr>
+
+<tr class="a">
+
+<td>hbase.provider.impl </td>
+
+<td>The name of the HBaseTableProvider implementation class. </td>
+
+<td>Optional </td>
+
+<td> </td>
+ </tr>
+ </tbody>
+</table></div>
+<div class="section">
+<h3><a name="Errors"></a>Errors</h3>
+<p>The most common result of incorrect PROFILE_GET arguments or Client configuration parameters is an empty result set, rather than an error. The Client cannot effectively validate the arguments, because the Profiler configuration parameters may be changed and the profile itself does not store them. The person doing the querying must carry forward the knowledge of the Profiler configuration parameters from the time of profile creation, and use corresponding PROFILE_GET arguments and Client configuration parameters when querying the data.</p></div>
+<div class="section">
+<h3><a name="Examples"></a>Examples</h3>
+<p>Retrieve all values of ‘snort-alerts’ from ‘10.0.0.1’ over the past 4 hours.</p>
+
+<div class="source">
+<div class="source">
+<pre>PROFILE_GET('snort-alerts', '10.0.0.1', 4, 'HOURS')
+</pre></div></div>
+<p>Retrieve all values of ‘snort-alerts’ from ‘10.0.0.1’ over the past 2 days.</p>
+
+<div class="source">
+<div class="source">
+<pre>PROFILE_GET('snort-alerts', '10.0.0.1', 2, 'DAYS')
+</pre></div></div>
+<p>If the profile had been defined to group the data by weekday versus weekend, then the following example would apply:</p>
+<p>Retrieve all values of ‘snort-alerts’ from ‘10.0.0.1’ that occurred on ‘weekdays’ over the past month.</p>
+
+<div class="source">
+<div class="source">
+<pre>PROFILE_GET('snort-alerts', '10.0.0.1', 1, 'MONTHS', ['weekdays'] )
+</pre></div></div>
+<p>The client may need to use a configuration different from the current Client configuration settings. For example, perhaps you are on a cluster shared with other analysts, and need to access a profile that was constructed 2 months ago using different period duration, while they are accessing more recent profiles constructed with the currently configured period duration. For this situation, you may use the <tt>config_overrides</tt> argument:</p>
+<p>Retrieve all values of ‘snort-alerts’ from ‘10.0.0.1’ over the past 2 days, with no <tt>groupBy</tt>, and overriding the usual global client configuration parameters for window duration.</p>
+
+<div class="source">
+<div class="source">
+<pre>PROFILE_GET('profile1', 'entity1', 2, 'DAYS', [], {'profiler.client.period.duration' : '2', 'profiler.client.period.duration.units' : 'MINUTES'})
+</pre></div></div>
+<p>Retrieve all values of ‘snort-alerts’ from ‘10.0.0.1’ that occurred on ‘weekdays’ over the past month, overriding the usual global client configuration parameters for window duration.</p>
+
+<div class="source">
+<div class="source">
+<pre>PROFILE_GET('profile1', 'entity1', 1, 'MONTHS', ['weekdays'], {'profiler.client.period.duration' : '2', 'profiler.client.period.duration.units' : 'MINUTES'})
+</pre></div></div></div></div>
+<div class="section">
+<h2><a name="Getting_Started"></a>Getting Started</h2>
+<p>These instructions step through the process of using the Stellar Client API on a live cluster. These instructions assume that the ‘Getting Started’ instructions included with the <a href="../metron-profiler/index.html">Metron Profiler</a> have been followed. This will create a Profile called ‘test’ whose data will be retrieved with the Stellar Client API.</p>
+<p>To validate that everything is working, login to the server hosting Metron. We will use the Stellar Shell to replicate the execution environment of Stellar running in a Storm topology, like Metron’s Parser or Enrichment topology. Replace ‘node1:2181’ with the URL to a Zookeeper Broker. </p>
+
+<div class="source">
+<div class="source">
+<pre>[root@node1 0.3.1]# bin/stellar -z node1:2181
+Stellar, Go!
+Please note that functions are loading lazily in the background and will be unavailable until loaded fully.
+{es.clustername=metron, es.ip=node1, es.port=9300, es.date.format=yyyy.MM.dd.HH}
+
+[Stellar]>>> ?PROFILE_GET
+Functions loaded, you may refer to functions now...
+PROFILE_GET
+Description: Retrieves a series of values from a stored profile.
+
+Arguments:
+ profile - The name of the profile.
+ entity - The name of the entity.
+ durationAgo - How long ago should values be retrieved from?
+ units - The units of 'durationAgo'.
+ groups_list - Optional, must correspond to the 'groupBy' list used in profile creation - List (in square brackets) of
+ groupBy values used to filter the profile. Default is the empty list, meaning groupBy was not used when
+ creating the profile.
+ config_overrides - Optional - Map (in curly braces) of name:value pairs, each overriding the global config parameter
+ of the same name. Default is the empty Map, meaning no overrides.
+
+Returns: The selected profile measurements.
+
+[Stellar]>>> PROFILE_GET('test','192.168.138.158', 1, 'HOURS')
+[12078.0, 8921.0, 12131.0]
+</pre></div></div>
+<p>The client API call above has retrieved the past hour of the ‘test’ profile for the entity ‘192.168.138.158’.</p></div>
+ </div>
+ </div>
+ </div>
+
+ <hr/>
+
+ <footer>
+ <div class="container-fluid">
+ <div class="row span12">Copyright © 2017.
+ All Rights Reserved.
+
+ </div>
+
+
+
+ </div>
+ </footer>
+ </body>
+</html>
Added: dev/incubator/metron/0.3.1-RC2-incubating/book-site/metron-analytics/metron-profiler/index.html
==============================================================================
--- dev/incubator/metron/0.3.1-RC2-incubating/book-site/metron-analytics/metron-profiler/index.html (added)
+++ dev/incubator/metron/0.3.1-RC2-incubating/book-site/metron-analytics/metron-profiler/index.html Wed Feb 8 13:49:53 2017
@@ -0,0 +1,779 @@
+<!DOCTYPE html>
+<!--
+ | Generated by Apache Maven Doxia at 2017-02-08
+ | Rendered using Apache Maven Fluido Skin 1.3.0
+-->
+<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
+ <head>
+ <meta charset="UTF-8" />
+ <meta name="viewport" content="width=device-width, initial-scale=1.0" />
+ <meta name="Date-Revision-yyyymmdd" content="20170208" />
+ <meta http-equiv="Content-Language" content="en" />
+ <title>Metron – Metron Profiler</title>
+ <link rel="stylesheet" href="../../css/apache-maven-fluido-1.3.0.min.css" />
+ <link rel="stylesheet" href="../../css/site.css" />
+ <link rel="stylesheet" href="../../css/print.css" media="print" />
+
+
+ <script type="text/javascript" src="../../js/apache-maven-fluido-1.3.0.min.js"></script>
+
+
+
+<script type="text/javascript">$( document ).ready( function() { $( '.carousel' ).carousel( { interval: 3500 } ) } );</script>
+
+ </head>
+ <body class="topBarDisabled">
+
+
+
+
+ <div class="container-fluid">
+ <div id="banner">
+ <div class="pull-left">
+ <a href="http://metron.incubator.apache.org/" id="bannerLeft">
+ <img src="../../images/metron-logo.png" alt="Apache Metron - Incubating" width="148px" height="48px"/>
+ </a>
+ </div>
+ <div class="pull-right"> <a href="http://incubator.apache.org/" id="bannerRight">
+ <img src="../../images/ApacheIncubating_Logo.png" alt="Apache Incubating" width="192px" height="48px"/>
+ </a>
+ </div>
+ <div class="clear"><hr/></div>
+ </div>
+
+ <div id="breadcrumbs">
+ <ul class="breadcrumb">
+
+
+ <li class="">
+ <a href="http://www.apache.org" class="externalLink" title="Apache">
+ Apache</a>
+ </li>
+ <li class="divider ">/</li>
+ <li class="">
+ <a href="http://metron.incubator.apache.org/" class="externalLink" title="Metron-Incubating">
+ Metron-Incubating</a>
+ </li>
+ <li class="divider ">/</li>
+ <li class="">
+ <a href="../../index.html" title="Documentation">
+ Documentation</a>
+ </li>
+ <li class="divider ">/</li>
+ <li class="">Metron Profiler</li>
+
+
+
+ <li id="publishDate" class="pull-right">Last Published: 2017-02-08</li> <li class="divider pull-right">|</li>
+ <li id="projectVersion" class="pull-right">Version: 0.3.1</li>
+
+ </ul>
+ </div>
+
+
+ <div class="row-fluid">
+ <div id="leftColumn" class="span3">
+ <div class="well sidebar-nav">
+
+
+ <ul class="nav nav-list">
+ <li class="nav-header">User Documentation</li>
+
+ <li>
+
+ <a href="../../index.html" title="Metron">
+ <i class="icon-chevron-down"></i>
+ Metron</a>
+ <ul class="nav nav-list">
+
+ <li>
+
+ <a href="../../Upgrading.html" title="Upgrading">
+ <i class="none"></i>
+ Upgrading</a>
+ </li>
+
+ <li>
+
+ <a href="../../metron-analytics/index.html" title="Analytics">
+ <i class="icon-chevron-down"></i>
+ Analytics</a>
+ <ul class="nav nav-list">
+
+ <li>
+
+ <a href="../../metron-analytics/metron-maas-service/index.html" title="Maas-service">
+ <i class="none"></i>
+ Maas-service</a>
+ </li>
+
+ <li class="active">
+
+ <a href="#"><i class="none"></i>Profiler</a>
+ </li>
+
+ <li>
+
+ <a href="../../metron-analytics/metron-profiler-client/index.html" title="Profiler-client">
+ <i class="none"></i>
+ Profiler-client</a>
+ </li>
+
+ <li>
+
+ <a href="../../metron-analytics/metron-statistics/index.html" title="Statistics">
+ <i class="icon-chevron-right"></i>
+ Statistics</a>
+ </li>
+ </ul>
+ </li>
+
+ <li>
+
+ <a href="../../metron-deployment/index.html" title="Deployment">
+ <i class="icon-chevron-right"></i>
+ Deployment</a>
+ </li>
+
+ <li>
+
+ <a href="../../metron-docker/index.html" title="Docker">
+ <i class="none"></i>
+ Docker</a>
+ </li>
+
+ <li>
+
+ <a href="../../metron-platform/index.html" title="Platform">
+ <i class="icon-chevron-right"></i>
+ Platform</a>
+ </li>
+
+ <li>
+
+ <a href="../../metron-sensors/index.html" title="Sensors">
+ <i class="icon-chevron-right"></i>
+ Sensors</a>
+ </li>
+ </ul>
+ </li>
+ </ul>
+
+
+
+ <hr class="divider" />
+
+ <div id="poweredBy">
+ <div class="clear"></div>
+ <div class="clear"></div>
+ <div class="clear"></div>
+ <a href="http://maven.apache.org/" title="Built by Maven" class="poweredBy">
+ <img class="builtBy" alt="Built by Maven" src="../../images/logos/maven-feather.png" />
+ </a>
+ </div>
+ </div>
+ </div>
+
+
+ <div id="bodyColumn" class="span9" >
+
+ <h1>Metron Profiler</h1>
+<p><a name="Metron_Profiler"></a></p>
+<p>The Profiler is a feature extraction mechanism that can generate a profile describing the behavior of an entity. An entity might be a server, user, subnet or application. Once a profile has been generated defining what normal behavior looks-like, models can be built that identify anomalous behavior. </p>
+<p>This is achieved by summarizing the streaming telemetry data consumed by Metron over sliding windows. A summary statistic is applied to the data received within a given window. Collecting this summary across many windows results in a time series that is useful for analysis.</p>
+<p>Any field contained within a message can be used to generate a profile. A profile can even be produced by combining fields that originate in different data sources. A user has considerable power to transform the data used in a profile by leveraging the Stellar language. A user only need configure the desired profiles and ensure that the Profiler topology is running.</p>
+
+<ul>
+
+<li><a href="#Getting_Started">Getting Started</a></li>
+
+<li><a href="#Creating_Profiles">Creating Profiles</a></li>
+
+<li><a href="#Configuring_the_Profiler">Configuring the Profiler</a></li>
+
+<li><a href="#Examples">Examples</a></li>
+
+<li><a href="#Implementation">Implementation</a></li>
+</ul>
+<div class="section">
+<h2><a name="Getting_Started"></a>Getting Started</h2>
+<p>This section will describe the steps required to get your first profile running.</p>
+
+<ol style="list-style-type: decimal">
+
+<li>
+<p>Stand-up a Metron environment. For this example, we will use the ‘Quick Dev’ environment. Follow the instructions included with <a href="../../metron-deployment/vagrant/quick-dev-platform/index.html">Quick Dev</a> or build your own.</p></li>
+
+<li>
+<p>Create a table within HBase that will store the profile data. The table name and column family must match the <a href="#configuring-the-profiler">Profiler’s configuration</a>.</p>
+
+<div class="source">
+<div class="source">
+<pre>$ /usr/hdp/current/hbase-client/bin/hbase shell
+hbase(main):001:0> create 'profiler', 'P'
+</pre></div></div></li>
+
+<li>
+<p>Edit the configuration file located at <tt>$METRON_HOME/config/profiler.properties</tt>. Change the kafka.zk and kafka.broker values from “node1” to the appropriate host name. Keep the same port numbers:</p>
+
+<div class="source">
+<div class="source">
+<pre>kafka.zk=node1:2181
+kafka.broker=node1:6667
+</pre></div></div></li>
+
+<li>
+<p>Define the profile in a file located at <tt>$METRON_HOME/config/zookeeper/profiler.json</tt>. The following example JSON will create a profile that simply counts the number of messages per <tt>ip_src_addr</tt>, during each sampling interval.</p>
+
+<div class="source">
+<div class="source">
+<pre>{
+ "profiles": [
+ {
+ "profile": "test",
+ "foreach": "ip_src_addr",
+ "init": { "count": "0" },
+ "update": { "count": "count + 1" },
+ "result": "count"
+ }
+ ]
+}
+</pre></div></div></li>
+
+<li>
+<p>Upload the profile definition to Zookeeper. (As always, change “node1” to the actual hostname.)</p>
+
+<div class="source">
+<div class="source">
+<pre>$ cd $METRON_HOME
+$ bin/zk_load_configs.sh -m PUSH -i config/zookeeper/ -z node1:2181
+</pre></div></div></li>
+
+<li>
+<p>Start the Profiler topology.</p>
+
+<div class="source">
+<div class="source">
+<pre>$ bin/start_profiler_topology.sh
+</pre></div></div></li>
+
+<li>
+<p>Ensure that test messages are being sent to the Profiler’s input topic in Kafka. The Profiler will consume messages from the <tt>inputTopic</tt> defined in the <a href="#configuring-the-profiler">Profiler’s configuration</a>.</p></li>
+
+<li>
+<p>Check the HBase table to validate that the Profiler is writing the profile. Remember that the Profiler is flushing the profile every 15 minutes. You will need to wait at least this long to start seeing profile data in HBase.</p>
+
+<div class="source">
+<div class="source">
+<pre>$ /usr/hdp/current/hbase-client/bin/hbase shell
+hbase(main):001:0> count 'profiler'
+</pre></div></div></li>
+
+<li>
+<p>Use the Profiler Client to read the profile data. The below example <tt>PROFILE_GET</tt> command will read data written by the sample profile given above, if 10.0.0.1 is one of the input values for <tt>ip_src_addr</tt>. More information on configuring and using the client can be found <a href="../metron-profiler-client/index.html">here</a>. It is assumed that the <tt>PROFILE_GET</tt> client is correctly configured before using it.</p>
+<p>``` $ bin/stellar -z node1:2181</p>
+<p>[Stellar]>>> PROFILE_GET( “test”, “10.0.0.1”, PROFILE_FIXED(30, “MINUTES”)) [451, 448] ```</p></li>
+</ol></div>
+<div class="section">
+<h2><a name="Creating_Profiles"></a>Creating Profiles</h2>
+<p>The Profiler specification requires a JSON-formatted set of elements, many of which can contain Stellar code. The specification contains the following elements. (For the impatient, skip ahead to the <a href="#Examples">Examples</a>.) The specification for the Profiler topology is stored in Zookeeper at <tt>/metron/topology/profiler</tt>. These properties also exist in the local filesystem at <tt>$METRON_HOME/config/zookeeper/profiler.json</tt>. The values can be changed on disk and then uploaded to Zookeeper using <tt>$METRON_HOME/bin/zk_load_configs.sh</tt>.</p>
+
+<table border="0" class="table table-striped">
+ <thead>
+
+<tr class="a">
+
+<th>Name </th>
+
+<th> </th>
+
+<th>Description</th>
+ </tr>
+ </thead>
+ <tbody>
+
+<tr class="b">
+
+<td><a href="#profile">profile</a> </td>
+
+<td>Required </td>
+
+<td>Unique name identifying the profile.</td>
+ </tr>
+
+<tr class="a">
+
+<td><a href="#foreach">foreach</a> </td>
+
+<td>Required </td>
+
+<td>A separate profile is maintained “for each” of these.</td>
+ </tr>
+
+<tr class="b">
+
+<td><a href="#onlyif">onlyif</a> </td>
+
+<td>Optional </td>
+
+<td>Boolean expression that determines if a message should be applied to the profile.</td>
+ </tr>
+
+<tr class="a">
+
+<td><a href="#groupBy">groupBy</a> </td>
+
+<td>Optional </td>
+
+<td>One or more Stellar expressions used to group the profile measurements when persisted.</td>
+ </tr>
+
+<tr class="b">
+
+<td><a href="#init">init</a> </td>
+
+<td>Optional </td>
+
+<td>One or more expressions executed at the start of a window period.</td>
+ </tr>
+
+<tr class="a">
+
+<td><a href="#update">update</a> </td>
+
+<td>Required </td>
+
+<td>One or more expressions executed when a message is applied to the profile.</td>
+ </tr>
+
+<tr class="b">
+
+<td><a href="#result">result</a> </td>
+
+<td>Required </td>
+
+<td>A Stellar expression that is executed when the window period expires.</td>
+ </tr>
+
+<tr class="a">
+
+<td><a href="#expires">expires</a> </td>
+
+<td>Optional </td>
+
+<td>Profile data is purged after this period of time, specified in milliseconds.</td>
+ </tr>
+ </tbody>
+</table>
+<div class="section">
+<h3><a name="profile"></a><tt>profile</tt></h3>
+<p><i>Required</i></p>
+<p>A unique name identifying the profile. The field is treated as a string. </p></div>
+<div class="section">
+<h3><a name="foreach"></a><tt>foreach</tt></h3>
+<p><i>Required</i></p>
+<p>A separate profile is maintained ‘for each’ of these. This is effectively the entity that the profile is describing. The field is expected to contain a Stellar expression whose result is the entity name. </p>
+<p>For example, if <tt>ip_src_addr</tt> then a separate profile would be maintained for each unique IP source address in the data; 10.0.0.1, 10.0.0.2, etc.</p></div>
+<div class="section">
+<h3><a name="onlyif"></a><tt>onlyif</tt></h3>
+<p><i>Optional</i></p>
+<p>An expression that determines if a message should be applied to the profile. A Stellar expression that returns a Boolean is expected. A message is only applied to a profile if this expression is true. This allows a profile to filter the messages that get applied to it. </p></div>
+<div class="section">
+<h3><a name="groupBy"></a><tt>groupBy</tt></h3>
+<p><i>Optional</i></p>
+<p>One or more Stellar expressions used to group the profile measurements when persisted. This is intended to sort the Profile data to allow for a contiguous scan when accessing subsets of the data. </p>
+<p>The ‘groupBy’ expressions can refer to any field within a <tt>org.apache.metron.profiler.ProfileMeasurement</tt>. A common use case would be grouping by day of week. This allows a contiguous scan to access all profile data for Mondays only. Using the following definition would achieve this. </p>
+
+<div class="source">
+<div class="source">
+<pre>"groupBy": [ "DAY_OF_WEEK()" ]
+</pre></div></div></div>
+<div class="section">
+<h3><a name="init"></a><tt>init</tt></h3>
+<p><i>Optional</i></p>
+<p>One or more expressions executed at the start of a window period. A map is expected where the key is the variable name and the value is a Stellar expression. The map can contain zero or more variable:expression pairs. At the start of each window period, each expression is executed once and stored in the given variable. Note that constant init values such as “0” must be in quotes regardless of their type, as the init value must be a string to be executed by Stellar.</p>
+
+<div class="source">
+<div class="source">
+<pre>"init": {
+ "var1": "0",
+ "var2": "1"
+}
+</pre></div></div></div>
+<div class="section">
+<h3><a name="update"></a><tt>update</tt></h3>
+<p><i>Required</i></p>
+<p>One or more expressions executed when a message is applied to the profile. A map is expected where the key is the variable name and the value is a Stellar expression. The map can include 0 or more variables/expressions. When each message is applied to the profile, the expression is executed and stored in a variable with the given name.</p>
+
+<div class="source">
+<div class="source">
+<pre>"update": {
+ "var1": "var1 + 1",
+ "var2": "var2 + 1"
+}
+</pre></div></div></div>
+<div class="section">
+<h3><a name="result"></a><tt>result</tt></h3>
+<p><i>Required</i></p>
+<p>A Stellar expression that is executed when the window period expires. The expression is expected to summarize the messages that were applied to the profile over the window period, using the state accumulated by the updates. The result will typically be a single numeric value, but it may be any serializable object, as shown in Example 4 below. </p></div>
+<div class="section">
+<h3><a name="expires"></a><tt>expires</tt></h3>
+<p><i>Optional</i></p>
+<p>A numeric value that defines how many days the profile data is retained. After this time, the data expires and is no longer accessible. If no value is defined, the data does not expire.</p></div></div>
+<div class="section">
+<h2><a name="Configuring_the_Profiler"></a>Configuring the Profiler</h2>
+<p>The Profiler runs as an independent Storm topology. The configuration for the Profiler topology is stored in local filesystem at <tt>$METRON_HOME/config/profiler.properties</tt>. The values can be changed on disk and then the Profiler topology must be restarted.</p>
+
+<table border="0" class="table table-striped">
+ <thead>
+
+<tr class="a">
+
+<th>Setting </th>
+
+<th>Description </th>
+ </tr>
+ </thead>
+ <tbody>
+
+<tr class="b">
+
+<td>profiler.workers </td>
+
+<td>The number of worker processes to create for the topology. </td>
+ </tr>
+
+<tr class="a">
+
+<td>profiler.executors </td>
+
+<td>The number of executors to spawn per component. </td>
+ </tr>
+
+<tr class="b">
+
+<td>profiler.input.topic </td>
+
+<td>The name of the Kafka topic from which to consume data. </td>
+ </tr>
+
+<tr class="a">
+
+<td>profiler.period.duration </td>
+
+<td>The duration of each profile period. This value should be defined along with <tt>profiler.period.duration.units</tt>. </td>
+ </tr>
+
+<tr class="b">
+
+<td>profiler.period.duration.units </td>
+
+<td>The units used to specify the <tt>profiler.period.duration</tt>. </td>
+ </tr>
+
+<tr class="a">
+
+<td>profiler.ttl </td>
+
+<td>If a message has not been applied to a Profile in this period of time, the Profile will be forgotten and its resources will be cleaned up. This value should be defined along with <tt>profiler.ttl.units</tt>. </td>
+ </tr>
+
+<tr class="b">
+
+<td>profiler.ttl.units </td>
+
+<td>The units used to specify the <tt>profiler.ttl</tt>. </td>
+ </tr>
+
+<tr class="a">
+
+<td>profiler.hbase.salt.divisor </td>
+
+<td>A salt is prepended to the row key to help prevent hotspotting. This constant is used to generate the salt. Ideally, this constant should be roughly equal to the number of nodes in the Hbase cluster. </td>
+ </tr>
+
+<tr class="b">
+
+<td>profiler.hbase.table </td>
+
+<td>The name of the HBase table that profiles are written to. </td>
+ </tr>
+
+<tr class="a">
+
+<td>profiler.hbase.column.family </td>
+
+<td>The column family used to store profiles. </td>
+ </tr>
+
+<tr class="b">
+
+<td>profiler.hbase.batch </td>
+
+<td>The number of puts that are written in a single batch. </td>
+ </tr>
+
+<tr class="a">
+
+<td>profiler.hbase.flush.interval.seconds </td>
+
+<td>The maximum number of seconds between batch writes to HBase. </td>
+ </tr>
+ </tbody>
+</table>
+<p>After altering the configuration, start the Profiler.</p>
+
+<div class="source">
+<div class="source">
+<pre>$ $METRON_HOME/start_profiler_topology.sh
+</pre></div></div></div>
+<div class="section">
+<h2><a name="Examples"></a>Examples</h2>
+<p>The following examples are intended to highlight the functionality provided by the Profiler. Each shows the configuration that would be required to generate the profile. </p>
+<p>These examples assume a fictitious input message stream that looks something like the following.</p>
+
+<div class="source">
+<div class="source">
+<pre>{
+ "ip_src_addr": "10.0.0.1",
+ "protocol": "HTTPS",
+ "length": "10",
+ "bytes_in": "234"
+},
+{
+ "ip_src_addr": "10.0.0.2",
+ "protocol": "HTTP",
+ "length": "20",
+ "bytes_in": "390"
+},
+{
+ "ip_src_addr": "10.0.0.3",
+ "protocol": "DNS",
+ "length": "30",
+ "bytes_in": "560"
+}
+</pre></div></div>
+<div class="section">
+<h3><a name="Example_1"></a>Example 1</h3>
+<p>The total number of bytes of HTTP data for each host. The following configuration would be used to generate this profile.</p>
+
+<div class="source">
+<div class="source">
+<pre>{
+ "profiles": [
+ {
+ "profile": "example1",
+ "foreach": "ip_src_addr",
+ "onlyif": "protocol == 'HTTP'",
+ "init": {
+ "total_bytes": 0.0
+ },
+ "update": {
+ "total_bytes": "total_bytes + bytes_in"
+ },
+ "result": "total_bytes",
+ "expires": 30
+ }
+ ]
+}
+</pre></div></div>
+<p>This creates a profile…</p>
+
+<ul>
+
+<li>Named ‘example1’</li>
+
+<li>That for each IP source address</li>
+
+<li>Only if the ‘protocol’ field equals ‘HTTP’</li>
+
+<li>Initializes a counter ‘total_bytes’ to zero</li>
+
+<li>Adds to ‘total_bytes’ the value of the message’s ‘bytes_in’ field</li>
+
+<li>Returns ‘total_bytes’ as the result</li>
+
+<li>The profile data will expire in 30 days</li>
+</ul></div>
+<div class="section">
+<h3><a name="Example_2"></a>Example 2</h3>
+<p>The ratio of DNS traffic to HTTP traffic for each host. The following configuration would be used to generate this profile.</p>
+
+<div class="source">
+<div class="source">
+<pre>{
+ "profiles": [
+ {
+ "profile": "example2",
+ "foreach": "ip_src_addr",
+ "onlyif": "protocol == 'DNS' or protocol == 'HTTP'",
+ "init": {
+ "num_dns": 1.0,
+ "num_http": 1.0
+ },
+ "update": {
+ "num_dns": "num_dns + (if protocol == 'DNS' then 1 else 0)",
+ "num_http": "num_http + (if protocol == 'HTTP' then 1 else 0)"
+ },
+ "result": "num_dns / num_http"
+ }
+ ]
+}
+</pre></div></div>
+<p>This creates a profile…</p>
+
+<ul>
+
+<li>Named ‘example2’</li>
+
+<li>That for each IP source address</li>
+
+<li>Only if the ‘protocol’ field equals ‘HTTP’ or ‘DNS’</li>
+
+<li>Accumulates the number of DNS requests</li>
+
+<li>Accumulates the number of HTTP requests</li>
+
+<li>Returns the ratio of these as the result</li>
+</ul></div>
+<div class="section">
+<h3><a name="Example_3"></a>Example 3</h3>
+<p>The average of the <tt>length</tt> field of HTTP traffic. The following configuration would be used to generate this profile.</p>
+
+<div class="source">
+<div class="source">
+<pre>{
+ "profiles": [
+ {
+ "profile": "example3",
+ "foreach": "ip_src_addr",
+ "onlyif": "protocol == 'HTTP'",
+ "update": { "s": "STATS_ADD(s, length)" },
+ "result": "STATS_MEAN(s)"
+ }
+ ]
+}
+</pre></div></div>
+<p>This creates a profile…</p>
+
+<ul>
+
+<li>Named ‘example3’</li>
+
+<li>That for each IP source address</li>
+
+<li>Only if the ‘protocol’ field is ‘HTTP’</li>
+
+<li>Adds the <tt>length</tt> field from each message</li>
+
+<li>Calculates the average as the result</li>
+</ul></div>
+<div class="section">
+<h3><a name="Example_4"></a>Example 4</h3>
+<p>It is important to note that the Profiler can persist any serializable Object, not just numeric values. An alternative to the previous example could take advantage of this. </p>
+<p>Instead of storing the mean of the lengths, the profile could store a statistical summarization of the lengths. This summary can then be used at a later time to calculate the mean, min, max, percentiles, or any other sensible metric. This provides a much greater degree of flexibility.</p>
+
+<div class="source">
+<div class="source">
+<pre>{
+ "profiles": [
+ {
+ "profile": "example4",
+ "foreach": "ip_src_addr",
+ "onlyif": "protocol == 'HTTP'",
+ "update": { "s": "STATS_ADD(s, length)" },
+ "result": "s"
+ }
+ ]
+}
+</pre></div></div>
+<p>The following Stellar REPL session shows how you might use this summary to calculate different metrics with the same underlying profile data. It is assumed that the PROFILE_GET client is configured as described <a href="../metron-profiler-client/index.html">here</a>.</p>
+<p>Retrieve the last 30 minutes of profile measurements for a specific host.</p>
+
+<div class="source">
+<div class="source">
+<pre>$ bin/stellar -z node1:2181
+
+[Stellar]>>> stats := PROFILE_GET( "example4", "10.0.0.1", PROFILE_FIXED(30, "MINUTES"))
+[Stellar]>>> stats
+[org.apache.metron.common.math.stats.OnlineStatisticsProvider@79fe4ab9, ...]
+</pre></div></div>
+<p>Calculate different metrics with the same profile data.</p>
+
+<div class="source">
+<div class="source">
+<pre>[Stellar]>>> STATS_MEAN( GET_FIRST( stats))
+15979.0625
+
+[Stellar]>>> STATS_PERCENTILE( GET_FIRST(stats), 90)
+30310.958
+</pre></div></div>
+<p>Merge all of the profile measurements over the past 30 minutes into a single summary and calculate the 90th percentile.</p>
+
+<div class="source">
+<div class="source">
+<pre>[Stellar]>>> merged := STATS_MERGE( stats)
+[Stellar]>>> STATS_PERCENTILE(merged, 90)
+29810.992
+</pre></div></div>
+<p>More information on accessing profile data can be found in the <a href="../metron-profiler-client/index.html">Profiler Client</a>.</p>
+<p>More information on using the <a href="../../metron-platform/metron-common/index.html"><tt>STATS_*</tt> functions in Stellar can be found here</a>.</p></div></div>
+<div class="section">
+<h2><a name="Implementation"></a>Implementation</h2></div>
+<div class="section">
+<h2><a name="Key_Classes"></a>Key Classes</h2>
+
+<ul>
+
+<li>
+<p><tt>ProfileMeasurement</tt> - Represents a single data point within a Profile. A Profile is effectively a time series. To this end a Profile is composed of many ProfileMeasurement values which in aggregate form a time series.</p></li>
+
+<li>
+<p><tt>ProfilePeriod</tt> - The Profiler captures one <tt>ProfileMeasurement</tt> each <tt>ProfilePeriod</tt>. A <tt>ProfilePeriod</tt> will occur at fixed, deterministic points in time. This allows for efficient retrieval of profile data.</p></li>
+
+<li>
+<p><tt>RowKeyBuilder</tt> - Builds row keys that can be used to read or write profile data to HBase.</p></li>
+
+<li>
+<p><tt>ColumnBuilder</tt> - Defines the columns of data stored with a profile measurement.</p></li>
+
+<li>
+<p><tt>ProfileHBaseMapper</tt> - Defines for the <tt>HBaseBolt</tt> how profile measurements are stored in HBase. This class leverages a <tt>RowKeyBuilder</tt> and <tt>ColumnBuilder</tt>.</p></li>
+</ul></div>
+<div class="section">
+<h2><a name="Storm_Topology"></a>Storm Topology</h2>
+<p>The Profiler is implemented as a Storm topology using the following bolts and spouts.</p>
+
+<ul>
+
+<li>
+<p><tt>KafkaSpout</tt> - A spout that consumes messages from a single Kafka topic. In most cases, the Profiler topology will consume messages from the <tt>indexing</tt> topic. This topic contains fully enriched messages that are ready to be indexed. This ensures that profiles can take advantage of all the available data elements.</p></li>
+
+<li>
+<p><tt>ProfileSplitterBolt</tt> - The bolt responsible for filtering incoming messages and directing each to the one or more downstream bolts that are responsible for building a profile. Each message may be needed by 0, 1 or even many profiles. Each emitted tuple contains the ‘resolved’ entity name, the profile definition, and the input message.</p></li>
+
+<li>
+<p><tt>ProfileBuilderBolt</tt> - This bolt maintains all of the state required to build a profile. When the window period expires, the data is summarized as a <tt>ProfileMeasurement</tt>, all state is flushed, and the <tt>ProfileMeasurement</tt> is emitted. Each instance of this bolt is responsible for maintaining the state for a single Profile-Entity pair.</p></li>
+
+<li>
+<p><tt>HBaseBolt</tt> - A bolt that is responsible for writing to HBase. Most profiles will be flushed every 15 minutes or so. If each <tt>ProfileBuilderBolt</tt> were responsible for writing to HBase itself, there would be little to no opportunity to optimize these writes. By aggregating the writes from multiple Profile-Entity pairs these writes can be batched, for example.</p></li>
+</ul></div>
+ </div>
+ </div>
+ </div>
+
+ <hr/>
+
+ <footer>
+ <div class="container-fluid">
+ <div class="row span12">Copyright © 2017.
+ All Rights Reserved.
+
+ </div>
+
+
+
+ </div>
+ </footer>
+ </body>
+</html>