You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@metron.apache.org by ma...@apache.org on 2017/12/08 09:27:19 UTC
svn commit: r23442 [21/24] - in /dev/metron/0.4.2-RC1: ./ site-book/
site-book/css/ site-book/images/ site-book/images/logos/
site-book/images/profiles/ site-book/img/ site-book/js/
site-book/metron-analytics/ site-book/metron-analytics/metron-maas-ser...
Added: dev/metron/0.4.2-RC1/site-book/metron-sensors/fastcapa/index.html
==============================================================================
--- dev/metron/0.4.2-RC1/site-book/metron-sensors/fastcapa/index.html (added)
+++ dev/metron/0.4.2-RC1/site-book/metron-sensors/fastcapa/index.html Fri Dec 8 09:27:19 2017
@@ -0,0 +1,1101 @@
+<!DOCTYPE html>
+<!--
+ | Generated by Apache Maven Doxia at 2017-12-08
+ | Rendered using Apache Maven Fluido Skin 1.3.0
+-->
+<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
+ <head>
+ <meta charset="UTF-8" />
+ <meta name="viewport" content="width=device-width, initial-scale=1.0" />
+ <meta name="Date-Revision-yyyymmdd" content="20171208" />
+ <meta http-equiv="Content-Language" content="en" />
+ <title>Metron – Fastcapa</title>
+ <link rel="stylesheet" href="../../css/apache-maven-fluido-1.3.0.min.css" />
+ <link rel="stylesheet" href="../../css/site.css" />
+ <link rel="stylesheet" href="../../css/print.css" media="print" />
+
+
+ <script type="text/javascript" src="../../js/apache-maven-fluido-1.3.0.min.js"></script>
+
+
+
+<script type="text/javascript">$( document ).ready( function() { $( '.carousel' ).carousel( { interval: 3500 } ) } );</script>
+
+ </head>
+ <body class="topBarDisabled">
+
+
+
+
+ <div class="container-fluid">
+ <div id="banner">
+ <div class="pull-left">
+ <a href="http://metron.apache.org/" id="bannerLeft">
+ <img src="../../images/metron-logo.png" alt="Apache Metron" width="148px" height="48px"/>
+ </a>
+ </div>
+ <div class="pull-right"> </div>
+ <div class="clear"><hr/></div>
+ </div>
+
+ <div id="breadcrumbs">
+ <ul class="breadcrumb">
+
+
+ <li class="">
+ <a href="http://www.apache.org" class="externalLink" title="Apache">
+ Apache</a>
+ </li>
+ <li class="divider ">/</li>
+ <li class="">
+ <a href="http://metron.apache.org/" class="externalLink" title="Metron">
+ Metron</a>
+ </li>
+ <li class="divider ">/</li>
+ <li class="">
+ <a href="../../index.html" title="Documentation">
+ Documentation</a>
+ </li>
+ <li class="divider ">/</li>
+ <li class="">Fastcapa</li>
+
+
+
+ <li id="publishDate" class="pull-right">Last Published: 2017-12-08</li> <li class="divider pull-right">|</li>
+ <li id="projectVersion" class="pull-right">Version: 0.4.2</li>
+
+ </ul>
+ </div>
+
+
+ <div class="row-fluid">
+ <div id="leftColumn" class="span3">
+ <div class="well sidebar-nav">
+
+
+ <ul class="nav nav-list">
+ <li class="nav-header">User Documentation</li>
+
+ <li>
+
+ <a href="../../index.html" title="Metron">
+ <i class="icon-chevron-down"></i>
+ Metron</a>
+ <ul class="nav nav-list">
+
+ <li>
+
+ <a href="../../Upgrading.html" title="Upgrading">
+ <i class="none"></i>
+ Upgrading</a>
+ </li>
+
+ <li>
+
+ <a href="../../metron-analytics/index.html" title="Analytics">
+ <i class="icon-chevron-right"></i>
+ Analytics</a>
+ </li>
+
+ <li>
+
+ <a href="../../metron-contrib/metron-docker/index.html" title="Docker">
+ <i class="none"></i>
+ Docker</a>
+ </li>
+
+ <li>
+
+ <a href="../../metron-deployment/index.html" title="Deployment">
+ <i class="icon-chevron-right"></i>
+ Deployment</a>
+ </li>
+
+ <li>
+
+ <a href="../../metron-interface/metron-alerts/index.html" title="Alerts">
+ <i class="none"></i>
+ Alerts</a>
+ </li>
+
+ <li>
+
+ <a href="../../metron-interface/metron-config/index.html" title="Config">
+ <i class="none"></i>
+ Config</a>
+ </li>
+
+ <li>
+
+ <a href="../../metron-interface/metron-rest/index.html" title="Rest">
+ <i class="none"></i>
+ Rest</a>
+ </li>
+
+ <li>
+
+ <a href="../../metron-platform/index.html" title="Platform">
+ <i class="icon-chevron-right"></i>
+ Platform</a>
+ </li>
+
+ <li>
+
+ <a href="../../metron-sensors/index.html" title="Sensors">
+ <i class="icon-chevron-down"></i>
+ Sensors</a>
+ <ul class="nav nav-list">
+
+ <li class="active">
+
+ <a href="#"><i class="none"></i>Fastcapa</a>
+ </li>
+
+ <li>
+
+ <a href="../../metron-sensors/pycapa/index.html" title="Pycapa">
+ <i class="none"></i>
+ Pycapa</a>
+ </li>
+ </ul>
+ </li>
+
+ <li>
+
+ <a href="../../metron-stellar/stellar-3rd-party-example/index.html" title="Stellar-3rd-party-example">
+ <i class="none"></i>
+ Stellar-3rd-party-example</a>
+ </li>
+
+ <li>
+
+ <a href="../../metron-stellar/stellar-common/index.html" title="Stellar-common">
+ <i class="icon-chevron-right"></i>
+ Stellar-common</a>
+ </li>
+
+ <li>
+
+ <a href="../../use-cases/index.html" title="Use-cases">
+ <i class="icon-chevron-right"></i>
+ Use-cases</a>
+ </li>
+ </ul>
+ </li>
+ </ul>
+
+
+
+ <hr class="divider" />
+
+ <div id="poweredBy">
+ <div class="clear"></div>
+ <div class="clear"></div>
+ <div class="clear"></div>
+ <a href="http://maven.apache.org/" title="Built by Maven" class="poweredBy">
+ <img class="builtBy" alt="Built by Maven" src="../../images/logos/maven-feather.png" />
+ </a>
+ </div>
+ </div>
+ </div>
+
+
+ <div id="bodyColumn" class="span9" >
+
+ <h1>Fastcapa</h1>
+<p>Fastcapa is a probe that performs fast network packet capture by leveraging Linux kernel-bypass and user space networking technology. The probe will bind to a network interface, capture network packets, and send the raw packet data to Kafka. This provides a scalable mechanism for ingesting high-volumes of network packet data into a Hadoop-y cluster.</p>
+<p>Fastcapa leverages the Data Plane Development Kit (<a class="externalLink" href="http://dpdk.org/">DPDK</a>). DPDK is a set of libraries and drivers to perform fast packet processing in Linux user space. </p>
+
+<ul>
+
+<li><a href="#Quick_Start">Quick Start</a></li>
+
+<li><a href="#Requirements">Requirements</a></li>
+
+<li><a href="#Installation">Installation</a></li>
+
+<li><a href="#Usage">Usage</a>
+
+<ul>
+
+<li><a href="#Parameters">Parameters</a></li>
+
+<li><a href="#Output">Output</a></li>
+
+<li><a href="#Kerberos">Kerberos</a></li>
+ </ul></li>
+
+<li><a href="#How_It_Works">How It Works</a></li>
+
+<li><a href="#Performance">Performance</a></li>
+
+<li><a href="#FAQs">FAQs</a></li>
+</ul>
+<div class="section">
+<h2><a name="Quick_Start"></a>Quick Start</h2>
+<p>The quickest way to see Fastcapa in action is to use a Virtualbox environment on your local machine. The necessary files and instructions to do this are located at <a href="../../metron-deployment/vagrant/fastcapa-test-platform/index.html"><tt>metron-deployment/vagrant/fastcapa-vagrant</tt></a>. All you need to do is execute the following.</p>
+
+<div class="source">
+<div class="source">
+<pre>cd metron-deployment/vagrant/fastcapa-test-platform
+vagrant up
+</pre></div></div>
+<p>This environment sets up two nodes. One node produces network packets over a network interface. The second node uses Fastcapa to capture those packets and write them to a Kafka broker running on the first node. Basic validation is performed to ensure that Fastcapa is able to land packet data in Kafka.</p></div>
+<div class="section">
+<h2><a name="Requirements"></a>Requirements</h2>
+<p>The following system requirements must be met to run the Fastcapa probe.</p>
+
+<ul>
+
+<li>Linux kernel >= 2.6.34</li>
+
+<li>A <a class="externalLink" href="http://dpdk.org/doc/nics">DPDK supported ethernet device; NIC</a>.</li>
+
+<li>Port(s) on the ethernet device that can be dedicated for exclusive use by Fastcapa</li>
+</ul></div>
+<div class="section">
+<h2><a name="Installation"></a>Installation</h2>
+<p>The process of installing Fastcapa has a fair number of steps and involves building DPDK, loading specific kernel modules, enabling huge page memory, and binding compatible network interface cards.</p>
+<div class="section">
+<h3><a name="Automated_Installation"></a>Automated Installation</h3>
+<p>The best documentation is code that actually does this for you. An Ansible role that performs the entire installation procedure can be found at <a href="../../metron-deployment/roles/fastcapa/index.html"><tt>metron-deployment/roles/fastcapa</tt></a>. Use this to install Fastcapa or as a guide for manual installation. The automated installation assumes CentOS 7.1 and is directly tested against <a class="externalLink" href="https://atlas.hashicorp.com/bento/boxes/centos-7.1">bento/centos-7.1</a>.</p></div>
+<div class="section">
+<h3><a name="Manual_Installation"></a>Manual Installation</h3>
+<p>The following manual installation steps assume that they are executed on CentOS 7.1. Some minor differences may result if you use a different Linux distribution.</p>
+
+<ul>
+
+<li><a href="#Enable_Transparent_Huge_Pages">Enable Transparent Huge Pages</a></li>
+
+<li><a href="#Install_DPDK">Install DPDK</a></li>
+
+<li><a href="#Install_Librdkafka">Install Librdkafka</a></li>
+
+<li><a href="#Install_Fastcapa">Install Fastcapa</a></li>
+</ul>
+<div class="section">
+<h4><a name="Enable_Transparent_Huge_Pages"></a>Enable Transparent Huge Pages</h4>
+<p>The probe performs its own memory management by leveraging transparent huge pages. In Linux, Transparent Huge Pages (THP) can be enabled either dynamically or on boot. It is recommended that these be allocated on boot to increase the chance that a larger, physically contiguous chunk of memory can be allocated.</p>
+<p>The size of THPs that are supported will vary based on your CPU. These typically include 2 MB and 1 GB THPs. For better performance, allocate 1 GB THPs if supported by your CPU.</p>
+
+<ol style="list-style-type: decimal">
+
+<li>
+<p>Ensure that your CPU supports 1 GB THPs. A CPU flag <tt>pdpe1gb</tt> indicates whether or not the CPU supports 1 GB THPs.</p>
+
+<div class="source">
+<div class="source">
+<pre>grep --color=always pdpe1gb /proc/cpuinfo | uniq
+</pre></div></div></li>
+
+<li>
+<p>Add the following boot parameters to the Linux kernel. Edit <tt>/etc/default/grub</tt> and add the additional kernel parameters to the line starting with <tt>GRUB_CMDLINE_LINUX</tt>.</p>
+
+<div class="source">
+<div class="source">
+<pre>GRUB_CMDLINE_LINUX=... default_hugepagesz=1G hugepagesz=1G hugepages=16
+</pre></div></div></li>
+
+<li>
+<p>Rebuild the grub configuration then reboot. The location of the Grub configuration file will differ across Linux distributions.</p>
+
+<div class="source">
+<div class="source">
+<pre>cp /etc/grub2-efi.cfg /etc/grub2-efi.cfg.orig
+/sbin/grub2-mkconfig -o /etc/grub2-efi.cfg
+</pre></div></div></li>
+
+<li>
+<p>Once the host has been rebooted, ensure that the THPs were successfully allocated.</p>
+
+<div class="source">
+<div class="source">
+<pre>$ grep HugePage /proc/meminfo
+AnonHugePages: 933888 kB
+HugePages_Total: 16
+HugePages_Free: 0
+HugePages_Rsvd: 0
+HugePages_Surp: 0
+</pre></div></div>
+<p>The total number of huge pages that you have been allocated should be distributed fairly evenly across each NUMA node. In this example, a total of 16 were requested and 8 have been assigned on each of the 2 NUMA nodes.</p>
+
+<div class="source">
+<div class="source">
+<pre>$ cat /sys/devices/system/node/node*/hugepages/hugepages-1048576kB/nr_hugepages
+8
+8
+</pre></div></div></li>
+
+<li>
+<p>Once the THPs have been reserved, they need to be mounted to make them available to the probe.</p>
+
+<div class="source">
+<div class="source">
+<pre>cp /etc/fstab /etc/fstab.orig
+mkdir -p /mnt/huge_1GB
+echo "nodev /mnt/huge_1GB hugetlbfs pagesize=1GB 0 0" >> /etc/fstab
+mount -fav
+</pre></div></div></li>
+</ol></div>
+<div class="section">
+<h4><a name="Install_DPDK"></a>Install DPDK</h4>
+
+<ol style="list-style-type: decimal">
+
+<li>
+<p>Install the required dependencies.</p>
+
+<div class="source">
+<div class="source">
+<pre>yum -y install "@Development tools"
+yum -y install pciutils net-tools glib2 glib2-devel git
+yum -y install kernel kernel-devel kernel-headers
+</pre></div></div></li>
+
+<li>
+<p>Decide where DPDK will be installed.</p>
+
+<div class="source">
+<div class="source">
+<pre>export DPDK_HOME=/usr/local/dpdk/
+</pre></div></div></li>
+
+<li>
+<p>Download, build and install DPDK.</p>
+
+<div class="source">
+<div class="source">
+<pre>wget http://fast.dpdk.org/rel/dpdk-16.11.1.tar.xz -O - | tar -xJ
+cd dpdk-stable-16.11.1/
+make config install T=x86_64-native-linuxapp-gcc DESTDIR=$DPDK_HOME
+</pre></div></div></li>
+
+<li>
+<p>Find the PCI address of the ethernet device that you plan on using to capture network packets. In the following example I plan on binding <tt>enp9s0f0</tt> which has a PCI address of <tt>09:00.0</tt>.</p>
+
+<div class="source">
+<div class="source">
+<pre>$ lspci | grep "VIC Ethernet"
+09:00.0 Ethernet controller: Cisco Systems Inc VIC Ethernet NIC (rev a2)
+0a:00.0 Ethernet controller: Cisco Systems Inc VIC Ethernet NIC (rev a2)
+</pre></div></div></li>
+
+<li>
+<p>Bind the device. Replace the device name and PCI address with what is appropriate for your environment.</p>
+
+<div class="source">
+<div class="source">
+<pre>ifdown enp9s0f0
+modprobe uio_pci_generic
+$DPDK_HOME/sbin/dpdk-devbind --bind=uio_pci_generic "09:00.0"
+</pre></div></div></li>
+
+<li>
+<p>Ensure that the device was bound. It should be shown as a ‘network device using DPDK-compatible driver.’</p>
+
+<div class="source">
+<div class="source">
+<pre>$ dpdk-devbind --status
+Network devices using DPDK-compatible driver
+============================================
+0000:09:00.0 'VIC Ethernet NIC' drv=uio_pci_generic unused=enic
+Network devices using kernel driver
+===================================
+0000:01:00.0 'I350 Gigabit Network Connection' if=eno1 drv=igb unused=uio_pci_generic
+</pre></div></div></li>
+</ol></div>
+<div class="section">
+<h4><a name="Install_Librdkafka"></a>Install Librdkafka</h4>
+<p>The probe has been tested with <a class="externalLink" href="https://github.com/edenhill/librdkafka/releases/tag/v0.9.4">Librdkafka 0.9.4</a>.</p>
+
+<ol style="list-style-type: decimal">
+
+<li>
+<p>Choose an installation path. In this example, the libs will actually be installed at <tt>/usr/local/lib</tt>; note that <tt>lib</tt> is appended to the prefix.</p>
+
+<div class="source">
+<div class="source">
+<pre>export RDK_PREFIX=/usr/local
+</pre></div></div></li>
+
+<li>
+<p>Download, build and install.</p>
+
+<div class="source">
+<div class="source">
+<pre>wget https://github.com/edenhill/librdkafka/archive/v0.9.4.tar.gz -O - | tar -xz
+cd librdkafka-0.9.4/
+./configure --prefix=$RDK_PREFIX
+make
+make install
+</pre></div></div></li>
+
+<li>
+<p>Ensure that the installation location is on the search path for the runtime shared library loader.</p>
+
+<div class="source">
+<div class="source">
+<pre>export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$RDK_PREFIX/lib
+</pre></div></div></li>
+</ol></div>
+<div class="section">
+<h4><a name="Install_Fastcapa"></a>Install Fastcapa</h4>
+
+<ol style="list-style-type: decimal">
+
+<li>
+<p>Set the required environment variables.</p>
+
+<div class="source">
+<div class="source">
+<pre>export RTE_SDK=$DPDK_HOME/share/dpdk/
+export RTE_TARGET=x86_64-native-linuxapp-gcc
+export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$RDK_HOME
+</pre></div></div></li>
+
+<li>
+<p>Build Fastcapa. The resulting binary will be placed at <tt>build/app/fastcapa</tt>.</p>
+
+<div class="source">
+<div class="source">
+<pre>cd metron/metron-sensors/fastcapa
+make
+</pre></div></div></li>
+</ol></div></div></div>
+<div class="section">
+<h2><a name="Usage"></a>Usage</h2>
+<p>Follow these steps to run Fastcapa.</p>
+
+<ol style="list-style-type: decimal">
+
+<li>
+<p>Create a configuration file that at a minimum specifies your Kafka broker. An example configuration file, <tt>conf/fastcapa.conf</tt>, is available that documents other useful parameters.</p>
+
+<div class="source">
+<div class="source">
+<pre>[kafka-global]
+metadata.broker.list = kafka-broker1:9092
+</pre></div></div></li>
+
+<li>
+<p>Bind the capture device. This is only needed if the device is not already bound. In this example, the device <tt>enp9s0f0</tt> with a PCI address of <tt>09:00:0</tt> is bound. Use values specific to your environment.</p>
+
+<div class="source">
+<div class="source">
+<pre>ifdown enp9s0f0
+modprobe uio_pci_generic
+$DPDK_HOME/sbin/dpdk-devbind --bind=uio_pci_generic "09:00.0"
+</pre></div></div></li>
+
+<li>
+<p>Run Fastcapa.</p>
+
+<div class="source">
+<div class="source">
+<pre>fastcapa -c 0x03 --huge-dir /mnt/huge_1GB -- -p 0x01 -t pcap -c /etc/fastcapa.conf
+</pre></div></div></li>
+
+<li>
+<p>Terminate Fastcapa with <tt>SIGINT</tt> or by entering <tt>CTRL-C</tt>. The probe will cleanly shut down all of the workers and allow the backlog of packets to drain. To terminate the process without clearing the queue, send a <tt>SIGKILL</tt> or be entering <tt>killall -9 fastcapa</tt>.</p></li>
+</ol>
+<div class="section">
+<h3><a name="Parameters"></a>Parameters</h3>
+<p>Fastcapa accepts three sets of parameters. </p>
+
+<ol style="list-style-type: decimal">
+
+<li>Command-line parameters passed directly to DPDK’s Environmental Abstraction Layer (EAL)</li>
+
+<li>Command-line parameters that define how Fastcapa will interact with DPDK. These parametera are separated on the command line by a double-dash (<tt>--</tt>).</li>
+
+<li>A configuration file that define how Fastcapa interacts with Librdkafka.</li>
+</ol>
+<div class="section">
+<h4><a name="Environmental_Abstraction_Layer_Parameters"></a>Environmental Abstraction Layer Parameters</h4>
+<p>The most commonly used EAL parameter involves specifying which logical CPU cores should be used for processing. This can be specified in any of the following ways.</p>
+
+<div class="source">
+<div class="source">
+<pre> -c COREMASK Hexadecimal bitmask of cores to run on
+ -l CORELIST List of cores to run on
+ The argument format is <c1>[-c2][,c3[-c4],...]
+ where c1, c2, etc are core indexes between 0 and 128
+ --lcores COREMAP Map lcore set to physical cpu set
+ The argument format is
+ '<lcores[@cpus]>[<,lcores[@cpus]>...]'
+ lcores and cpus list are grouped by '(' and ')'
+ Within the group, '-' is used for range separator,
+ ',' is used for single number separator.
+ '( )' can be omitted for single element group,
+ '@' can be omitted if cpus and lcores have the same value
+</pre></div></div>
+<p>To get more information about other EAL parameters, run the following.</p>
+
+<div class="source">
+<div class="source">
+<pre>fastcapa -h
+</pre></div></div></div>
+<div class="section">
+<h4><a name="Fastcapa-Core_Parameters"></a>Fastcapa-Core Parameters</h4>
+
+<table border="0" class="table table-striped">
+ <thead>
+
+<tr class="a">
+
+<th>Name </th>
+
+<th>Command </th>
+
+<th>Description </th>
+
+<th>Default </th>
+ </tr>
+ </thead>
+ <tbody>
+
+<tr class="b">
+
+<td>Port Mask </td>
+
+<td>-p PORT_MASK </td>
+
+<td>A bit mask identifying which ports to bind. </td>
+
+<td>0x01 </td>
+ </tr>
+
+<tr class="a">
+
+<td>Receive Burst Size </td>
+
+<td>-b RX_BURST_SIZE </td>
+
+<td>The max number of packets processed by a receive worker. </td>
+
+<td>32 </td>
+ </tr>
+
+<tr class="b">
+
+<td>Transmit Burst Size </td>
+
+<td>-w TX_BURST_SIZE </td>
+
+<td>The max number of packets processed by a transmit worker. </td>
+
+<td>256 </td>
+ </tr>
+
+<tr class="a">
+
+<td>Receive Descriptors </td>
+
+<td>-d NB_RX_DESC </td>
+
+<td>The number of descriptors for each receive queue (the size of the receive queue.) Limited by the ethernet device in use. </td>
+
+<td>1024 </td>
+ </tr>
+
+<tr class="b">
+
+<td>Transmission Ring Size </td>
+
+<td>-x TX_RING_SIZE </td>
+
+<td>The size of each transmission ring. This must be a power of 2. </td>
+
+<td>2048 </td>
+ </tr>
+
+<tr class="a">
+
+<td>Number Receive Queues </td>
+
+<td>-q NB_RX_QUEUE </td>
+
+<td>Number of receive queues to use for each port. Limited by the ethernet device in use. </td>
+
+<td>2 </td>
+ </tr>
+
+<tr class="b">
+
+<td>Kafka Topic </td>
+
+<td>-t KAFKA_TOPIC </td>
+
+<td>The name of the Kafka topic. </td>
+
+<td>pcap </td>
+ </tr>
+
+<tr class="a">
+
+<td>Configuration File </td>
+
+<td>-c KAFKA_CONF </td>
+
+<td>Path to a file containing configuration values. </td>
+
+<td> </td>
+ </tr>
+
+<tr class="b">
+
+<td>Stats </td>
+
+<td>-s KAFKA_STATS </td>
+
+<td>Appends performance metrics in the form of JSON strings to the specified file. </td>
+
+<td> </td>
+ </tr>
+ </tbody>
+</table>
+<p>To get more information about the Fastcapa specific parameters, run the following. Note that this puts the <tt>-h</tt> after the double-dash <tt>--</tt>.</p>
+
+<div class="source">
+<div class="source">
+<pre>fastcapa -- -h
+
+fastcapa [EAL options] -- [APP options]
+ -p PORT_MASK bitmask of ports to bind [0x01]
+ -b RX_BURST_SIZE burst size of receive worker [32]
+ -w TX_BURST_SIZE burst size of transmit worker [256]
+ -d NB_RX_DESC num of descriptors for receive ring [1024]
+ -x TX_RING_SIZE size of tx rings (must be a power of 2) [2048]
+ -q NB_RX_QUEUE num of receive queues for each device [1]
+ -t KAFKA_TOPIC name of the kafka topic [pcap]
+ -c KAFKA_CONF file containing configs for kafka client
+ -s KAFKA_STATS append kafka client stats to a file
+ -h print this help message
+</pre></div></div></div>
+<div class="section">
+<h4><a name="Fastcapa-Kafka_Configuration_File"></a>Fastcapa-Kafka Configuration File</h4>
+<p>The path to the configuration file is specified with the <tt>-c</tt> command line argument. The file can contain any global or topic-specific, producer-focused <a class="externalLink" href="https://github.com/edenhill/librdkafka/blob/master/CONFIGURATION.md">configuration values accepted by Librdkafka</a>. </p>
+<p>The configuration file is a <tt>.ini</tt>-like Glib configuration file. The global configuration values should be placed under a <tt>[kafka-global]</tt> header and topic-specific values should be placed under <tt>[kafka-topic]</tt>.</p>
+<p>A minimally viable configuration file would only need to include the Kafka broker to connect to.</p>
+
+<div class="source">
+<div class="source">
+<pre>[kafka-global]
+metadata.broker.list = kafka-broker1:9092, kafka-broker2:9092
+</pre></div></div>
+<p>The configuration parameters that are important for either basic functioning or performance tuning of Fastcapa include the following.</p>
+<p>Global configuration values that should be located under the <tt>[kafka-global]</tt> header.</p>
+
+<table border="0" class="table table-striped">
+ <thead>
+
+<tr class="a">
+
+<th><i>Name</i> </th>
+
+<th><i>Description</i> </th>
+
+<th><i>Default</i> </th>
+ </tr>
+ </thead>
+ <tbody>
+
+<tr class="b">
+
+<td>metadata.broker.list </td>
+
+<td>Initial list of brokers as a CSV list of broker host or host:port </td>
+
+<td> </td>
+ </tr>
+
+<tr class="a">
+
+<td>client.id </td>
+
+<td>Client identifier. </td>
+
+<td> </td>
+ </tr>
+
+<tr class="b">
+
+<td>queue.buffering.max.messages </td>
+
+<td>Maximum number of messages allowed on the producer queue </td>
+
+<td>100000 </td>
+ </tr>
+
+<tr class="a">
+
+<td>queue.buffering.max.ms </td>
+
+<td>Maximum time, in milliseconds, for buffering data on the producer queue </td>
+
+<td>1000 </td>
+ </tr>
+
+<tr class="b">
+
+<td>message.copy.max.bytes </td>
+
+<td>Maximum size for message to be copied to buffer. Messages larger than this will be passed by reference (zero-copy) at the expense of larger iovecs. </td>
+
+<td>65535 </td>
+ </tr>
+
+<tr class="a">
+
+<td>batch.num.messages </td>
+
+<td>Maximum number of messages batched in one MessageSet </td>
+
+<td>10000 </td>
+ </tr>
+
+<tr class="b">
+
+<td>statistics.interval.ms </td>
+
+<td>How often statistics are emitted; 0 = never </td>
+
+<td>0 </td>
+ </tr>
+
+<tr class="a">
+
+<td>compression.codec </td>
+
+<td>Compression codec to use for compressing message sets; {none, gzip, snappy, lz4 } </td>
+
+<td>none </td>
+ </tr>
+ </tbody>
+</table>
+<p>Topic configuration values that should be located under the <tt>[kafka-topic]</tt> header.</p>
+
+<table border="0" class="table table-striped">
+ <thead>
+
+<tr class="a">
+
+<th><i>Name</i> </th>
+
+<th><i>Description</i> </th>
+
+<th><i>Default</i> </th>
+ </tr>
+ </thead>
+ <tbody>
+
+<tr class="b">
+
+<td>compression.codec </td>
+
+<td>Compression codec to use for compressing message sets; {none, gzip, snappy, lz4 } </td>
+
+<td>none </td>
+ </tr>
+
+<tr class="a">
+
+<td>request.required.acks </td>
+
+<td>How many acknowledgements the leader broker must receive from ISR brokers before responding to the request; { 0 = no ack, 1 = leader ack, -1 = all ISRs } </td>
+
+<td>1 </td>
+ </tr>
+
+<tr class="b">
+
+<td>message.timeout.ms </td>
+
+<td>Local message timeout. This value is only enforced locally and limits the time a produced message waits for successful delivery. A time of 0 is infinite. </td>
+
+<td>300000 </td>
+ </tr>
+
+<tr class="a">
+
+<td>queue.buffering.max.kbytes </td>
+
+<td>Maximum total message size sum allowed on the producer queue </td>
+
+<td> </td>
+ </tr>
+ </tbody>
+</table></div></div>
+<div class="section">
+<h3><a name="Output"></a>Output</h3>
+<p>When running the probe some basic counters are output to stdout. Of course during normal operation these values will be much larger.</p>
+
+<div class="source">
+<div class="source">
+<pre> ------ in ------ --- queued --- ----- out ----- ---- drops ----
+[nic] 8 - - -
+[rx] 8 0 8 0
+[tx] 8 0 8 0
+[kaf] 8 1 7 0
+</pre></div></div>
+
+<ul>
+
+<li><tt>[nic]</tt> + <tt>in</tt>: The ethernet device is reporting that it has seen 8 packets.</li>
+
+<li><tt>[rx]</tt> + <tt>in</tt>: The receive workers have consumed 8 packets from the device.</li>
+
+<li><tt>[rx]</tt> + <tt>out</tt>: The receive workers have enqueued 8 packets onto the transmission rings.</li>
+
+<li><tt>[rx]</tt> + <tt>drops</tt>: If the transmission rings become full it will prevent the receive workers from enqueuing additional packets. The excess packets are dropped. This value will never decrease.</li>
+
+<li><tt>[tx]</tt> + <tt>in</tt>: The transmission workers have consumed 8 packets.</li>
+
+<li><tt>[tx]</tt> + <tt>out</tt>: The transmission workers have packaged 8 packets into Kafka messages.</li>
+
+<li><tt>[tx]</tt> + <tt>drops</tt>: If the Kafka client library accepted fewer packets than expected. This value can increase or decrease over time as additional packets are acknowledged by the Kafka client library at a later point in time.</li>
+
+<li><tt>[kaf]</tt> + <tt>in</tt>: The Kafka client library has received 8 packets.</li>
+
+<li><tt>[kaf]</tt> + <tt>out</tt>: A total of 7 packets has successfully reached Kafka.</li>
+
+<li><tt>[kaf]</tt> + <tt>queued</tt>: There is 1 packet within the <tt>rdkafka</tt> queue waiting to be sent.</li>
+</ul></div>
+<div class="section">
+<h3><a name="Kerberos"></a>Kerberos</h3>
+<p>The probe can be used in a Kerberized environment. Follow these additional steps to use Fastcapa with Kerberos. The following assumptions have been made. These may need altered to fit your environment.</p>
+
+<ul>
+
+<li>The Kafka broker is at <tt>kafka1:6667</tt></li>
+
+<li>Zookeeper is at <tt>zookeeper1:2181</tt></li>
+
+<li>The Kafka security protocol is <tt>SASL_PLAINTEXT</tt></li>
+
+<li>The keytab used is located at <tt>/etc/security/keytabs/metron.headless.keytab</tt></li>
+
+<li>The service principal is <tt>metron@EXAMPLE.COM</tt></li>
+</ul>
+
+<ol style="list-style-type: decimal">
+
+<li>
+<p>Build Librdkafka with SASL support (<tt>--enable-sasl</tt>).</p>
+
+<div class="source">
+<div class="source">
+<pre>wget https://github.com/edenhill/librdkafka/archive/v0.9.4.tar.gz -O - | tar -xz
+cd librdkafka-0.9.4/
+./configure --prefix=$RDK_PREFIX --enable-sasl
+make
+make install
+</pre></div></div></li>
+
+<li>
+<p>Validate Librdkafka does indeed support SASL. Run the following command and ensure that <tt>sasl</tt> is returned as a built-in feature.</p>
+
+<div class="source">
+<div class="source">
+<pre>$ examples/rdkafka_example -X builtin.features
+builtin.features = gzip,snappy,ssl,sasl,regex
+</pre></div></div>
+<p>If it is not, ensure that you have <tt>libsasl</tt> or <tt>libsasl2</tt> installed. On CentOS, this can be installed with the following command.</p>
+
+<div class="source">
+<div class="source">
+<pre>yum install -y cyrus-sasl cyrus-sasl-devel cyrus-sasl-gssapi
+</pre></div></div></li>
+
+<li>
+<p>Grant access to your Kafka topic. In this example, it is simply named <tt>pcap</tt>.</p>
+
+<div class="source">
+<div class="source">
+<pre>$KAFKA_HOME/bin/kafka-acls.sh --authorizer kafka.security.auth.SimpleAclAuthorizer \
+ --authorizer-properties zookeeper.connect=zookeeper1:2181 \
+ --add --allow-principal User:metron --topic pcap
+</pre></div></div></li>
+
+<li>
+<p>Obtain a Kerberos ticket.</p>
+
+<div class="source">
+<div class="source">
+<pre>kinit -kt /etc/security/keytabs/metron.headless.keytab metron@EXAMPLE.COM
+</pre></div></div></li>
+
+<li>
+<p>Add the following additional configuration values to your Fastcapa configuration file.</p>
+
+<div class="source">
+<div class="source">
+<pre>security.protocol = SASL_PLAINTEXT
+sasl.kerberos.keytab = /etc/security/keytabs/metron.headless.keytab
+sasl.kerberos.principal = metron@EXAMPLE.COM
+</pre></div></div></li>
+
+<li>
+<p>Now run Fastcapa as you normally would. It should have no problem landing packets in your kerberized Kafka broker.</p></li>
+</ol></div></div>
+<div class="section">
+<h2><a name="How_It_Works"></a>How It Works</h2>
+<p>The probe leverages a poll-mode, burst-oriented mechanism to capture packets from a network interface and transmit them efficiently to a Kafka topic. Each packet is wrapped within a single Kafka message and the current timestamp, as epoch microseconds in network byte order, is attached as the message’s key.</p>
+<p>The probe leverages Receive Side Scaling (RSS), a feature provided by some ethernet devices that allows processing of received data to occur across multiple processes and logical cores. It does this by running a hash function on each packet, whose value assigns the packet to one, of possibly many, receive queues. The total number and size of these receive queues are limited by the ethernet device in use. More capable ethernet devices will offer a greater number and greater sized receive queues. </p>
+
+<ul>
+
+<li>Increasing the number of receive queues allows for greater parallelization of receive side processing.</li>
+
+<li>Increasing the size of each receive queue can allow the probe to handle larger, temporary spikes of network packets that can often occur.</li>
+</ul>
+<p>A set of receive workers, each assigned to a unique logical core, are responsible for fetching packets from the receive queues. There can only be one receive worker for each receive queue. The receive workers continually poll the receive queues and attempt to fetch multiple packets on each request. The maximum number of packets fetched in one request is known as the ‘burst size’. If the receive worker actually receives ‘burst size’ packets, then it is likely that the queue is under pressure and more packets are available. In this case the worker immediately fetches another ‘burst size’ set of packets. It repeats this process up to a fixed number of times while the queue is under pressure.</p>
+<p>The receive workers then enqueue the received packets into a fixed size ring buffer known as a transmit ring. There is always one transmit ring for each receive queue. A set of transmit workers then dequeue packets from the transmit rings. There can be one or more transmit workers assigned to any single transmit ring. Each transmit worker has its own unique connection to Kafka.</p>
+
+<ul>
+
+<li>Increasing the number of transmit workers allows for greater parallelization when writing data to Kafka.</li>
+
+<li>Increasing the size of the transmit rings allows the probe to better handle temporary interruptions and latency when writing to Kafka.</li>
+</ul>
+<p>After receiving the network packets from the transmit worker, the Kafka client library internally maintains its own send queue of messages. Multiple threads are then responsible for managing this queue and creating batches of messages that are sent in bulk to a Kafka broker. No control is exercised over this additional send queue and its worker threads, which can be an impediment to performance. This is an opportunity for improvement that can be addressed as follow-on work.</p></div>
+<div class="section">
+<h2><a name="Performance"></a>Performance</h2>
+<p>Beyond tuning the parameters previously described, the following should be carefully considered to achieve maximum performance.</p>
+<div class="section">
+<h3><a name="Kafka_Partitions"></a>Kafka Partitions</h3>
+<p>Parallelizing access to a topic in Kafka is achieved by defining multiple partitions. The greater the number of partitions, the more parallelizable access to that topic becomes. To achieve high throughput it is important to ensure that the Kafka topic in use has a large number of partitions, evenly distributed across each of the nodes in your Kafka cluster.</p>
+<p>The specific number of partitions needed will differ for each environment, but at least 128 partitions has been shown to significantly increase performance in some environments.</p></div>
+<div class="section">
+<h3><a name="Physical_Layout"></a>Physical Layout</h3>
+<p>It is important to note the physical layout of the hardware when assigning worker cores to the probe. The worker cores should be on the same NUMA node or socket as the ethernet device itself. Assigning logical cores across NUMA boundaries can significantly impede performance.</p>
+<p>The following commands can help identify logical cores that are located on the same NUMA node or socket as the ethernet device itself. These commands should be run when the device is still managed by the kernel itself; before binding the interface.</p>
+
+<div class="source">
+<div class="source">
+<pre>$ cat /sys/class/net/enp9s0f0/device/local_cpulist
+0-7,16-23
+</pre></div></div>
+<p>The following command can be used to better understand the physical layout of the CPU in relation to NUMA nodes.</p>
+
+<div class="source">
+<div class="source">
+<pre>$ lscpu
+...
+NUMA node0 CPU(s): 0-7,16-23
+NUMA node1 CPU(s): 8-15,24-31
+</pre></div></div>
+<p>In this example <tt>enp9s0f0</tt> is located on NUMA node 0 and is local to the logical cores 0-7 and 16-23. You should choose worker cores from this list.</p></div>
+<div class="section">
+<h3><a name="CPU_Isolation"></a>CPU Isolation</h3>
+<p>Once you have chosen the logical cores to use that are local to the ethernet device, it also beneficial to isolate those cores so that the Linux kernel scheduler does not attempt to run tasks on them. This can be done using the <tt>isolcpus</tt> kernel boot parameter.</p>
+
+<div class="source">
+<div class="source">
+<pre>isolcpus=0,1,2,3,4,5,6,7
+</pre></div></div></div>
+<div class="section">
+<h3><a name="Device_Limitations"></a>Device Limitations</h3>
+<p>Check the output of running the probe to ensure that there are no device limitations that you are not aware of. While you may have specified 16 receive queues on the command line, your device may not support that number. This is especially true for the number of receive queues and descriptors.</p>
+<p>The following example shows the output when the number of receive descriptors requested is greater than what can be supported by the device. In many cases the probe will not terminate, but will choose the maximum allowable value and continue. This behavior is specific to the underlying device driver in use.</p>
+
+<div class="source">
+<div class="source">
+<pre>PMD: rte_enic_pmd: Rq 0 Scatter rx mode enabled
+PMD: rte_enic_pmd: Rq 0 Scatter rx mode not being used
+PMD: rte_enic_pmd: Number of rx_descs too high, adjusting to maximum
+PMD: rte_enic_pmd: Using 512 rx descriptors (sop 512, data 0)
+PMD: rte_enic_pmd: Rq 1 Scatter rx mode enabled
+PMD: rte_enic_pmd: Rq 1 Scatter rx mode not being used
+PMD: rte_enic_pmd: Number of rx_descs too high, adjusting to maximum
+PMD: rte_enic_pmd: Using 512 rx descriptors (sop 512, data 0)
+PMD: rte_enic_pmd: TX Queues - effective number of descs:32
+PMD: rte_enic_pmd: vNIC resources used: wq 1 rq 4 cq 3 intr 0
+</pre></div></div></div>
+<div class="section">
+<h3><a name="More_Information"></a>More Information</h3>
+<p>More information on this topic can be found in <a class="externalLink" href="http://dpdk.org/doc/guides/linux_gsg/nic_perf_intel_platform.html">DPDK’s Getting Started Guide</a>.</p></div></div>
+<div class="section">
+<h2><a name="FAQs"></a>FAQs</h2>
+<div class="section">
+<h3><a name="No_free_hugepages_reported"></a>No free hugepages reported</h3>
+<p>Problem: When executing <tt>fastcapa</tt> it fails with the following error message.</p>
+
+<div class="source">
+<div class="source">
+<pre>EAL: No free hugepages reported in hugepages-1048576kB
+PANIC in rte_eal_init():
+Cannot get hugepage information
+</pre></div></div>
+<p>Solution: This can occur if any process that has been allocated THPs crashes and fails to return the resources.</p>
+
+<ul>
+
+<li>
+<p>Delete the THP files that are not in use.</p>
+
+<div class="source">
+<div class="source">
+<pre>rm -f /mnt/huge_1GB/rtemap_*
+</pre></div></div></li>
+
+<li>
+<p>If the first option does not work, re-mount the <tt>hugetlbfs</tt> file system.</p>
+
+<div class="source">
+<div class="source">
+<pre>umount -a -t hugetlbfs
+mount -a
+</pre></div></div></li>
+</ul></div>
+<div class="section">
+<h3><a name="No_ethernet_ports_detected"></a>No ethernet ports detected</h3>
+<p>Problem: When executing <tt>fastcapa</tt> it fails with the following error message.</p>
+
+<div class="source">
+<div class="source">
+<pre>EAL: Error - exiting with code: 1
+ Cause: No ethernet ports detected.
+</pre></div></div>
+
+<ul>
+
+<li>Solution 1: The <tt>uio_pci_generic</tt> kernel module has not been loaded.</li>
+</ul>
+
+<div class="source">
+<div class="source">
+<pre>modprobe uio_pci_generic
+</pre></div></div>
+
+<ul>
+
+<li>Solution 2: Ensure that the ethernet device is bound. Re-bind if necessary.</li>
+</ul>
+
+<div class="source">
+<div class="source">
+<pre> dpdk-devbind --unbind "09:00.0"
+ dpdk-devbind --bind=uio_pci_generic "09:00.0"
+ dpdk-devbind --status
+</pre></div></div></div></div>
+ </div>
+ </div>
+ </div>
+
+ <hr/>
+
+ <footer>
+ <div class="container-fluid">
+ <div class="row span12">Copyright © 2017
+ <a href="https://www.apache.org">The Apache Software Foundation</a>.
+ All Rights Reserved.
+
+ </div>
+
+
+
+ </div>
+ </footer>
+ </body>
+</html>
Added: dev/metron/0.4.2-RC1/site-book/metron-sensors/index.html
==============================================================================
--- dev/metron/0.4.2-RC1/site-book/metron-sensors/index.html (added)
+++ dev/metron/0.4.2-RC1/site-book/metron-sensors/index.html Fri Dec 8 09:27:19 2017
@@ -0,0 +1,238 @@
+<!DOCTYPE html>
+<!--
+ | Generated by Apache Maven Doxia at 2017-12-08
+ | Rendered using Apache Maven Fluido Skin 1.3.0
+-->
+<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
+ <head>
+ <meta charset="UTF-8" />
+ <meta name="viewport" content="width=device-width, initial-scale=1.0" />
+ <meta name="Date-Revision-yyyymmdd" content="20171208" />
+ <meta http-equiv="Content-Language" content="en" />
+ <title>Metron – Metron Sensors</title>
+ <link rel="stylesheet" href="../css/apache-maven-fluido-1.3.0.min.css" />
+ <link rel="stylesheet" href="../css/site.css" />
+ <link rel="stylesheet" href="../css/print.css" media="print" />
+
+
+ <script type="text/javascript" src="../js/apache-maven-fluido-1.3.0.min.js"></script>
+
+
+
+<script type="text/javascript">$( document ).ready( function() { $( '.carousel' ).carousel( { interval: 3500 } ) } );</script>
+
+ </head>
+ <body class="topBarDisabled">
+
+
+
+
+ <div class="container-fluid">
+ <div id="banner">
+ <div class="pull-left">
+ <a href="http://metron.apache.org/" id="bannerLeft">
+ <img src="../images/metron-logo.png" alt="Apache Metron" width="148px" height="48px"/>
+ </a>
+ </div>
+ <div class="pull-right"> </div>
+ <div class="clear"><hr/></div>
+ </div>
+
+ <div id="breadcrumbs">
+ <ul class="breadcrumb">
+
+
+ <li class="">
+ <a href="http://www.apache.org" class="externalLink" title="Apache">
+ Apache</a>
+ </li>
+ <li class="divider ">/</li>
+ <li class="">
+ <a href="http://metron.apache.org/" class="externalLink" title="Metron">
+ Metron</a>
+ </li>
+ <li class="divider ">/</li>
+ <li class="">
+ <a href="../index.html" title="Documentation">
+ Documentation</a>
+ </li>
+ <li class="divider ">/</li>
+ <li class="">Metron Sensors</li>
+
+
+
+ <li id="publishDate" class="pull-right">Last Published: 2017-12-08</li> <li class="divider pull-right">|</li>
+ <li id="projectVersion" class="pull-right">Version: 0.4.2</li>
+
+ </ul>
+ </div>
+
+
+ <div class="row-fluid">
+ <div id="leftColumn" class="span3">
+ <div class="well sidebar-nav">
+
+
+ <ul class="nav nav-list">
+ <li class="nav-header">User Documentation</li>
+
+ <li>
+
+ <a href="../index.html" title="Metron">
+ <i class="icon-chevron-down"></i>
+ Metron</a>
+ <ul class="nav nav-list">
+
+ <li>
+
+ <a href="../Upgrading.html" title="Upgrading">
+ <i class="none"></i>
+ Upgrading</a>
+ </li>
+
+ <li>
+
+ <a href="../metron-analytics/index.html" title="Analytics">
+ <i class="icon-chevron-right"></i>
+ Analytics</a>
+ </li>
+
+ <li>
+
+ <a href="../metron-contrib/metron-docker/index.html" title="Docker">
+ <i class="none"></i>
+ Docker</a>
+ </li>
+
+ <li>
+
+ <a href="../metron-deployment/index.html" title="Deployment">
+ <i class="icon-chevron-right"></i>
+ Deployment</a>
+ </li>
+
+ <li>
+
+ <a href="../metron-interface/metron-alerts/index.html" title="Alerts">
+ <i class="none"></i>
+ Alerts</a>
+ </li>
+
+ <li>
+
+ <a href="../metron-interface/metron-config/index.html" title="Config">
+ <i class="none"></i>
+ Config</a>
+ </li>
+
+ <li>
+
+ <a href="../metron-interface/metron-rest/index.html" title="Rest">
+ <i class="none"></i>
+ Rest</a>
+ </li>
+
+ <li>
+
+ <a href="../metron-platform/index.html" title="Platform">
+ <i class="icon-chevron-right"></i>
+ Platform</a>
+ </li>
+
+ <li class="active">
+
+ <a href="#"><i class="icon-chevron-down"></i>Sensors</a>
+ <ul class="nav nav-list">
+
+ <li>
+
+ <a href="../metron-sensors/fastcapa/index.html" title="Fastcapa">
+ <i class="none"></i>
+ Fastcapa</a>
+ </li>
+
+ <li>
+
+ <a href="../metron-sensors/pycapa/index.html" title="Pycapa">
+ <i class="none"></i>
+ Pycapa</a>
+ </li>
+ </ul>
+ </li>
+
+ <li>
+
+ <a href="../metron-stellar/stellar-3rd-party-example/index.html" title="Stellar-3rd-party-example">
+ <i class="none"></i>
+ Stellar-3rd-party-example</a>
+ </li>
+
+ <li>
+
+ <a href="../metron-stellar/stellar-common/index.html" title="Stellar-common">
+ <i class="icon-chevron-right"></i>
+ Stellar-common</a>
+ </li>
+
+ <li>
+
+ <a href="../use-cases/index.html" title="Use-cases">
+ <i class="icon-chevron-right"></i>
+ Use-cases</a>
+ </li>
+ </ul>
+ </li>
+ </ul>
+
+
+
+ <hr class="divider" />
+
+ <div id="poweredBy">
+ <div class="clear"></div>
+ <div class="clear"></div>
+ <div class="clear"></div>
+ <a href="http://maven.apache.org/" title="Built by Maven" class="poweredBy">
+ <img class="builtBy" alt="Built by Maven" src="../images/logos/maven-feather.png" />
+ </a>
+ </div>
+ </div>
+ </div>
+
+
+ <div id="bodyColumn" class="span9" >
+
+ <div class="section">
+<h2><a name="Metron_Sensors"></a>Metron Sensors</h2>
+
+<ul>
+
+<li>
+<p><a class="externalLink" href="https://github.com/apache/metron-bro-plugin-kafka"><tt>metron-bro-plugin-kafka</tt></a>: Provides integration between <a class="externalLink" href="https://www.bro.org/">Bro</a> and Kafka. A Bro plugin that sends logging output to Kafka. This provides a convenient means for tools in the Hadoop ecosystem, such as Storm, Spark, and others to process the data generated by Bro.</p></li>
+
+<li>
+<p><a href="fastcapa/index.html"><tt>fastcapa</tt></a>: Performs fast network packet capture by leveraging Linux kernel-bypass and user space networking technology. The probe will bind to a network interface, capture network packets, and send the raw packet data to Kafka. This provides a scalable mechanism for ingesting high-volumes of network packet data.</p></li>
+
+<li>
+<p><a href="pycapa/index.html"><tt>pycapa</tt></a>: Performs lightweight network packet capture, retrieves network packets from Kafka, generates <tt>libpcap</tt>-compliant files, and enables integration with third-party tools like Wireshark.</p></li>
+</ul></div>
+ </div>
+ </div>
+ </div>
+
+ <hr/>
+
+ <footer>
+ <div class="container-fluid">
+ <div class="row span12">Copyright © 2017
+ <a href="https://www.apache.org">The Apache Software Foundation</a>.
+ All Rights Reserved.
+
+ </div>
+
+
+
+ </div>
+ </footer>
+ </body>
+</html>
Added: dev/metron/0.4.2-RC1/site-book/metron-sensors/pycapa/index.html
==============================================================================
--- dev/metron/0.4.2-RC1/site-book/metron-sensors/pycapa/index.html (added)
+++ dev/metron/0.4.2-RC1/site-book/metron-sensors/pycapa/index.html Fri Dec 8 09:27:19 2017
@@ -0,0 +1,574 @@
+<!DOCTYPE html>
+<!--
+ | Generated by Apache Maven Doxia at 2017-12-08
+ | Rendered using Apache Maven Fluido Skin 1.3.0
+-->
+<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
+ <head>
+ <meta charset="UTF-8" />
+ <meta name="viewport" content="width=device-width, initial-scale=1.0" />
+ <meta name="Date-Revision-yyyymmdd" content="20171208" />
+ <meta http-equiv="Content-Language" content="en" />
+ <title>Metron – Pycapa</title>
+ <link rel="stylesheet" href="../../css/apache-maven-fluido-1.3.0.min.css" />
+ <link rel="stylesheet" href="../../css/site.css" />
+ <link rel="stylesheet" href="../../css/print.css" media="print" />
+
+
+ <script type="text/javascript" src="../../js/apache-maven-fluido-1.3.0.min.js"></script>
+
+
+
+<script type="text/javascript">$( document ).ready( function() { $( '.carousel' ).carousel( { interval: 3500 } ) } );</script>
+
+ </head>
+ <body class="topBarDisabled">
+
+
+
+
+ <div class="container-fluid">
+ <div id="banner">
+ <div class="pull-left">
+ <a href="http://metron.apache.org/" id="bannerLeft">
+ <img src="../../images/metron-logo.png" alt="Apache Metron" width="148px" height="48px"/>
+ </a>
+ </div>
+ <div class="pull-right"> </div>
+ <div class="clear"><hr/></div>
+ </div>
+
+ <div id="breadcrumbs">
+ <ul class="breadcrumb">
+
+
+ <li class="">
+ <a href="http://www.apache.org" class="externalLink" title="Apache">
+ Apache</a>
+ </li>
+ <li class="divider ">/</li>
+ <li class="">
+ <a href="http://metron.apache.org/" class="externalLink" title="Metron">
+ Metron</a>
+ </li>
+ <li class="divider ">/</li>
+ <li class="">
+ <a href="../../index.html" title="Documentation">
+ Documentation</a>
+ </li>
+ <li class="divider ">/</li>
+ <li class="">Pycapa</li>
+
+
+
+ <li id="publishDate" class="pull-right">Last Published: 2017-12-08</li> <li class="divider pull-right">|</li>
+ <li id="projectVersion" class="pull-right">Version: 0.4.2</li>
+
+ </ul>
+ </div>
+
+
+ <div class="row-fluid">
+ <div id="leftColumn" class="span3">
+ <div class="well sidebar-nav">
+
+
+ <ul class="nav nav-list">
+ <li class="nav-header">User Documentation</li>
+
+ <li>
+
+ <a href="../../index.html" title="Metron">
+ <i class="icon-chevron-down"></i>
+ Metron</a>
+ <ul class="nav nav-list">
+
+ <li>
+
+ <a href="../../Upgrading.html" title="Upgrading">
+ <i class="none"></i>
+ Upgrading</a>
+ </li>
+
+ <li>
+
+ <a href="../../metron-analytics/index.html" title="Analytics">
+ <i class="icon-chevron-right"></i>
+ Analytics</a>
+ </li>
+
+ <li>
+
+ <a href="../../metron-contrib/metron-docker/index.html" title="Docker">
+ <i class="none"></i>
+ Docker</a>
+ </li>
+
+ <li>
+
+ <a href="../../metron-deployment/index.html" title="Deployment">
+ <i class="icon-chevron-right"></i>
+ Deployment</a>
+ </li>
+
+ <li>
+
+ <a href="../../metron-interface/metron-alerts/index.html" title="Alerts">
+ <i class="none"></i>
+ Alerts</a>
+ </li>
+
+ <li>
+
+ <a href="../../metron-interface/metron-config/index.html" title="Config">
+ <i class="none"></i>
+ Config</a>
+ </li>
+
+ <li>
+
+ <a href="../../metron-interface/metron-rest/index.html" title="Rest">
+ <i class="none"></i>
+ Rest</a>
+ </li>
+
+ <li>
+
+ <a href="../../metron-platform/index.html" title="Platform">
+ <i class="icon-chevron-right"></i>
+ Platform</a>
+ </li>
+
+ <li>
+
+ <a href="../../metron-sensors/index.html" title="Sensors">
+ <i class="icon-chevron-down"></i>
+ Sensors</a>
+ <ul class="nav nav-list">
+
+ <li>
+
+ <a href="../../metron-sensors/fastcapa/index.html" title="Fastcapa">
+ <i class="none"></i>
+ Fastcapa</a>
+ </li>
+
+ <li class="active">
+
+ <a href="#"><i class="none"></i>Pycapa</a>
+ </li>
+ </ul>
+ </li>
+
+ <li>
+
+ <a href="../../metron-stellar/stellar-3rd-party-example/index.html" title="Stellar-3rd-party-example">
+ <i class="none"></i>
+ Stellar-3rd-party-example</a>
+ </li>
+
+ <li>
+
+ <a href="../../metron-stellar/stellar-common/index.html" title="Stellar-common">
+ <i class="icon-chevron-right"></i>
+ Stellar-common</a>
+ </li>
+
+ <li>
+
+ <a href="../../use-cases/index.html" title="Use-cases">
+ <i class="icon-chevron-right"></i>
+ Use-cases</a>
+ </li>
+ </ul>
+ </li>
+ </ul>
+
+
+
+ <hr class="divider" />
+
+ <div id="poweredBy">
+ <div class="clear"></div>
+ <div class="clear"></div>
+ <div class="clear"></div>
+ <a href="http://maven.apache.org/" title="Built by Maven" class="poweredBy">
+ <img class="builtBy" alt="Built by Maven" src="../../images/logos/maven-feather.png" />
+ </a>
+ </div>
+ </div>
+ </div>
+
+
+ <div id="bodyColumn" class="span9" >
+
+ <div class="section">
+<h2><a name="Pycapa"></a>Pycapa</h2>
+
+<ul>
+
+<li><a href="#Overview">Overview</a></li>
+
+<li><a href="#Installation">Installation</a></li>
+
+<li><a href="#Usage">Usage</a>
+
+<ul>
+
+<li><a href="#Parameters">Parameters</a></li>
+
+<li><a href="#Examples">Examples</a></li>
+
+<li><a href="#Kerberos">Kerberos</a></li>
+ </ul></li>
+
+<li><a href="#FAQs">FAQs</a></li>
+</ul>
+<h1>Overview</h1>
+<p>Pycapa performs network packet capture, both off-the-wire and from a Kafka topic, which is useful for the testing and development of <a class="externalLink" href="https://github.com/apache/metron">Apache Metron</a>. It is not intended for production use. The tool will capture packets from a specified interface and push them into a Kafka Topic. The tool can also do the reverse. It can consume packets from Kafka and reconstruct each network packet. This can then be used to create a <a class="externalLink" href="https://wiki.wireshark.org/Development/LibpcapFileFormat">libpcap-compliant file</a> or even to feed directly into a tool like Wireshark to monitor ongoing activity.</p>
+<h1>Installation</h1>
+<p>General notes on the installation of Pycapa.</p>
+
+<ul>
+
+<li>Python 2.7 is required.</li>
+
+<li>The following package dependencies are required and can be installed automatically with <tt>pip</tt>.
+
+<ul>
+
+<li><a class="externalLink" href="https://github.com/confluentinc/confluent-kafka-python">confluent-kafka-python</a></li>
+
+<li><a class="externalLink" href="https://github.com/CoreSecurity/pcapy">pcapy</a></li>
+ </ul></li>
+
+<li>These instructions can be used directly on CentOS 7+.</li>
+
+<li>Other Linux distributions that come with Python 2.7 can use these instructions with some minor modifications.</li>
+
+<li>Older distributions, like CentOS 6, that come with Python 2.6 installed, should install Python 2.7 within a virtual environment and then run Pycapa from within the virtual environment.</li>
+</ul>
+
+<ol style="list-style-type: decimal">
+
+<li>
+<p>Install system dependencies including the core development tools, Python libraries and header files, and Libpcap libraries and header files. On CentOS 7+, you can install these requirements with the following command.</p>
+
+<div class="source">
+<div class="source">
+<pre> yum -y install "@Development tools" python-devel libpcap-devel
+</pre></div></div></li>
+
+<li>
+<p>Install Librdkafka at your chosen $PREFIX.</p>
+
+<div class="source">
+<div class="source">
+<pre> export PREFIX=/usr
+ wget https://github.com/edenhill/librdkafka/archive/v0.9.4.tar.gz -O - | tar -xz
+ cd librdkafka-0.9.4/
+ ./configure --prefix=$PREFIX
+ make
+ make install
+</pre></div></div></li>
+
+<li>
+<p>Add Librdkafka to the dynamic library load path.</p>
+
+<div class="source">
+<div class="source">
+<pre>echo "$PREFIX/lib" >> /etc/ld.so.conf.d/pycapa.conf
+ldconfig -v
+</pre></div></div></li>
+
+<li>
+<p>Install Pycapa. This assumes that you already have the Metron source code on the host.</p>
+
+<div class="source">
+<div class="source">
+<pre>cd metron/metron-sensors/pycapa
+pip install -r requirements.txt
+python setup.py install
+</pre></div></div></li>
+</ol>
+<h1>Usage</h1>
+<p>Pycapa has two primary runtime modes.</p>
+
+<ul>
+
+<li>
+<p><b>Producer Mode</b>: Pycapa can capture packets from a network interface and forward those packets to a Kafka topic. Pycapa embeds the raw network packet data in the Kafka message body. The message key contains the timestamp indicating when the packet was captured in microseconds from the epoch, in network byte order.</p></li>
+
+<li>
+<p><b>Consumer Mode</b>: Pycapa can also perform the reverse operation. It can consume packets from Kafka and reconstruct each network packet. This can then be used to create a <a class="externalLink" href="https://wiki.wireshark.org/Development/LibpcapFileFormat">libpcap-compliant file</a> or even to feed directly into a tool like Wireshark to monitor activity.</p></li>
+</ul>
+<div class="section">
+<h3><a name="Parameters"></a>Parameters</h3>
+
+<div class="source">
+<div class="source">
+<pre>$ pycapa --help
+usage: pycapa [-h] [-p] [-c] [-k KAFKA_BROKERS] [-t KAFKA_TOPIC]
+ [-o {begin,end,stored}] [-i NETWORK_IFACE] [-m MAX_PACKETS]
+ [-pp PRETTY_PRINT] [-ll LOG_LEVEL] [-X KAFKA_CONFIGS]
+ [-s SNAPLEN]
+
+optional arguments:
+ -h, --help show this help message and exit
+ -p, --producer sniff packets and send to kafka
+ -c, --consumer read packets from kafka
+ -k KAFKA_BROKERS, --kafka-broker KAFKA_BROKERS
+ kafka broker(s) as host:port
+ -t KAFKA_TOPIC, --kafka-topic KAFKA_TOPIC
+ kafka topic
+ -o {begin,end,stored}, --kafka-offset {begin,end,stored}
+ kafka offset to consume from; default=end
+ -i NETWORK_IFACE, --interface NETWORK_IFACE
+ network interface to listen on
+ -m MAX_PACKETS, --max-packets MAX_PACKETS
+ stop after this number of packets
+ -pp PRETTY_PRINT, --pretty-print PRETTY_PRINT
+ pretty print every X packets
+ -ll LOG_LEVEL, --log-level LOG_LEVEL
+ set the log level; DEBUG, INFO, WARN
+ -X KAFKA_CONFIGS define a kafka client parameter; key=value
+ -s SNAPLEN, --snaplen SNAPLEN
+ capture only the first X bytes of each packet;
+ default=65535
+</pre></div></div></div>
+<div class="section">
+<h3><a name="Examples"></a>Examples</h3>
+<div class="section">
+<h4><a name="Example_1"></a>Example 1</h4>
+<p>Capture 10 packets from the <tt>eth0</tt> network interface and forward those to a Kafka topic called <tt>pcap</tt> running on <tt>localhost:9092</tt>. The process will not terminate until all messages have been delivered to Kafka.</p>
+
+<div class="source">
+<div class="source">
+<pre>$ pycapa --producer \
+ --interface eth0 \
+ --kafka-broker localhost:9092 \
+ --kafka-topic pcap \
+ --max-packets 10
+INFO:root:Connecting to Kafka; {'bootstrap.servers': 'localhost:9092', 'group.id': 'AWBHMIAESAHJ'}
+INFO:root:Starting packet capture
+INFO:root:Waiting for '6' message(s) to flush
+INFO:root:'10' packet(s) in, '10' packet(s) out
+</pre></div></div></div>
+<div class="section">
+<h4><a name="Example_2"></a>Example 2</h4>
+<p>Capture packets until SIGINT is received (the interrupt signal sent when entering CTRL-C in the console.) In this example, nothing will be reported as packets are captured and delivered to Kafka. Simply wait a few seconds, then type CTRL-C and the number of packets will be reported.</p>
+
+<div class="source">
+<div class="source">
+<pre>$ pycapa --producer \
+ --interface en0 \
+ --kafka-broker localhost:9092 \
+ --kafka-topic pcap
+INFO:root:Connecting to Kafka; {'bootstrap.servers': 'localhost:9092', 'group.id': 'EULLGDOMZDCT'}
+INFO:root:Starting packet capture
+^C
+INFO:root:Clean shutdown process started
+INFO:root:Waiting for '2' message(s) to flush
+INFO:root:'21' packet(s) in, '21' packet(s) out
+</pre></div></div></div>
+<div class="section">
+<h4><a name="Example_3"></a>Example 3</h4>
+<p>While capturing packets, output diagnostic information every 5 packets. Diagnostics will report when packets have been received from the network interface and when they have been successfully delivered to Kafka.</p>
+
+<div class="source">
+<div class="source">
+<pre>$ pycapa --producer \
+ --interface eth0 \
+ --kafka-broker localhost:9092 \
+ --kafka-topic pcap \
+ --pretty-print 5
+ INFO:root:Connecting to Kafka; {'bootstrap.servers': 'localhost:9092', 'group.id': 'UAWINMBDNQEH'}
+ INFO:root:Starting packet capture
+ Packet received[5]
+ Packet delivered[5]: date=2017-05-08 14:48:54.474031 topic=pcap partition=0 offset=29086 len=42
+ Packet received[10]
+ Packet received[15]
+ Packet delivered[10]: date=2017-05-08 14:48:58.879710 topic=pcap partition=0 offset=0 len=187
+ Packet delivered[15]: date=2017-05-08 14:48:59.633127 topic=pcap partition=0 offset=0 len=43
+ Packet received[20]
+ Packet delivered[20]: date=2017-05-08 14:49:01.949628 topic=pcap partition=0 offset=29101 len=134
+ Packet received[25]
+ ^C
+ INFO:root:Clean shutdown process started
+ Packet delivered[25]: date=2017-05-08 14:49:03.589940 topic=pcap partition=0 offset=0 len=142
+ INFO:root:Waiting for '1' message(s) to flush
+ INFO:root:'27' packet(s) in, '27' packet(s) out
+
+</pre></div></div></div>
+<div class="section">
+<h4><a name="Example_4"></a>Example 4</h4>
+<p>Consume 10 packets and create a libpcap-compliant pcap file.</p>
+
+<div class="source">
+<div class="source">
+<pre> $ pycapa --consumer \
+ --kafka-broker localhost:9092 \
+ --kafka-topic pcap \
+ --max-packets 10 \
+ > out.pcap
+ $ tshark -r out.pcap
+ 1 0.000000 199.193.204.147 → 192.168.0.3 TLSv1.2 151 Application Data
+ 2 0.000005 199.193.204.147 → 192.168.0.3 TLSv1.2 1191 Application Data
+ 3 0.000088 192.168.0.3 → 199.193.204.147 TCP 66 54788 → 443 [ACK] Seq=1 Ack=86 Win=4093 Len=0 TSval=961284465 TSecr=943744612
+ 4 0.000089 192.168.0.3 → 199.193.204.147 TCP 66 54788 → 443 [ACK] Seq=1 Ack=1211 Win=4058 Len=0 TSval=961284465 TSecr=943744612
+ 5 0.948788 192.168.0.3 → 192.30.253.125 TCP 54 54671 → 443 [ACK] Seq=1 Ack=1 Win=4096 Len=0
+ 6 1.005175 192.30.253.125 → 192.168.0.3 TCP 66 [TCP ACKed unseen segment] 443 → 54671 [ACK] Seq=1 Ack=2 Win=31 Len=0 TSval=2658544467 TSecr=961240339
+ 7 1.636312 fe80::1286:8cff:fe0e:65df → ff02::1 ICMPv6 134 Router Advertisement from 10:86:8c:0e:65:df
+ 8 2.253052 192.175.27.112 → 192.168.0.3 TLSv1.2 928 Application Data
+ 9 2.253140 192.168.0.3 → 192.175.27.112 TCP 66 55078 → 443 [ACK] Seq=1 Ack=863 Win=4069 Len=0 TSval=961286699 TSecr=967172238
+ 10 2.494769 192.168.0.3 → 224.0.0.251 MDNS 82 Standard query 0x0000 PTR _googlecast._tcp.local, "QM" question
+</pre></div></div></div>
+<div class="section">
+<h4><a name="Example_5"></a>Example 5</h4>
+<p>Consume 10 packets from the Kafka topic <tt>pcap</tt> running on <tt>localhost:9092</tt>, then pipe those into Wireshark for DPI.</p>
+
+<div class="source">
+<div class="source">
+<pre>$ pycapa --consumer \
+ --kafka-broker localhost:9092 \
+ --kafka-topic pcap \
+ --max-packets 10 \
+ | tshark -i -
+Capturing on 'Standard input'
+ 1 0.000000 ArrisGro_0e:65:df → Apple_bf:0d:43 ARP 56 Who has 192.168.0.3? Tell 192.168.0.1
+ 2 0.000044 Apple_bf:0d:43 → ArrisGro_0e:65:df ARP 42 192.168.0.3 is at ac:bc:32:bf:0d:43
+ 3 0.203495 fe80::1286:8cff:fe0e:65df → ff02::1 ICMPv6 134 Router Advertisement from 10:86:8c:0e:65:df
+ 4 2.031988 192.168.0.3 → 96.27.183.249 TCP 54 55110 → 443 [ACK] Seq=1 Ack=1 Win=4108 Len=0
+ 5 2.035816 192.30.253.125 → 192.168.0.3 TLSv1.2 97 Application Data
+ 6 2.035892 192.168.0.3 → 192.30.253.125 TCP 66 54671 → 443 [ACK] Seq=1 Ack=32 Win=4095 Len=0 TSval=961120495 TSecr=2658503052
+ 7 2.035994 192.168.0.3 → 192.30.253.125 TLSv1.2 101 Application Data
+ 8 2.053866 96.27.183.249 → 192.168.0.3 TCP 66 [TCP ACKed unseen segment] 443 → 55110 [ACK] Seq=1 Ack=2 Win=243 Len=0 TSval=728145145 TSecr=961030381
+ 9 2.083872 192.30.253.125 → 192.168.0.3 TCP 66 443 → 54671 [ACK] Seq=32 Ack=36 Win=31 Len=0 TSval=2658503087 TSecr=961120495
+ 10 3.173189 fe80::1286:8cff:fe0e:65df → ff02::1 ICMPv6 134 Router Advertisement from 10:86:8c:0e:65:df
+10 packets captured
+</pre></div></div></div></div>
+<div class="section">
+<h3><a name="Kerberos"></a>Kerberos</h3>
+<p>The probe can be used in a Kerberized environment. Follow these additional steps to use Pycapa with Kerberos. The following assumptions have been made. These may need altered to fit your environment.</p>
+
+<ul>
+
+<li>The Kafka broker is at <tt>kafka1:6667</tt></li>
+
+<li>Zookeeper is at <tt>zookeeper1:2181</tt></li>
+
+<li>The Kafka security protocol is <tt>SASL_PLAINTEXT</tt></li>
+
+<li>The keytab used is located at <tt>/etc/security/keytabs/metron.headless.keytab</tt></li>
+
+<li>The service principal is <tt>metron@EXAMPLE.COM</tt></li>
+</ul>
+
+<ol style="list-style-type: decimal">
+
+<li>
+<p>Build Librdkafka with SASL support (<tt>--enable-sasl</tt>) and install at your chosen $PREFIX.</p>
+
+<div class="source">
+<div class="source">
+<pre>wget https://github.com/edenhill/librdkafka/archive/v0.9.4.tar.gz -O - | tar -xz
+cd librdkafka-0.9.4/
+./configure --prefix=$PREFIX --enable-sasl
+make
+make install
+</pre></div></div></li>
+
+<li>
+<p>Validate Librdkafka does indeed support SASL. Run the following command and ensure that <tt>sasl</tt> is returned as a built-in feature.</p>
+
+<div class="source">
+<div class="source">
+<pre>$ examples/rdkafka_example -X builtin.features
+builtin.features = gzip,snappy,ssl,sasl,regex
+</pre></div></div>
+<p>If it is not, ensure that you have <tt>libsasl</tt> or <tt>libsasl2</tt> installed. On CentOS, this can be installed with the following command.</p>
+
+<div class="source">
+<div class="source">
+<pre>yum install -y cyrus-sasl cyrus-sasl-devel cyrus-sasl-gssapi
+</pre></div></div></li>
+
+<li>
+<p>Grant access to your Kafka topic. In this example the topic is simply named <tt>pcap</tt>.</p>
+
+<div class="source">
+<div class="source">
+<pre>${KAFKA_HOME}/bin/kafka-acls.sh \
+ --authorizer kafka.security.auth.SimpleAclAuthorizer \
+ --authorizer-properties zookeeper.connect=zookeeper1:2181 \
+ --add \
+ --allow-principal User:metron \
+ --topic pcap
+${KAFKA_HOME}/bin/kafka-acls.sh \
+ --authorizer kafka.security.auth.SimpleAclAuthorizer \
+ --authorizer-properties zookeeper.connect=zookeeper1:2181 \
+ --add \
+ --allow-principal User:metron \
+ --group pycapa
+</pre></div></div></li>
+
+<li>
+<p>Use Pycapa as you normally would, but append the following three additional parameters</p>
+
+<ul>
+
+<li><tt>security.protocol</tt></li>
+
+<li><tt>sasl.kerberos.keytab</tt></li>
+
+<li><tt>sasl.kerberos.principal</tt></li>
+ </ul>
+
+<div class="source">
+<div class="source">
+<pre> $ pycapa --producer \
+ --interface eth0 \
+ --kafka-broker kafka1:6667 \
+ --kafka-topic pcap --max-packets 10 \
+ -X security.protocol=SASL_PLAINTEXT \
+ -X sasl.kerberos.keytab=/etc/security/keytabs/metron.headless.keytab \
+ -X sasl.kerberos.principal=metron-metron@METRONEXAMPLE.COM
+ INFO:root:Connecting to Kafka; {'sasl.kerberos.principal': 'metron-metron@METRONEXAMPLE.COM', 'group.id': 'ORNLVWJZZUAA', 'security.protocol': 'SASL_PLAINTEXT', 'sasl.kerberos.keytab': '/etc/security/keytabs/metron.headless.keytab', 'bootstrap.servers': 'kafka1:6667'}
+ INFO:root:Starting packet capture
+ INFO:root:Waiting for '1' message(s) to flush
+ INFO:root:'10' packet(s) in, '10' packet(s) out
+</pre></div></div></li>
+</ol>
+<h1>FAQs</h1></div>
+<div class="section">
+<h3><a name="How_do_I_get_more_logs"></a>How do I get more logs?</h3>
+<p>Use the following two command-line arguments to get detailed logging.</p>
+
+<div class="source">
+<div class="source">
+<pre>-X debug=all --log-level DEBUG
+</pre></div></div></div>
+<div class="section">
+<h3><a name="When_I_run_Pycapa_against_a_Kafka_broker_with_Kerberos_enabled_why_do_I_get_an_error_like_No_such_configuration_property:_sasl.kerberos.principal"></a>When I run Pycapa against a Kafka broker with Kerberos enabled, why do I get an error like “No such configuration property: ‘sasl.kerberos.principal’”?</h3>
+<p>This can be a confusing error message because <tt>sasl.kerberos.principal</tt> is indeed a valid property for librdkafka as defined <a class="externalLink" href="https://github.com/edenhill/librdkafka/blob/master/CONFIGURATION.md">here</a>. This is most likely because Pycapa is running against a version of Librdkafka without SASL support enabled. This might happen if you have accidentally installed multiple versions of Librdkafka and Pycapa is unexpectedly using the version without SASL support enabled.</p>
+<p>Bottom Line: Make sure that Pycapa is running against a version of Librdkafka with SASL support enabled.</p></div></div>
+ </div>
+ </div>
+ </div>
+
+ <hr/>
+
+ <footer>
+ <div class="container-fluid">
+ <div class="row span12">Copyright © 2017
+ <a href="https://www.apache.org">The Apache Software Foundation</a>.
+ All Rights Reserved.
+
+ </div>
+
+
+
+ </div>
+ </footer>
+ </body>
+</html>