You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@flume.apache.org by bu...@apache.org on 2015/06/01 21:50:48 UTC
svn commit: r953472 [1/5] - in /websites/staging/flume/trunk/content: ./
.doctrees/ .doctrees/releases/ _sources/ _sources/releases/ releases/
Author: buildbot
Date: Mon Jun 1 19:50:47 2015
New Revision: 953472
Log:
Staging update by buildbot for flume
Added:
websites/staging/flume/trunk/content/.doctrees/releases/1.6.0.doctree (with props)
websites/staging/flume/trunk/content/_sources/releases/1.6.0.txt
websites/staging/flume/trunk/content/releases/1.6.0.html
Modified:
websites/staging/flume/trunk/content/ (props changed)
websites/staging/flume/trunk/content/.doctrees/FlumeDeveloperGuide.doctree
websites/staging/flume/trunk/content/.doctrees/FlumeUserGuide.doctree
websites/staging/flume/trunk/content/.doctrees/download.doctree
websites/staging/flume/trunk/content/.doctrees/environment.pickle
websites/staging/flume/trunk/content/.doctrees/index.doctree
websites/staging/flume/trunk/content/.doctrees/releases/index.doctree
websites/staging/flume/trunk/content/FlumeDeveloperGuide.html
websites/staging/flume/trunk/content/FlumeUserGuide.html
websites/staging/flume/trunk/content/_sources/FlumeDeveloperGuide.txt
websites/staging/flume/trunk/content/_sources/FlumeUserGuide.txt
websites/staging/flume/trunk/content/_sources/download.txt
websites/staging/flume/trunk/content/_sources/index.txt
websites/staging/flume/trunk/content/_sources/releases/index.txt
websites/staging/flume/trunk/content/documentation.html
websites/staging/flume/trunk/content/download.html
websites/staging/flume/trunk/content/index.html
websites/staging/flume/trunk/content/project-reports.html
websites/staging/flume/trunk/content/releases/1.5.2.html
websites/staging/flume/trunk/content/releases/index.html
websites/staging/flume/trunk/content/searchindex.js
Propchange: websites/staging/flume/trunk/content/
------------------------------------------------------------------------------
--- cms:source-revision (original)
+++ cms:source-revision Mon Jun 1 19:50:47 2015
@@ -1 +1 @@
-1657863
+1682982
Modified: websites/staging/flume/trunk/content/.doctrees/FlumeDeveloperGuide.doctree
==============================================================================
Binary files - no diff available.
Modified: websites/staging/flume/trunk/content/.doctrees/FlumeUserGuide.doctree
==============================================================================
Binary files - no diff available.
Modified: websites/staging/flume/trunk/content/.doctrees/download.doctree
==============================================================================
Binary files - no diff available.
Modified: websites/staging/flume/trunk/content/.doctrees/environment.pickle
==============================================================================
Binary files - no diff available.
Modified: websites/staging/flume/trunk/content/.doctrees/index.doctree
==============================================================================
Binary files - no diff available.
Added: websites/staging/flume/trunk/content/.doctrees/releases/1.6.0.doctree
==============================================================================
Binary file - no diff available.
Propchange: websites/staging/flume/trunk/content/.doctrees/releases/1.6.0.doctree
------------------------------------------------------------------------------
svn:mime-type = application/octet-stream
Modified: websites/staging/flume/trunk/content/.doctrees/releases/index.doctree
==============================================================================
Binary files - no diff available.
Modified: websites/staging/flume/trunk/content/FlumeDeveloperGuide.html
==============================================================================
--- websites/staging/flume/trunk/content/FlumeDeveloperGuide.html (original)
+++ websites/staging/flume/trunk/content/FlumeDeveloperGuide.html Mon Jun 1 19:50:47 2015
@@ -7,7 +7,7 @@
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
- <title>Flume 1.5.2 Developer Guide — Apache Flume</title>
+ <title>Flume 1.6.0 Developer Guide — Apache Flume</title>
<link rel="stylesheet" href="_static/flume.css" type="text/css" />
<link rel="stylesheet" href="_static/pygments.css" type="text/css" />
@@ -27,7 +27,7 @@
<link rel="top" title="Apache Flume" href="index.html" />
<link rel="up" title="Documentation" href="documentation.html" />
<link rel="next" title="Releases" href="releases/index.html" />
- <link rel="prev" title="Flume 1.5.2 User Guide" href="FlumeUserGuide.html" />
+ <link rel="prev" title="Flume 1.6.0 User Guide" href="FlumeUserGuide.html" />
</head>
<body>
<div class="header">
@@ -59,8 +59,8 @@
<div class="bodywrapper">
<div class="body">
- <div class="section" id="flume-1-5-2-developer-guide">
-<h1>Flume 1.5.2 Developer Guide<a class="headerlink" href="#flume-1-5-2-developer-guide" title="Permalink to this headline">¶</a></h1>
+ <div class="section" id="flume-1-6-0-developer-guide">
+<h1>Flume 1.6.0 Developer Guide<a class="headerlink" href="#flume-1-6-0-developer-guide" title="Permalink to this headline">¶</a></h1>
<div class="section" id="introduction">
<h2>Introduction<a class="headerlink" href="#introduction" title="Permalink to this headline">¶</a></h2>
<div class="section" id="overview">
@@ -293,6 +293,112 @@ properties:</p>
</pre></div>
</div>
</div>
+<div class="section" id="secure-rpc-client-thrift">
+<h5>Secure RPC client - Thrift<a class="headerlink" href="#secure-rpc-client-thrift" title="Permalink to this headline">¶</a></h5>
+<p>As of Flume 1.6.0, Thrift source and sink supports kerberos based authentication.
+The client needs to use the getThriftInstance method of <tt class="docutils literal"><span class="pre">SecureRpcClientFactory</span></tt>
+to get hold of a <tt class="docutils literal"><span class="pre">SecureThriftRpcClient</span></tt>. <tt class="docutils literal"><span class="pre">SecureThriftRpcClient</span></tt> extends
+<tt class="docutils literal"><span class="pre">ThriftRpcClient</span></tt> which implements the <tt class="docutils literal"><span class="pre">RpcClient</span></tt> interface. The kerberos
+authentication module resides in flume-ng-auth module which is
+required in classpath, when using the <tt class="docutils literal"><span class="pre">SecureRpcClientFactory</span></tt>. Both the client
+principal and the client keytab should be passed in as parameters through the
+properties and they reflect the credentials of the client to authenticate
+against the kerberos KDC. In addition, the server principal of the destination
+Thrift source to which this client is connecting to, should also be provided.
+The following example shows how to use the <tt class="docutils literal"><span class="pre">SecureRpcClientFactory</span></tt>
+within a user’s data-generating application:</p>
+<div class="highlight-java"><div class="highlight"><pre><span class="kn">import</span> <span class="nn">org.apache.flume.Event</span><span class="o">;</span>
+<span class="kn">import</span> <span class="nn">org.apache.flume.EventDeliveryException</span><span class="o">;</span>
+<span class="kn">import</span> <span class="nn">org.apache.flume.event.EventBuilder</span><span class="o">;</span>
+<span class="kn">import</span> <span class="nn">org.apache.flume.api.SecureRpcClientFactory</span><span class="o">;</span>
+<span class="kn">import</span> <span class="nn">org.apache.flume.api.RpcClientConfigurationConstants</span><span class="o">;</span>
+<span class="kn">import</span> <span class="nn">org.apache.flume.api.RpcClient</span><span class="o">;</span>
+<span class="kn">import</span> <span class="nn">java.nio.charset.Charset</span><span class="o">;</span>
+<span class="kn">import</span> <span class="nn">java.util.Properties</span><span class="o">;</span>
+
+<span class="kd">public</span> <span class="kd">class</span> <span class="nc">MyApp</span> <span class="o">{</span>
+ <span class="kd">public</span> <span class="kd">static</span> <span class="kt">void</span> <span class="nf">main</span><span class="o">(</span><span class="n">String</span><span class="o">[]</span> <span class="n">args</span><span class="o">)</span> <span class="o">{</span>
+ <span class="n">MySecureRpcClientFacade</span> <span class="n">client</span> <span class="o">=</span> <span class="k">new</span> <span class="n">MySecureRpcClientFacade</span><span class="o">();</span>
+ <span class="c1">// Initialize client with the remote Flume agent's host, port</span>
+ <span class="n">Properties</span> <span class="n">props</span> <span class="o">=</span> <span class="k">new</span> <span class="n">Properties</span><span class="o">();</span>
+ <span class="n">props</span><span class="o">.</span><span class="na">setProperty</span><span class="o">(</span><span class="n">RpcClientConfigurationConstants</span><span class="o">.</span><span class="na">CONFIG_CLIENT_TYPE</span><span class="o">,</span> <span class="s">"thrift"</span><span class="o">);</span>
+ <span class="n">props</span><span class="o">.</span><span class="na">setProperty</span><span class="o">(</span><span class="s">"hosts"</span><span class="o">,</span> <span class="s">"h1"</span><span class="o">);</span>
+ <span class="n">props</span><span class="o">.</span><span class="na">setProperty</span><span class="o">(</span><span class="s">"hosts.h1"</span><span class="o">,</span> <span class="s">"client.example.org"</span><span class="o">+</span><span class="s">":"</span><span class="o">+</span> <span class="n">String</span><span class="o">.</span><span class="na">valueOf</span><span class="o">(</span><span class="mi">41414</span><span class="o">));</span>
+
+ <span class="c1">// Initialize client with the kerberos authentication related properties</span>
+ <span class="n">props</span><span class="o">.</span><span class="na">setProperty</span><span class="o">(</span><span class="s">"kerberos"</span><span class="o">,</span> <span class="s">"true"</span><span class="o">);</span>
+ <span class="n">props</span><span class="o">.</span><span class="na">setProperty</span><span class="o">(</span><span class="s">"client-principal"</span><span class="o">,</span> <span class="s">"flumeclient/client.example.org@EXAMPLE.ORG"</span><span class="o">);</span>
+ <span class="n">props</span><span class="o">.</span><span class="na">setProperty</span><span class="o">(</span><span class="s">"client-keytab"</span><span class="o">,</span> <span class="s">"/tmp/flumeclient.keytab"</span><span class="o">);</span>
+ <span class="n">props</span><span class="o">.</span><span class="na">setProperty</span><span class="o">(</span><span class="s">"server-principal"</span><span class="o">,</span> <span class="s">"flume/server.example.org@EXAMPLE.ORG"</span><span class="o">);</span>
+ <span class="n">client</span><span class="o">.</span><span class="na">init</span><span class="o">(</span><span class="n">props</span><span class="o">);</span>
+
+ <span class="c1">// Send 10 events to the remote Flume agent. That agent should be</span>
+ <span class="c1">// configured to listen with an AvroSource.</span>
+ <span class="n">String</span> <span class="n">sampleData</span> <span class="o">=</span> <span class="s">"Hello Flume!"</span><span class="o">;</span>
+ <span class="k">for</span> <span class="o">(</span><span class="kt">int</span> <span class="n">i</span> <span class="o">=</span> <span class="mi">0</span><span class="o">;</span> <span class="n">i</span> <span class="o"><</span> <span class="mi">10</span><span class="o">;</span> <span class="n">i</span><span class="o">++)</span> <span class="o">{</span>
+ <span class="n">client</span><span class="o">.</span><span class="na">sendDataToFlume</span><span class="o">(</span><span class="n">sampleData</span><span class="o">);</span>
+ <span class="o">}</span>
+
+ <span class="n">client</span><span class="o">.</span><span class="na">cleanUp</span><span class="o">();</span>
+ <span class="o">}</span>
+<span class="o">}</span>
+
+<span class="kd">class</span> <span class="nc">MySecureRpcClientFacade</span> <span class="o">{</span>
+ <span class="kd">private</span> <span class="n">RpcClient</span> <span class="n">client</span><span class="o">;</span>
+ <span class="kd">private</span> <span class="n">Properties</span> <span class="n">properties</span><span class="o">;</span>
+
+ <span class="kd">public</span> <span class="kt">void</span> <span class="nf">init</span><span class="o">(</span><span class="n">Properties</span> <span class="n">properties</span><span class="o">)</span> <span class="o">{</span>
+ <span class="c1">// Setup the RPC connection</span>
+ <span class="k">this</span><span class="o">.</span><span class="na">properties</span> <span class="o">=</span> <span class="n">properties</span><span class="o">;</span>
+ <span class="c1">// Create the ThriftSecureRpcClient instance by using SecureRpcClientFactory</span>
+ <span class="k">this</span><span class="o">.</span><span class="na">client</span> <span class="o">=</span> <span class="n">SecureRpcClientFactory</span><span class="o">.</span><span class="na">getThriftInstance</span><span class="o">(</span><span class="n">properties</span><span class="o">);</span>
+ <span class="o">}</span>
+
+ <span class="kd">public</span> <span class="kt">void</span> <span class="nf">sendDataToFlume</span><span class="o">(</span><span class="n">String</span> <span class="n">data</span><span class="o">)</span> <span class="o">{</span>
+ <span class="c1">// Create a Flume Event object that encapsulates the sample data</span>
+ <span class="n">Event</span> <span class="n">event</span> <span class="o">=</span> <span class="n">EventBuilder</span><span class="o">.</span><span class="na">withBody</span><span class="o">(</span><span class="n">data</span><span class="o">,</span> <span class="n">Charset</span><span class="o">.</span><span class="na">forName</span><span class="o">(</span><span class="s">"UTF-8"</span><span class="o">));</span>
+
+ <span class="c1">// Send the event</span>
+ <span class="k">try</span> <span class="o">{</span>
+ <span class="n">client</span><span class="o">.</span><span class="na">append</span><span class="o">(</span><span class="n">event</span><span class="o">);</span>
+ <span class="o">}</span> <span class="k">catch</span> <span class="o">(</span><span class="n">EventDeliveryException</span> <span class="n">e</span><span class="o">)</span> <span class="o">{</span>
+ <span class="c1">// clean up and recreate the client</span>
+ <span class="n">client</span><span class="o">.</span><span class="na">close</span><span class="o">();</span>
+ <span class="n">client</span> <span class="o">=</span> <span class="kc">null</span><span class="o">;</span>
+ <span class="n">client</span> <span class="o">=</span> <span class="n">SecureRpcClientFactory</span><span class="o">.</span><span class="na">getThriftInstance</span><span class="o">(</span><span class="n">properties</span><span class="o">);</span>
+ <span class="o">}</span>
+ <span class="o">}</span>
+
+ <span class="kd">public</span> <span class="kt">void</span> <span class="nf">cleanUp</span><span class="o">()</span> <span class="o">{</span>
+ <span class="c1">// Close the RPC connection</span>
+ <span class="n">client</span><span class="o">.</span><span class="na">close</span><span class="o">();</span>
+ <span class="o">}</span>
+<span class="o">}</span>
+</pre></div>
+</div>
+<p>The remote <tt class="docutils literal"><span class="pre">ThriftSource</span></tt> should be started in kerberos mode.
+Below is an example Flume agent configuration that’s waiting for a connection
+from MyApp:</p>
+<div class="highlight-properties"><div class="highlight"><pre><span class="na">a1.channels</span> <span class="o">=</span> <span class="s">c1</span>
+<span class="na">a1.sources</span> <span class="o">=</span> <span class="s">r1</span>
+<span class="na">a1.sinks</span> <span class="o">=</span> <span class="s">k1</span>
+
+<span class="na">a1.channels.c1.type</span> <span class="o">=</span> <span class="s">memory</span>
+
+<span class="na">a1.sources.r1.channels</span> <span class="o">=</span> <span class="s">c1</span>
+<span class="na">a1.sources.r1.type</span> <span class="o">=</span> <span class="s">thrift</span>
+<span class="na">a1.sources.r1.bind</span> <span class="o">=</span> <span class="s">0.0.0.0</span>
+<span class="na">a1.sources.r1.port</span> <span class="o">=</span> <span class="s">41414</span>
+<span class="na">a1.sources.r1.kerberos</span> <span class="o">=</span> <span class="s">true</span>
+<span class="na">a1.sources.r1.agent-principal</span> <span class="o">=</span> <span class="s">flume/server.example.org@EXAMPLE.ORG</span>
+<span class="na">a1.sources.r1.agent-keytab</span> <span class="o">=</span> <span class="s">/tmp/flume.keytab</span>
+
+
+<span class="na">a1.sinks.k1.channel</span> <span class="o">=</span> <span class="s">c1</span>
+<span class="na">a1.sinks.k1.type</span> <span class="o">=</span> <span class="s">logger</span>
+</pre></div>
+</div>
+</div>
<div class="section" id="failover-client">
<h5>Failover Client<a class="headerlink" href="#failover-client" title="Permalink to this headline">¶</a></h5>
<p>This class wraps the default Avro RPC client to provide failover handling
@@ -456,16 +562,16 @@ sources, sinks, and channels are allowed
is a special embedded source and events should be send to the source
via the put, putAll methods on the EmbeddedAgent object. Only File Channel
and Memory Channel are allowed as channels while Avro Sink is the only
-supported sink.</p>
+supported sink. Interceptors are also supported by the embedded agent.</p>
<p>Note: The embedded agent has a dependency on hadoop-core.jar.</p>
<p>Configuration of an Embedded Agent is similar to configuration of a
full Agent. The following is an exhaustive list of configration options:</p>
<p>Required properties are in <strong>bold</strong>.</p>
<table border="1" class="docutils">
<colgroup>
-<col width="11%" />
-<col width="9%" />
-<col width="80%" />
+<col width="20%" />
+<col width="15%" />
+<col width="65%" />
</colgroup>
<thead valign="bottom">
<tr class="row-odd"><th class="head">Property Name</th>
@@ -480,11 +586,13 @@ full Agent. The following is an exhausti
</tr>
<tr class="row-odd"><td><strong>channel.type</strong></td>
<td>–</td>
-<td>Either <tt class="docutils literal"><span class="pre">memory</span></tt> or <tt class="docutils literal"><span class="pre">file</span></tt> which correspond to MemoryChannel and FileChannel respectively.</td>
+<td>Either <tt class="docutils literal"><span class="pre">memory</span></tt> or <tt class="docutils literal"><span class="pre">file</span></tt> which correspond
+to MemoryChannel and FileChannel respectively.</td>
</tr>
<tr class="row-even"><td>channel.*</td>
<td>–</td>
-<td>Configuration options for the channel type requested, see MemoryChannel or FileChannel user guide for an exhaustive list.</td>
+<td>Configuration options for the channel type requested,
+see MemoryChannel or FileChannel user guide for an exhaustive list.</td>
</tr>
<tr class="row-odd"><td><strong>sinks</strong></td>
<td>–</td>
@@ -492,19 +600,34 @@ full Agent. The following is an exhausti
</tr>
<tr class="row-even"><td><strong>sink.type</strong></td>
<td>–</td>
-<td>Property name must match a name in the list of sinks. Value must be <tt class="docutils literal"><span class="pre">avro</span></tt></td>
+<td>Property name must match a name in the list of sinks.
+Value must be <tt class="docutils literal"><span class="pre">avro</span></tt></td>
</tr>
<tr class="row-odd"><td>sink.*</td>
<td>–</td>
-<td>Configuration options for the sink. See AvroSink user guide for an exhaustive list, however note AvroSink requires at least hostname and port.</td>
+<td>Configuration options for the sink.
+See AvroSink user guide for an exhaustive list,
+however note AvroSink requires at least hostname and port.</td>
</tr>
<tr class="row-even"><td><strong>processor.type</strong></td>
<td>–</td>
-<td>Either <tt class="docutils literal"><span class="pre">failover</span></tt> or <tt class="docutils literal"><span class="pre">load_balance</span></tt> which correspond to FailoverSinksProcessor and LoadBalancingSinkProcessor respectively.</td>
+<td>Either <tt class="docutils literal"><span class="pre">failover</span></tt> or <tt class="docutils literal"><span class="pre">load_balance</span></tt> which correspond
+to FailoverSinksProcessor and LoadBalancingSinkProcessor respectively.</td>
</tr>
<tr class="row-odd"><td>processor.*</td>
<td>–</td>
-<td>Configuration options for the sink processor selected. See FailoverSinksProcessor and LoadBalancingSinkProcessor user guide for an exhaustive list.</td>
+<td>Configuration options for the sink processor selected.
+See FailoverSinksProcessor and LoadBalancingSinkProcessor
+user guide for an exhaustive list.</td>
+</tr>
+<tr class="row-even"><td>source.interceptors</td>
+<td>–</td>
+<td>Space-separated list of interceptors</td>
+</tr>
+<tr class="row-odd"><td>source.interceptors.*</td>
+<td>–</td>
+<td>Configuration options for individual interceptors
+specified in the source.interceptors property</td>
</tr>
</tbody>
</table>
@@ -520,6 +643,10 @@ full Agent. The following is an exhausti
<span class="n">properties</span><span class="o">.</span><span class="na">put</span><span class="o">(</span><span class="s">"sink2.hostname"</span><span class="o">,</span> <span class="s">"collector2.apache.org"</span><span class="o">);</span>
<span class="n">properties</span><span class="o">.</span><span class="na">put</span><span class="o">(</span><span class="s">"sink2.port"</span><span class="o">,</span> <span class="s">"5565"</span><span class="o">);</span>
<span class="n">properties</span><span class="o">.</span><span class="na">put</span><span class="o">(</span><span class="s">"processor.type"</span><span class="o">,</span> <span class="s">"load_balance"</span><span class="o">);</span>
+<span class="n">properties</span><span class="o">.</span><span class="na">put</span><span class="o">(</span><span class="s">"source.interceptors"</span><span class="o">,</span> <span class="s">"i1"</span><span class="o">);</span>
+<span class="n">properties</span><span class="o">.</span><span class="na">put</span><span class="o">(</span><span class="s">"source.interceptors.i1.type"</span><span class="o">,</span> <span class="s">"static"</span><span class="o">);</span>
+<span class="n">properties</span><span class="o">.</span><span class="na">put</span><span class="o">(</span><span class="s">"source.interceptors.i1.key"</span><span class="o">,</span> <span class="s">"key1"</span><span class="o">);</span>
+<span class="n">properties</span><span class="o">.</span><span class="na">put</span><span class="o">(</span><span class="s">"source.interceptors.i1.value"</span><span class="o">,</span> <span class="s">"value1"</span><span class="o">);</span>
<span class="n">EmbeddedAgent</span> <span class="n">agent</span> <span class="o">=</span> <span class="k">new</span> <span class="n">EmbeddedAgent</span><span class="o">(</span><span class="s">"myagent"</span><span class="o">);</span>
@@ -806,7 +933,7 @@ mechanism that captures the new data and
<h3><a href="index.html">This Page</a></h3>
<ul>
-<li><a class="reference internal" href="#">Flume 1.5.2 Developer Guide</a><ul>
+<li><a class="reference internal" href="#">Flume 1.6.0 Developer Guide</a><ul>
<li><a class="reference internal" href="#introduction">Introduction</a><ul>
<li><a class="reference internal" href="#overview">Overview</a></li>
<li><a class="reference internal" href="#architecture">Architecture</a><ul>
@@ -824,6 +951,7 @@ mechanism that captures the new data and
<li><a class="reference internal" href="#client-sdk">Client SDK</a></li>
<li><a class="reference internal" href="#rpc-client-interface">RPC client interface</a></li>
<li><a class="reference internal" href="#rpc-clients-avro-and-thrift">RPC clients - Avro and Thrift</a></li>
+<li><a class="reference internal" href="#secure-rpc-client-thrift">Secure RPC client - Thrift</a></li>
<li><a class="reference internal" href="#failover-client">Failover Client</a></li>
<li><a class="reference internal" href="#loadbalancing-rpc-client">LoadBalancing RPC client</a></li>
</ul>