(logging-flume) branch trunk-site-stg-out updated: Add website content generated from `7597b4f15e54da7d9b562b7cd3a00fd5636d100a`

[pk...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

ppkarwasz pushed a commit to branch trunk-site-stg-out
in repository https://gitbox.apache.org/repos/asf/logging-flume.git


The following commit(s) were added to refs/heads/trunk-site-stg-out by this push:
     new 11372ea4 Add website content generated from `7597b4f15e54da7d9b562b7cd3a00fd5636d100a`
11372ea4 is described below

commit 11372ea496d336c2fbe5f49ccaa2e01278c3362d
Author: Piotr P. Karwasz <pk...@apache.org>
AuthorDate: Wed Jul 22 14:04:14 2026 +0200

    Add website content generated from `7597b4f15e54da7d9b562b7cd3a00fd5636d100a`
---
 404.html                                    |    91 +
 README.txt                                  |     1 -
 _/css/site.css                              |     3 +
 _/font/roboto-cyrillic-400-italic.woff2     |   Bin 0 -> 10292 bytes
 _/font/roboto-cyrillic-400-normal.woff2     |   Bin 0 -> 9628 bytes
 _/font/roboto-cyrillic-500-italic.woff2     |   Bin 0 -> 10640 bytes
 _/font/roboto-cyrillic-500-normal.woff2     |   Bin 0 -> 9840 bytes
 _/font/roboto-latin-400-italic.woff         |   Bin 0 -> 22048 bytes
 _/font/roboto-latin-400-italic.woff2        |   Bin 0 -> 17368 bytes
 _/font/roboto-latin-400-normal.woff         |   Bin 0 -> 20344 bytes
 _/font/roboto-latin-400-normal.woff2        |   Bin 0 -> 15744 bytes
 _/font/roboto-latin-500-italic.woff         |   Bin 0 -> 22092 bytes
 _/font/roboto-latin-500-italic.woff2        |   Bin 0 -> 17336 bytes
 _/font/roboto-latin-500-normal.woff         |   Bin 0 -> 20544 bytes
 _/font/roboto-latin-500-normal.woff2        |   Bin 0 -> 15920 bytes
 _/font/roboto-mono-latin-400-normal.woff    |   Bin 0 -> 15160 bytes
 _/font/roboto-mono-latin-400-normal.woff2   |   Bin 0 -> 12312 bytes
 _/font/roboto-mono-latin-500-normal.woff    |   Bin 0 -> 15364 bytes
 _/font/roboto-mono-latin-500-normal.woff2   |   Bin 0 -> 12532 bytes
 _/img/back.svg                              |     1 +
 _/img/caret.svg                             |     1 +
 _/img/chevron.svg                           |     1 +
 _/img/home-o.svg                            |     1 +
 _/img/home.svg                              |     1 +
 _/img/menu.svg                              |     1 +
 _/img/octicons-16.svg                       |     1 +
 _/js/site.js                                |     7 +
 _/js/vendor/highlight.js                    |     1 +
 flume/FlumeDeveloperGuide.html              |  1353 +++
 flume/FlumeUserGuide.html                   | 11727 ++++++++++++++++++++++++++
 flume/_images/DevGuide_image00.png          |   Bin 0 -> 11849 bytes
 flume/_images/DevGuide_image01.png          |   Bin 0 -> 17014 bytes
 flume/_images/UserGuide_image00.png         |   Bin 0 -> 14069 bytes
 flume/_images/UserGuide_image01.png         |   Bin 0 -> 23462 bytes
 flume/_images/UserGuide_image02.png         |   Bin 0 -> 37785 bytes
 flume/_images/UserGuide_image03.png         |   Bin 0 -> 13666 bytes
 flume/_images/feather-small.png             |   Bin 0 -> 31327 bytes
 flume/_images/flume-logo.png                |   Bin 0 -> 33341 bytes
 flume/documentation.html                    |   260 +
 flume/download.html                         |   286 +
 flume/flume-spring-boot/download.html       |   287 +
 flume/flume-spring-boot/index.html          |   235 +
 flume/flume-spring-boot/releases/2.0.0.html |   258 +
 flume/flume-spring-boot/releases/index.html |   239 +
 flume/flume-spring-boot/source.html         |   319 +
 flume/getinvolved.html                      |   243 +
 flume/index.html                            |  1356 +++
 flume/license.html                          |   375 +
 flume/mailinglists.html                     |   271 +
 flume/releases/1.0.0.html                   |   737 ++
 flume/releases/1.1.0.html                   |   456 +
 flume/releases/1.10.0.html                  |   419 +
 flume/releases/1.10.1.html                  |   286 +
 flume/releases/1.11.0.html                  |   324 +
 flume/releases/1.2.0.html                   |   831 ++
 flume/releases/1.3.0.html                   |   781 ++
 flume/releases/1.3.1.html                   |   312 +
 flume/releases/1.4.0.html                   |   869 ++
 flume/releases/1.5.0.1.html                 |   276 +
 flume/releases/1.5.0.html                   |   657 ++
 flume/releases/1.5.2.html                   |   292 +
 flume/releases/1.6.0.html                   |   631 ++
 flume/releases/1.7.0.html                   |   605 ++
 flume/releases/1.8.0.html                   |   454 +
 flume/releases/1.9.0.html                   |   503 ++
 flume/releases/index.html                   |   255 +
 flume/security.html                         |   412 +
 flume/source.html                           |   357 +
 flume/subprojects.html                      |   233 +
 flume/team.html                             |   503 ++
 flume/testing.html                          |   233 +
 index.html                                  |     9 +
 sitemap.xml                                 |   143 +
 73 files changed, 27896 insertions(+), 1 deletion(-)

diff --git a/404.html b/404.html
new file mode 100644
index 00000000..e6e6dc31
--- /dev/null
+++ b/404.html
@@ -0,0 +1,91 @@
+<!DOCTYPE html>
+<html lang="en">
+  <head>
+    <meta charset="utf-8">
+    <meta name="viewport" content="width=device-width,initial-scale=1">
+    <title>Page Not Found :: Apache Flume</title>
+    <meta name="generator" content="Antora 3.1.15">
+    <link rel="stylesheet" href="/flume/2.x/_/css/site.css">
+  </head>
+  <body class="status-404">
+<header class="header">
+  <nav class="navbar">
+    <div class="navbar-brand">
+      <a class="navbar-item" href="https://logging.apache.org/flume/2.x">Apache Flume</a>
+      <button class="navbar-burger" aria-controls="topbar-nav" aria-expanded="false" aria-label="Toggle main menu">
+        <span></span>
+        <span></span>
+        <span></span>
+      </button>
+    </div>
+    <div id="topbar-nav" class="navbar-menu">
+      <div class="navbar-end">
+        <a class="navbar-item" href="#">Home</a>
+        <div class="navbar-item has-dropdown is-hoverable">
+          <a class="navbar-link" href="#">Products</a>
+          <div class="navbar-dropdown">
+            <a class="navbar-item" href="#">Product A</a>
+            <a class="navbar-item" href="#">Product B</a>
+            <a class="navbar-item" href="#">Product C</a>
+          </div>
+        </div>
+        <div class="navbar-item has-dropdown is-hoverable">
+          <a class="navbar-link" href="#">Services</a>
+          <div class="navbar-dropdown">
+            <a class="navbar-item" href="#">Service A</a>
+            <a class="navbar-item" href="#">Service B</a>
+            <a class="navbar-item" href="#">Service C</a>
+          </div>
+        </div>
+        <div class="navbar-item">
+          <span class="control">
+            <a class="button is-primary" href="#">Download</a>
+          </span>
+        </div>
+      </div>
+    </div>
+  </nav>
+</header>
+<div class="body">
+<div class="nav-container">
+  <aside class="nav">
+    <div class="panels">
+<div class="nav-panel-explore is-active" data-panel="explore">
+  <ul class="components">
+    <li class="component">
+      <div class="title"><a href="/flume/2.x/flume/index.html">Apache Flume</a></div>
+    </li>
+  </ul>
+</div>
+    </div>
+  </aside>
+</div>
+<main class="article">
+<div class="toolbar" role="navigation">
+<button class="nav-toggle"></button>
+  <a href="/flume/2.x/flume/index.html" class="home-link"></a>
+<nav class="breadcrumbs" aria-label="breadcrumbs">
+</nav>
+</div>
+  <div class="content">
+<article class="doc">
+<h1 class="page">Page Not Found</h1>
+<div class="paragraph">
+<p>The page you&#8217;re looking for does not exist. It may have been moved. You can return to the <a href="/flume/2.x/flume/index.html">start page</a>, or follow one of the links in the navigation to the left.</p>
+</div>
+<div class="paragraph">
+<p>If you arrived on this page by clicking on a link, please notify the owner of the site that the link is broken.
+If you typed the URL of this page manually, please double check that you entered the address correctly.</p>
+</div>
+</article>
+  </div>
+</main>
+</div>
+<footer class="footer">
+  <p>This page was built using the Antora default UI.</p>
+  <p>The source code for this UI is licensed under the terms of the MPL-2.0 license.</p>
+</footer>
+<script id="site-script" src="/flume/2.x/_/js/site.js" data-ui-root-path="/flume/2.x/_"></script>
+<script async src="/flume/2.x/_/js/vendor/highlight.js"></script>
+  </body>
+</html>
diff --git a/README.txt b/README.txt
deleted file mode 100644
index d2a663e3..00000000
--- a/README.txt
+++ /dev/null
@@ -1 +0,0 @@
-Content for initializing an orphan branch for the website to be generated from `7597b4f15e54da7d9b562b7cd3a00fd5636d100a`
diff --git a/_/css/site.css b/_/css/site.css
new file mode 100644
index 00000000..05f73fff
--- /dev/null
+++ b/_/css/site.css
@@ -0,0 +1,3 @@
+@font-face{font-family:Roboto;font-style:normal;font-weight:400;src:url(../font/roboto-latin-400-normal.woff2) format("woff2"),url(../font/roboto-latin-400-normal.woff) format("woff");unicode-range:U+00??,U+0131,U+0152-0153,U+02bb-02bc,U+02c6,U+02da,U+02dc,U+2000-206f,U+2074,U+20ac,U+2122,U+2191,U+2193,U+2212,U+2215,U+feff,U+fffd}@font-face{font-family:Roboto;font-style:normal;font-weight:400;src:url(../font/roboto-cyrillic-400-normal.woff2) format("woff2");unicode-range:U+0301,U+0400-04 [...]
+
+/*! Adapted from the GitHub style by Vasily Polovnyov <va...@whiteants.net> */.hljs-comment,.hljs-quote{color:#998;font-style:italic}.hljs-keyword,.hljs-selector-tag,.hljs-subst{color:#333;font-weight:600}.hljs-literal,.hljs-number,.hljs-tag .hljs-attr,.hljs-template-variable,.hljs-variable{color:teal}.hljs-doctag,.hljs-string{color:#d14}.hljs-section,.hljs-selector-id,.hljs-title{color:#900;font-weight:600}.hljs-subst{font-weight:400}.hljs-class .hljs-title,.hljs-type{color:#458;font-wei [...]
\ No newline at end of file
diff --git a/_/font/roboto-cyrillic-400-italic.woff2 b/_/font/roboto-cyrillic-400-italic.woff2
new file mode 100644
index 00000000..dd587a2b
Binary files /dev/null and b/_/font/roboto-cyrillic-400-italic.woff2 differ
diff --git a/_/font/roboto-cyrillic-400-normal.woff2 b/_/font/roboto-cyrillic-400-normal.woff2
new file mode 100644
index 00000000..47da3629
Binary files /dev/null and b/_/font/roboto-cyrillic-400-normal.woff2 differ
diff --git a/_/font/roboto-cyrillic-500-italic.woff2 b/_/font/roboto-cyrillic-500-italic.woff2
new file mode 100644
index 00000000..cbe564b0
Binary files /dev/null and b/_/font/roboto-cyrillic-500-italic.woff2 differ
diff --git a/_/font/roboto-cyrillic-500-normal.woff2 b/_/font/roboto-cyrillic-500-normal.woff2
new file mode 100644
index 00000000..cb5834ff
Binary files /dev/null and b/_/font/roboto-cyrillic-500-normal.woff2 differ
diff --git a/_/font/roboto-latin-400-italic.woff b/_/font/roboto-latin-400-italic.woff
new file mode 100644
index 00000000..ebee16b9
Binary files /dev/null and b/_/font/roboto-latin-400-italic.woff differ
diff --git a/_/font/roboto-latin-400-italic.woff2 b/_/font/roboto-latin-400-italic.woff2
new file mode 100644
index 00000000..e1b7a79f
Binary files /dev/null and b/_/font/roboto-latin-400-italic.woff2 differ
diff --git a/_/font/roboto-latin-400-normal.woff b/_/font/roboto-latin-400-normal.woff
new file mode 100644
index 00000000..9eaa94f9
Binary files /dev/null and b/_/font/roboto-latin-400-normal.woff differ
diff --git a/_/font/roboto-latin-400-normal.woff2 b/_/font/roboto-latin-400-normal.woff2
new file mode 100644
index 00000000..020729ef
Binary files /dev/null and b/_/font/roboto-latin-400-normal.woff2 differ
diff --git a/_/font/roboto-latin-500-italic.woff b/_/font/roboto-latin-500-italic.woff
new file mode 100644
index 00000000..b6ad1c5b
Binary files /dev/null and b/_/font/roboto-latin-500-italic.woff differ
diff --git a/_/font/roboto-latin-500-italic.woff2 b/_/font/roboto-latin-500-italic.woff2
new file mode 100644
index 00000000..ae1933f3
Binary files /dev/null and b/_/font/roboto-latin-500-italic.woff2 differ
diff --git a/_/font/roboto-latin-500-normal.woff b/_/font/roboto-latin-500-normal.woff
new file mode 100644
index 00000000..d39bb52a
Binary files /dev/null and b/_/font/roboto-latin-500-normal.woff differ
diff --git a/_/font/roboto-latin-500-normal.woff2 b/_/font/roboto-latin-500-normal.woff2
new file mode 100644
index 00000000..29342a8d
Binary files /dev/null and b/_/font/roboto-latin-500-normal.woff2 differ
diff --git a/_/font/roboto-mono-latin-400-normal.woff b/_/font/roboto-mono-latin-400-normal.woff
new file mode 100644
index 00000000..be3eb4c4
Binary files /dev/null and b/_/font/roboto-mono-latin-400-normal.woff differ
diff --git a/_/font/roboto-mono-latin-400-normal.woff2 b/_/font/roboto-mono-latin-400-normal.woff2
new file mode 100644
index 00000000..f8894bab
Binary files /dev/null and b/_/font/roboto-mono-latin-400-normal.woff2 differ
diff --git a/_/font/roboto-mono-latin-500-normal.woff b/_/font/roboto-mono-latin-500-normal.woff
new file mode 100644
index 00000000..43ca6a1b
Binary files /dev/null and b/_/font/roboto-mono-latin-500-normal.woff differ
diff --git a/_/font/roboto-mono-latin-500-normal.woff2 b/_/font/roboto-mono-latin-500-normal.woff2
new file mode 100644
index 00000000..b4f2bf8c
Binary files /dev/null and b/_/font/roboto-mono-latin-500-normal.woff2 differ
diff --git a/_/img/back.svg b/_/img/back.svg
new file mode 100644
index 00000000..bf7d30e9
--- /dev/null
+++ b/_/img/back.svg
@@ -0,0 +1 @@
+<svg xmlns="http://www.w3.org/2000/svg" width="100" height="100" viewBox="0 0 100 100"><path d="M50.001 9.9L9.9 50 50 90.1l5.656-5.655-30.435-30.442H90.1v-8.006H25.222l30.435-30.44z"/></svg>
\ No newline at end of file
diff --git a/_/img/caret.svg b/_/img/caret.svg
new file mode 100644
index 00000000..1af41bc6
--- /dev/null
+++ b/_/img/caret.svg
@@ -0,0 +1 @@
+<svg xmlns="http://www.w3.org/2000/svg" width="30" height="30" viewBox="0 0 30 30"><path d="M10.187 3l14 12-14 12z" fill="#c1c1c1" stroke="#c1c1c1" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"/></svg>
\ No newline at end of file
diff --git a/_/img/chevron.svg b/_/img/chevron.svg
new file mode 100644
index 00000000..40e962af
--- /dev/null
+++ b/_/img/chevron.svg
@@ -0,0 +1 @@
+<svg xmlns="http://www.w3.org/2000/svg" width="30" height="30" viewBox="0 0 30 30"><path d="M3.67 6.59L1.455 8.615 15 23.375l13.545-14.76L26.33 6.59 15 18.76z" fill="#5d5d5d"/></svg>
\ No newline at end of file
diff --git a/_/img/home-o.svg b/_/img/home-o.svg
new file mode 100644
index 00000000..95d193b7
--- /dev/null
+++ b/_/img/home-o.svg
@@ -0,0 +1 @@
+<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 100 100"><g fill="none" stroke="#222" stroke-width="4"><path d="M50.02 13.576l-28.58 25.54-.02 47.345h21.506l.025-25.166H57.05l.025 25.166H78.62l.02-47.31-28.62-25.575z"/><path d="M63.815 25.904v-9.217h8.657V33.64zM21.439 39.116l-9.982 8.92m77.125 0l-9.943-8.885"/></g></svg>
\ No newline at end of file
diff --git a/_/img/home.svg b/_/img/home.svg
new file mode 100644
index 00000000..4e96b354
--- /dev/null
+++ b/_/img/home.svg
@@ -0,0 +1 @@
+<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 100 100"><g stroke="#222" stroke-width="4"><path d="M50.02 13.576l-28.58 25.54-.02 47.345h21.506l.025-25.166H57.05l.025 25.166H78.62l.02-47.31-28.62-25.575z" fill="#222" fill-rule="evenodd"/><path d="M63.815 25.904v-9.217h8.657V33.64z" fill="#222" fill-rule="evenodd"/><path d="M21.439 39.116l-9.982 8.92m77.125 0l-9.943-8.885" fill="none"/></g></svg>
\ No newline at end of file
diff --git a/_/img/menu.svg b/_/img/menu.svg
new file mode 100644
index 00000000..8b43b2e0
--- /dev/null
+++ b/_/img/menu.svg
@@ -0,0 +1 @@
+<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 100 100"><path d="M35 19.772h55" fill="none" stroke="#222" stroke-width="8" stroke-opacity=".941"/><path fill="#222" d="M10 12.272h15v15H10z"/><path d="M43 63.678h45" fill="none" stroke="#222" stroke-width="5" stroke-opacity=".941"/><path fill="#222" d="M23 58.677h10v10H23z"/><path d="M43 82.727h45" fill="none" stroke="#222" stroke-width="5" stroke-opacity=".941"/><path fill="#222" d="M23 77.727h10v10H23z"/><path d="M43 44.628h45" fill [...]
\ No newline at end of file
diff --git a/_/img/octicons-16.svg b/_/img/octicons-16.svg
new file mode 100644
index 00000000..c2215106
--- /dev/null
+++ b/_/img/octicons-16.svg
@@ -0,0 +1 @@
+<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 16 16"><desc>Octicons v11.2.0 by GitHub - https://primer.style/octicons/ - License: MIT</desc><symbol id="icon-clippy" viewBox="0 0 16 16"><path fill-rule="evenodd" d="M5.75 1a.75.75 0 00-.75.75v3c0 .414.336.75.75.75h4.5a.75.75 0 00.75-.75v-3a.75.75 0 00-.75-.75h-4.5zm.75 3V2.5h3V4h-3zm-2.874-.467a.75.75 0 00-.752-1.298A1.75 1.75 0 002 3.75v9.5c0 .966.784 1.75 1.75 1.75h8.5A1.75 1.75 0 0014 13.25v-9.5a1.75 1.75 0 00-.874-1.515.75.75 0  [...]
\ No newline at end of file
diff --git a/_/js/site.js b/_/js/site.js
new file mode 100644
index 00000000..916a120d
--- /dev/null
+++ b/_/js/site.js
@@ -0,0 +1,7 @@
+!function(){"use strict";var c,e,o,t,s,r,l=/^sect(\d)$/,i=document.querySelector(".nav-container"),a=document.querySelector(".nav-toggle");function n(){var e,t,n=window.location.hash;if(n&&(n.indexOf("%")&&(n=decodeURIComponent(n)),!(e=o.querySelector('.nav-link[href="'+n+'"]')))){n=document.getElementById(n.slice(1));if(n)for(var i=n,a=document.querySelector("article.doc");(i=i.parentNode)&&i!==a;){var c=i.id;if((c=c||(c=l.test(i.className))&&(i.firstElementChild||{}).id)&&(e=o.querySel [...]
+!function(){"use strict";var e=document.querySelector("aside.toc.sidebar");if(e){if(document.querySelector("body.-toc"))return e.parentNode.removeChild(e);var t=parseInt(e.dataset.levels||2,10);if(!(t<0)){var n="article.doc",d=document.querySelector(n);if(d){for(var o=[],i=0;i<=t;i++){var r=[n];if(i){for(var c=1;c<=i;c++)r.push((2===c?".sectionbody>":"")+".sect"+c);r.push("h"+(i+1)+"[id]"+(1<i?":not(.discrete)":""))}else r.push("h1[id].sect0");o.push(r.join(">"))}m=o.join(","),f=d.parent [...]
+!function(){"use strict";var o,i,c=document.querySelector("article.doc");function n(e){return e&&(~e.indexOf("%")?decodeURIComponent(e):e).slice(1)}function r(e){if(e){if(e.altKey||e.ctrlKey)return;window.location.hash="#"+this.id,e.preventDefault()}var t=function n(e,t){return c.contains(e)?n(e.offsetParent,e.offsetTop+t):t}(this,0)-o.getBoundingClientRect().bottom;!1===e&&i?window.scrollTo({left:0,top:t,behavior:"instant"}):window.scrollTo(0,t)}c&&(o=document.querySelector(".toolbar"), [...]
+!function(){"use strict";var t,e=document.querySelector(".page-versions .version-menu-toggle");e&&(t=document.querySelector(".page-versions"),e.addEventListener("click",function(e){t.classList.toggle("is-active"),e.stopPropagation()}),document.documentElement.addEventListener("click",function(){t.classList.remove("is-active")}))}();
+!function(){"use strict";var i=document.querySelector(".navbar-burger");i&&i.addEventListener("click",function(t){t.stopPropagation(),document.documentElement.classList.toggle("is-clipped--navbar"),i.setAttribute("aria-expanded",this.classList.toggle("is-active"));t=document.getElementById(this.getAttribute("aria-controls")||this.dataset.target);{var e;t.classList.toggle("is-active")&&(t.style.maxHeight="",e=window.innerHeight-Math.round(t.getBoundingClientRect().top),parseInt(window.get [...]
+!function(){"use strict";var o=/^\$ (\S[^\\\n]*(\\\n(?!\$ )[^\\\n]*)*)(?=\n|$)/gm,s=/( ) *\\\n *|\\\n( ?) */g,l=/ +$/gm,e=(document.getElementById("site-script")||{dataset:{}}).dataset,d=window.navigator.clipboard,r=e.svgAs,p=(null==e.uiRootPath?window:e).uiRootPath||".";[].slice.call(document.querySelectorAll(".doc pre.highlight, .doc .literalblock pre")).forEach(function(e){var t,n,a,c;if(e.classList.contains("highlight"))(i=(t=e.querySelector("code")).dataset.lang)&&"console"!==i&&((a [...]
+!function(){"use strict";function i(e,t,r){if(e.scrollHeight!==e.getBoundingClientRect().height)for(t=t.slice(1,-1).reverse();t.length&&e.scrollHeight>e.getBoundingClientRect().height;){var i,n=t.pop(),l=(n=n.querySelector("a")||n).innerText.trim();l.length>r&&(n.setAttribute("title",l),i=l.slice(0,i=r-3)+(" "===l.charAt(i)?" ":"")+"...",n.innerText=i)}}var e,t,r;!(r=document.querySelector(".toolbar"))||(e=Array.prototype.slice.call(r.querySelectorAll(".breadcrumbs li")||[])).length<3||( [...]
\ No newline at end of file
diff --git a/_/js/vendor/highlight.js b/_/js/vendor/highlight.js
new file mode 100644
index 00000000..45478769
--- /dev/null
+++ b/_/js/vendor/highlight.js
@@ -0,0 +1 @@
+!function(){var e,n,a={},t=(e=function(i){var a,g=[],s=Object.keys,M=Object.create(null),r=Object.create(null),O=!0,n=/^(no-?highlight|plain|text)$/i,l=/\blang(?:uage)?-([\w-]+)\b/i,t=/((^(<[^>]+>|\t|)+|(?:\n)))/gm,C="</span>",x="Could not find the language '{}', did you forget to load/include a language module?",S={classPrefix:"hljs-",tabReplace:null,useBR:!1,languages:undefined},o="of and for in not or if then".split(" ");function T(e){return e.replace(/&/g,"&amp;").replace(/</g,"&lt;" [...]
\ No newline at end of file
diff --git a/flume/FlumeDeveloperGuide.html b/flume/FlumeDeveloperGuide.html
new file mode 100644
index 00000000..a0f9a923
--- /dev/null
+++ b/flume/FlumeDeveloperGuide.html
@@ -0,0 +1,1353 @@
+<!DOCTYPE html>
+<html lang="en">
+  <head>
+    <meta charset="utf-8">
+    <meta name="viewport" content="width=device-width,initial-scale=1">
+    <title>Flume 1.11.0 Developer Guide :: Apache Flume</title>
+    <link rel="canonical" href="https://logging.apache.org/flume/2.x/flume/FlumeDeveloperGuide.html">
+    <meta name="generator" content="Antora 3.1.15">
+    <link rel="stylesheet" href="../_/css/site.css">
+  </head>
+  <body class="article">
+<header class="header">
+  <nav class="navbar">
+    <div class="navbar-brand">
+      <a class="navbar-item" href="https://logging.apache.org/flume/2.x">Apache Flume</a>
+      <button class="navbar-burger" aria-controls="topbar-nav" aria-expanded="false" aria-label="Toggle main menu">
+        <span></span>
+        <span></span>
+        <span></span>
+      </button>
+    </div>
+    <div id="topbar-nav" class="navbar-menu">
+      <div class="navbar-end">
+        <a class="navbar-item" href="#">Home</a>
+        <div class="navbar-item has-dropdown is-hoverable">
+          <a class="navbar-link" href="#">Products</a>
+          <div class="navbar-dropdown">
+            <a class="navbar-item" href="#">Product A</a>
+            <a class="navbar-item" href="#">Product B</a>
+            <a class="navbar-item" href="#">Product C</a>
+          </div>
+        </div>
+        <div class="navbar-item has-dropdown is-hoverable">
+          <a class="navbar-link" href="#">Services</a>
+          <div class="navbar-dropdown">
+            <a class="navbar-item" href="#">Service A</a>
+            <a class="navbar-item" href="#">Service B</a>
+            <a class="navbar-item" href="#">Service C</a>
+          </div>
+        </div>
+        <div class="navbar-item">
+          <span class="control">
+            <a class="button is-primary" href="#">Download</a>
+          </span>
+        </div>
+      </div>
+    </div>
+  </nav>
+</header>
+<div class="body">
+<div class="nav-container" data-component="flume" data-version="">
+  <aside class="nav">
+    <div class="panels">
+<div class="nav-panel-menu is-active" data-panel="menu">
+  <nav class="nav-menu">
+    <button class="nav-menu-toggle" aria-label="Toggle expand/collapse all" style="display: none"></button>
+    <h3 class="title"><a href="index.html">Apache Flume</a></h3>
+<ul class="nav-list">
+  <li class="nav-item" data-depth="0">
+<ul class="nav-list">
+  <li class="nav-item" data-depth="1">
+    <a class="nav-link" href="index.html">Home</a>
+  </li>
+  <li class="nav-item" data-depth="1">
+    <a class="nav-link" href="getinvolved.html">Get Involved</a>
+  </li>
+  <li class="nav-item" data-depth="1">
+    <a class="nav-link" href="download.html">Download</a>
+  </li>
+  <li class="nav-item" data-depth="1">
+    <a class="nav-link" href="security.html">Security</a>
+  </li>
+  <li class="nav-item" data-depth="1">
+    <button class="nav-item-toggle"></button>
+    <a class="nav-link" href="documentation.html">Documentation</a>
+<ul class="nav-list">
+  <li class="nav-item" data-depth="2">
+    <a class="nav-link" href="FlumeUserGuide.html">Flume User Guide</a>
+  </li>
+  <li class="nav-item is-current-page" data-depth="2">
+    <a class="nav-link" href="FlumeDeveloperGuide.html">Flume Developer Guide</a>
+  </li>
+</ul>
+  </li>
+  <li class="nav-item" data-depth="1">
+    <button class="nav-item-toggle"></button>
+    <a class="nav-link" href="releases/index.html">Releases</a>
+<ul class="nav-list">
+  <li class="nav-item" data-depth="2">
+    <a class="nav-link" href="releases/1.11.0.html">1.11.0</a>
+  </li>
+  <li class="nav-item" data-depth="2">
+    <a class="nav-link" href="releases/1.10.1.html">1.10.1</a>
+  </li>
+  <li class="nav-item" data-depth="2">
+    <a class="nav-link" href="releases/1.10.0.html">1.10.0</a>
+  </li>
+  <li class="nav-item" data-depth="2">
+    <a class="nav-link" href="releases/1.9.0.html">1.9.0</a>
+  </li>
+  <li class="nav-item" data-depth="2">
+    <a class="nav-link" href="releases/1.8.0.html">1.8.0</a>
+  </li>
+  <li class="nav-item" data-depth="2">
+    <a class="nav-link" href="releases/1.7.0.html">1.7.0</a>
+  </li>
+  <li class="nav-item" data-depth="2">
+    <a class="nav-link" href="releases/1.6.0.html">1.6.0</a>
+  </li>
+  <li class="nav-item" data-depth="2">
+    <a class="nav-link" href="releases/1.5.2.html">1.5.2</a>
+  </li>
+  <li class="nav-item" data-depth="2">
+    <a class="nav-link" href="releases/1.5.0.1.html">1.5.0.1</a>
+  </li>
+  <li class="nav-item" data-depth="2">
+    <a class="nav-link" href="releases/1.5.0.html">1.5.0</a>
+  </li>
+  <li class="nav-item" data-depth="2">
+    <a class="nav-link" href="releases/1.4.0.html">1.4.0</a>
+  </li>
+  <li class="nav-item" data-depth="2">
+    <a class="nav-link" href="releases/1.3.1.html">1.3.1</a>
+  </li>
+  <li class="nav-item" data-depth="2">
+    <a class="nav-link" href="releases/1.3.0.html">1.3.0</a>
+  </li>
+  <li class="nav-item" data-depth="2">
+    <a class="nav-link" href="releases/1.2.0.html">1.2.0</a>
+  </li>
+  <li class="nav-item" data-depth="2">
+    <a class="nav-link" href="releases/1.1.0.html">1.1.0</a>
+  </li>
+  <li class="nav-item" data-depth="2">
+    <a class="nav-link" href="releases/1.0.0.html">1.0.0</a>
+  </li>
+</ul>
+  </li>
+  <li class="nav-item" data-depth="1">
+    <button class="nav-item-toggle"></button>
+    <a class="nav-link" href="flume-spring-boot/index.html">Flume Spring Boot</a>
+<ul class="nav-list">
+  <li class="nav-item" data-depth="2">
+    <a class="nav-link" href="flume-spring-boot/download.html">Download</a>
+  </li>
+  <li class="nav-item" data-depth="2">
+    <button class="nav-item-toggle"></button>
+    <a class="nav-link" href="flume-spring-boot/releases/index.html">Releases</a>
+<ul class="nav-list">
+  <li class="nav-item" data-depth="3">
+    <a class="nav-link" href="flume-spring-boot/releases/2.0.0.html">2.0.0</a>
+  </li>
+</ul>
+  </li>
+  <li class="nav-item" data-depth="2">
+    <a class="nav-link" href="flume-spring-boot/source.html">Source</a>
+  </li>
+</ul>
+  </li>
+  <li class="nav-item" data-depth="1">
+    <a class="nav-link" href="team.html">Team</a>
+  </li>
+  <li class="nav-item" data-depth="1">
+    <a class="nav-link" href="mailinglists.html">Mailing Lists</a>
+  </li>
+  <li class="nav-item" data-depth="1">
+    <a class="nav-link" href="source.html">Source Repository</a>
+  </li>
+  <li class="nav-item" data-depth="1">
+    <a class="nav-link" href="subprojects.html">Subprojects</a>
+  </li>
+  <li class="nav-item" data-depth="1">
+    <a class="nav-link" href="testing.html">Testing</a>
+  </li>
+  <li class="nav-item" data-depth="1">
+    <a class="nav-link" href="license.html">License</a>
+  </li>
+</ul>
+  </li>
+</ul>
+  </nav>
+</div>
+<div class="nav-panel-explore" data-panel="explore">
+  <div class="context">
+    <span class="title">Apache Flume</span>
+    <span class="version"></span>
+  </div>
+  <ul class="components">
+    <li class="component is-current">
+      <div class="title"><a href="index.html">Apache Flume</a></div>
+    </li>
+  </ul>
+</div>
+    </div>
+  </aside>
+</div>
+<main class="article">
+<div class="toolbar" role="navigation">
+<button class="nav-toggle"></button>
+  <a href="index.html" class="home-link"></a>
+<nav class="breadcrumbs" aria-label="breadcrumbs">
+  <ul>
+    <li><a href="index.html">Apache Flume</a></li>
+    <li><a href="documentation.html">Documentation</a></li>
+    <li><a href="FlumeDeveloperGuide.html">Flume Developer Guide</a></li>
+  </ul>
+</nav>
+<div class="edit-this-page"><a href="file:///home/piotr/workspace/logging/flume/flume-1.x/target/generated-site/antora/modules/ROOT/pages/FlumeDeveloperGuide.adoc">Edit this Page</a></div>
+</div>
+  <div class="content">
+<aside class="toc sidebar" data-title="Contents" data-levels="2">
+  <div class="toc-menu"></div>
+</aside>
+<article class="doc">
+<h1 class="page">Flume 1.11.0 Developer Guide</h1>
+<div class="sect1">
+<h2 id="_overview"><a class="anchor" href="#_overview"></a>Overview</h2>
+<div class="sectionbody">
+<div class="paragraph">
+<p>Apache Flume is a distributed, reliable, and available system for
+efficiently collecting, aggregating and moving large amounts of log
+data from many different sources to a centralized data store.</p>
+</div>
+<div class="paragraph">
+<p>Apache Flume is a top-level project at the Apache Software Foundation.
+There are currently two release code lines available, versions 0.9.x and 1.x.
+This documentation applies to the 1.x codeline.
+For the 0.9.x codeline, please see the <a href="https://archive.cloudera.com/cdh/3/flume/DeveloperGuide/">Flume 0.9.x Developer Guide</a>.</p>
+</div>
+</div>
+</div>
+<div class="sect1">
+<h2 id="_architecture"><a class="anchor" href="#_architecture"></a>Architecture</h2>
+<div class="sectionbody">
+<div class="sect2">
+<h3 id="_data_flow_model"><a class="anchor" href="#_data_flow_model"></a>Data flow model</h3>
+<div class="paragraph">
+<p>An <code>Event</code> is a unit of data that flows through a Flume agent. The <code>Event</code>
+flows from <code>Source</code> to <code>Channel</code> to <code>Sink</code>, and is represented by an
+implementation of the <code>Event</code> interface. An <code>Event</code> carries a payload (byte
+array) that is accompanied by an optional set of headers (string attributes).
+A Flume agent is a process (JVM) that hosts the components that allow
+`Event`s to flow from an external source to a external destination.</p>
+</div>
+<div class="imageblock">
+<div class="content">
+<img src="_images/DevGuide_image00.png" alt="Agent component diagram">
+</div>
+</div>
+<div class="paragraph">
+<p>A <code>Source</code> consumes <code>Event`s having a specific format, and those
+`Event`s are delivered to the `Source</code> by an external source like a web
+server. For example, an <code>AvroSource</code> can be used to receive Avro <code>Event`s
+from clients or from other Flume agents in the flow. When a `Source</code> receives
+an <code>Event</code>, it stores it into one or more <code>Channel`s. The `Channel</code> is
+a passive store that holds the <code>Event</code> until that <code>Event</code> is consumed by a
+<code>Sink</code>. One type of <code>Channel</code> available in Flume is the <code>FileChannel</code>
+which uses the local filesystem as its backing store. A <code>Sink</code> is responsible
+for removing an <code>Event</code> from the <code>Channel</code> and putting it into an external
+repository like HDFS (in the case of an <code>HDFSEventSink</code>) or forwarding it to
+the <code>Source</code> at the next hop of the flow. The <code>Source</code> and <code>Sink</code> within
+the given agent run asynchronously with the <code>Event`s staged in the
+`Channel</code>.</p>
+</div>
+</div>
+<div class="sect2">
+<h3 id="_reliability"><a class="anchor" href="#_reliability"></a>Reliability</h3>
+<div class="paragraph">
+<p>An <code>Event</code> is staged in a Flume agent&#8217;s <code>Channel</code>. Then it&#8217;s the
+<code>Sink&#8217;s responsibility to deliver the `Event</code> to the next agent or
+terminal repository (like HDFS) in the flow. The <code>Sink</code> removes an <code>Event</code>
+from the <code>Channel</code> only after the <code>Event</code> is stored into the <code>Channel</code> of
+the next agent or stored in the terminal repository. This is how the single-hop
+message delivery semantics in Flume provide end-to-end reliability of the flow.
+Flume uses a transactional approach to guarantee the reliable delivery of the
+<code>Event`s. The `Source`s and `Sink`s encapsulate the
+storage/retrieval of the `Event`s in a `Transaction</code> provided by the
+<code>Channel</code>. This ensures that the set of <code>Event`s are reliably passed from
+point to point in the flow. In the case of a multi-hop flow, the `Sink</code> from
+the previous hop and the <code>Source</code> of the next hop both have their
+<code>Transaction`s open to ensure that the `Event</code> data is safely stored in
+the <code>Channel</code> of the next hop.</p>
+</div>
+</div>
+</div>
+</div>
+<div class="sect1">
+<h2 id="_building_flume"><a class="anchor" href="#_building_flume"></a>Building Flume</h2>
+<div class="sectionbody">
+<div class="sect2">
+<h3 id="_getting_the_source"><a class="anchor" href="#_getting_the_source"></a>Getting the source</h3>
+<div class="paragraph">
+<p>Check-out the code using Git. Click here for
+<a href="https://git-wip-us.apache.org/repos/asf/flume.git">the git repository root</a>
+or at <a href="https://github.com/apache/flume.git">GitHub</a>.</p>
+</div>
+<div class="paragraph">
+<p>The Flume 1.x development happens under the branch "trunk" so this command line
+can be used:</p>
+</div>
+<div class="quoteblock">
+<blockquote>
+<div class="paragraph">
+<p>git clone <a href="https://git-wip-us.apache.org/repos/asf/flume.git" class="bare">https://git-wip-us.apache.org/repos/asf/flume.git</a></p>
+</div>
+</blockquote>
+</div>
+<div class="paragraph">
+<p>or</p>
+</div>
+<div class="quoteblock">
+<blockquote>
+<div class="paragraph">
+<p>git clone <a href="https://github.com/apache/flume.git" class="bare">https://github.com/apache/flume.git</a></p>
+</div>
+</blockquote>
+</div>
+</div>
+<div class="sect2">
+<h3 id="_compiletest_flume"><a class="anchor" href="#_compiletest_flume"></a>Compile/test Flume</h3>
+<div class="paragraph">
+<p>The Flume build is mavenized. You can compile Flume using the standard Maven
+commands:</p>
+</div>
+<div class="olist arabic">
+<ol class="arabic">
+<li>
+<p>Compile only: <code>mvn clean compile</code></p>
+</li>
+<li>
+<p>Compile and run unit tests: <code>mvn clean test</code></p>
+</li>
+<li>
+<p>Run individual test(s): <code>mvn clean test -Dtest=&lt;Test1&gt;,&lt;Test2&gt;,&#8230;&#8203; -DfailIfNoTests=false</code></p>
+</li>
+<li>
+<p>Create tarball package: <code>mvn clean install</code></p>
+</li>
+<li>
+<p>Create tarball package (skip unit tests): <code>mvn clean install -DskipTests</code></p>
+</li>
+</ol>
+</div>
+</div>
+<div class="sect2">
+<h3 id="_updating_protocol_buffer_version"><a class="anchor" href="#_updating_protocol_buffer_version"></a>Updating Protocol Buffer Version</h3>
+<div class="paragraph">
+<p>File channel has a dependency on Protocol Buffer. When updating the version of Protocol Buffer
+used by Flume, it is necessary to regenerate the data access classes using the protoc compiler
+that is part of Protocol Buffer as follows.</p>
+</div>
+<div class="olist arabic">
+<ol class="arabic">
+<li>
+<p>Update version of Protocol Buffer in pom.xml</p>
+</li>
+<li>
+<p>Add Apache license header to any of the generated files that are missing it</p>
+</li>
+<li>
+<p>Rebuild and test Flume: <code>cd ../..; mvn clean install</code></p>
+</li>
+</ol>
+</div>
+</div>
+</div>
+</div>
+<div class="sect1">
+<h2 id="_developing_custom_components"><a class="anchor" href="#_developing_custom_components"></a>Developing custom components</h2>
+<div class="sectionbody">
+<div class="sect2">
+<h3 id="_client"><a class="anchor" href="#_client"></a>Client</h3>
+<div class="paragraph">
+<p>The client operates at the point of origin of events and delivers them to a
+Flume agent. Clients typically operate in the process space of the application
+they are consuming data from. Flume currently supports Avro, log4j, syslog,
+and Http POST (with a JSON body) as ways to transfer data from a external
+source. Additionally, there’s an <code>ExecSource</code> that can consume the output of a
+local process as input to Flume.</p>
+</div>
+<div class="paragraph">
+<p>It’s quite possible to have a use case where these existing options are not
+sufficient. In this case you can build a custom mechanism to send data to
+Flume. There are two ways of achieving this. The first option is to create a
+custom client that communicates with one of Flume’s existing <code>Source`s like
+`AvroSource</code> or <code>SyslogTcpSource</code>. Here the client should convert its data
+into messages understood by these Flume <code>Source`s. The other option is to
+write a custom Flume `Source</code> that directly talks with your existing client
+application using some IPC or RPC protocol, and then converts the client data
+into Flume <code>Event`s to be sent downstream. Note that all events stored
+within the `Channel</code> of a Flume agent must exist as Flume `Event`s.</p>
+</div>
+<div class="sect3">
+<h4 id="_client_sdk"><a class="anchor" href="#_client_sdk"></a>Client SDK</h4>
+<div class="paragraph">
+<p>Though Flume contains a number of built-in mechanisms (i.e. `Source`s) to
+ingest data, often one wants the ability to communicate with Flume directly from
+a custom application. The Flume Client SDK is a library that enables
+applications to connect to Flume and send data into Flume’s data flow over RPC.</p>
+</div>
+</div>
+<div class="sect3">
+<h4 id="_rpc_client_interface"><a class="anchor" href="#_rpc_client_interface"></a>RPC client interface</h4>
+<div class="paragraph">
+<p>An implementation of Flume&#8217;s RpcClient interface encapsulates the RPC mechanism
+supported by Flume. The user&#8217;s application can simply call the Flume Client
+SDK&#8217;s <code>append(Event)</code> or <code>appendBatch(List&lt;Event&gt;)</code> to send data and not
+worry about the underlying message exchange details. The user can provide the
+required <code>Event</code> arg by either directly implementing the <code>Event</code> interface,
+by using a convenience implementation such as the SimpleEvent class, or by using
+<code>EventBuilder&#8217;s overloaded `withBody()</code> static helper methods.</p>
+</div>
+</div>
+<div class="sect3">
+<h4 id="_rpc_clients_avro_and_thrift"><a class="anchor" href="#_rpc_clients_avro_and_thrift"></a>RPC clients - Avro and Thrift</h4>
+<div class="paragraph">
+<p>As of Flume 1.4.0, Avro is the default RPC protocol. The
+<code>NettyAvroRpcClient</code> and <code>ThriftRpcClient</code> implement the <code>RpcClient</code>
+interface. The client needs to create this object with the host and port of
+the target Flume agent, and can then use the <code>RpcClient</code> to send data into
+the agent. The following example shows how to use the Flume Client SDK API
+within a user&#8217;s data-generating application:</p>
+</div>
+<div class="listingblock">
+<div class="content">
+<pre class="highlightjs highlight"><code class="language-java hljs" data-lang="java">import org.apache.flume.Event;
+import org.apache.flume.EventDeliveryException;
+import org.apache.flume.api.RpcClient;
+import org.apache.flume.api.RpcClientFactory;
+import org.apache.flume.event.EventBuilder;
+import java.nio.charset.Charset;
+
+public class MyApp {
+  public static void main(String[] args) {
+    MyRpcClientFacade client = new MyRpcClientFacade();
+    // Initialize client with the remote Flume agent's host and port
+    client.init("host.example.org", 41414);
+
+    // Send 10 events to the remote Flume agent. That agent should be
+    // configured to listen with an AvroSource.
+    String sampleData = "Hello Flume!";
+    for (int i = 0; i &lt; 10; i++) {
+      client.sendDataToFlume(sampleData);
+    }
+
+    client.cleanUp();
+  }
+}
+
+class MyRpcClientFacade {
+  private RpcClient client;
+  private String hostname;
+  private int port;
+
+  public void init(String hostname, int port) {
+    // Setup the RPC connection
+    this.hostname = hostname;
+    this.port = port;
+    this.client = RpcClientFactory.getDefaultInstance(hostname, port);
+    // Use the following method to create a thrift client (instead of the above line):
+    // this.client = RpcClientFactory.getThriftInstance(hostname, port);
+  }
+
+  public void sendDataToFlume(String data) {
+    // Create a Flume Event object that encapsulates the sample data
+    Event event = EventBuilder.withBody(data, Charset.forName("UTF-8"));
+
+    // Send the event
+    try {
+      client.append(event);
+    } catch (EventDeliveryException e) {
+      // clean up and recreate the client
+      client.close();
+      client = null;
+      client = RpcClientFactory.getDefaultInstance(hostname, port);
+      // Use the following method to create a thrift client (instead of the above line):
+      // this.client = RpcClientFactory.getThriftInstance(hostname, port);
+    }
+  }
+
+  public void cleanUp() {
+    // Close the RPC connection
+    client.close();
+  }
+
+}</code></pre>
+</div>
+</div>
+<div class="paragraph">
+<p>The remote Flume agent needs to have an <code>AvroSource</code> (or a
+<code>ThriftSource</code> if you are using a Thrift client) listening on some port.
+Below is an example Flume agent configuration that&#8217;s waiting for a connection
+from MyApp:</p>
+</div>
+<div class="listingblock">
+<div class="content">
+<pre class="highlightjs highlight"><code class="language-properties hljs" data-lang="properties">a1.channels = c1
+a1.sources = r1
+a1.sinks = k1
+
+a1.channels.c1.type = memory
+
+a1.sources.r1.channels = c1
+a1.sources.r1.type = avro
+# For using a thrift source set the following instead of the above line.
+# a1.source.r1.type = thrift
+a1.sources.r1.bind = 0.0.0.0
+a1.sources.r1.port = 41414
+
+a1.sinks.k1.channel = c1
+a1.sinks.k1.type = logger</code></pre>
+</div>
+</div>
+<div class="paragraph">
+<p>For more flexibility, the default Flume client implementations
+(<code>NettyAvroRpcClient</code> and <code>ThriftRpcClient</code>) can be configured with these
+properties:</p>
+</div>
+<div class="listingblock">
+<div class="content">
+<pre class="highlightjs highlight"><code class="language-properties hljs" data-lang="properties">client.type = default (for avro) or thrift (for thrift)
+
+hosts = h1                           # default client accepts only 1 host
+                                     # (additional hosts will be ignored)
+
+hosts.h1 = host1.example.org:41414   # host and port must both be specified
+                                     # (neither has a default)
+
+batch-size = 100                     # Must be &gt;=1 (default: 100)
+
+connect-timeout = 20000              # Must be &gt;=1000 (default: 20000)
+
+request-timeout = 20000              # Must be &gt;=1000 (default: 20000)</code></pre>
+</div>
+</div>
+</div>
+<div class="sect3">
+<h4 id="_secure_rpc_client_thrift"><a class="anchor" href="#_secure_rpc_client_thrift"></a>Secure RPC client - Thrift</h4>
+<div class="paragraph">
+<p>As of Flume 1.6.0, Thrift source and sink supports kerberos based authentication.
+The client needs to use the getThriftInstance method of <code>SecureRpcClientFactory</code>
+to get hold of a <code>SecureThriftRpcClient</code>. <code>SecureThriftRpcClient</code> extends
+<code>ThriftRpcClient</code> which implements the <code>RpcClient</code> interface. The kerberos
+authentication module resides in flume-ng-auth module which is
+required in classpath, when using the <code>SecureRpcClientFactory</code>. Both the client
+principal and the client keytab should be passed in as parameters through the
+properties and they reflect the credentials of the client to authenticate
+against the kerberos KDC. In addition, the server principal of the destination
+Thrift source to which this client is connecting to, should also be provided.
+The following example shows how to use the <code>SecureRpcClientFactory</code>
+within a user&#8217;s data-generating application:</p>
+</div>
+<div class="listingblock">
+<div class="content">
+<pre class="highlightjs highlight"><code class="language-java hljs" data-lang="java">import org.apache.flume.Event;
+import org.apache.flume.EventDeliveryException;
+import org.apache.flume.event.EventBuilder;
+import org.apache.flume.api.SecureRpcClientFactory;
+import org.apache.flume.api.RpcClientConfigurationConstants;
+import org.apache.flume.api.RpcClient;
+import java.nio.charset.Charset;
+import java.util.Properties;
+
+public class MyApp {
+  public static void main(String[] args) {
+    MySecureRpcClientFacade client = new MySecureRpcClientFacade();
+    // Initialize client with the remote Flume agent's host, port
+    Properties props = new Properties();
+    props.setProperty(RpcClientConfigurationConstants.CONFIG_CLIENT_TYPE, "thrift");
+    props.setProperty("hosts", "h1");
+    props.setProperty("hosts.h1", "client.example.org"+":"+ String.valueOf(41414));
+
+    // Initialize client with the kerberos authentication related properties
+    props.setProperty("kerberos", "true");
+    props.setProperty("client-principal", "flumeclient/client.example.org@EXAMPLE.ORG");
+    props.setProperty("client-keytab", "/tmp/flumeclient.keytab");
+    props.setProperty("server-principal", "flume/server.example.org@EXAMPLE.ORG");
+    client.init(props);
+
+    // Send 10 events to the remote Flume agent. That agent should be
+    // configured to listen with an AvroSource.
+    String sampleData = "Hello Flume!";
+    for (int i = 0; i &lt; 10; i++) {
+      client.sendDataToFlume(sampleData);
+    }
+
+    client.cleanUp();
+  }
+}
+
+class MySecureRpcClientFacade {
+  private RpcClient client;
+  private Properties properties;
+
+  public void init(Properties properties) {
+    // Setup the RPC connection
+    this.properties = properties;
+    // Create the ThriftSecureRpcClient instance by using SecureRpcClientFactory
+    this.client = SecureRpcClientFactory.getThriftInstance(properties);
+  }
+
+  public void sendDataToFlume(String data) {
+    // Create a Flume Event object that encapsulates the sample data
+    Event event = EventBuilder.withBody(data, Charset.forName("UTF-8"));
+
+    // Send the event
+    try {
+      client.append(event);
+    } catch (EventDeliveryException e) {
+      // clean up and recreate the client
+      client.close();
+      client = null;
+      client = SecureRpcClientFactory.getThriftInstance(properties);
+    }
+  }
+
+  public void cleanUp() {
+    // Close the RPC connection
+    client.close();
+  }
+}</code></pre>
+</div>
+</div>
+<div class="paragraph">
+<p>The remote <code>ThriftSource</code> should be started in kerberos mode.
+Below is an example Flume agent configuration that&#8217;s waiting for a connection
+from MyApp:</p>
+</div>
+<div class="listingblock">
+<div class="content">
+<pre class="highlightjs highlight"><code class="language-properties hljs" data-lang="properties">a1.channels = c1
+a1.sources = r1
+a1.sinks = k1
+
+a1.channels.c1.type = memory
+
+a1.sources.r1.channels = c1
+a1.sources.r1.type = thrift
+a1.sources.r1.bind = 0.0.0.0
+a1.sources.r1.port = 41414
+a1.sources.r1.kerberos = true
+a1.sources.r1.agent-principal = flume/server.example.org@EXAMPLE.ORG
+a1.sources.r1.agent-keytab = /tmp/flume.keytab
+
+
+a1.sinks.k1.channel = c1
+a1.sinks.k1.type = logger</code></pre>
+</div>
+</div>
+</div>
+<div class="sect3">
+<h4 id="_failover_client"><a class="anchor" href="#_failover_client"></a>Failover Client</h4>
+<div class="paragraph">
+<p>This class wraps the default Avro RPC client to provide failover handling
+capability to clients. This takes a whitespace-separated list of &lt;host&gt;:&lt;port&gt;
+representing the Flume agents that make-up a failover group. The Failover RPC
+Client currently does not support thrift. If there’s a
+communication error with the currently selected host (i.e. agent) agent,
+then the failover client automatically fails-over to the next host in the list.
+For example:</p>
+</div>
+<div class="listingblock">
+<div class="content">
+<pre class="highlightjs highlight"><code class="language-java hljs" data-lang="java">// Setup properties for the failover
+Properties props = new Properties();
+props.put("client.type", "default_failover");
+
+// List of hosts (space-separated list of user-chosen host aliases)
+props.put("hosts", "h1 h2 h3");
+
+// host/port pair for each host alias
+String host1 = "host1.example.org:41414";
+String host2 = "host2.example.org:41414";
+String host3 = "host3.example.org:41414";
+props.put("hosts.h1", host1);
+props.put("hosts.h2", host2);
+props.put("hosts.h3", host3);
+
+// create the client with failover properties
+RpcClient client = RpcClientFactory.getInstance(props);</code></pre>
+</div>
+</div>
+<div class="paragraph">
+<p>For more flexibility, the failover Flume client implementation
+(<code>FailoverRpcClient</code>) can be configured with these properties:</p>
+</div>
+<div class="listingblock">
+<div class="content">
+<pre class="highlightjs highlight"><code class="language-properties hljs" data-lang="properties">client.type = default_failover
+
+hosts = h1 h2 h3                     # at least one is required, but 2 or
+                                     # more makes better sense
+
+hosts.h1 = host1.example.org:41414
+
+hosts.h2 = host2.example.org:41414
+
+hosts.h3 = host3.example.org:41414
+
+max-attempts = 3                     # Must be &gt;=0 (default: number of hosts
+                                     # specified, 3 in this case). A '0'
+                                     # value doesn't make much sense because
+                                     # it will just cause an append call to
+                                     # immmediately fail. A '1' value means
+                                     # that the failover client will try only
+                                     # once to send the Event, and if it
+                                     # fails then there will be no failover
+                                     # to a second client, so this value
+                                     # causes the failover client to
+                                     # degenerate into just a default client.
+                                     # It makes sense to set this value to at
+                                     # least the number of hosts that you
+                                     # specified.
+
+batch-size = 100                     # Must be &gt;=1 (default: 100)
+
+connect-timeout = 20000              # Must be &gt;=1000 (default: 20000)
+
+request-timeout = 20000              # Must be &gt;=1000 (default: 20000)</code></pre>
+</div>
+</div>
+</div>
+<div class="sect3">
+<h4 id="_loadbalancing_rpc_client"><a class="anchor" href="#_loadbalancing_rpc_client"></a>LoadBalancing RPC client</h4>
+<div class="paragraph">
+<p>The Flume Client SDK also supports an RpcClient which load-balances among
+multiple hosts. This type of client takes a whitespace-separated list of
+&lt;host&gt;:&lt;port&gt; representing the Flume agents that make-up a load-balancing group.
+This client can be configured with a load balancing strategy that either
+randomly selects one of the configured hosts, or selects a host in a round-robin
+fashion. You can also specify your own custom class that implements the
+<code>LoadBalancingRpcClient$HostSelector</code> interface so that a custom selection
+order is used. In that case, the FQCN of the custom class needs to be specified
+as the value of the <code>host-selector</code> property. The LoadBalancing RPC Client
+currently does not support thrift.</p>
+</div>
+<div class="paragraph">
+<p>If <code>backoff</code> is enabled then the client will temporarily blacklist
+hosts that fail, causing them to be excluded from being selected as a failover
+host until a given timeout. When the timeout elapses, if the host is still
+unresponsive then this is considered a sequential failure, and the timeout is
+increased exponentially to avoid potentially getting stuck in long waits on
+unresponsive hosts.</p>
+</div>
+<div class="paragraph">
+<p>The maximum backoff time can be configured by setting <code>maxBackoff</code> (in
+milliseconds). The maxBackoff default is 30 seconds (specified in the
+<code>OrderSelector</code> class that&#8217;s the superclass of both load balancing
+strategies). The backoff timeout will increase exponentially with each
+sequential failure up to the maximum possible backoff timeout.
+The maximum possible backoff is limited to 65536 seconds (about 18.2 hours).
+For example:</p>
+</div>
+<div class="listingblock">
+<div class="content">
+<pre class="highlightjs highlight"><code class="language-java hljs" data-lang="java">// Setup properties for the load balancing
+Properties props = new Properties();
+props.put("client.type", "default_loadbalance");
+
+// List of hosts (space-separated list of user-chosen host aliases)
+props.put("hosts", "h1 h2 h3");
+
+// host/port pair for each host alias
+String host1 = "host1.example.org:41414";
+String host2 = "host2.example.org:41414";
+String host3 = "host3.example.org:41414";
+props.put("hosts.h1", host1);
+props.put("hosts.h2", host2);
+props.put("hosts.h3", host3);
+
+props.put("host-selector", "random"); // For random host selection
+// props.put("host-selector", "round_robin"); // For round-robin host
+//                                            // selection
+props.put("backoff", "true"); // Disabled by default.
+
+props.put("maxBackoff", "10000"); // Defaults 0, which effectively
+                                  // becomes 30000 ms
+
+// Create the client with load balancing properties
+RpcClient client = RpcClientFactory.getInstance(props);</code></pre>
+</div>
+</div>
+<div class="paragraph">
+<p>For more flexibility, the load-balancing Flume client implementation
+(<code>LoadBalancingRpcClient</code>) can be configured with these properties:</p>
+</div>
+<div class="listingblock">
+<div class="content">
+<pre class="highlightjs highlight"><code class="language-properties hljs" data-lang="properties">client.type = default_loadbalance
+
+hosts = h1 h2 h3                     # At least 2 hosts are required
+
+hosts.h1 = host1.example.org:41414
+
+hosts.h2 = host2.example.org:41414
+
+hosts.h3 = host3.example.org:41414
+
+backoff = false                      # Specifies whether the client should
+                                     # back-off from (i.e. temporarily
+                                     # blacklist) a failed host
+                                     # (default: false).
+
+maxBackoff = 0                       # Max timeout in millis that a will
+                                     # remain inactive due to a previous
+                                     # failure with that host (default: 0,
+                                     # which effectively becomes 30000)
+
+host-selector = round_robin          # The host selection strategy used
+                                     # when load-balancing among hosts
+                                     # (default: round_robin).
+                                     # Other values are include "random"
+                                     # or the FQCN of a custom class
+                                     # that implements
+                                     # LoadBalancingRpcClient$HostSelector
+
+batch-size = 100                     # Must be &gt;=1 (default: 100)
+
+connect-timeout = 20000              # Must be &gt;=1000 (default: 20000)
+
+request-timeout = 20000              # Must be &gt;=1000 (default: 20000)</code></pre>
+</div>
+</div>
+</div>
+</div>
+<div class="sect2">
+<h3 id="_embedded_agent"><a class="anchor" href="#_embedded_agent"></a>Embedded agent</h3>
+<div class="paragraph">
+<p>Flume has an embedded agent api which allows users to embed an agent in their
+application. This agent is meant to be lightweight and as such not all
+sources, sinks, and channels are allowed. Specifically the source used
+is a special embedded source and events should be send to the source
+via the put, putAll methods on the EmbeddedAgent object. Only File Channel
+and Memory Channel are allowed as channels while Avro Sink is the only
+supported sink. Interceptors are also supported by the embedded agent.</p>
+</div>
+<div class="paragraph">
+<p>Note: The embedded agent has a dependency on hadoop-core.jar.</p>
+</div>
+<div class="paragraph">
+<p>Configuration of an Embedded Agent is similar to configuration of a
+full Agent. The following is an exhaustive list of configration options:</p>
+</div>
+<div class="paragraph">
+<p>Required properties are in <strong>bold</strong>.</p>
+</div>
+<table class="tableblock frame-all grid-all stretch">
+<colgroup>
+<col style="width: 33.3333%;">
+<col style="width: 33.3333%;">
+<col style="width: 33.3334%;">
+</colgroup>
+<thead>
+<tr>
+<th class="tableblock halign-left valign-top">Property Name</th>
+<th class="tableblock halign-left valign-top">Default</th>
+<th class="tableblock halign-left valign-top">Description</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">source.type</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">embedded</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">The only available source is the embedded source.</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock"><strong>channel.type</strong></p></td>
+<td class="tableblock halign-left valign-top"><div class="content"><div class="dlist">
+<dl>
+<dt class="hdlist1">&#8201;&#8212;&#8201;</dt>
+<dd>
+<p>to MemoryChannel</p>
+</dd>
+</dl>
+</div></div></td>
+<td class="tableblock halign-left valign-top"><div class="content"><div class="dlist">
+<dl>
+<dt class="hdlist1">Either <code>memory</code> or <code>file</code> which correspond</dt>
+<dd>
+<p>and FileChannel respectively.</p>
+</dd>
+</dl>
+</div></div></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">channel.*</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Configuration options for the channel type requested,</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">se</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">e MemoryChannel or</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">FileChannel user guide for an exhaustive list.</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock"><strong>sinks</strong></p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">List of sink names</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock"><strong>sink.type</strong></p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Property name must match a name in the list of sinks.</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Va</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">lue must be ``avro</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">``</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">sink.*</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Configuration options for the sink.</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Se</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">e AvroSink user gu</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">ide for an exhaustive list,</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">ho</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">wever note AvroSin</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">k requires at least hostname and port.</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock"><strong>processor.type</strong></p></td>
+<td class="tableblock halign-left valign-top"><div class="content"><div class="dlist">
+<dl>
+<dt class="hdlist1">&#8201;&#8212;&#8201;</dt>
+<dd>
+<p>to FailoverS</p>
+</dd>
+</dl>
+</div></div></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Either <code>failover</code> or <code>load_balance</code> which correspond
+inksProcessor and LoadBalancingSinkProcessor respectively.</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">processor.*</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Configuration options for the sink processor selected.</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Se</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">e FailoverSinksPro</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">cessor and LoadBalancingSinkProcessor</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">us</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">er guide for an ex</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">haustive list.</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">source.interceptors</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Space-separated list of interceptors</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">source.interceptors.*</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Configuration options for individual interceptors</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">sp</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">ecified in the sou</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">rce.interceptors property</p></td>
+</tr>
+</tbody>
+</table>
+<div class="paragraph">
+<p>Below is an example of how to use the agent:</p>
+</div>
+<div class="listingblock">
+<div class="content">
+<pre class="highlightjs highlight"><code class="language-java hljs" data-lang="java">Map&lt;String, String&gt; properties = new HashMap&lt;String, String&gt;();
+properties.put("channel.type", "memory");
+properties.put("channel.capacity", "200");
+properties.put("sinks", "sink1 sink2");
+properties.put("sink1.type", "avro");
+properties.put("sink2.type", "avro");
+properties.put("sink1.hostname", "collector1.apache.org");
+properties.put("sink1.port", "5564");
+properties.put("sink2.hostname", "collector2.apache.org");
+properties.put("sink2.port",  "5565");
+properties.put("processor.type", "load_balance");
+properties.put("source.interceptors", "i1");
+properties.put("source.interceptors.i1.type", "static");
+properties.put("source.interceptors.i1.key", "key1");
+properties.put("source.interceptors.i1.value", "value1");
+
+EmbeddedAgent agent = new EmbeddedAgent("myagent");
+
+agent.configure(properties);
+agent.start();
+
+List&lt;Event&gt; events = Lists.newArrayList();
+
+events.add(event);
+events.add(event);
+events.add(event);
+events.add(event);
+
+agent.putAll(events);
+
+...
+
+agent.stop();</code></pre>
+</div>
+</div>
+</div>
+<div class="sect2">
+<h3 id="_transaction_interface"><a class="anchor" href="#_transaction_interface"></a>Transaction interface</h3>
+<div class="paragraph">
+<p>The <code>Transaction</code> interface is the basis of reliability for Flume. All the
+major components (ie. <code>Source`s, `Sink`s and `Channel`s) must use a
+Flume `Transaction</code>.</p>
+</div>
+<div class="imageblock">
+<div class="content">
+<img src="_images/DevGuide_image01.png" alt="Transaction sequence diagram">
+</div>
+</div>
+<div class="paragraph">
+<p>A <code>Transaction</code> is implemented within a <code>Channel</code> implementation. Each
+<code>Source</code> and <code>Sink</code> that is connected to a <code>Channel</code> must obtain a
+<code>Transaction</code> object. The <code>Source`s use a `ChannelProcessor</code>
+to manage the <code>Transaction`s, the `Sink`s manage them explicitly via
+their configured `Channel</code>. The operation to stage an
+<code>Event</code> (put it into a <code>Channel</code>) or extract an <code>Event</code> (take it out of a
+<code>Channel</code>) is done inside an active <code>Transaction</code>. For example:</p>
+</div>
+<div class="listingblock">
+<div class="content">
+<pre class="highlightjs highlight"><code class="language-java hljs" data-lang="java">Channel ch = new MemoryChannel();
+Transaction txn = ch.getTransaction();
+txn.begin();
+try {
+  // This try clause includes whatever Channel operations you want to do
+
+  Event eventToStage = EventBuilder.withBody("Hello Flume!",
+                       Charset.forName("UTF-8"));
+  ch.put(eventToStage);
+  // Event takenEvent = ch.take();
+  // ...
+  txn.commit();
+} catch (Throwable t) {
+  txn.rollback();
+
+  // Log exception, handle individual exceptions as needed
+
+  // re-throw all Errors
+  if (t instanceof Error) {
+    throw (Error)t;
+  }
+} finally {
+  txn.close();
+}</code></pre>
+</div>
+</div>
+<div class="paragraph">
+<p>Here we get hold of a <code>Transaction</code> from a <code>Channel</code>. After <code>begin()</code>
+returns, the <code>Transaction</code> is now active/open and the <code>Event</code> is then put
+into the <code>Channel</code>. If the put is successful, then the <code>Transaction</code> is
+committed and closed.</p>
+</div>
+</div>
+<div class="sect2">
+<h3 id="_sink"><a class="anchor" href="#_sink"></a>Sink</h3>
+<div class="paragraph">
+<p>The purpose of a <code>Sink</code> to extract <code>Event`s from the `Channel</code> and
+forward them to the next Flume Agent in the flow or store them in an external
+repository. A <code>Sink</code> is associated with exactly one <code>Channel`s, as
+configured in the Flume properties file. There’s one `SinkRunner</code> instance
+associated with every configured <code>Sink</code>, and when the Flume framework calls
+<code>SinkRunner.start()</code>, a new thread is created to drive the <code>Sink</code> (using
+<code>SinkRunner.PollingRunner</code> as the thread&#8217;s <code>Runnable</code>). This thread manages
+the <code>Sink</code>’s lifecycle. The <code>Sink</code> needs to implement the <code>start()</code> and
+<code>stop()</code> methods that are part of the <code>LifecycleAware</code> interface. The
+<code>Sink.start()</code> method should initialize the <code>Sink</code> and bring it to a state
+where it can forward the <code>Event`s to its next destination. The
+`Sink.process()</code> method should do the core processing of extracting the
+<code>Event</code> from the <code>Channel</code> and forwarding it. The <code>Sink.stop()</code> method
+should do the necessary cleanup (e.g. releasing resources). The <code>Sink</code>
+implementation also needs to implement the <code>Configurable</code> interface for
+processing its own configuration settings. For example:</p>
+</div>
+<div class="listingblock">
+<div class="content">
+<pre class="highlightjs highlight"><code class="language-java hljs" data-lang="java">public class MySink extends AbstractSink implements Configurable {
+  private String myProp;
+
+  @Override
+  public void configure(Context context) {
+    String myProp = context.getString("myProp", "defaultValue");
+
+    // Process the myProp value (e.g. validation)
+
+    // Store myProp for later retrieval by process() method
+    this.myProp = myProp;
+  }
+
+  @Override
+  public void start() {
+    // Initialize the connection to the external repository (e.g. HDFS) that
+    // this Sink will forward Events to ..
+  }
+
+  @Override
+  public void stop () {
+    // Disconnect from the external respository and do any
+    // additional cleanup (e.g. releasing resources or nulling-out
+    // field values) ..
+  }
+
+  @Override
+  public Status process() throws EventDeliveryException {
+    Status status = null;
+
+    // Start transaction
+    Channel ch = getChannel();
+    Transaction txn = ch.getTransaction();
+    txn.begin();
+    try {
+      // This try clause includes whatever Channel operations you want to do
+
+      Event event = ch.take();
+
+      // Send the Event to the external repository.
+      // storeSomeData(e);
+
+      txn.commit();
+      status = Status.READY;
+    } catch (Throwable t) {
+      txn.rollback();
+
+      // Log exception, handle individual exceptions as needed
+
+      status = Status.BACKOFF;
+
+      // re-throw all Errors
+      if (t instanceof Error) {
+        throw (Error)t;
+      }
+    }
+    return status;
+  }
+}</code></pre>
+</div>
+</div>
+</div>
+<div class="sect2">
+<h3 id="_source"><a class="anchor" href="#_source"></a>Source</h3>
+<div class="paragraph">
+<p>The purpose of a <code>Source</code> is to receive data from an external client and store
+it into the configured <code>Channel`s. A `Source</code> can get an instance of its own
+<code>ChannelProcessor</code> to process an <code>Event</code>, commited within a <code>Channel</code>
+local transaction, in serial. In the case of an exception, required
+`Channel`s will propagate the exception, all `Channel`s will rollback their
+transaction, but events processed previously on other `Channel`s will remain
+committed.</p>
+</div>
+<div class="paragraph">
+<p>Similar to the <code>SinkRunner.PollingRunner</code> <code>Runnable</code>, there’s
+a <code>PollingRunner</code> <code>Runnable</code> that executes on a thread created when the
+Flume framework calls <code>PollableSourceRunner.start()</code>. Each configured
+<code>PollableSource</code> is associated with its own thread that runs a
+<code>PollingRunner</code>. This thread manages the <code>PollableSource</code>’s lifecycle,
+such as starting and stopping. A <code>PollableSource</code> implementation must
+implement the <code>start()</code> and <code>stop()</code> methods that are declared in the
+<code>LifecycleAware</code> interface. The runner of a <code>PollableSource</code> invokes that
+<code>Source&#8217;s `process()</code> method. The <code>process()</code> method should check for
+new data and store it into the <code>Channel</code> as Flume `Event`s.</p>
+</div>
+<div class="paragraph">
+<p>Note that there are actually two types of <code>Source`s. The `PollableSource</code>
+was already mentioned. The other is the <code>EventDrivenSource</code>. The
+<code>EventDrivenSource</code>, unlike the <code>PollableSource</code>, must have its own callback
+mechanism that captures the new data and stores it into the <code>Channel</code>. The
+<code>EventDrivenSource`s are not each driven by their own thread like the
+`PollableSource`s are. Below is an example of a custom `PollableSource</code>:</p>
+</div>
+<div class="listingblock">
+<div class="content">
+<pre class="highlightjs highlight"><code class="language-java hljs" data-lang="java">public class MySource extends AbstractSource implements Configurable, PollableSource {
+  private String myProp;
+
+  @Override
+  public void configure(Context context) {
+    String myProp = context.getString("myProp", "defaultValue");
+
+    // Process the myProp value (e.g. validation, convert to another type, ...)
+
+    // Store myProp for later retrieval by process() method
+    this.myProp = myProp;
+  }
+
+  @Override
+  public void start() {
+    // Initialize the connection to the external client
+  }
+
+  @Override
+  public void stop () {
+    // Disconnect from external client and do any additional cleanup
+    // (e.g. releasing resources or nulling-out field values) ..
+  }
+
+  @Override
+  public Status process() throws EventDeliveryException {
+    Status status = null;
+
+    try {
+      // This try clause includes whatever Channel/Event operations you want to do
+
+      // Receive new data
+      Event e = getSomeData();
+
+      // Store the Event into this Source's associated Channel(s)
+      getChannelProcessor().processEvent(e);
+
+      status = Status.READY;
+    } catch (Throwable t) {
+      // Log exception, handle individual exceptions as needed
+
+      status = Status.BACKOFF;
+
+      // re-throw all Errors
+      if (t instanceof Error) {
+        throw (Error)t;
+      }
+    } finally {
+      txn.close();
+    }
+    return status;
+  }
+}</code></pre>
+</div>
+</div>
+</div>
+<div class="sect2">
+<h3 id="_channel"><a class="anchor" href="#_channel"></a>Channel</h3>
+<div class="paragraph">
+<p>TBD</p>
+</div>
+</div>
+<div class="sect2">
+<h3 id="_initializable"><a class="anchor" href="#_initializable"></a>Initializable</h3>
+<div class="paragraph">
+<p>As of Flume 1.10.0 Sources, Sinks, and Channels may implement the Intitializable interface. Doing so
+allows the component to have access the materialized configuration before any of the components have been
+started. While this ability is quite useful when using the standard configuration, it is less useful when
+configuring using Spring Boot as Spring&#8217;s autowiring generally can be used to accomplish the same thing.</p>
+</div>
+<div class="paragraph">
+<p>This example shows a Sink being configured with the name of a Source. While initializing it will
+retrieve the Source from the configuration and save it. During event processing a new event will be
+sent to the Source, presumably after the event has be modified in some way.</p>
+</div>
+<div class="listingblock">
+<div class="content">
+<pre class="highlightjs highlight"><code class="language-java hljs" data-lang="java">public class NullInitSink extends NullSink implements Initializable {
+
+  private static final Logger logger = LoggerFactory.getLogger(NullInitSink.class);
+  private String sourceName = null;
+  private EventProcessor eventProcessor = null;
+  private long total = 0;
+
+  public NullInitSink() {
+    super();
+  }
+
+  @Override
+  public void configure(Context context) {
+    sourceName = context.getString("targetSource");
+    super.configure(context);
+
+  }
+
+  @Override
+  public void initialize(MaterializedConfiguration configuration) {
+    logger.debug("Locating source for event publishing");
+    for (Map.Entry&lt;String, SourceRunner&gt;  entry : configuration.getSourceRunners().entrySet()) {
+      if (entry.getKey().equals(sourceName)) {
+        Source source = entry.getValue().getSource();
+        if (source instanceof EventProcessor) {
+          eventProcessor = (EventProcessor) source;
+          logger.debug("Found event processor {}", source.getName());
+          return;
+        }
+      }
+    }
+    logger.warn("No Source named {} found for republishing events.", sourceName);
+  }
+
+  @Override
+  public Status process() throws EventDeliveryException {
+    Status status = Status.READY;
+
+    Channel channel = getChannel();
+    Transaction transaction = channel.getTransaction();
+    Event event = null;
+    CounterGroup counterGroup = getCounterGroup();
+    long batchSize = getBatchSize();
+    long eventCounter = counterGroup.get("events.success");
+
+    try {
+      transaction.begin();
+      int i = 0;
+      for (i = 0; i &lt; batchSize; i++) {
+        event = channel.take();
+        if (event != null) {
+          long id = Long.parseLong(new String(event.getBody()));
+          total += id;
+          event.getHeaders().put("Total", Long.toString(total));
+          eventProcessor.processEvent(event);
+          logger.info("Null sink {} successful processed event {}", getName(), id);
+        } else {
+          status = Status.BACKOFF;
+          break;
+        }
+      }
+      transaction.commit();
+      counterGroup.addAndGet("events.success", (long) Math.min(batchSize, i));
+      counterGroup.incrementAndGet("transaction.success");
+    } catch (Exception ex) {
+      transaction.rollback();
+      counterGroup.incrementAndGet("transaction.failed");
+      logger.error("Failed to deliver event. Exception follows.", ex);
+      throw new EventDeliveryException("Failed to deliver event: " + event, ex);
+    } finally {
+      transaction.close();
+    }
+
+    return status;
+  }
+}</code></pre>
+</div>
+</div>
+</div>
+</div>
+</div>
+</article>
+  </div>
+</main>
+</div>
+<footer class="footer">
+  <p>This page was built using the Antora default UI.</p>
+  <p>The source code for this UI is licensed under the terms of the MPL-2.0 license.</p>
+</footer>
+<script id="site-script" src="../_/js/site.js" data-ui-root-path="../_"></script>
+<script async src="../_/js/vendor/highlight.js"></script>
+  </body>
+</html>
diff --git a/flume/FlumeUserGuide.html b/flume/FlumeUserGuide.html
new file mode 100644
index 00000000..d3908f59
--- /dev/null
+++ b/flume/FlumeUserGuide.html
@@ -0,0 +1,11727 @@
+<!DOCTYPE html>
+<html lang="en">
+  <head>
+    <meta charset="utf-8">
+    <meta name="viewport" content="width=device-width,initial-scale=1">
+    <title>Flume 1.11.0 User Guide :: Apache Flume</title>
+    <link rel="canonical" href="https://logging.apache.org/flume/2.x/flume/FlumeUserGuide.html">
+    <meta name="generator" content="Antora 3.1.15">
+    <link rel="stylesheet" href="../_/css/site.css">
+  </head>
+  <body class="article">
+<header class="header">
+  <nav class="navbar">
+    <div class="navbar-brand">
+      <a class="navbar-item" href="https://logging.apache.org/flume/2.x">Apache Flume</a>
+      <button class="navbar-burger" aria-controls="topbar-nav" aria-expanded="false" aria-label="Toggle main menu">
+        <span></span>
+        <span></span>
+        <span></span>
+      </button>
+    </div>
+    <div id="topbar-nav" class="navbar-menu">
+      <div class="navbar-end">
+        <a class="navbar-item" href="#">Home</a>
+        <div class="navbar-item has-dropdown is-hoverable">
+          <a class="navbar-link" href="#">Products</a>
+          <div class="navbar-dropdown">
+            <a class="navbar-item" href="#">Product A</a>
+            <a class="navbar-item" href="#">Product B</a>
+            <a class="navbar-item" href="#">Product C</a>
+          </div>
+        </div>
+        <div class="navbar-item has-dropdown is-hoverable">
+          <a class="navbar-link" href="#">Services</a>
+          <div class="navbar-dropdown">
+            <a class="navbar-item" href="#">Service A</a>
+            <a class="navbar-item" href="#">Service B</a>
+            <a class="navbar-item" href="#">Service C</a>
+          </div>
+        </div>
+        <div class="navbar-item">
+          <span class="control">
+            <a class="button is-primary" href="#">Download</a>
+          </span>
+        </div>
+      </div>
+    </div>
+  </nav>
+</header>
+<div class="body">
+<div class="nav-container" data-component="flume" data-version="">
+  <aside class="nav">
+    <div class="panels">
+<div class="nav-panel-menu is-active" data-panel="menu">
+  <nav class="nav-menu">
+    <button class="nav-menu-toggle" aria-label="Toggle expand/collapse all" style="display: none"></button>
+    <h3 class="title"><a href="index.html">Apache Flume</a></h3>
+<ul class="nav-list">
+  <li class="nav-item" data-depth="0">
+<ul class="nav-list">
+  <li class="nav-item" data-depth="1">
+    <a class="nav-link" href="index.html">Home</a>
+  </li>
+  <li class="nav-item" data-depth="1">
+    <a class="nav-link" href="getinvolved.html">Get Involved</a>
+  </li>
+  <li class="nav-item" data-depth="1">
+    <a class="nav-link" href="download.html">Download</a>
+  </li>
+  <li class="nav-item" data-depth="1">
+    <a class="nav-link" href="security.html">Security</a>
+  </li>
+  <li class="nav-item" data-depth="1">
+    <button class="nav-item-toggle"></button>
+    <a class="nav-link" href="documentation.html">Documentation</a>
+<ul class="nav-list">
+  <li class="nav-item is-current-page" data-depth="2">
+    <a class="nav-link" href="FlumeUserGuide.html">Flume User Guide</a>
+  </li>
+  <li class="nav-item" data-depth="2">
+    <a class="nav-link" href="FlumeDeveloperGuide.html">Flume Developer Guide</a>
+  </li>
+</ul>
+  </li>
+  <li class="nav-item" data-depth="1">
+    <button class="nav-item-toggle"></button>
+    <a class="nav-link" href="releases/index.html">Releases</a>
+<ul class="nav-list">
+  <li class="nav-item" data-depth="2">
+    <a class="nav-link" href="releases/1.11.0.html">1.11.0</a>
+  </li>
+  <li class="nav-item" data-depth="2">
+    <a class="nav-link" href="releases/1.10.1.html">1.10.1</a>
+  </li>
+  <li class="nav-item" data-depth="2">
+    <a class="nav-link" href="releases/1.10.0.html">1.10.0</a>
+  </li>
+  <li class="nav-item" data-depth="2">
+    <a class="nav-link" href="releases/1.9.0.html">1.9.0</a>
+  </li>
+  <li class="nav-item" data-depth="2">
+    <a class="nav-link" href="releases/1.8.0.html">1.8.0</a>
+  </li>
+  <li class="nav-item" data-depth="2">
+    <a class="nav-link" href="releases/1.7.0.html">1.7.0</a>
+  </li>
+  <li class="nav-item" data-depth="2">
+    <a class="nav-link" href="releases/1.6.0.html">1.6.0</a>
+  </li>
+  <li class="nav-item" data-depth="2">
+    <a class="nav-link" href="releases/1.5.2.html">1.5.2</a>
+  </li>
+  <li class="nav-item" data-depth="2">
+    <a class="nav-link" href="releases/1.5.0.1.html">1.5.0.1</a>
+  </li>
+  <li class="nav-item" data-depth="2">
+    <a class="nav-link" href="releases/1.5.0.html">1.5.0</a>
+  </li>
+  <li class="nav-item" data-depth="2">
+    <a class="nav-link" href="releases/1.4.0.html">1.4.0</a>
+  </li>
+  <li class="nav-item" data-depth="2">
+    <a class="nav-link" href="releases/1.3.1.html">1.3.1</a>
+  </li>
+  <li class="nav-item" data-depth="2">
+    <a class="nav-link" href="releases/1.3.0.html">1.3.0</a>
+  </li>
+  <li class="nav-item" data-depth="2">
+    <a class="nav-link" href="releases/1.2.0.html">1.2.0</a>
+  </li>
+  <li class="nav-item" data-depth="2">
+    <a class="nav-link" href="releases/1.1.0.html">1.1.0</a>
+  </li>
+  <li class="nav-item" data-depth="2">
+    <a class="nav-link" href="releases/1.0.0.html">1.0.0</a>
+  </li>
+</ul>
+  </li>
+  <li class="nav-item" data-depth="1">
+    <button class="nav-item-toggle"></button>
+    <a class="nav-link" href="flume-spring-boot/index.html">Flume Spring Boot</a>
+<ul class="nav-list">
+  <li class="nav-item" data-depth="2">
+    <a class="nav-link" href="flume-spring-boot/download.html">Download</a>
+  </li>
+  <li class="nav-item" data-depth="2">
+    <button class="nav-item-toggle"></button>
+    <a class="nav-link" href="flume-spring-boot/releases/index.html">Releases</a>
+<ul class="nav-list">
+  <li class="nav-item" data-depth="3">
+    <a class="nav-link" href="flume-spring-boot/releases/2.0.0.html">2.0.0</a>
+  </li>
+</ul>
+  </li>
+  <li class="nav-item" data-depth="2">
+    <a class="nav-link" href="flume-spring-boot/source.html">Source</a>
+  </li>
+</ul>
+  </li>
+  <li class="nav-item" data-depth="1">
+    <a class="nav-link" href="team.html">Team</a>
+  </li>
+  <li class="nav-item" data-depth="1">
+    <a class="nav-link" href="mailinglists.html">Mailing Lists</a>
+  </li>
+  <li class="nav-item" data-depth="1">
+    <a class="nav-link" href="source.html">Source Repository</a>
+  </li>
+  <li class="nav-item" data-depth="1">
+    <a class="nav-link" href="subprojects.html">Subprojects</a>
+  </li>
+  <li class="nav-item" data-depth="1">
+    <a class="nav-link" href="testing.html">Testing</a>
+  </li>
+  <li class="nav-item" data-depth="1">
+    <a class="nav-link" href="license.html">License</a>
+  </li>
+</ul>
+  </li>
+</ul>
+  </nav>
+</div>
+<div class="nav-panel-explore" data-panel="explore">
+  <div class="context">
+    <span class="title">Apache Flume</span>
+    <span class="version"></span>
+  </div>
+  <ul class="components">
+    <li class="component is-current">
+      <div class="title"><a href="index.html">Apache Flume</a></div>
+    </li>
+  </ul>
+</div>
+    </div>
+  </aside>
+</div>
+<main class="article">
+<div class="toolbar" role="navigation">
+<button class="nav-toggle"></button>
+  <a href="index.html" class="home-link"></a>
+<nav class="breadcrumbs" aria-label="breadcrumbs">
+  <ul>
+    <li><a href="index.html">Apache Flume</a></li>
+    <li><a href="documentation.html">Documentation</a></li>
+    <li><a href="FlumeUserGuide.html">Flume User Guide</a></li>
+  </ul>
+</nav>
+<div class="edit-this-page"><a href="file:///home/piotr/workspace/logging/flume/flume-1.x/target/generated-site/antora/modules/ROOT/pages/FlumeUserGuide.adoc">Edit this Page</a></div>
+</div>
+  <div class="content">
+<aside class="toc sidebar" data-title="Contents" data-levels="2">
+  <div class="toc-menu"></div>
+</aside>
+<article class="doc">
+<h1 class="page">Flume 1.11.0 User Guide</h1>
+<div class="sect1">
+<h2 id="_introduction"><a class="anchor" href="#_introduction"></a>Introduction</h2>
+<div class="sectionbody">
+<div class="sect2">
+<h3 id="_overview"><a class="anchor" href="#_overview"></a>Overview</h3>
+<div class="paragraph">
+<p>Apache Flume is a distributed, reliable, and available system for efficiently
+collecting, aggregating and moving large amounts of log data from many
+different sources to a centralized data store.</p>
+</div>
+<div class="paragraph">
+<p>The use of Apache Flume is not only restricted to log data aggregation.
+Since data sources are customizable, Flume can be used to transport massive quantities
+of event data including but not limited to network traffic data, social-media-generated data,
+email messages and pretty much any data source possible.</p>
+</div>
+<div class="paragraph">
+<p>Apache Flume is a top level project at the Apache Software Foundation.</p>
+</div>
+</div>
+<div class="sect2">
+<h3 id="_system_requirements"><a class="anchor" href="#_system_requirements"></a>System Requirements</h3>
+<div class="olist arabic">
+<ol class="arabic">
+<li>
+<p>Java Runtime Environment - Java 1.8 or later</p>
+</li>
+<li>
+<p>Memory - Sufficient memory for configurations used by sources, channels or sinks</p>
+</li>
+<li>
+<p>Disk Space - Sufficient disk space for configurations used by channels or sinks</p>
+</li>
+<li>
+<p>Directory Permissions - Read/Write permissions for directories used by agent</p>
+</li>
+</ol>
+</div>
+</div>
+<div class="sect2">
+<h3 id="_architecture"><a class="anchor" href="#_architecture"></a>Architecture</h3>
+<div class="sect3">
+<h4 id="_data_flow_model"><a class="anchor" href="#_data_flow_model"></a>Data flow model</h4>
+<div class="paragraph">
+<p>A Flume event is defined as a unit of data flow having a byte payload and an
+optional set of string attributes. A Flume agent is a (JVM) process that hosts
+the components through which events flow from an external source to the next
+destination (hop).</p>
+</div>
+<div class="imageblock">
+<div class="content">
+<img src="_images/UserGuide_image00.png" alt="Agent component diagram">
+</div>
+</div>
+<div class="paragraph">
+<p>A Flume source consumes events delivered to it by an external source like a web
+server. The external source sends events to Flume in a format that is
+recognized by the target Flume source. For example, an Avro Flume source can be
+used to receive Avro events from Avro clients or other Flume agents in the flow
+that send events from an Avro sink. A similar flow can be defined using
+a Thrift Flume Source to receive events from a Thrift Sink or a Flume
+Thrift Rpc Client or Thrift clients written in any language generated from
+the Flume thrift protocol.When a Flume source receives an event, it
+stores it into one or more channels. The channel is a passive store that keeps
+the event until it&#8217;s consumed by a Flume sink. The file channel is one example&#8201;&#8212;&#8201;it is backed by the local filesystem. The sink removes the event
+from the channel and puts it into an external repository like HDFS (via Flume
+HDFS sink) or forwards it to the Flume source of the next Flume agent (next
+hop) in the flow. The source and sink within the given agent run asynchronously
+with the events staged in the channel.</p>
+</div>
+</div>
+<div class="sect3">
+<h4 id="_complex_flows"><a class="anchor" href="#_complex_flows"></a>Complex flows</h4>
+<div class="paragraph">
+<p>Flume allows a user to build multi-hop flows where events travel through
+multiple agents before reaching the final destination. It also allows fan-in
+and fan-out flows, contextual routing and backup routes (fail-over) for failed
+hops.</p>
+</div>
+</div>
+<div class="sect3">
+<h4 id="_reliability"><a class="anchor" href="#_reliability"></a>Reliability</h4>
+<div class="paragraph">
+<p>The events are staged in a channel on each agent. The events are then delivered
+to the next agent or terminal repository (like HDFS) in the flow. The events
+are removed from a channel only after they are stored in the channel of next
+agent or in the terminal repository. This is a how the single-hop message
+delivery semantics in Flume provide end-to-end reliability of the flow.</p>
+</div>
+<div class="paragraph">
+<p>Flume uses a transactional approach to guarantee the reliable delivery of the
+events. The sources and sinks encapsulate in a transaction the
+storage/retrieval, respectively, of the events placed in or provided by a
+transaction provided by the channel. This ensures that the set of events are
+reliably passed from point to point in the flow. In the case of a multi-hop
+flow, the sink from the previous hop and the source from the next hop both have
+their transactions running to ensure that the data is safely stored in the
+channel of the next hop.</p>
+</div>
+</div>
+<div class="sect3">
+<h4 id="_recoverability"><a class="anchor" href="#_recoverability"></a>Recoverability</h4>
+<div class="paragraph">
+<p>The events are staged in the channel, which manages recovery from failure.
+Flume supports a durable file channel which is backed by the local file system.
+There&#8217;s also a memory channel which simply stores the events in an in-memory
+queue, which is faster but any events still left in the memory channel when an
+agent process dies can&#8217;t be recovered.</p>
+</div>
+<div class="paragraph">
+<p>Flume&#8217;s <span class="title-ref">KafkaChannel</span> uses Apache Kafka to stage events. Using a replicated
+Kafka topic as a channel helps avoiding event loss in case of a disk failure.</p>
+</div>
+</div>
+</div>
+</div>
+</div>
+<div class="sect1">
+<h2 id="_standard_setup"><a class="anchor" href="#_standard_setup"></a>Standard Setup</h2>
+<div class="sectionbody">
+<div class="paragraph">
+<p>This section documents how to configure and wire Flume components using Flume&#8217;s
+long-standing configuration techniques using properties files. See the following
+section for creating a Flume application using Spring Boot.</p>
+</div>
+<div class="sect2">
+<h3 id="_setting_up_an_agent"><a class="anchor" href="#_setting_up_an_agent"></a>Setting up an agent</h3>
+<div class="paragraph">
+<p>Flume agent configuration is stored in one or more configuration files that
+follow the Java properties file format. Configurations for one or more agents
+can be specified in these configuration files. The configuration includes
+properties of each source, sink and channel in an agent and how they are wired
+together to form data flows.</p>
+</div>
+<div class="sect3">
+<h4 id="_configuring_individual_components"><a class="anchor" href="#_configuring_individual_components"></a>Configuring individual components</h4>
+<div class="paragraph">
+<p>Each component (source, sink or channel) in the flow has a name, type, and set
+of properties that are specific to the type and instantiation. For example, an
+Avro source needs a hostname (or IP address) and a port number to receive data
+from. A memory channel can have max queue size ("capacity"), and an HDFS sink
+needs to know the file system URI, path to create files, frequency of file
+rotation ("hdfs.rollInterval") etc. All such attributes of a component needs to
+be set in the properties file of the hosting Flume agent.</p>
+</div>
+</div>
+<div class="sect3">
+<h4 id="_wiring_the_pieces_together"><a class="anchor" href="#_wiring_the_pieces_together"></a>Wiring the pieces together</h4>
+<div class="paragraph">
+<p>The agent needs to know what individual components to load and how they are
+connected in order to constitute the flow. This is done by listing the names of
+each of the sources, sinks and channels in the agent, and then specifying the
+connecting channel for each sink and source. For example, an agent flows events
+from an Avro source called avroWeb to HDFS sink hdfs-cluster1 via a file
+channel called file-channel. The configuration file will contain names of these
+components and file-channel as a shared channel for both avroWeb source and
+hdfs-cluster1 sink.</p>
+</div>
+</div>
+<div class="sect3">
+<h4 id="_starting_an_agent"><a class="anchor" href="#_starting_an_agent"></a>Starting an agent</h4>
+<div class="paragraph">
+<p>An agent is started using a shell script called flume-ng which is located in
+the bin directory of the Flume distribution. You need to specify the agent
+name, the config directory, and the config file on the command line:</p>
+</div>
+<div class="literalblock">
+<div class="content">
+<pre>$ bin/flume-ng agent -n $agent_name -c conf -f conf/flume-conf.properties.template</pre>
+</div>
+</div>
+<div class="paragraph">
+<p>Now the agent will start running source and sinks configured in the given
+properties file.</p>
+</div>
+</div>
+<div class="sect3">
+<h4 id="_a_simple_example"><a class="anchor" href="#_a_simple_example"></a>A simple example</h4>
+<div class="paragraph">
+<p>Here, we give an example configuration file, describing a single-node Flume deployment.
+This configuration lets a user generate events and subsequently logs them to the console.</p>
+</div>
+<div class="listingblock">
+<div class="content">
+<pre class="highlightjs highlight"><code class="language-properties hljs" data-lang="properties"># example.conf: A single-node Flume configuration
+
+# Name the components on this agent
+a1.sources = r1
+a1.sinks = k1
+a1.channels = c1
+
+# Describe/configure the source
+a1.sources.r1.type = netcat
+a1.sources.r1.bind = localhost
+a1.sources.r1.port = 44444
+
+# Describe the sink
+a1.sinks.k1.type = logger
+
+# Use a channel which buffers events in memory
+a1.channels.c1.type = memory
+a1.channels.c1.capacity = 1000
+a1.channels.c1.transactionCapacity = 100
+
+# Bind the source and sink to the channel
+a1.sources.r1.channels = c1
+a1.sinks.k1.channel = c1</code></pre>
+</div>
+</div>
+<div class="paragraph">
+<p>This configuration defines a single agent named a1. a1 has a source that listens for data on port 44444, a channel
+that buffers event data in memory, and a sink that logs event data to the console. The configuration file names the
+various components, then describes their types and configuration parameters. A given configuration file might define
+several named agents; when a given Flume process is launched a flag is passed telling it which named agent to manifest.</p>
+</div>
+<div class="paragraph">
+<p>Given this configuration file, we can start Flume as follows:</p>
+</div>
+<div class="literalblock">
+<div class="content">
+<pre>$ bin/flume-ng agent --conf conf --conf-file example.conf --name a1</pre>
+</div>
+</div>
+<div class="paragraph">
+<p>Note that in a full deployment we would typically include one more option: <code>--conf=&lt;conf-dir&gt;</code>.
+The <code>&lt;conf-dir&gt;</code> directory would include a shell script <em>flume-env.sh</em> and potentially a log4j configuration file.</p>
+</div>
+<div class="paragraph">
+<p>From a separate terminal, we can then telnet port 44444 and send Flume an event:</p>
+</div>
+<div class="listingblock">
+<div class="content">
+<pre class="highlightjs highlight"><code class="language-properties hljs" data-lang="properties">$ telnet localhost 44444
+Trying 127.0.0.1...
+Connected to localhost.localdomain (127.0.0.1).
+Escape character is '^]'.
+Hello world! &lt;ENTER&gt;
+OK</code></pre>
+</div>
+</div>
+<div class="paragraph">
+<p>The original Flume terminal will output the event in a log message.</p>
+</div>
+<div class="listingblock">
+<div class="content">
+<pre class="highlightjs highlight"><code class="language-properties hljs" data-lang="properties">12/06/19 15:32:19 INFO source.NetcatSource: Source starting
+12/06/19 15:32:19 INFO source.NetcatSource: Created serverSocket:sun.nio.ch.ServerSocketChannelImpl[/127.0.0.1:44444]
+12/06/19 15:32:34 INFO sink.LoggerSink: Event: { headers:{} body: 48 65 6C 6C 6F 20 77 6F 72 6C 64 21 0D          Hello world!. }</code></pre>
+</div>
+</div>
+<div class="paragraph">
+<p>Congratulations - you&#8217;ve successfully configured and deployed a Flume agent! Subsequent sections cover agent configuration in much more detail.</p>
+</div>
+</div>
+<div class="sect3">
+<h4 id="_configuration_from_uris"><a class="anchor" href="#_configuration_from_uris"></a>Configuration from URIs</h4>
+<div class="paragraph">
+<p>As of version 1.10.0 Flume supports being configured using URIs instead of just from local files. Direct support
+for HTTP(S), file, and classpath URIs is included. The HTTP support includes support for authentication using
+basic authorization but other authorization mechanisms may be supported by specifying the fully qualified name
+of the class that implements the AuthorizationProvider interface using the --auth-provider option. HTTP also
+supports reloading of configuration files using polling if the target server properly responds to the If-Modified-Since
+header.</p>
+</div>
+<div class="paragraph">
+<p>To specify credentials for HTTP authentication add:</p>
+</div>
+<div class="literalblock">
+<div class="content">
+<pre>--conf-user userid --conf-password password</pre>
+</div>
+</div>
+<div class="paragraph">
+<p>to the startup command.</p>
+</div>
+</div>
+<div class="sect3">
+<h4 id="_multiple_configuration_files"><a class="anchor" href="#_multiple_configuration_files"></a>Multiple Configuration Files</h4>
+<div class="paragraph">
+<p>As of version 1.10.0 Flume supports being configured from multiple configuration files instead of just one.
+This more easily allows values to be overridden or added based on specific environments. Each file should
+be configured using its own --conf-file or --conf-uri option. However, all files should either be provided
+with --conf-file or with --conf-uri. If --conf-file and --conf-uri appear together as options all --conf-uri
+configurations will be processed before any of the --conf-file configurations are merged.</p>
+</div>
+<div class="paragraph">
+<p>For example, a configuration of:</p>
+</div>
+<div class="literalblock">
+<div class="content">
+<pre>$ bin/flume-ng agent --conf conf --conf-file example.conf --conf-uri http://localhost:80/flume.conf --conf-uri http://localhost:80/override.conf --name a1</pre>
+</div>
+</div>
+<div class="paragraph">
+<p>will cause flume.conf to be read first, override.conf to be merged with it and finally example.conf would be
+merged last. If it is desirec to have example.conf be the base configuration it should be specified using the
+--conf-uri option either as:</p>
+</div>
+<div class="literalblock">
+<div class="content">
+<pre>--conf-uri classpath://example.conf
+or
+--conf-uri file:///example.conf</pre>
+</div>
+</div>
+<div class="paragraph">
+<p>depending on how it should be accessed.</p>
+</div>
+</div>
+<div class="sect3">
+<h4 id="_using_environment_variables_system_properies_or_other_properties_configuration_files"><a class="anchor" href="#_using_environment_variables_system_properies_or_other_properties_configuration_files"></a>Using environment variables, system properies, or other properties configuration files</h4>
+<div class="paragraph">
+<p>Flume has the ability to substitute environment variables in the configuration. For example:</p>
+</div>
+<div class="literalblock">
+<div class="content">
+<pre>a1.sources = r1
+a1.sources.r1.type = netcat
+a1.sources.r1.bind = 0.0.0.0
+a1.sources.r1.port = ${env:NC_PORT}
+a1.sources.r1.channels = c1</pre>
+</div>
+</div>
+<div class="paragraph">
+<p>NB: it currently works for values only, not for keys. (Ie. only on the "right side" of the <span class="title-ref">=</span> mark of the config lines.)</p>
+</div>
+<div class="paragraph">
+<p>As of version 1.10.0 Flume resolves configuration values using Apache Commons Text&#8217;s StringSubstitutor
+class using the default set of Lookups along with a lookup that uses the configuration files as a
+source for replacement values.</p>
+</div>
+<div class="dlist">
+<dl>
+<dt class="hdlist1">For example</dt>
+<dd>
+<p>$ NC_PORT=44444 bin/flume-ng agent --conf conf --conf-file example.conf --name a1</p>
+</dd>
+</dl>
+</div>
+<div class="paragraph">
+<p>Note the above is just an example, environment variables can be configured in other ways, including being set in <span class="title-ref">conf/flume-env.sh</span>.</p>
+</div>
+<div class="paragraph">
+<p>As noted, system properties are also supported, so the configuration:</p>
+</div>
+<div class="literalblock">
+<div class="content">
+<pre>a1.sources = r1
+a1.sources.r1.type = netcat
+a1.sources.r1.bind = 0.0.0.0
+a1.sources.r1.port = ${sys:NC_PORT}
+a1.sources.r1.channels = c1</pre>
+</div>
+</div>
+<div class="paragraph">
+<p>could be used and the startup command could be:</p>
+</div>
+<div class="literalblock">
+<div class="content">
+<pre>$ bin/flume-ng agent --conf conf --conf-file example.conf --name a1 -DNC_PORT=44444</pre>
+</div>
+</div>
+<div class="paragraph">
+<p>Furthermore, because multiple configuration files are allowed the first file could contain:</p>
+</div>
+<div class="literalblock">
+<div class="content">
+<pre>a1.sources = r1
+a1.sources.r1.type = netcat
+a1.sources.r1.bind = 0.0.0.0
+a1.sources.r1.port = ${NC_PORT}
+a1.sources.r1.channels = c1</pre>
+</div>
+</div>
+<div class="paragraph">
+<p>and the override file could contain:</p>
+</div>
+<div class="literalblock">
+<div class="content">
+<pre>NC_PORT = 44444</pre>
+</div>
+</div>
+<div class="paragraph">
+<p>In this case the startup command could be:</p>
+</div>
+<div class="literalblock">
+<div class="content">
+<pre>$ bin/flume-ng agent --conf conf --conf-file example.conf --conf-file override.conf --name a1</pre>
+</div>
+</div>
+<div class="paragraph">
+<p>Note that the method for specifying environment variables as was done in prior versions will stil work
+but has been deprecated in favor of using ${env:varName}.</p>
+</div>
+</div>
+<div class="sect3">
+<h4 id="_using_a_command_options_file"><a class="anchor" href="#_using_a_command_options_file"></a>Using a command options file</h4>
+<div class="paragraph">
+<p>Instead of specifying all the command options on the command line as of version 1.10.0 command
+options may be placed in either /etc/flume/flume.opts or flume.opts on the classpath. An example
+might be:</p>
+</div>
+<div class="literalblock">
+<div class="content">
+<pre>conf-file = example.conf
+conf-file = override.conf
+name = a1</pre>
+</div>
+</div>
+</div>
+<div class="sect3">
+<h4 id="_logging_raw_data"><a class="anchor" href="#_logging_raw_data"></a>Logging raw data</h4>
+<div class="paragraph">
+<p>Logging the raw stream of data flowing through the ingest pipeline is not desired behavior in
+many production environments because this may result in leaking sensitive data or security related
+configurations, such as secret keys, to Flume log files.
+By default, Flume will not log such information. On the other hand, if the data pipeline is broken,
+Flume will attempt to provide clues for debugging the problem.</p>
+</div>
+<div class="paragraph">
+<p>One way to debug problems with event pipelines is to set up an additional <a href="#memory-channel">Memory Channel</a>
+connected to a <a href="#logger-sink">Logger Sink</a>, which will output all event data to the Flume logs.
+In some situations, however, this approach is insufficient.</p>
+</div>
+<div class="paragraph">
+<p>In order to enable logging of event- and configuration-related data, some Java system properties
+must be set in addition to log4j properties.</p>
+</div>
+<div class="paragraph">
+<p>To enable configuration-related logging, set the Java system property
+<code>-Dorg.apache.flume.log.printconfig=true</code>. This can either be passed on the command line or by
+setting this in the <code>JAVA_OPTS</code> variable in <em>flume-env.sh</em>.</p>
+</div>
+<div class="paragraph">
+<p>To enable data logging, set the Java system property <code>-Dorg.apache.flume.log.rawdata=true</code>
+in the same way described above. For most components, the log4j logging level must also be set to
+DEBUG or TRACE to make event-specific logging appear in the Flume logs.</p>
+</div>
+<div class="paragraph">
+<p>Here is an example of enabling both configuration logging and raw data logging:</p>
+</div>
+<div class="literalblock">
+<div class="content">
+<pre>$ bin/flume-ng agent --conf conf --conf-file example.conf --name a1 -Dorg.apache.flume.log.printconfig=true -Dorg.apache.flume.log.rawdata=true</pre>
+</div>
+</div>
+</div>
+<div class="sect3">
+<h4 id="_zookeeper_based_configuration"><a class="anchor" href="#_zookeeper_based_configuration"></a>Zookeeper based Configuration</h4>
+<div class="paragraph">
+<p>Flume supports Agent configurations via Zookeeper. <em>This is an experimental feature.</em> The configuration file needs to be uploaded
+in the Zookeeper, under a configurable prefix. The configuration file is stored in Zookeeper Node data.
+Following is how the Zookeeper Node tree would look like for agents a1 and a2</p>
+</div>
+<div class="listingblock">
+<div class="content">
+<pre class="highlightjs highlight"><code class="language-properties hljs" data-lang="properties">- /flume
+ |- /a1 [Agent config file]
+ |- /a2 [Agent config file]</code></pre>
+</div>
+</div>
+<div class="paragraph">
+<p>Once the configuration file is uploaded, start the agent with following options</p>
+</div>
+<div class="quoteblock">
+<blockquote>
+<div class="paragraph">
+<p>$ bin/flume-ng agent --conf conf -z zkhost:2181,zkhost1:2181 -p /flume --name a1</p>
+</div>
+</blockquote>
+</div>
+<table class="tableblock frame-all grid-all stretch">
+<colgroup>
+<col style="width: 33.3333%;">
+<col style="width: 33.3333%;">
+<col style="width: 33.3334%;">
+</colgroup>
+<thead>
+<tr>
+<th class="tableblock halign-left valign-top">Argument Name</th>
+<th class="tableblock halign-left valign-top">Default</th>
+<th class="tableblock halign-left valign-top">Description</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock"><strong>z</strong></p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Zookeeper connection string. Comma separated list of hostname:port</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock"><strong>p</strong></p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">/flume</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Base Path in Zookeeper to store Agent configurations</p></td>
+</tr>
+</tbody>
+</table>
+</div>
+<div class="sect3">
+<h4 id="_installing_third_party_plugins"><a class="anchor" href="#_installing_third_party_plugins"></a>Installing third-party plugins</h4>
+<div class="paragraph">
+<p>Flume has a fully plugin-based architecture. While Flume ships with many
+out-of-the-box sources, channels, sinks, serializers, and the like, many
+implementations exist which ship separately from Flume.</p>
+</div>
+<div class="paragraph">
+<p>While it has always been possible to include custom Flume components by
+adding their jars to the FLUME_CLASSPATH variable in the flume-env.sh file,
+Flume now supports a special directory called <code>plugins.d</code> which automatically
+picks up plugins that are packaged in a specific format. This allows for easier
+management of plugin packaging issues as well as simpler debugging and
+troubleshooting of several classes of issues, especially library dependency
+conflicts.</p>
+</div>
+<div class="sect4">
+<h5 id="_the_plugins_d_directory"><a class="anchor" href="#_the_plugins_d_directory"></a>The plugins.d directory</h5>
+<div class="paragraph">
+<p>The <code>plugins.d</code> directory is located at <code>$FLUME_HOME/plugins.d</code>. At startup
+time, the <code>flume-ng</code> start script looks in the <code>plugins.d</code> directory for
+plugins that conform to the below format and includes them in proper paths when
+starting up <code>java</code>.</p>
+</div>
+</div>
+<div class="sect4">
+<h5 id="_directory_layout_for_plugins"><a class="anchor" href="#_directory_layout_for_plugins"></a>Directory layout for plugins</h5>
+<div class="paragraph">
+<p>Each plugin (subdirectory) within <code>plugins.d</code> can have up to three
+sub-directories:</p>
+</div>
+<div class="olist arabic">
+<ol class="arabic">
+<li>
+<p>lib - the plugin&#8217;s jar(s)</p>
+</li>
+<li>
+<p>libext - the plugin&#8217;s dependency jar(s)</p>
+</li>
+<li>
+<p>native - any required native libraries, such as <code>.so</code> files</p>
+</li>
+</ol>
+</div>
+<div class="paragraph">
+<p>Example of two plugins within the plugins.d directory:</p>
+</div>
+<div class="listingblock">
+<div class="content">
+<pre class="highlightjs highlight"><code class="language-none hljs" data-lang="none">plugins.d/
+plugins.d/custom-source-1/
+plugins.d/custom-source-1/lib/my-source.jar
+plugins.d/custom-source-1/libext/spring-core-2.5.6.jar
+plugins.d/custom-source-2/
+plugins.d/custom-source-2/lib/custom.jar
+plugins.d/custom-source-2/native/gettext.so</code></pre>
+</div>
+</div>
+</div>
+</div>
+</div>
+<div class="sect2">
+<h3 id="_data_ingestion"><a class="anchor" href="#_data_ingestion"></a>Data ingestion</h3>
+<div class="paragraph">
+<p>Flume supports a number of mechanisms to ingest data from external sources.</p>
+</div>
+<div class="sect3">
+<h4 id="_rpc"><a class="anchor" href="#_rpc"></a>RPC</h4>
+<div class="paragraph">
+<p>An Avro client included in the Flume distribution can send a given file to
+Flume Avro source using avro RPC mechanism:</p>
+</div>
+<div class="literalblock">
+<div class="content">
+<pre>$ bin/flume-ng avro-client -H localhost -p 41414 -F /usr/logs/log.10</pre>
+</div>
+</div>
+<div class="paragraph">
+<p>The above command will send the contents of /usr/logs/log.10 to to the Flume
+source listening on that ports.</p>
+</div>
+</div>
+<div class="sect3">
+<h4 id="_executing_commands"><a class="anchor" href="#_executing_commands"></a>Executing commands</h4>
+<div class="paragraph">
+<p>There&#8217;s an exec source that executes a given command and consumes the output. A
+single 'line' of output ie. text followed by carriage return ('\r') or line
+feed ('\n') or both together.</p>
+</div>
+</div>
+<div class="sect3">
+<h4 id="_network_streams"><a class="anchor" href="#_network_streams"></a>Network streams</h4>
+<div class="paragraph">
+<p>Flume supports the following mechanisms to read data from popular log stream
+types, such as:</p>
+</div>
+<div class="olist arabic">
+<ol class="arabic">
+<li>
+<p>Avro</p>
+</li>
+<li>
+<p>Thrift</p>
+</li>
+<li>
+<p>Syslog</p>
+</li>
+<li>
+<p>Netcat</p>
+</li>
+</ol>
+</div>
+</div>
+</div>
+<div class="sect2">
+<h3 id="_setting_multi_agent_flow"><a class="anchor" href="#_setting_multi_agent_flow"></a>Setting multi-agent flow</h3>
+<div class="imageblock">
+<div class="content">
+<img src="_images/UserGuide_image03.png" alt="Two agents communicating over Avro RPC">
+</div>
+</div>
+<div class="paragraph">
+<p>In order to flow the data across multiple agents or hops, the sink of the
+previous agent and source of the current hop need to be avro type with the sink
+pointing to the hostname (or IP address) and port of the source.</p>
+</div>
+</div>
+<div class="sect2">
+<h3 id="_consolidation"><a class="anchor" href="#_consolidation"></a>Consolidation</h3>
+<div class="paragraph">
+<p>A very common scenario in log collection is a large number of log producing
+clients sending data to a few consumer agents that are attached to the storage
+subsystem. For example, logs collected from hundreds of web servers sent to a
+dozen of agents that write to HDFS cluster.</p>
+</div>
+<div class="imageblock">
+<div class="content">
+<img src="_images/UserGuide_image02.png" alt="A fan-in flow using Avro RPC to consolidate events in one place">
+</div>
+</div>
+<div class="paragraph">
+<p>This can be achieved in Flume by configuring a number of first tier agents with
+an avro sink, all pointing to an avro source of single agent (Again you could
+use the thrift sources/sinks/clients in such a scenario). This source
+on the second tier agent consolidates the received events into a single
+channel which is consumed by a sink to its final destination.</p>
+</div>
+</div>
+<div class="sect2">
+<h3 id="_multiplexing_the_flow"><a class="anchor" href="#_multiplexing_the_flow"></a>Multiplexing the flow</h3>
+<div class="paragraph">
+<p>Flume supports multiplexing the event flow to one or more destinations. This is
+achieved by defining a flow multiplexer that can replicate or selectively route
+an event to one or more channels.</p>
+</div>
+<div class="imageblock">
+<div class="content">
+<img src="_images/UserGuide_image01.png" alt="A fan-out flow using a (multiplexing) channel selector">
+</div>
+</div>
+<div class="paragraph">
+<p>The above example shows a source from agent "foo" fanning out the flow to three
+different channels. This fan out can be replicating or multiplexing. In case of
+replicating flow, each event is sent to all three channels. For the
+multiplexing case, an event is delivered to a subset of available channels when
+an event&#8217;s attribute matches a preconfigured value. For example, if an event
+attribute called "txnType" is set to "customer", then it should go to channel1
+and channel3, if it&#8217;s "vendor" then it should go to channel2, otherwise
+channel3. The mapping can be set in the agent&#8217;s configuration file.</p>
+</div>
+</div>
+</div>
+</div>
+<div class="sect1">
+<h2 id="_spring_boot_setup"><a class="anchor" href="#_spring_boot_setup"></a>Spring Boot Setup</h2>
+<div class="sectionbody">
+<div class="paragraph">
+<p>Apache Flume provides the flume-spring-boot module to provide support for packaging and
+configuring an application using Spring Boot. Version 2.0.0 or greater of flume-spring-boot
+should be used.</p>
+</div>
+<div class="paragraph">
+<p>Flume "normally" follows a paradigm where each component implements the Configurable
+interface and must implement the configure method to configure itself by retrieving
+the configuration attributes from its Context&#8217;s properties. In contrast,
+Spring and Spring Boot normally rely on dependency injection. where the configuration
+values are set into the object being configured either as constructor parameters or
+via setter methods.</p>
+</div>
+<div class="paragraph">
+<p>Flume&#8217;s integration with Spring Boot provides support to configure Flume components
+in the application&#8217;s properties via the normal application.yml. Unlike Flume&#8217;s
+default method of configuring however, only component attributes are specified in
+the application.yml. Wiring components together, and thus defining the flows, is
+handled via Spring&#8217;s Java configuration.</p>
+</div>
+<div class="sect2">
+<h3 id="_creating_the_application"><a class="anchor" href="#_creating_the_application"></a>Creating the application</h3>
+<div class="paragraph">
+<p>Flume&#8217;s Spring Boot support provides the main class to be configured with spring as
+<span class="title-ref">org.apache.flume.spring.boot.FlumeApplication</span>. Flume applications that use Spring
+Boot should configure the Spring Boot Maven plugin with that as the main class as in:</p>
+</div>
+<div class="literalblock">
+<div class="content">
+<pre>&lt;execution&gt;
+  &lt;id&gt;repackage&lt;/id&gt;
+  &lt;goals&gt;
+    &lt;goal&gt;repackage&lt;/goal&gt;
+  &lt;/goals&gt;
+  &lt;configuration&gt;
+    &lt;executable&gt;true&lt;/executable&gt;
+    &lt;mainClass&gt;org.apache.flume.spring.boot.FlumeApplication&lt;/mainClass&gt;
+  &lt;/configuration&gt;
+&lt;/execution&gt;</pre>
+</div>
+</div>
+</div>
+<div class="sect2">
+<h3 id="_component_scanning"><a class="anchor" href="#_component_scanning"></a>Component Scanning</h3>
+<div class="paragraph">
+<p>Spring Boot will automatically locate all the Spring components provided by Flume.
+However, in order for the Flume application to be configured Spring needs the
+configuration and package names used by the application in order for Spring to locate
+these components.This is accomplished in the application by providing a META-INF/spring.factories
+file that enables auto configuration for one class that will then provide the component
+scanning information for the rest of the application. For example:</p>
+</div>
+<div class="paragraph">
+<p>META-INF/spring.factories:</p>
+</div>
+<div class="literalblock">
+<div class="content">
+<pre>org.springframework.boot.autoconfigure.EnableAutoConfiguration=com.sample.myapp.config.AppConfig</pre>
+</div>
+</div>
+<div class="paragraph">
+<p>com.sample.config.AppConfig.java:</p>
+</div>
+<div class="literalblock">
+<div class="content">
+<pre>package com.sample.myapp.config;
+
+import org.springframework.context.annotation.ComponentScan;
+import org.springframework.context.annotation.Configuration;
+
+@Configuration
+@ComponentScan(basePackages="com.sample.myapp")
+public class MyConfiguration {
+
+}</pre>
+</div>
+</div>
+<div class="paragraph">
+<p>This will result in all classes in the <span class="title-ref">com.sample.myapp</span> package and
+it`s sub-packages being scanned by Spring for components to be included. Note
+that classes found there may also use Spring&#8217;s <span class="title-ref">@Import</span> annotation to include
+classes in other packages.</p>
+</div>
+</div>
+<div class="sect2">
+<h3 id="_component_wiring"><a class="anchor" href="#_component_wiring"></a>Component Wiring</h3>
+<div class="paragraph">
+<p>Flume&#8217;s Spring Boot support will automatically gather all the defined Channels,
+SourceRunners, and SinkRunners and start them. To do that they all must first
+be created as Spring Singletons using the Spring <span class="title-ref">@Bean</span> annotation in a class
+containing the <span class="title-ref">@Configuration</span> annotation on the class declaration and then
+initialize them just as the "normal" FlumeApplication class does. To define
+these components an application should provide a Configuration class that
+creates these Flume components. An example configuration that generates
+sequence numbers, writes them to a MemoryChannel and then consumes these events
+without publishing them anywhere would look like:</p>
+</div>
+<div class="literalblock">
+<div class="content">
+<pre>@Configuration
+@ComponentScan(basePackages="com.sample.myapp")
+public class AppConfig extends AbstractFlumeConfiguration {
+
+  @Bean
+  @ConfigurationProperties(prefix = "flume.sources.source1")
+  public Map&lt;String, String&gt; source1Properties() {
+    return new HashMap&lt;&gt;();
+  }
+
+  @Bean
+  @ConfigurationProperties(prefix = "flume.channels.channel1")
+  public Map&lt;String, String&gt; channel1Properties() {
+    return new HashMap&lt;&gt;();
+  }
+
+  @Bean
+  public Channel memoryChannel(Map&lt;String, String&gt; channel1Properties) {
+    return configureChannel("channel1", MemoryChannel.class, channel1Properties);
+  }
+
+  @Bean
+  public SourceRunner seqSource(Channel memoryChannel, Map&lt;String, String&gt; source1Properties) {
+    ChannelSelector selector = new ReplicatingChannelSelector();
+    selector.setChannels(listOf(memoryChannel));
+    return configureSource("source1", SequenceGeneratorSource.class, selector,
+        source1Properties);
+  }
+
+  @Bean
+  public SinkRunner nullSink(Channel memoryChannel) {
+    Sink sink = configureSink("null", NullSink.class, memoryChannel,null);
+    return createSinkRunner(configureSinkProcessor(null, DefaultSinkProcessor.class,
+        listOf(sink)));
+  }
+}</pre>
+</div>
+</div>
+<div class="paragraph">
+<p>The configuration for this configuration might look like:</p>
+</div>
+<div class="literalblock">
+<div class="content">
+<pre>spring:
+  application:
+    name: flume-test
+
+server:
+  port: 41414
+
+flume:
+  metrics: http
+  sources:
+    source1:
+      totalEvents: 10000
+  channels:
+    channel1:
+      capacity: 10000</pre>
+</div>
+</div>
+<div class="paragraph">
+<p>This would result in an application named "flume-test" that listens on port 41414 for the /metrics endpoint.
+10,000 events would be written to the channel. These events would be consumed by the NullSink. The Configuration
+class should extend AbstractFlumeConfiguration, as shown here, to be able to use the helper classes that
+construct the appropriate Flume components.</p>
+</div>
+<div class="paragraph">
+<p>Note that all Spring Boot facilities are available to Flume applications configured this way.</p>
+</div>
+<div class="paragraph">
+<p>SinkGroups and Sinks may also be configured in a similar fashion as in:</p>
+</div>
+<div class="literalblock">
+<div class="content">
+<pre>flume:
+  sinkGroups:
+    rrobin:
+       backoff: true
+       selector: round_robin
+       "selector.maxTimeOut": 30000
+
+  sinks:
+    avroSinks:
+      avroSink1:
+        hostname: 192.168.10.10
+        port: 4141
+        batch-size: 100
+        compression-type: deflate
+      avroSink2:
+        hostname: 192.168.10.11
+        port: 4141
+        batch-size: 100
+        compression-type: deflate</pre>
+</div>
+</div>
+<div class="paragraph">
+<p>These would be configured in the Java config with:</p>
+</div>
+<div class="literalblock">
+<div class="content">
+<pre>@Bean
+@ConfigurationProperties(prefix = "flume.sink-groups.rrobin")
+public Map&lt;String, String&gt; rrobinProperties() {
+    return new HashMap&lt;&gt;();
+}
+
+@Bean
+@ConfigurationProperties(prefix = "flume.sinks.avro-sinks")
+public Map&lt;String, AvroSinkConfiguration&gt; avroSinksProperties() {
+    return new HashMap&lt;&gt;();
+}
+
+@Bean
+public List&lt;Sink&gt; avroSinks(final Channel avroFileChannel,
+    final Map&lt;String, AvroSinkConfiguration&gt; avroSinksProperties) {
+    List&lt;Sink&gt; sinks = new ArrayList&lt;&gt;();
+    for (Map.Entry&lt;String, AvroSinkConfiguration&gt; entry : avroSinksProperties.entrySet()) {
+        sinks.add(configureSink(entry.getKey(), AvroSink.class, avroFileChannel,
+            entry.getValue().getProperties()));
+    }
+    return sinks;
+}
+
+@Bean
+public SinkRunner avroSinkRunner(final Map&lt;String, String&gt; rrobinProperties, final List&lt;Sink&gt; avroSinks) {
+    return createSinkRunner(
+        configureSinkProcessor(rrobinProperties, LoadBalancingSinkProcessor.class, avroSinks));
+}</pre>
+</div>
+</div>
+<div class="paragraph">
+<p>Note that the attribute names specified for the sources, channels, and sink groups must match the attribute names
+specified for the components in other portions of this documentation.</p>
+</div>
+<div class="paragraph">
+<p>It is important to note that a concrete class was used to capture the data for the Avro Sinks. When a simple Map
+is used Spring seems to get confused with the nested Maps. The <span class="title-ref">AvroSinkConfiguration</span> class would look like:</p>
+</div>
+<div class="literalblock">
+<div class="content">
+<pre>public class AvroSinkConfiguration {
+
+   private String hostName;
+   private int port;
+   private int batchSize;
+   private String compressionType;
+
+   public String getHostName() {
+      return hostName;
+   }
+
+   public void setHostName(String hostName) {
+      this.hostName = hostName;
+   }
+
+   public int getPort() {
+      return port;
+   }
+
+   public void setPort(int port) {
+      this.port = port;
+   }
+
+   public int getBatchSize() {
+      return batchSize;
+   }
+
+   public void setBatchSize(int batchSize) {
+      this.batchSize = batchSize;
+   }
+
+   public String getCompressionType() {
+      return compressionType;
+   }
+
+   public void setCompressionType(String compressionType) {
+      this.compressionType = compressionType;
+   }
+
+   public Map&lt;String, String&gt; getProperties() {
+      Map&lt;String, String&gt; map = new HashMap&lt;&gt;();
+      map.put("hostname", hostName);
+      map.put("port", Integer.toString(port));
+      map.put("batchSize", Integer.toString(batchSize));
+      map.put(compressionType, compressionType);
+      return map;
+   }
+}</pre>
+</div>
+</div>
+</div>
+</div>
+</div>
+<div class="sect1">
+<h2 id="_configuration"><a class="anchor" href="#_configuration"></a>Configuration</h2>
+<div class="sectionbody">
+<div class="paragraph">
+<p>As mentioned in the earlier section, standard Flume agent configuration is
+read from a file that resembles a Java property file format with hierarchical property
+settings.</p>
+</div>
+<div class="sect2">
+<h3 id="_defining_the_flow"><a class="anchor" href="#_defining_the_flow"></a>Defining the flow</h3>
+<div class="paragraph">
+<p>To define the flow within a single agent, you need to link the sources and
+sinks via a channel. You need to list the sources, sinks and channels for the
+given agent, and then point the source and sink to a channel. A source instance
+can specify multiple channels, but a sink instance can only specify one channel.
+The format is as follows:</p>
+</div>
+<div class="listingblock">
+<div class="content">
+<pre class="highlightjs highlight"><code class="language-properties hljs" data-lang="properties"># list the sources, sinks and channels for the agent
+&lt;Agent&gt;.sources = &lt;Source&gt;
+&lt;Agent&gt;.sinks = &lt;Sink&gt;
+&lt;Agent&gt;.channels = &lt;Channel1&gt; &lt;Channel2&gt;
+
+# set channel for source
+&lt;Agent&gt;.sources.&lt;Source&gt;.channels = &lt;Channel1&gt; &lt;Channel2&gt; ...
+
+# set channel for sink
+&lt;Agent&gt;.sinks.&lt;Sink&gt;.channel = &lt;Channel1&gt;</code></pre>
+</div>
+</div>
+<div class="paragraph">
+<p>For example, an agent named agent_foo is reading data from an external avro client and sending
+it to HDFS via a memory channel. The config file weblog.config could look like:</p>
+</div>
+<div class="listingblock">
+<div class="content">
+<pre class="highlightjs highlight"><code class="language-properties hljs" data-lang="properties"># list the sources, sinks and channels for the agent
+agent_foo.sources = avro-appserver-src-1
+agent_foo.sinks = hdfs-sink-1
+agent_foo.channels = mem-channel-1
+
+# set channel for source
+agent_foo.sources.avro-appserver-src-1.channels = mem-channel-1
+
+# set channel for sink
+agent_foo.sinks.hdfs-sink-1.channel = mem-channel-1</code></pre>
+</div>
+</div>
+<div class="paragraph">
+<p>This will make the events flow from avro-AppSrv-source to hdfs-Cluster1-sink
+through the memory channel mem-channel-1. When the agent is started with the
+weblog.config as its config file, it will instantiate that flow.</p>
+</div>
+</div>
+<div class="sect2">
+<h3 id="_configuring_individual_components_2"><a class="anchor" href="#_configuring_individual_components_2"></a>Configuring individual components</h3>
+<div class="paragraph">
+<p>After defining the flow, you need to set properties of each source, sink and
+channel. This is done in the same hierarchical namespace fashion where you set
+the component type and other values for the properties specific to each
+component:</p>
+</div>
+<div class="listingblock">
+<div class="content">
+<pre class="highlightjs highlight"><code class="language-properties hljs" data-lang="properties"># properties for sources
+&lt;Agent&gt;.sources.&lt;Source&gt;.&lt;someProperty&gt; = &lt;someValue&gt;
+
+# properties for channels
+&lt;Agent&gt;.channels.&lt;Channel&gt;.&lt;someProperty&gt; = &lt;someValue&gt;
+
+# properties for sinks
+&lt;Agent&gt;.sinks.&lt;Sink&gt;.&lt;someProperty&gt; = &lt;someValue&gt;</code></pre>
+</div>
+</div>
+<div class="paragraph">
+<p>The property "type" needs to be set for each component for Flume to understand
+what kind of object it needs to be. Each source, sink and channel type has its
+own set of properties required for it to function as intended. All those need
+to be set as needed. In the previous example, we have a flow from
+avro-AppSrv-source to hdfs-Cluster1-sink through the memory channel
+mem-channel-1. Here&#8217;s an example that shows configuration of each of those
+components:</p>
+</div>
+<div class="listingblock">
+<div class="content">
+<pre class="highlightjs highlight"><code class="language-properties hljs" data-lang="properties">agent_foo.sources = avro-AppSrv-source
+agent_foo.sinks = hdfs-Cluster1-sink
+agent_foo.channels = mem-channel-1
+
+# set channel for sources, sinks
+
+# properties of avro-AppSrv-source
+agent_foo.sources.avro-AppSrv-source.type = avro
+agent_foo.sources.avro-AppSrv-source.bind = localhost
+agent_foo.sources.avro-AppSrv-source.port = 10000
+
+# properties of mem-channel-1
+agent_foo.channels.mem-channel-1.type = memory
+agent_foo.channels.mem-channel-1.capacity = 1000
+agent_foo.channels.mem-channel-1.transactionCapacity = 100
+
+# properties of hdfs-Cluster1-sink
+agent_foo.sinks.hdfs-Cluster1-sink.type = hdfs
+agent_foo.sinks.hdfs-Cluster1-sink.hdfs.path = hdfs://namenode/flume/webdata
+
+#...</code></pre>
+</div>
+</div>
+</div>
+<div class="sect2">
+<h3 id="_adding_multiple_flows_in_an_agent"><a class="anchor" href="#_adding_multiple_flows_in_an_agent"></a>Adding multiple flows in an agent</h3>
+<div class="paragraph">
+<p>A single Flume agent can contain several independent flows. You can list
+multiple sources, sinks and channels in a config. These components can be
+linked to form multiple flows:</p>
+</div>
+<div class="listingblock">
+<div class="content">
+<pre class="highlightjs highlight"><code class="language-properties hljs" data-lang="properties"># list the sources, sinks and channels for the agent
+&lt;Agent&gt;.sources = &lt;Source1&gt; &lt;Source2&gt;
+&lt;Agent&gt;.sinks = &lt;Sink1&gt; &lt;Sink2&gt;
+&lt;Agent&gt;.channels = &lt;Channel1&gt; &lt;Channel2&gt;</code></pre>
+</div>
+</div>
+<div class="paragraph">
+<p>Then you can link the sources and sinks to their corresponding channels (for
+sources) or channel (for sinks) to setup two different flows. For example, if
+you need to setup two flows in an agent, one going from an external avro client
+to external HDFS and another from output of a tail to avro sink, then here&#8217;s a
+config to do that:</p>
+</div>
+<div class="listingblock">
+<div class="content">
+<pre class="highlightjs highlight"><code class="language-properties hljs" data-lang="properties"># list the sources, sinks and channels in the agent
+agent_foo.sources = avro-AppSrv-source1 exec-tail-source2
+agent_foo.sinks = hdfs-Cluster1-sink1 avro-forward-sink2
+agent_foo.channels = mem-channel-1 file-channel-2
+
+# flow #1 configuration
+agent_foo.sources.avro-AppSrv-source1.channels = mem-channel-1
+agent_foo.sinks.hdfs-Cluster1-sink1.channel = mem-channel-1
+
+# flow #2 configuration
+agent_foo.sources.exec-tail-source2.channels = file-channel-2
+agent_foo.sinks.avro-forward-sink2.channel = file-channel-2</code></pre>
+</div>
+</div>
+</div>
+<div class="sect2">
+<h3 id="_configuring_a_multi_agent_flow"><a class="anchor" href="#_configuring_a_multi_agent_flow"></a>Configuring a multi agent flow</h3>
+<div class="paragraph">
+<p>To setup a multi-tier flow, you need to have an avro/thrift sink of first hop
+pointing to avro/thrift source of the next hop. This will result in the first
+Flume agent forwarding events to the next Flume agent. For example, if you are
+periodically sending files (1 file per event) using avro client to a local
+Flume agent, then this local agent can forward it to another agent that has the
+mounted for storage.</p>
+</div>
+<div class="paragraph">
+<p>Weblog agent config:</p>
+</div>
+<div class="listingblock">
+<div class="content">
+<pre class="highlightjs highlight"><code class="language-properties hljs" data-lang="properties"># list sources, sinks and channels in the agent
+agent_foo.sources = avro-AppSrv-source
+agent_foo.sinks = avro-forward-sink
+agent_foo.channels = file-channel
+
+# define the flow
+agent_foo.sources.avro-AppSrv-source.channels = file-channel
+agent_foo.sinks.avro-forward-sink.channel = file-channel
+
+# avro sink properties
+agent_foo.sinks.avro-forward-sink.type = avro
+agent_foo.sinks.avro-forward-sink.hostname = 10.1.1.100
+agent_foo.sinks.avro-forward-sink.port = 10000
+
+# configure other pieces
+#...</code></pre>
+</div>
+</div>
+<div class="paragraph">
+<p>HDFS agent config:</p>
+</div>
+<div class="listingblock">
+<div class="content">
+<pre class="highlightjs highlight"><code class="language-properties hljs" data-lang="properties"># list sources, sinks and channels in the agent
+agent_foo.sources = avro-collection-source
+agent_foo.sinks = hdfs-sink
+agent_foo.channels = mem-channel
+
+# define the flow
+agent_foo.sources.avro-collection-source.channels = mem-channel
+agent_foo.sinks.hdfs-sink.channel = mem-channel
+
+# avro source properties
+agent_foo.sources.avro-collection-source.type = avro
+agent_foo.sources.avro-collection-source.bind = 10.1.1.100
+agent_foo.sources.avro-collection-source.port = 10000
+
+# configure other pieces
+#...</code></pre>
+</div>
+</div>
+<div class="paragraph">
+<p>Here we link the avro-forward-sink from the weblog agent to the
+avro-collection-source of the hdfs agent. This will result in the events coming
+from the external appserver source eventually getting stored in HDFS.</p>
+</div>
+</div>
+<div class="sect2">
+<h3 id="_fan_out_flow"><a class="anchor" href="#_fan_out_flow"></a>Fan out flow</h3>
+<div class="paragraph">
+<p>As discussed in previous section, Flume supports fanning out the flow from one
+source to multiple channels. There are two modes of fan out, replicating and
+multiplexing. In the replicating flow, the event is sent to all the configured
+channels. In case of multiplexing, the event is sent to only a subset of
+qualifying channels. To fan out the flow, one needs to specify a list of
+channels for a source and the policy for the fanning it out. This is done by
+adding a channel "selector" that can be replicating or multiplexing. Then
+further specify the selection rules if it&#8217;s a multiplexer. If you don&#8217;t specify
+a selector, then by default it&#8217;s replicating:</p>
+</div>
+<div class="listingblock">
+<div class="content">
+<pre class="highlightjs highlight"><code class="language-properties hljs" data-lang="properties"># List the sources, sinks and channels for the agent
+&lt;Agent&gt;.sources = &lt;Source1&gt;
+&lt;Agent&gt;.sinks = &lt;Sink1&gt; &lt;Sink2&gt;
+&lt;Agent&gt;.channels = &lt;Channel1&gt; &lt;Channel2&gt;
+
+# set list of channels for source (separated by space)
+&lt;Agent&gt;.sources.&lt;Source1&gt;.channels = &lt;Channel1&gt; &lt;Channel2&gt;
+
+# set channel for sinks
+&lt;Agent&gt;.sinks.&lt;Sink1&gt;.channel = &lt;Channel1&gt;
+&lt;Agent&gt;.sinks.&lt;Sink2&gt;.channel = &lt;Channel2&gt;
+
+&lt;Agent&gt;.sources.&lt;Source1&gt;.selector.type = replicating</code></pre>
+</div>
+</div>
+<div class="paragraph">
+<p>The multiplexing select has a further set of properties to bifurcate the flow.
+This requires specifying a mapping of an event attribute to a set for channel.
+The selector checks for each configured attribute in the event header. If it
+matches the specified value, then that event is sent to all the channels mapped
+to that value. If there&#8217;s no match, then the event is sent to set of channels
+configured as default:</p>
+</div>
+<div class="listingblock">
+<div class="content">
+<pre class="highlightjs highlight"><code class="language-properties hljs" data-lang="properties"># Mapping for multiplexing selector
+&lt;Agent&gt;.sources.&lt;Source1&gt;.selector.type = multiplexing
+&lt;Agent&gt;.sources.&lt;Source1&gt;.selector.header = &lt;someHeader&gt;
+&lt;Agent&gt;.sources.&lt;Source1&gt;.selector.mapping.&lt;Value1&gt; = &lt;Channel1&gt;
+&lt;Agent&gt;.sources.&lt;Source1&gt;.selector.mapping.&lt;Value2&gt; = &lt;Channel1&gt; &lt;Channel2&gt;
+&lt;Agent&gt;.sources.&lt;Source1&gt;.selector.mapping.&lt;Value3&gt; = &lt;Channel2&gt;
+#...
+
+&lt;Agent&gt;.sources.&lt;Source1&gt;.selector.default = &lt;Channel2&gt;</code></pre>
+</div>
+</div>
+<div class="paragraph">
+<p>The mapping allows overlapping the channels for each value.</p>
+</div>
+<div class="paragraph">
+<p>The following example has a single flow that multiplexed to two paths. The
+agent named agent_foo has a single avro source and two channels linked to two sinks:</p>
+</div>
+<div class="listingblock">
+<div class="content">
+<pre class="highlightjs highlight"><code class="language-properties hljs" data-lang="properties"># list the sources, sinks and channels in the agent
+agent_foo.sources = avro-AppSrv-source1
+agent_foo.sinks = hdfs-Cluster1-sink1 avro-forward-sink2
+agent_foo.channels = mem-channel-1 file-channel-2
+
+# set channels for source
+agent_foo.sources.avro-AppSrv-source1.channels = mem-channel-1 file-channel-2
+
+# set channel for sinks
+agent_foo.sinks.hdfs-Cluster1-sink1.channel = mem-channel-1
+agent_foo.sinks.avro-forward-sink2.channel = file-channel-2
+
+# channel selector configuration
+agent_foo.sources.avro-AppSrv-source1.selector.type = multiplexing
+agent_foo.sources.avro-AppSrv-source1.selector.header = State
+agent_foo.sources.avro-AppSrv-source1.selector.mapping.CA = mem-channel-1
+agent_foo.sources.avro-AppSrv-source1.selector.mapping.AZ = file-channel-2
+agent_foo.sources.avro-AppSrv-source1.selector.mapping.NY = mem-channel-1 file-channel-2
+agent_foo.sources.avro-AppSrv-source1.selector.default = mem-channel-1</code></pre>
+</div>
+</div>
+<div class="paragraph">
+<p>The selector checks for a header called "State". If the value is "CA" then its
+sent to mem-channel-1, if its "AZ" then it goes to file-channel-2 or if its
+"NY" then both. If the "State" header is not set or doesn&#8217;t match any of the
+three, then it goes to mem-channel-1 which is designated as 'default'.</p>
+</div>
+<div class="paragraph">
+<p>The selector also supports optional channels. To specify optional channels for
+a header, the config parameter 'optional' is used in the following way:</p>
+</div>
+<div class="listingblock">
+<div class="content">
+<pre class="highlightjs highlight"><code class="language-properties hljs" data-lang="properties"># channel selector configuration
+agent_foo.sources.avro-AppSrv-source1.selector.type = multiplexing
+agent_foo.sources.avro-AppSrv-source1.selector.header = State
+agent_foo.sources.avro-AppSrv-source1.selector.mapping.CA = mem-channel-1
+agent_foo.sources.avro-AppSrv-source1.selector.mapping.AZ = file-channel-2
+agent_foo.sources.avro-AppSrv-source1.selector.mapping.NY = mem-channel-1 file-channel-2
+agent_foo.sources.avro-AppSrv-source1.selector.optional.CA = mem-channel-1 file-channel-2
+agent_foo.sources.avro-AppSrv-source1.selector.mapping.AZ = file-channel-2
+agent_foo.sources.avro-AppSrv-source1.selector.default = mem-channel-1</code></pre>
+</div>
+</div>
+<div class="paragraph">
+<p>The selector will attempt to write to the required channels first and will fail
+the transaction if even one of these channels fails to consume the events. The
+transaction is reattempted on <strong>all</strong> of the channels. Once all required
+channels have consumed the events, then the selector will attempt to write to
+the optional channels. A failure by any of the optional channels to consume the
+event is simply ignored and not retried.</p>
+</div>
+<div class="paragraph">
+<p>If there is an overlap between the optional channels and required channels for a
+specific header, the channel is considered to be required, and a failure in the
+channel will cause the entire set of required channels to be retried. For
+instance, in the above example, for the header "CA" mem-channel-1 is considered
+to be a required channel even though it is marked both as required and optional,
+and a failure to write to this channel will cause that
+event to be retried on <strong>all</strong> channels configured for the selector.</p>
+</div>
+<div class="paragraph">
+<p>Note that if a header does not have any required channels, then the event will
+be written to the default channels and will be attempted to be written to the
+optional channels for that header. Specifying optional channels will still cause
+the event to be written to the default channels, if no required channels are
+specified. If no channels are designated as default and there are no required,
+the selector will attempt to write the events to the optional channels. Any
+failures are simply ignored in that case.</p>
+</div>
+</div>
+<div class="sect2">
+<h3 id="_ssltls_support"><a class="anchor" href="#_ssltls_support"></a>SSL/TLS support</h3>
+<div class="paragraph">
+<p>Several Flume components support the SSL/TLS protocols in order to communicate with other systems
+securely.</p>
+</div>
+<table class="tableblock frame-all grid-all stretch">
+<colgroup>
+<col style="width: 50%;">
+<col style="width: 50%;">
+</colgroup>
+<thead>
+<tr>
+<th class="tableblock halign-left valign-top">Component</th>
+<th class="tableblock halign-left valign-top">SSL server or client</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Avro Source</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">server</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Avro Sink</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">client</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Thrift Source</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">server</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Thrift Sink</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">client</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Kafka Source</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">client</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Kafka Channel</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">client</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Kafka Sink</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">client</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">HTTP Source</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">server</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">JMS Source</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">client</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Syslog TCP Source</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">server</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Multiport Syslog TCP Source</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">server</p></td>
+</tr>
+</tbody>
+</table>
+<div class="paragraph">
+<p>The SSL compatible components have several configuration parameters to set up SSL, like
+enable SSL flag, keystore / truststore parameters (location, password, type) and additional
+SSL parameters (eg. disabled protocols).</p>
+</div>
+<div class="paragraph">
+<p>Enabling SSL for a component is always specified at component level in the agent configuration file.
+So some components may be configured to use SSL while others not (even with the same component type).</p>
+</div>
+<div class="paragraph">
+<p>The keystore / truststore setup can be specified at component level or globally.</p>
+</div>
+<div class="paragraph">
+<p>In case of the component level setup, the keystore / truststore is configured in the agent
+configuration file through component specific parameters. The advantage of this method is that the
+components can use different keystores (if this would be needed). The disadvantage is that the
+keystore parameters must be copied for each component in the agent configuration file.
+The component level setup is optional, but if it is defined, it has higher precedence than
+the global parameters.</p>
+</div>
+<div class="paragraph">
+<p>With the global setup, it is enough to define the keystore / truststore parameters once
+and use the same settings for all components, which means less and more centralized configuration.</p>
+</div>
+<div class="paragraph">
+<p>The global setup can be configured either through system properties or through environment variables.</p>
+</div>
+<table class="tableblock frame-all grid-all stretch">
+<colgroup>
+<col style="width: 33.3333%;">
+<col style="width: 33.3333%;">
+<col style="width: 33.3334%;">
+</colgroup>
+<thead>
+<tr>
+<th class="tableblock halign-left valign-top">System property</th>
+<th class="tableblock halign-left valign-top">Environment variable</th>
+<th class="tableblock halign-left valign-top">Description</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">javax.net.ssl.keyStore</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">FLUME_SSL_KEYSTORE_PATH</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Keystore location</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">javax.net.ssl.keyStorePassword</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">FLUME_SSL_KEYSTORE_PASSWORD</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Keystore password</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">javax.net.ssl.keyStoreType</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">FLUME_SSL_KEYSTORE_TYPE</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Keystore type (by default JKS)</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">javax.net.ssl.trustStore</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">FLUME_SSL_TRUSTSTORE_PATH</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Truststore location</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">javax.net.ssl.trustStorePassword</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">FLUME_SSL_TRUSTSTORE_PASSWORD</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Truststore password</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">javax.net.ssl.trustStoreType</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">FLUME_SSL_TRUSTSTORE_TYPE</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Truststore type (by default JKS)</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">flume.ssl.include.protocols</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">FLUME_SSL_INCLUDE_PROTOCOLS</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Protocols to include when calculating enabled protocols. A comma (,) separated list.
+Excluded protocols will be excluded from this list if provided.</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">flume.ssl.exclude.protocols</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">FLUME_SSL_EXCLUDE_PROTOCOLS</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Protocols to exclude when calculating enabled protocols. A comma (,) separated list.</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">flume.ssl.include.cipherSuites</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">FLUME_SSL_INCLUDE_CIPHERSUITES</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Cipher suites to include when calculating enabled cipher suites. A comma (,) separated list.
+Excluded cipher suites will be excluded from this list if provided.</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">flume.ssl.exclude.cipherSuites</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">FLUME_SSL_EXCLUDE_CIPHERSUITES</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Cipher suites to exclude when calculating enabled cipher suites. A comma (,) separated list.</p></td>
+</tr>
+</tbody>
+</table>
+<div class="paragraph">
+<p>The SSL system properties can either be passed on the command line or by setting the <code>JAVA_OPTS</code>
+environment variable in <em>conf/flume-env.sh</em>. (Although, using the command line is inadvisable because
+the commands including the passwords will be saved to the command history.)</p>
+</div>
+<div class="listingblock">
+<div class="content">
+<pre class="highlightjs highlight"><code class="language-properties hljs" data-lang="properties">export JAVA_OPTS="$JAVA_OPTS -Djavax.net.ssl.keyStore=/path/to/keystore.jks"
+export JAVA_OPTS="$JAVA_OPTS -Djavax.net.ssl.keyStorePassword=password"</code></pre>
+</div>
+</div>
+<div class="paragraph">
+<p>Flume uses the system properties defined in JSSE (Java Secure Socket Extension), so this is
+a standard way for setting up SSL. On the other hand, specifying passwords in system properties
+means that the passwords can be seen in the process list. For cases where it is not acceptable,
+it is also be possible to define the parameters in environment variables. Flume initializes
+the JSSE system properties from the corresponding environment variables internally in this case.</p>
+</div>
+<div class="paragraph">
+<p>The SSL environment variables can either be set in the shell environment before
+starting Flume or in <em>conf/flume-env.sh</em>. (Although, using the command line is inadvisable because
+the commands including the passwords will be saved to the command history.)</p>
+</div>
+<div class="listingblock">
+<div class="content">
+<pre class="highlightjs highlight"><code class="language-properties hljs" data-lang="properties">export FLUME_SSL_KEYSTORE_PATH=/path/to/keystore.jks
+export FLUME_SSL_KEYSTORE_PASSWORD=password</code></pre>
+</div>
+</div>
+<div class="paragraph">
+<p><strong>Please note:</strong></p>
+</div>
+<div class="ulist">
+<ul>
+<li>
+<p>SSL must be enabled at component level. Specifying the global SSL parameters alone will not
+have any effect.</p>
+</li>
+<li>
+<p>If the global SSL parameters are specified at multiple levels, the priority is the
+following (from higher to lower):</p>
+<div class="ulist">
+<ul>
+<li>
+<p>component parameters in agent config</p>
+</li>
+<li>
+<p>system properties</p>
+</li>
+<li>
+<p>environment variables</p>
+</li>
+</ul>
+</div>
+</li>
+<li>
+<p>If SSL is enabled for a component, but the SSL parameters are not specified in any of the ways
+described above, then</p>
+<div class="ulist">
+<ul>
+<li>
+<p>in case of keystores: configuration error</p>
+</li>
+<li>
+<p>in case of truststores: the default truststore will be used (<code>jssecacerts</code> / <code>cacerts</code> in Oracle JDK)</p>
+</li>
+</ul>
+</div>
+</li>
+<li>
+<p>The trustore password is optional in all cases. If not specified, then no integrity check will be
+performed on the truststore when it is opened by the JDK.</p>
+</li>
+</ul>
+</div>
+</div>
+<div class="sect2">
+<h3 id="_source_and_sink_batch_sizes_and_channel_transaction_capacities"><a class="anchor" href="#_source_and_sink_batch_sizes_and_channel_transaction_capacities"></a>Source and sink batch sizes and channel transaction capacities</h3>
+<div class="paragraph">
+<p>Sources and sinks can have a batch size parameter that determines the maximum number of events they
+process in one batch. This happens within a channel transaction that has an upper limit called
+transaction capacity. Batch size must be smaller than the channel&#8217;s transaction capacity.
+There is an explicit check to prevent incompatible settings. This check happens
+whenever the configuration is read.</p>
+</div>
+</div>
+<div class="sect2">
+<h3 id="_flume_sources"><a class="anchor" href="#_flume_sources"></a>Flume Sources</h3>
+<div class="sect3">
+<h4 id="_avro_source"><a class="anchor" href="#_avro_source"></a>Avro Source</h4>
+<div class="paragraph">
+<p>Listens on Avro port and receives events from external Avro client streams.
+When paired with the built-in Avro Sink on another (previous hop) Flume agent,
+it can create tiered collection topologies.
+Required properties are in <strong>bold</strong>.</p>
+</div>
+<table class="tableblock frame-all grid-all stretch">
+<colgroup>
+<col style="width: 33.3333%;">
+<col style="width: 33.3333%;">
+<col style="width: 33.3334%;">
+</colgroup>
+<thead>
+<tr>
+<th class="tableblock halign-left valign-top">Property Name</th>
+<th class="tableblock halign-left valign-top">Default</th>
+<th class="tableblock halign-left valign-top">Description</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock"><strong>channels</strong></p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock"><strong>type</strong></p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">The component type name, needs to be <code>avro</code></p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock"><strong>bind</strong></p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">hostname or IP address to listen on</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock"><strong>port</strong></p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Port # to bind to</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">threads</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Maximum number of worker threads to spawn</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">selector.type</p></td>
+<td class="tableblock halign-left valign-top"></td>
+<td class="tableblock halign-left valign-top"></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">selector.*</p></td>
+<td class="tableblock halign-left valign-top"></td>
+<td class="tableblock halign-left valign-top"></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">interceptors</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Space-separated list of interceptors</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">interceptors.*</p></td>
+<td class="tableblock halign-left valign-top"></td>
+<td class="tableblock halign-left valign-top"></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">compression-type</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">none</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">This can be "none" or "deflate". The compression-type must match the compression-type of matching AvroSource</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">ssl</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">false</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Set this to true to enable SSL encryption. If SSL is enabled,
+you must also specify a "keystore" and a "keystore-password",
+either through component level parameters (see below)
+or as global SSL parameters (see <a href="#ssltls-support">SSL/TLS support</a> section).</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">keystore</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">This is the path to a Java keystore file.
+If not specified here, then the global keystore will be used
+(if defined, otherwise configuration error).</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">keystore-password</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">The password for the Java keystore.
+If not specified here, then the global keystore password will be used
+(if defined, otherwise configuration error).</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">keystore-type</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">JKS</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">The type of the Java keystore. This can be "JKS" or "PKCS12".
+If not specified here, then the global keystore type will be used
+(if defined, otherwise the default is JKS).</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">exclude-protocols</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">SSLv3</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Space-separated list of SSL/TLS protocols to exclude.
+SSLv3 will always be excluded in addition to the protocols specified.</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">include-protocols</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Space-separated list of SSL/TLS protocols to include.
+The enabled protocols will be the included protocols without the excluded protocols.
+If included-protocols is empty, it includes every supported protocols.</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">exclude-cipher-suites</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Space-separated list of cipher suites to exclude.</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">include-cipher-suites</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Space-separated list of cipher suites to include.
+The enabled cipher suites will be the included cipher suites without the excluded cipher suites.
+If included-cipher-suites is empty, it includes every supported cipher suites.</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">ipFilter</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">false</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Set this to true to enable ipFiltering for netty</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">ipFilterRules</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Define N netty ipFilter pattern rules with this config.</p></td>
+</tr>
+</tbody>
+</table>
+<div class="paragraph">
+<p>Example for agent named a1:</p>
+</div>
+<div class="listingblock">
+<div class="content">
+<pre class="highlightjs highlight"><code class="language-properties hljs" data-lang="properties">a1.sources = r1
+a1.channels = c1
+a1.sources.r1.type = avro
+a1.sources.r1.channels = c1
+a1.sources.r1.bind = 0.0.0.0
+a1.sources.r1.port = 4141</code></pre>
+</div>
+</div>
+<div class="paragraph">
+<p>Example of ipFilterRules</p>
+</div>
+<div class="paragraph">
+<p>ipFilterRules defines N netty ipFilters separated by a comma a pattern rule must be in this format.</p>
+</div>
+<div class="paragraph">
+<p>&lt;'allow' or deny&gt;:&lt;'ip' or 'name' for computer name&gt;:&lt;pattern&gt;
+or
+allow/deny:ip/name:pattern</p>
+</div>
+<div class="paragraph">
+<p>example: ipFilterRules=allow:ip:127.*,allow:name:localhost,deny:ip:*</p>
+</div>
+<div class="paragraph">
+<p>Note that the first rule to match will apply as the example below shows from a client on the localhost</p>
+</div>
+<div class="paragraph">
+<p>This will Allow the client on localhost be deny clients from any other ip "allow:name:localhost,deny:ip:*"
+This will deny the client on localhost be allow clients from any other ip "deny:name:localhost,allow:ip:*"</p>
+</div>
+</div>
+<div class="sect3">
+<h4 id="_thrift_source"><a class="anchor" href="#_thrift_source"></a>Thrift Source</h4>
+<div class="paragraph">
+<p>Listens on Thrift port and receives events from external Thrift client streams.
+When paired with the built-in ThriftSink on another (previous hop) Flume agent,
+it can create tiered collection topologies.
+Thrift source can be configured to start in secure mode by enabling kerberos authentication.
+agent-principal and agent-keytab are the properties used by the
+Thrift source to authenticate to the kerberos KDC.
+Required properties are in <strong>bold</strong>.</p>
+</div>
+<table class="tableblock frame-all grid-all stretch">
+<colgroup>
+<col style="width: 33.3333%;">
+<col style="width: 33.3333%;">
+<col style="width: 33.3334%;">
+</colgroup>
+<thead>
+<tr>
+<th class="tableblock halign-left valign-top">Property Name</th>
+<th class="tableblock halign-left valign-top">Default</th>
+<th class="tableblock halign-left valign-top">Description</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock"><strong>channels</strong></p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock"><strong>type</strong></p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">The component type name, needs to be <code>thrift</code></p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock"><strong>bind</strong></p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">hostname or IP address to listen on</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock"><strong>port</strong></p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Port # to bind to</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">threads</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Maximum number of worker threads to spawn</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">selector.type</p></td>
+<td class="tableblock halign-left valign-top"></td>
+<td class="tableblock halign-left valign-top"></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">selector.*</p></td>
+<td class="tableblock halign-left valign-top"></td>
+<td class="tableblock halign-left valign-top"></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">interceptors</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Space separated list of interceptors</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">interceptors.*</p></td>
+<td class="tableblock halign-left valign-top"></td>
+<td class="tableblock halign-left valign-top"></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">ssl</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">false</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Set this to true to enable SSL encryption. If SSL is enabled,
+you must also specify a "keystore" and a "keystore-password",
+either through component level parameters (see below)
+or as global SSL parameters (see <a href="#ssltls-support">SSL/TLS support</a> section)</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">keystore</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">This is the path to a Java keystore file.
+If not specified here, then the global keystore will be used
+(if defined, otherwise configuration error).</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">keystore-password</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">The password for the Java keystore.
+If not specified here, then the global keystore password will be used
+(if defined, otherwise configuration error).</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">keystore-type</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">JKS</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">The type of the Java keystore. This can be "JKS" or "PKCS12".
+If not specified here, then the global keystore type will be used
+(if defined, otherwise the default is JKS).</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">exclude-protocols</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">SSLv3</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Space-separated list of SSL/TLS protocols to exclude.
+SSLv3 will always be excluded in addition to the protocols specified.</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">include-protocols</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Space-separated list of SSL/TLS protocols to include.
+The enabled protocols will be the included protocols without the excluded protocols.
+If included-protocols is empty, it includes every supported protocols.</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">exclude-cipher-suites</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Space-separated list of cipher suites to exclude.</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">include-cipher-suites</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Space-separated list of cipher suites to include.
+The enabled cipher suites will be the included cipher suites without the excluded cipher suites.</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">kerberos</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">false</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Set to true to enable kerberos authentication. In kerberos mode, agent-principal and agent-keytab are required for successful authentication. The Thrift source in secure mode, will accept connections only from Thrift clients that have kerberos enabled and are successfully authenticated to the kerberos KDC.</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">agent-principal</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">The kerberos principal used by the Thrift Source to authenticate to the kerberos KDC.</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">agent-keytab</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">—-</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">The keytab location used by the Thrift Source in combination with the agent-principal to authenticate to the kerberos KDC.</p></td>
+</tr>
+</tbody>
+</table>
+<div class="paragraph">
+<p>Example for agent named a1:</p>
+</div>
+<div class="listingblock">
+<div class="content">
+<pre class="highlightjs highlight"><code class="language-properties hljs" data-lang="properties">a1.sources = r1
+a1.channels = c1
+a1.sources.r1.type = thrift
+a1.sources.r1.channels = c1
+a1.sources.r1.bind = 0.0.0.0
+a1.sources.r1.port = 4141</code></pre>
+</div>
+</div>
+</div>
+<div class="sect3">
+<h4 id="_exec_source"><a class="anchor" href="#_exec_source"></a>Exec Source</h4>
+<div class="paragraph">
+<p>Exec source runs a given Unix command on start-up and expects that process to
+continuously produce data on standard out (stderr is simply discarded, unless
+property logStdErr is set to true). If the process exits for any reason, the source also exits and
+will produce no further data. This means configurations such as <code>cat [named pipe]</code>
+or <code>tail -F [file]</code> are going to produce the desired results where as <code>date</code>
+will probably not - the former two commands produce streams of data where as the
+latter produces a single event and exits.</p>
+</div>
+<div class="paragraph">
+<p>Required properties are in <strong>bold</strong>.</p>
+</div>
+<table class="tableblock frame-all grid-all stretch">
+<colgroup>
+<col style="width: 33.3333%;">
+<col style="width: 33.3333%;">
+<col style="width: 33.3334%;">
+</colgroup>
+<thead>
+<tr>
+<th class="tableblock halign-left valign-top">Property Name</th>
+<th class="tableblock halign-left valign-top">Default</th>
+<th class="tableblock halign-left valign-top">Description</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock"><strong>channels</strong></p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock"><strong>type</strong></p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">The component type name, needs to be <code>exec</code></p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock"><strong>command</strong></p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">The command to execute</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">shell</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">A shell invocation used to run the command. e.g. /bin/sh -c. Required only for commands relying on shell features like wildcards, back ticks, pipes etc.</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">restartThrottle</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">10000</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Amount of time (in millis) to wait before attempting a restart</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">restart</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">false</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Whether the executed cmd should be restarted if it dies</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">logStdErr</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">false</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Whether the command&#8217;s stderr should be logged</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">batchSize</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">20</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">The max number of lines to read and send to the channel at a time</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">batchTimeout</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">3000</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Amount of time (in milliseconds) to wait, if the buffer size was not reached, before data is pushed downstream</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">selector.type</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">replicating</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">replicating or multiplexing</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">selector.*</p></td>
+<td class="tableblock halign-left valign-top"></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Depends on the selector.type value</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">interceptors</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Space-separated list of interceptors</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">interceptors.*</p></td>
+<td class="tableblock halign-left valign-top"></td>
+<td class="tableblock halign-left valign-top"></td>
+</tr>
+</tbody>
+</table>
+<div class="admonitionblock warning">
+<table>
+<tr>
+<td class="icon">
+<i class="fa icon-warning" title="Warning"></i>
+</td>
+<td class="content">
+<div class="paragraph">
+<p>The problem with ExecSource and other asynchronous sources is that the
+source can not guarantee that if there is a failure to put the event
+into the Channel the client knows about it. In such cases, the data will
+be lost. As a for instance, one of the most commonly requested features
+is the <code>tail -F [file]</code>-like use case where an application writes
+to a log file on disk and Flume tails the file, sending each line as an
+event. While this is possible, there&#8217;s an obvious problem; what happens
+if the channel fills up and Flume can&#8217;t send an event? Flume has no way
+of indicating to the application writing the log file that it needs to
+retain the log or that the event hasn&#8217;t been sent, for some reason. If
+this doesn&#8217;t make sense, you need only know this: Your application can
+never guarantee data has been received when using a unidirectional
+asynchronous interface such as ExecSource! As an extension of this
+warning - and to be completely clear - there is absolutely zero guarantee
+of event delivery when using this source. For stronger reliability
+guarantees, consider the Spooling Directory Source, Taildir Source or direct integration
+with Flume via the SDK.</p>
+</div>
+</td>
+</tr>
+</table>
+</div>
+<div class="paragraph">
+<p>Example for agent named a1:</p>
+</div>
+<div class="listingblock">
+<div class="content">
+<pre class="highlightjs highlight"><code class="language-properties hljs" data-lang="properties">a1.sources = r1
+a1.channels = c1
+a1.sources.r1.type = exec
+a1.sources.r1.command = tail -F /var/log/secure
+a1.sources.r1.channels = c1</code></pre>
+</div>
+</div>
+<div class="paragraph">
+<p>The 'shell' config is used to invoke the 'command' through a command shell (such as Bash
+or Powershell). The 'command' is passed as an argument to 'shell' for execution. This
+allows the 'command' to use features from the shell such as wildcards, back ticks, pipes,
+loops, conditionals etc. In the absence of the 'shell' config, the 'command' will be
+invoked directly. Common values for 'shell' : '/bin/sh -c', '/bin/ksh -c',
+'cmd /c', 'powershell -Command', etc.</p>
+</div>
+<div class="listingblock">
+<div class="content">
+<pre class="highlightjs highlight"><code class="language-properties hljs" data-lang="properties">a1.sources.tailsource-1.type = exec
+a1.sources.tailsource-1.shell = /bin/bash -c
+a1.sources.tailsource-1.command = for i in /path/*.txt; do cat $i; done</code></pre>
+</div>
+</div>
+</div>
+<div class="sect3">
+<h4 id="_jms_source"><a class="anchor" href="#_jms_source"></a>JMS Source</h4>
+<div class="paragraph">
+<p>JMS Source reads messages from a JMS destination such as a queue or topic. Being a JMS
+application it should work with any JMS provider but has only been tested with ActiveMQ.
+The JMS source provides configurable batch size, message selector, user/pass, and message
+to flume event converter. Note that the vendor provided JMS jars should be included in the
+Flume classpath using plugins.d directory (preferred), --classpath on command line, or
+via FLUME_CLASSPATH variable in flume-env.sh.</p>
+</div>
+<div class="paragraph">
+<p>Required properties are in <strong>bold</strong>.</p>
+</div>
+<table class="tableblock frame-all grid-all stretch">
+<colgroup>
+<col style="width: 33.3333%;">
+<col style="width: 33.3333%;">
+<col style="width: 33.3334%;">
+</colgroup>
+<thead>
+<tr>
+<th class="tableblock halign-left valign-top">Property Name</th>
+<th class="tableblock halign-left valign-top">Default</th>
+<th class="tableblock halign-left valign-top">Description</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock"><strong>channels</strong></p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock"><strong>type</strong></p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">The component type name, needs to be <code>jms</code></p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock"><strong>initialContextFactory</strong></p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Inital Context Factory, e.g: org.apache.activemq.jndi.ActiveMQInitialContextFactory</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock"><strong>connectionFactory</strong></p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">The JNDI name the connection factory should appear as</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock"><strong>providerURL</strong></p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">The JMS provider URL</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock"><strong>destinationName</strong></p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Destination name</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock"><strong>destinationType</strong></p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Destination type (queue or topic)</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">messageSelector</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Message selector to use when creating the consumer</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">userName</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Username for the destination/provider</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">passwordFile</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">File containing the password for the destination/provider</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">batchSize</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">100</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Number of messages to consume in one batch</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">converter.type</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">DEFAULT</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Class to use to convert messages to flume events. See below.</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">converter.*</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Converter properties.</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">converter.charset</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">UTF-8</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Default converter only. Charset to use when converting JMS TextMessages to byte arrays.</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">createDurableSubscription</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">false</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Whether to create durable subscription. Durable subscription can only be used with
+destinationType topic. If true, "clientId" and "durableSubscriptionName"
+have to be specified.</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">clientId</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">JMS client identifier set on Connection right after it is created.
+Required for durable subscriptions.</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">durableSubscriptionName</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Name used to identify the durable subscription. Required for durable subscriptions.</p></td>
+</tr>
+</tbody>
+</table>
+<div class="sect4">
+<h5 id="_jms_message_converter"><a class="anchor" href="#_jms_message_converter"></a>JMS message converter</h5>
+<div class="paragraph">
+<p>The JMS source allows pluggable converters, though it&#8217;s likely the default converter will work
+for most purposes. The default converter is able to convert Bytes, Text, and Object messages
+to FlumeEvents. In all cases, the properties in the message are added as headers to the
+FlumeEvent.</p>
+</div>
+<div class="dlist">
+<dl>
+<dt class="hdlist1">BytesMessage</dt>
+<dd>
+<p>Bytes of message are copied to body of the FlumeEvent. Cannot convert more than 2GB
+of data per message.</p>
+</dd>
+<dt class="hdlist1">TextMessage</dt>
+<dd>
+<p>Text of message is converted to a byte array and copied to the body of the
+FlumeEvent. The default converter uses UTF-8 by default but this is configurable.</p>
+</dd>
+<dt class="hdlist1">ObjectMessage</dt>
+<dd>
+<p>Object is written out to a ByteArrayOutputStream wrapped in an ObjectOutputStream and
+the resulting array is copied to the body of the FlumeEvent.</p>
+</dd>
+</dl>
+</div>
+<div class="paragraph">
+<p>Example for agent named a1:</p>
+</div>
+<div class="listingblock">
+<div class="content">
+<pre class="highlightjs highlight"><code class="language-properties hljs" data-lang="properties">a1.sources = r1
+a1.channels = c1
+a1.sources.r1.type = jms
+a1.sources.r1.channels = c1
+a1.sources.r1.initialContextFactory = org.apache.activemq.jndi.ActiveMQInitialContextFactory
+a1.sources.r1.connectionFactory = GenericConnectionFactory
+a1.sources.r1.providerURL = tcp://mqserver:61616
+a1.sources.r1.destinationName = BUSINESS_DATA
+a1.sources.r1.destinationType = QUEUE</code></pre>
+</div>
+</div>
+</div>
+<div class="sect4">
+<h5 id="_ssl_and_jms_source"><a class="anchor" href="#_ssl_and_jms_source"></a>SSL and JMS Source</h5>
+<div class="paragraph">
+<p>JMS client implementations typically support to configure SSL/TLS via some Java system properties defined by JSSE
+(Java Secure Socket Extension). Specifying these system properties for Flume&#8217;s JVM, JMS Source (or more precisely the
+JMS client implementation used by the JMS Source) can connect to the JMS server through SSL (of course only when the JMS
+server has also been set up to use SSL).
+It should work with any JMS provider and has been tested with ActiveMQ, IBM MQ and Oracle WebLogic.</p>
+</div>
+<div class="paragraph">
+<p>The following sections describe the SSL configuration steps needed on the Flume side only. You can find more detailed
+descriptions about the server side setup of the different JMS providers and also full working configuration examples on
+Flume Wiki.</p>
+</div>
+<div class="paragraph">
+<p><strong>SSL transport / server authentication:</strong></p>
+</div>
+<div class="paragraph">
+<p>If the JMS server uses self-signed certificate or its certificate is signed by a non-trusted CA (eg. the company&#8217;s own
+CA), then a truststore (containing the right certificate) needs to be set up and passed to Flume. It can be done via
+the global SSL parameters. For more details about the global SSL setup, see the <a href="#ssltls-support">SSL/TLS support</a> section.</p>
+</div>
+<div class="paragraph">
+<p>Some JMS providers require SSL specific JNDI Initial Context Factory and/or Provider URL settings when using SSL (eg.
+ActiveMQ uses ssl:// URL prefix instead of tcp://).
+In this case the source properties (<code>initialContextFactory</code> and/or <code>providerURL</code>) have to be adjusted in the agent
+config file.</p>
+</div>
+<div class="paragraph">
+<p><strong>Client certificate authentication (two-way SSL):</strong></p>
+</div>
+<div class="paragraph">
+<p>JMS Source can authenticate to the JMS server through client certificate authentication instead of the usual
+user/password login (when SSL is used and the JMS server is configured to accept this kind of authentication).</p>
+</div>
+<div class="paragraph">
+<p>The keystore containing Flume&#8217;s key used for the authentication needs to be configured via the global SSL parameters
+again. For more details about the global SSL setup, see the <a href="#ssltls-support">SSL/TLS support</a> section.</p>
+</div>
+<div class="paragraph">
+<p>The keystore should contain only one key (if multiple keys are present, then the first one will be used).
+The key password must be the same as the keystore password.</p>
+</div>
+<div class="paragraph">
+<p>In case of client certificate authentication, it is not needed to specify the <code>userName</code> / <code>passwordFile</code> properties
+for the JMS Source in the Flume agent config file.</p>
+</div>
+<div class="paragraph">
+<p><strong>Please note:</strong></p>
+</div>
+<div class="paragraph">
+<p>There are no component level configuration parameters for JMS Source unlike in case of other components.
+No enable SSL flag either.
+SSL setup is controlled by JNDI/Provider URL settings (ultimately the JMS server settings) and by the presence / absence
+of the truststore / keystore.</p>
+</div>
+</div>
+</div>
+<div class="sect3">
+<h4 id="_spooling_directory_source"><a class="anchor" href="#_spooling_directory_source"></a>Spooling Directory Source</h4>
+<div class="paragraph">
+<p>This source lets you ingest data by placing files to be ingested into a
+"spooling" directory on disk.
+This source will watch the specified directory for new files, and will parse
+events out of new files as they appear.
+The event parsing logic is pluggable.
+After a given file has been fully read
+into the channel, completion by default is indicated by renaming the file or it can be deleted or the trackerDir is used
+to keep track of processed files.</p>
+</div>
+<div class="paragraph">
+<p>Unlike the Exec source, this source is reliable and will not miss data, even if
+Flume is restarted or killed. In exchange for this reliability, only immutable,
+uniquely-named files must be dropped into the spooling directory. Flume tries
+to detect these problem conditions and will fail loudly if they are violated:</p>
+</div>
+<div class="olist arabic">
+<ol class="arabic">
+<li>
+<p>If a file is written to after being placed into the spooling directory,
+Flume will print an error to its log file and stop processing.</p>
+</li>
+<li>
+<p>If a file name is reused at a later time, Flume will print an error to its
+log file and stop processing.</p>
+</li>
+</ol>
+</div>
+<div class="paragraph">
+<p>To avoid the above issues, it may be useful to add a unique identifier
+(such as a timestamp) to log file names when they are moved into the spooling
+directory.</p>
+</div>
+<div class="paragraph">
+<p>Despite the reliability guarantees of this source, there are still
+cases in which events may be duplicated if certain downstream failures occur.
+This is consistent with the guarantees offered by other Flume components.</p>
+</div>
+<table class="tableblock frame-all grid-all stretch">
+<colgroup>
+<col style="width: 33.3333%;">
+<col style="width: 33.3333%;">
+<col style="width: 33.3334%;">
+</colgroup>
+<thead>
+<tr>
+<th class="tableblock halign-left valign-top">Property Name</th>
+<th class="tableblock halign-left valign-top">Default</th>
+<th class="tableblock halign-left valign-top">Description</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock"><strong>channels</strong></p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock"><strong>type</strong></p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">The component type name, needs to be <code>spooldir</code>.</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock"><strong>spoolDir</strong></p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">The directory from which to read files from.</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">fileSuffix</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">.COMPLETED</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Suffix to append to completely ingested files</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">deletePolicy</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">never</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">When to delete completed files: <code>never</code> or <code>immediate</code></p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">fileHeader</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">false</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Whether to add a header storing the absolute path filename.</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">fileHeaderKey</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">file</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Header key to use when appending absolute path filename to event header.</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">basenameHeader</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">false</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Whether to add a header storing the basename of the file.</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">basenameHeaderKey</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">basename</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Header Key to use when appending basename of file to event header.</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">includePattern</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">^.*$</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Regular expression specifying which files to include.
+It can used together with <code>ignorePattern</code>.
+If a file matches both <code>ignorePattern</code> and <code>includePattern</code> regex,
+the file is ignored.</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">ignorePattern</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">^$</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Regular expression specifying which files to ignore (skip).
+It can used together with <code>includePattern</code>.
+If a file matches both <code>ignorePattern</code> and <code>includePattern</code> regex,
+the file is ignored.</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">trackerDir</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">.flumespool</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Directory to store metadata related to processing of files.
+If this path is not an absolute path, then it is interpreted as relative to the spoolDir.</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">trackingPolicy</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">rename</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">The tracking policy defines how file processing is tracked. It can be "rename" or
+"tracker_dir". This parameter is only effective if the deletePolicy is "never".
+"rename" - After processing files they get renamed according to the fileSuffix parameter.
+"tracker_dir" - Files are not renamed but a new empty file is created in the trackerDir.
+The new tracker file name is derived from the ingested one plus the fileSuffix.</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">consumeOrder</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">oldest</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">In which order files in the spooling directory will be consumed <code>oldest</code>,
+<code>youngest</code> and <code>random</code>. In case of <code>oldest</code> and <code>youngest</code>, the last modified
+time of the files will be used to compare the files. In case of a tie, the file
+with smallest lexicographical order will be consumed first. In case of <code>random</code> any
+file will be picked randomly. When using <code>oldest</code> and <code>youngest</code> the whole
+directory will be scanned to pick the oldest/youngest file, which might be slow if there
+are a large number of files, while using <code>random</code> may cause old files to be consumed
+very late if new files keep coming in the spooling directory.</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">pollDelay</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">500</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Delay (in milliseconds) used when polling for new files.</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">recursiveDirectorySearch</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">false</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Whether to monitor sub directories for new files to read.</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">maxBackoff</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">4000</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">The maximum time (in millis) to wait between consecutive attempts to
+write to the channel(s) if the channel is full. The source will start at
+a low backoff and increase it exponentially each time the channel throws a
+ChannelException, up to the value specified by this parameter.</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">batchSize</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">100</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Granularity at which to batch transfer to the channel</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">inputCharset</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">UTF-8</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Character set used by deserializers that treat the input file as text.</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">decodeErrorPolicy</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock"><code>FAIL</code></p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">What to do when we see a non-decodable character in the input file.
+<code>FAIL</code>: Throw an exception and fail to parse the file.
+<code>REPLACE</code>: Replace the unparseable character with the "replacement character" char,
+typically Unicode U&#43;FFFD.
+<code>IGNORE</code>: Drop the unparseable character sequence.</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">deserializer</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock"><code>LINE</code></p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Specify the deserializer used to parse the file into events.
+Defaults to parsing each line as an event. The class specified must implement
+<code>EventDeserializer.Builder</code>.</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">deserializer.*</p></td>
+<td class="tableblock halign-left valign-top"></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Varies per event deserializer.</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">bufferMaxLines</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">(Obsolete) This option is now ignored.</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">bufferMaxLineLength</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">5000</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">(Deprecated) Maximum length of a line in the commit buffer. Use deserializer.maxLineLength instead.</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">selector.type</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">replicating</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">replicating or multiplexing</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">selector.*</p></td>
+<td class="tableblock halign-left valign-top"></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Depends on the selector.type value</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">interceptors</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Space-separated list of interceptors</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">interceptors.*</p></td>
+<td class="tableblock halign-left valign-top"></td>
+<td class="tableblock halign-left valign-top"></td>
+</tr>
+</tbody>
+</table>
+<div class="paragraph">
+<p>Example for an agent named agent-1:</p>
+</div>
+<div class="listingblock">
+<div class="content">
+<pre class="highlightjs highlight"><code class="language-properties hljs" data-lang="properties">a1.channels = ch-1
+a1.sources = src-1
+
+a1.sources.src-1.type = spooldir
+a1.sources.src-1.channels = ch-1
+a1.sources.src-1.spoolDir = /var/log/apache/flumeSpool
+a1.sources.src-1.fileHeader = true</code></pre>
+</div>
+</div>
+<div class="sect4">
+<h5 id="_event_deserializers"><a class="anchor" href="#_event_deserializers"></a>Event Deserializers</h5>
+<div class="paragraph">
+<p>The following event deserializers ship with Flume.</p>
+</div>
+<div class="sect5">
+<h6 id="_line"><a class="anchor" href="#_line"></a>LINE</h6>
+<div class="paragraph">
+<p>This deserializer generates one event per line of text input.</p>
+</div>
+<table class="tableblock frame-all grid-all stretch">
+<colgroup>
+<col style="width: 33.3333%;">
+<col style="width: 33.3333%;">
+<col style="width: 33.3334%;">
+</colgroup>
+<thead>
+<tr>
+<th class="tableblock halign-left valign-top">Property Name</th>
+<th class="tableblock halign-left valign-top">Default</th>
+<th class="tableblock halign-left valign-top">Description</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">deserializer.maxLineLength</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">2048</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Maximum number of characters to include in a single event.
+If a line exceeds this length, it is truncated, and the
+remaining characters on the line will appear in a
+subsequent event.</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">deserializer.outputCharset</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">UTF-8</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Charset to use for encoding events put into the channel.</p></td>
+</tr>
+</tbody>
+</table>
+</div>
+<div class="sect5">
+<h6 id="_avro"><a class="anchor" href="#_avro"></a>AVRO</h6>
+<div class="paragraph">
+<p>This deserializer is able to read an Avro container file, and it generates
+one event per Avro record in the file.
+Each event is annotated with a header that indicates the schema used.
+The body of the event is the binary Avro record data, not
+including the schema or the rest of the container file elements.</p>
+</div>
+<div class="paragraph">
+<p>Note that if the spool directory source must retry putting one of these events
+onto a channel (for example, because the channel is full), then it will reset
+and retry from the most recent Avro container file sync point. To reduce
+potential event duplication in such a failure scenario, write sync markers more
+frequently in your Avro input files.</p>
+</div>
+<table class="tableblock frame-all grid-all stretch">
+<colgroup>
+<col style="width: 33.3333%;">
+<col style="width: 33.3333%;">
+<col style="width: 33.3334%;">
+</colgroup>
+<thead>
+<tr>
+<th class="tableblock halign-left valign-top">Property Name</th>
+<th class="tableblock halign-left valign-top">Default</th>
+<th class="tableblock halign-left valign-top">Description</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">deserializer.schemaType</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">HASH</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">How the schema is represented. By default, or when the value <code>HASH</code>
+is specified, the Avro schema is hashed and
+the hash is stored in every event in the event header
+"flume.avro.schema.hash". If <code>LITERAL</code> is specified, the JSON-encoded
+schema itself is stored in every event in the event header
+"flume.avro.schema.literal". Using <code>LITERAL</code> mode is relatively
+inefficient compared to <code>HASH</code> mode.</p></td>
+</tr>
+</tbody>
+</table>
+</div>
+<div class="sect5">
+<h6 id="_blobdeserializer"><a class="anchor" href="#_blobdeserializer"></a>BlobDeserializer</h6>
+<div class="paragraph">
+<p>This deserializer reads a Binary Large Object (BLOB) per event, typically one BLOB per file. For example a PDF or JPG file. Note that this approach is not suitable for very large objects because the entire BLOB is buffered in RAM.</p>
+</div>
+<table class="tableblock frame-all grid-all stretch">
+<colgroup>
+<col style="width: 33.3333%;">
+<col style="width: 33.3333%;">
+<col style="width: 33.3334%;">
+</colgroup>
+<thead>
+<tr>
+<th class="tableblock halign-left valign-top">Property Name</th>
+<th class="tableblock halign-left valign-top">Default</th>
+<th class="tableblock halign-left valign-top">Description</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock"><strong>deserializer</strong></p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">The FQCN of this class: <code>org.apache.flume.sink.solr.morphline.BlobDeserializer$Builder</code></p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">deserializer.maxBlobLength</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">100000000</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">The maximum number of bytes to read and buffer for a given request</p></td>
+</tr>
+</tbody>
+</table>
+</div>
+</div>
+</div>
+<div class="sect3">
+<h4 id="_taildir_source"><a class="anchor" href="#_taildir_source"></a>Taildir Source</h4>
+<div class="admonitionblock note">
+<table>
+<tr>
+<td class="icon">
+<i class="fa icon-note" title="Note"></i>
+</td>
+<td class="content">
+<div class="paragraph">
+<p><strong>This source is provided as a preview feature. It does not work on Windows.</strong></p>
+</div>
+</td>
+</tr>
+</table>
+</div>
+<div class="paragraph">
+<p>Watch the specified files, and tail them in nearly real-time once detected new lines appended to the each files.
+If the new lines are being written, this source will retry reading them in wait for the completion of the write.</p>
+</div>
+<div class="paragraph">
+<p>This source is reliable and will not miss data even when the tailing files rotate.
+It periodically writes the last read position of each files on the given position file in JSON format.
+If Flume is stopped or down for some reason, it can restart tailing from the position written on the existing position file.</p>
+</div>
+<div class="paragraph">
+<p>In other use case, this source can also start tailing from the arbitrary position for each files using the given position file.
+When there is no position file on the specified path, it will start tailing from the first line of each files by default.</p>
+</div>
+<div class="paragraph">
+<p>Files will be consumed in order of their modification time. File with the oldest modification time will be consumed first.</p>
+</div>
+<div class="paragraph">
+<p>This source does not rename or delete or do any modifications to the file being tailed.
+Currently this source does not support tailing binary files. It reads text files line by line.</p>
+</div>
+<table class="tableblock frame-all grid-all stretch">
+<colgroup>
+<col style="width: 33.3333%;">
+<col style="width: 33.3333%;">
+<col style="width: 33.3334%;">
+</colgroup>
+<thead>
+<tr>
+<th class="tableblock halign-left valign-top">Property Name</th>
+<th class="tableblock halign-left valign-top">Default</th>
+<th class="tableblock halign-left valign-top">Description</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock"><strong>channels</strong></p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock"><strong>type</strong></p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">The component type name, needs to be <code>TAILDIR</code>.</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock"><strong>filegroups</strong></p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Space-separated list of file groups. Each file group indicates a set of files to be tailed.</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock"><strong>filegroups.&lt;filegroupName&gt;</strong></p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Absolute path of the file group. Regular expression (and not file system patterns) can be used for filename only.</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">positionFile</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">~/.flume/taildir_position.json</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">File in JSON format to record the inode, the absolute path and the last position of each tailing file.</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">headers.&lt;filegroupName&gt;.&lt;headerKey&gt;</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Header value which is the set with header key. Multiple headers can be specified for one file group.</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">byteOffsetHeader</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">false</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Whether to add the byte offset of a tailed line to a header called 'byteoffset'.</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">skipToEnd</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">false</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Whether to skip the position to EOF in the case of files not written on the position file.</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">idleTimeout</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">120000</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Time (ms) to close inactive files. If the closed file is appended new lines to, this source will automatically re-open it.</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">writePosInterval</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">3000</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Interval time (ms) to write the last position of each file on the position file.</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">batchSize</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">100</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Max number of lines to read and send to the channel at a time. Using the default is usually fine.</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">maxBatchCount</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Long.MAX_VALUE</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Controls the number of batches being read consecutively from the same file.
+If the source is tailing multiple files and one of them is written at a fast rate,
+it can prevent other files to be processed, because the busy file would be read in an endless loop.
+In this case lower this value.</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">backoffSleepIncrement</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">1000</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">The increment for time delay before reattempting to poll for new data, when the last attempt did not find any new data.</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">maxBackoffSleep</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">5000</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">The max time delay between each reattempt to poll for new data, when the last attempt did not find any new data.</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">cachePatternMatching</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">true</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Listing directories and applying the filename regex pattern may be time consuming for directories
+containing thousands of files. Caching the list of matching files can improve performance.
+The order in which files are consumed will also be cached.
+Requires that the file system keeps track of modification times with at least a 1-second granularity.</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">fileHeader</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">false</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Whether to add a header storing the absolute path filename.</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">fileHeaderKey</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">file</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Header key to use when appending absolute path filename to event header.</p></td>
+</tr>
+</tbody>
+</table>
+<div class="paragraph">
+<p>Example for agent named a1:</p>
+</div>
+<div class="listingblock">
+<div class="content">
+<pre class="highlightjs highlight"><code class="language-properties hljs" data-lang="properties">a1.sources = r1
+a1.channels = c1
+a1.sources.r1.type = TAILDIR
+a1.sources.r1.channels = c1
+a1.sources.r1.positionFile = /var/log/flume/taildir_position.json
+a1.sources.r1.filegroups = f1 f2
+a1.sources.r1.filegroups.f1 = /var/log/test1/example.log
+a1.sources.r1.headers.f1.headerKey1 = value1
+a1.sources.r1.filegroups.f2 = /var/log/test2/.*log.*
+a1.sources.r1.headers.f2.headerKey1 = value2
+a1.sources.r1.headers.f2.headerKey2 = value2-2
+a1.sources.r1.fileHeader = true
+a1.sources.ri.maxBatchCount = 1000</code></pre>
+</div>
+</div>
+</div>
+<div class="sect3">
+<h4 id="_twitter_1_firehose_source_experimental"><a class="anchor" href="#_twitter_1_firehose_source_experimental"></a>Twitter 1% firehose Source (experimental)</h4>
+<div class="admonitionblock warning">
+<table>
+<tr>
+<td class="icon">
+<i class="fa icon-warning" title="Warning"></i>
+</td>
+<td class="content">
+<div class="paragraph">
+<p>This source is highly experimental and may change between minor versions of Flume.
+Use at your own risk.</p>
+</div>
+</td>
+</tr>
+</table>
+</div>
+<div class="paragraph">
+<p>Experimental source that connects via Streaming API to the 1% sample twitter
+firehose, continuously downloads tweets, converts them to Avro format and
+sends Avro events to a downstream Flume sink. Requires the consumer and
+access tokens and secrets of a Twitter developer account.
+Required properties are in <strong>bold</strong>.</p>
+</div>
+<table class="tableblock frame-all grid-all stretch">
+<colgroup>
+<col style="width: 33.3333%;">
+<col style="width: 33.3333%;">
+<col style="width: 33.3334%;">
+</colgroup>
+<thead>
+<tr>
+<th class="tableblock halign-left valign-top">Property Name</th>
+<th class="tableblock halign-left valign-top">Default</th>
+<th class="tableblock halign-left valign-top">Description</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock"><strong>channels</strong></p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock"><strong>type</strong></p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">The component type name, needs to be <code>org.apache.flume.source.twitter.TwitterSource</code></p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock"><strong>consumerKey</strong></p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">OAuth consumer key</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock"><strong>consumerSecret</strong></p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">OAuth consumer secret</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock"><strong>accessToken</strong></p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">OAuth access token</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock"><strong>accessTokenSecret</strong></p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">OAuth token secret</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">maxBatchSize</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">1000</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Maximum number of twitter messages to put in a single batch</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">maxBatchDurationMillis</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">1000</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Maximum number of milliseconds to wait before closing a batch</p></td>
+</tr>
+</tbody>
+</table>
+<div class="paragraph">
+<p>Example for agent named a1:</p>
+</div>
+<div class="listingblock">
+<div class="content">
+<pre class="highlightjs highlight"><code class="language-properties hljs" data-lang="properties">a1.sources = r1
+a1.channels = c1
+a1.sources.r1.type = org.apache.flume.source.twitter.TwitterSource
+a1.sources.r1.channels = c1
+a1.sources.r1.consumerKey = YOUR_TWITTER_CONSUMER_KEY
+a1.sources.r1.consumerSecret = YOUR_TWITTER_CONSUMER_SECRET
+a1.sources.r1.accessToken = YOUR_TWITTER_ACCESS_TOKEN
+a1.sources.r1.accessTokenSecret = YOUR_TWITTER_ACCESS_TOKEN_SECRET
+a1.sources.r1.maxBatchSize = 10
+a1.sources.r1.maxBatchDurationMillis = 200</code></pre>
+</div>
+</div>
+</div>
+<div class="sect3">
+<h4 id="_kafka_source"><a class="anchor" href="#_kafka_source"></a>Kafka Source</h4>
+<div class="paragraph">
+<p>Kafka Source is an Apache Kafka consumer that reads messages from Kafka topics.
+If you have multiple Kafka sources running, you can configure them with the same Consumer Group
+so each will read a unique set of partitions for the topics. This currently supports Kafka server releases 0.10.1.0 or higher. Testing was done up to 2.0.1 that was the highest avilable version at the time of the release.</p>
+</div>
+<table class="tableblock frame-all grid-all stretch">
+<colgroup>
+<col style="width: 33.3333%;">
+<col style="width: 33.3333%;">
+<col style="width: 33.3334%;">
+</colgroup>
+<thead>
+<tr>
+<th class="tableblock halign-left valign-top">Property Name</th>
+<th class="tableblock halign-left valign-top">Default</th>
+<th class="tableblock halign-left valign-top">Description</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock"><strong>channels</strong></p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock"><strong>type</strong></p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">The component type name, needs to be <code>org.apache.flume.source.kafka.KafkaSource</code></p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock"><strong>kafka.bootstrap.servers</strong></p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">List of brokers in the Kafka cluster used by the source</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">kafka.consumer.group.id</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">flume</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Unique identified of consumer group. Setting the same id in multiple sources or agents
+indicates that they are part of the same consumer group</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock"><strong>kafka.topics</strong></p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Comma-separated list of topics the Kafka consumer will read messages from.</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock"><strong>kafka.topics.regex</strong></p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Regex that defines set of topics the source is subscribed on. This property has higher priority
+than <code>kafka.topics</code> and overrides <code>kafka.topics</code> if exists.</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">batchSize</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">1000</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Maximum number of messages written to Channel in one batch</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">batchDurationMillis</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">1000</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Maximum time (in ms) before a batch will be written to Channel
+The batch will be written whenever the first of size and time will be reached.</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">backoffSleepIncrement</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">1000</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Initial and incremental wait time that is triggered when a Kafka Topic appears to be empty.
+Wait period will reduce aggressive pinging of an empty Kafka Topic. One second is ideal for
+ingestion use cases but a lower value may be required for low latency operations with
+interceptors.</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">maxBackoffSleep</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">5000</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Maximum wait time that is triggered when a Kafka Topic appears to be empty. Five seconds is
+ideal for ingestion use cases but a lower value may be required for low latency operations
+with interceptors.</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">useFlumeEventFormat</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">false</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">By default events are taken as bytes from the Kafka topic directly into the event body. Set to
+true to read events as the Flume Avro binary format. Used in conjunction with the same property
+on the KafkaSink or with the parseAsFlumeEvent property on the Kafka Channel this will preserve
+any Flume headers sent on the producing side.</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">setTopicHeader</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">true</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">When set to true, stores the topic of the retrieved message into a header, defined by the
+<code>topicHeader</code> property.</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">topicHeader</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">topic</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Defines the name of the header in which to store the name of the topic the message was received
+from, if the <code>setTopicHeader</code> property is set to <code>true</code>. Care should be taken if combining
+with the Kafka Sink <code>topicHeader</code> property so as to avoid sending the message back to the same
+topic in a loop.</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">timestampHeader</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">If present the Kafka message timestamp value will be copied into the specified Flume header name.</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">header.NAME</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Used to identify which headers from the Kafka message should be added as a FLume header. The value
+of NAME should match the Flume header name and the value should be the name of the header to be used
+as the Kafka header name.</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">kafka.consumer.security.protocol</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">PLAINTEXT</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Set to SASL_PLAINTEXT, SASL_SSL or SSL if writing to Kafka using some level of security. See below for additional info on secure setup.</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock"><em>more consumer security props</em></p></td>
+<td class="tableblock halign-left valign-top"></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">If using SASL_PLAINTEXT, SASL_SSL or SSL refer to <a href="https://kafka.apache.org/documentation.html#security">Kafka security</a> for additional
+properties that need to be set on consumer.</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Other Kafka Consumer Properties</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">These properties are used to configure the Kafka Consumer. Any consumer property supported
+by Kafka can be used. The only requirement is to prepend the property name with the prefix
+<code>kafka.consumer</code>.
+For example: <code>kafka.consumer.auto.offset.reset</code></p></td>
+</tr>
+</tbody>
+</table>
+<div class="admonitionblock note">
+<table>
+<tr>
+<td class="icon">
+<i class="fa icon-note" title="Note"></i>
+</td>
+<td class="content">
+<div class="paragraph">
+<p>The Kafka Source overrides two Kafka consumer parameters:
+auto.commit.enable is set to "false" by the source and every batch is committed. Kafka source guarantees at least once
+strategy of messages retrieval. The duplicates can be present when the source starts.
+The Kafka Source also provides defaults for the key.deserializer(org.apache.kafka.common.serialization.StringSerializer)
+and value.deserializer(org.apache.kafka.common.serialization.ByteArraySerializer). Modification of these parameters is not recommended.</p>
+</div>
+</td>
+</tr>
+</table>
+</div>
+<div class="paragraph">
+<p>Deprecated Properties</p>
+</div>
+<table class="tableblock frame-all grid-all stretch">
+<colgroup>
+<col style="width: 33.3333%;">
+<col style="width: 33.3333%;">
+<col style="width: 33.3334%;">
+</colgroup>
+<thead>
+<tr>
+<th class="tableblock halign-left valign-top">Property Name</th>
+<th class="tableblock halign-left valign-top">Default</th>
+<th class="tableblock halign-left valign-top">Description</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">topic</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Use kafka.topics</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">groupId</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">flume</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Use kafka.consumer.group.id</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">zookeeperConnect</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Is no longer supported by Kafka consumer client since 0.9.x. Use kafka.bootstrap.servers
+to establish connection with Kafka cluster</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">migrateZookeeperOffsets</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">true</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">When no Kafka stored offset is found, look up the offsets in Zookeeper and commit them to Kafka.
+This should be true to support seamless Kafka client migration from older versions of Flume.
+Once migrated this can be set to false, though that should generally not be required.
+If no Zookeeper offset is found, the Kafka configuration kafka.consumer.auto.offset.reset
+defines how offsets are handled.
+Check <a href="https://kafka.apache.org/documentation.html#consumerconfigs">Kafka documentation</a>
+for details</p></td>
+</tr>
+</tbody>
+</table>
+<div class="paragraph">
+<p>Example for topic subscription by comma-separated topic list.</p>
+</div>
+<div class="listingblock">
+<div class="content">
+<pre class="highlightjs highlight"><code class="language-properties hljs" data-lang="properties">tier1.sources.source1.type = org.apache.flume.source.kafka.KafkaSource
+tier1.sources.source1.channels = channel1
+tier1.sources.source1.batchSize = 5000
+tier1.sources.source1.batchDurationMillis = 2000
+tier1.sources.source1.kafka.bootstrap.servers = localhost:9092
+tier1.sources.source1.kafka.topics = test1, test2
+tier1.sources.source1.kafka.consumer.group.id = custom.g.id</code></pre>
+</div>
+</div>
+<div class="paragraph">
+<p>Example for topic subscription by regex</p>
+</div>
+<div class="listingblock">
+<div class="content">
+<pre class="highlightjs highlight"><code class="language-properties hljs" data-lang="properties">tier1.sources.source1.type = org.apache.flume.source.kafka.KafkaSource
+tier1.sources.source1.channels = channel1
+tier1.sources.source1.kafka.bootstrap.servers = localhost:9092
+tier1.sources.source1.kafka.topics.regex = ^topic[0-9]$
+# the default kafka.consumer.group.id=flume is used</code></pre>
+</div>
+</div>
+<div class="paragraph">
+<p><strong>Security and Kafka Source:</strong></p>
+</div>
+<div class="paragraph">
+<p>Secure authentication as well as data encryption is supported on the communication channel between Flume and Kafka.
+For secure authentication SASL/GSSAPI (Kerberos V5) or SSL (even though the parameter is named SSL, the actual protocol is a TLS implementation) can be used from Kafka version 0.9.0.</p>
+</div>
+<div class="paragraph">
+<p>As of now data encryption is solely provided by SSL/TLS.</p>
+</div>
+<div class="paragraph">
+<p>Setting <code>kafka.consumer.security.protocol</code> to any of the following value means:</p>
+</div>
+<div class="ulist">
+<ul>
+<li>
+<p><strong>SASL_PLAINTEXT</strong> - Kerberos or plaintext authentication with no data encryption</p>
+</li>
+<li>
+<p><strong>SASL_SSL</strong> - Kerberos or plaintext authentication with data encryption</p>
+</li>
+<li>
+<p><strong>SSL</strong> - TLS based encryption with optional authentication.</p>
+</li>
+</ul>
+</div>
+<div class="admonitionblock warning">
+<table>
+<tr>
+<td class="icon">
+<i class="fa icon-warning" title="Warning"></i>
+</td>
+<td class="content">
+<div class="paragraph">
+<p>There is a performance degradation when SSL is enabled,
+the magnitude of which depends on the CPU type and the JVM implementation.
+Reference: <a href="https://kafka.apache.org/documentation#security_overview">Kafka security overview</a>
+and the jira for tracking this issue:
+<a href="https://issues.apache.org/jira/browse/KAFKA-2561">KAFKA-2561</a></p>
+</div>
+</td>
+</tr>
+</table>
+</div>
+<div class="paragraph">
+<p><strong>TLS and Kafka Source:</strong></p>
+</div>
+<div class="paragraph">
+<p>Please read the steps described in <a href="https://kafka.apache.org/documentation#security_configclients">Configuring Kafka Clients SSL</a>
+to learn about additional configuration settings for fine tuning for example any of the following:
+security provider, cipher suites, enabled protocols, truststore or keystore types.</p>
+</div>
+<div class="paragraph">
+<p>Example configuration with server side authentication and data encryption.</p>
+</div>
+<div class="listingblock">
+<div class="content">
+<pre class="highlightjs highlight"><code class="language-properties hljs" data-lang="properties">a1.sources.source1.type = org.apache.flume.source.kafka.KafkaSource
+a1.sources.source1.kafka.bootstrap.servers = kafka-1:9093,kafka-2:9093,kafka-3:9093
+a1.sources.source1.kafka.topics = mytopic
+a1.sources.source1.kafka.consumer.group.id = flume-consumer
+a1.sources.source1.kafka.consumer.security.protocol = SSL
+# optional, the global truststore can be used alternatively
+a1.sources.source1.kafka.consumer.ssl.truststore.location=/path/to/truststore.jks
+a1.sources.source1.kafka.consumer.ssl.truststore.password=&lt;password to access the truststore&gt;</code></pre>
+</div>
+</div>
+<div class="paragraph">
+<p>Specifying the truststore is optional here, the global truststore can be used instead.
+For more details about the global SSL setup, see the <a href="#ssltls-support">SSL/TLS support</a> section.</p>
+</div>
+<div class="paragraph">
+<p>Note: By default the property <code>ssl.endpoint.identification.algorithm</code>
+is not defined, so hostname verification is not performed.
+In order to enable hostname verification, set the following properties</p>
+</div>
+<div class="listingblock">
+<div class="content">
+<pre class="highlightjs highlight"><code class="language-properties hljs" data-lang="properties">a1.sources.source1.kafka.consumer.ssl.endpoint.identification.algorithm=HTTPS</code></pre>
+</div>
+</div>
+<div class="paragraph">
+<p>Once enabled, clients will verify the server&#8217;s fully qualified domain name (FQDN)
+against one of the following two fields:</p>
+</div>
+<div class="olist arabic">
+<ol class="arabic">
+<li>
+<p>Common Name (CN) <a href="https://tools.ietf.org/html/rfc6125#section-2.3" class="bare">https://tools.ietf.org/html/rfc6125#section-2.3</a></p>
+</li>
+<li>
+<p>Subject Alternative Name (SAN) <a href="https://tools.ietf.org/html/rfc5280#section-4.2.1.6" class="bare">https://tools.ietf.org/html/rfc5280#section-4.2.1.6</a></p>
+</li>
+</ol>
+</div>
+<div class="paragraph">
+<p>If client side authentication is also required then additionally the following needs to be added to Flume agent
+configuration or the global SSL setup can be used (see <a href="#ssltls-support">SSL/TLS support</a> section).
+Each Flume agent has to have its client certificate which has to be trusted by Kafka brokers either
+individually or by their signature chain. Common example is to sign each client certificate by a single Root CA
+which in turn is trusted by Kafka brokers.</p>
+</div>
+<div class="listingblock">
+<div class="content">
+<pre class="highlightjs highlight"><code class="language-properties hljs" data-lang="properties"># optional, the global keystore can be used alternatively
+a1.sources.source1.kafka.consumer.ssl.keystore.location=/path/to/client.keystore.jks
+a1.sources.source1.kafka.consumer.ssl.keystore.password=&lt;password to access the keystore&gt;</code></pre>
+</div>
+</div>
+<div class="paragraph">
+<p>If keystore and key use different password protection then <code>ssl.key.password</code> property will
+provide the required additional secret for both consumer keystores:</p>
+</div>
+<div class="listingblock">
+<div class="content">
+<pre class="highlightjs highlight"><code class="language-properties hljs" data-lang="properties">a1.sources.source1.kafka.consumer.ssl.key.password=&lt;password to access the key&gt;</code></pre>
+</div>
+</div>
+<div class="paragraph">
+<p><strong>Kerberos and Kafka Source:</strong></p>
+</div>
+<div class="paragraph">
+<p>To use Kafka source with a Kafka cluster secured with Kerberos, set the <code>consumer.security.protocol</code> properties noted above for consumer.
+The Kerberos keytab and principal to be used with Kafka brokers is specified in a JAAS file&#8217;s "KafkaClient" section. "Client" section describes the Zookeeper connection if needed.
+See <a href="https://kafka.apache.org/documentation.html#security_sasl_clientconfig">Kafka doc</a>
+for information on the JAAS file contents. The location of this JAAS file and optionally the system wide kerberos configuration can be specified via JAVA_OPTS in flume-env.sh:</p>
+</div>
+<div class="listingblock">
+<div class="content">
+<pre class="highlightjs highlight"><code class="language-properties hljs" data-lang="properties">JAVA_OPTS="$JAVA_OPTS -Djava.security.krb5.conf=/path/to/krb5.conf"
+JAVA_OPTS="$JAVA_OPTS -Djava.security.auth.login.config=/path/to/flume_jaas.conf"</code></pre>
+</div>
+</div>
+<div class="paragraph">
+<p>Example secure configuration using SASL_PLAINTEXT:</p>
+</div>
+<div class="listingblock">
+<div class="content">
+<pre class="highlightjs highlight"><code class="language-properties hljs" data-lang="properties">a1.sources.source1.type = org.apache.flume.source.kafka.KafkaSource
+a1.sources.source1.kafka.bootstrap.servers = kafka-1:9093,kafka-2:9093,kafka-3:9093
+a1.sources.source1.kafka.topics = mytopic
+a1.sources.source1.kafka.consumer.group.id = flume-consumer
+a1.sources.source1.kafka.consumer.security.protocol = SASL_PLAINTEXT
+a1.sources.source1.kafka.consumer.sasl.mechanism = GSSAPI
+a1.sources.source1.kafka.consumer.sasl.kerberos.service.name = kafka</code></pre>
+</div>
+</div>
+<div class="paragraph">
+<p>Example secure configuration using SASL_SSL:</p>
+</div>
+<div class="listingblock">
+<div class="content">
+<pre class="highlightjs highlight"><code class="language-properties hljs" data-lang="properties">a1.sources.source1.type = org.apache.flume.source.kafka.KafkaSource
+a1.sources.source1.kafka.bootstrap.servers = kafka-1:9093,kafka-2:9093,kafka-3:9093
+a1.sources.source1.kafka.topics = mytopic
+a1.sources.source1.kafka.consumer.group.id = flume-consumer
+a1.sources.source1.kafka.consumer.security.protocol = SASL_SSL
+a1.sources.source1.kafka.consumer.sasl.mechanism = GSSAPI
+a1.sources.source1.kafka.consumer.sasl.kerberos.service.name = kafka
+# optional, the global truststore can be used alternatively
+a1.sources.source1.kafka.consumer.ssl.truststore.location=/path/to/truststore.jks
+a1.sources.source1.kafka.consumer.ssl.truststore.password=&lt;password to access the truststore&gt;</code></pre>
+</div>
+</div>
+<div class="paragraph">
+<p>Sample JAAS file. For reference of its content please see client config sections of the desired authentication mechanism (GSSAPI/PLAIN)
+in Kafka documentation of <a href="https://kafka.apache.org/documentation#security_sasl_clientconfig">SASL configuration</a>.
+Since the Kafka Source may also connect to Zookeeper for offset migration, the "Client" section was also added to this example.
+This won&#8217;t be needed unless you require offset migration, or you require this section for other secure components.
+Also please make sure that the operating system user of the Flume processes has read privileges on the jaas and keytab files.</p>
+</div>
+<div class="listingblock">
+<div class="content">
+<pre class="highlightjs highlight"><code class="language-javascript hljs" data-lang="javascript">Client {
+  com.sun.security.auth.module.Krb5LoginModule required
+  useKeyTab=true
+  storeKey=true
+  keyTab="/path/to/keytabs/flume.keytab"
+  principal="flume/flumehost1.example.com@YOURKERBEROSREALM";
+};
+
+KafkaClient {
+  com.sun.security.auth.module.Krb5LoginModule required
+  useKeyTab=true
+  storeKey=true
+  keyTab="/path/to/keytabs/flume.keytab"
+  principal="flume/flumehost1.example.com@YOURKERBEROSREALM";
+};</code></pre>
+</div>
+</div>
+</div>
+<div class="sect3">
+<h4 id="_netcat_tcp_source"><a class="anchor" href="#_netcat_tcp_source"></a>NetCat TCP Source</h4>
+<div class="paragraph">
+<p>A netcat-like source that listens on a given port and turns each line of text
+into an event. Acts like <code>nc -k -l [host] [port]</code>. In other words,
+it opens a specified port and listens for data. The expectation is that the
+supplied data is newline separated text. Each line of text is turned into a
+Flume event and sent via the connected channel.</p>
+</div>
+<div class="paragraph">
+<p>Required properties are in <strong>bold</strong>.</p>
+</div>
+<table class="tableblock frame-all grid-all stretch">
+<colgroup>
+<col style="width: 33.3333%;">
+<col style="width: 33.3333%;">
+<col style="width: 33.3334%;">
+</colgroup>
+<thead>
+<tr>
+<th class="tableblock halign-left valign-top">Property Name</th>
+<th class="tableblock halign-left valign-top">Default</th>
+<th class="tableblock halign-left valign-top">Description</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock"><strong>channels</strong></p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock"><strong>type</strong></p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">The component type name, needs to be <code>netcat</code></p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock"><strong>bind</strong></p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Host name or IP address to bind to</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock"><strong>port</strong></p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Port # to bind to</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">max-line-length</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">512</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Max line length per event body (in bytes)</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">ack-every-event</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">true</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Respond with an "OK" for every event received</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">selector.type</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">replicating</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">replicating or multiplexing</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">selector.*</p></td>
+<td class="tableblock halign-left valign-top"></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Depends on the selector.type value</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">interceptors</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Space-separated list of interceptors</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">interceptors.*</p></td>
+<td class="tableblock halign-left valign-top"></td>
+<td class="tableblock halign-left valign-top"></td>
+</tr>
+</tbody>
+</table>
+<div class="paragraph">
+<p>Example for agent named a1:</p>
+</div>
+<div class="listingblock">
+<div class="content">
+<pre class="highlightjs highlight"><code class="language-properties hljs" data-lang="properties">a1.sources = r1
+a1.channels = c1
+a1.sources.r1.type = netcat
+a1.sources.r1.bind = 0.0.0.0
+a1.sources.r1.port = 6666
+a1.sources.r1.channels = c1</code></pre>
+</div>
+</div>
+</div>
+<div class="sect3">
+<h4 id="_netcat_udp_source"><a class="anchor" href="#_netcat_udp_source"></a>NetCat UDP Source</h4>
+<div class="paragraph">
+<p>As per the original Netcat (TCP) source, this source that listens on a given
+port and turns each line of text into an event and sent via the connected channel.
+Acts like <code>nc -u -k -l [host] [port]</code>.</p>
+</div>
+<div class="paragraph">
+<p>Required properties are in <strong>bold</strong>.</p>
+</div>
+<table class="tableblock frame-all grid-all stretch">
+<colgroup>
+<col style="width: 33.3333%;">
+<col style="width: 33.3333%;">
+<col style="width: 33.3334%;">
+</colgroup>
+<thead>
+<tr>
+<th class="tableblock halign-left valign-top">Property Name</th>
+<th class="tableblock halign-left valign-top">Default</th>
+<th class="tableblock halign-left valign-top">Description</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock"><strong>channels</strong></p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock"><strong>type</strong></p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">The component type name, needs to be <code>netcatudp</code></p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock"><strong>bind</strong></p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Host name or IP address to bind to</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock"><strong>port</strong></p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Port # to bind to</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">remoteAddressHeader</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">selector.type</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">replicating</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">replicating or multiplexing</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">selector.*</p></td>
+<td class="tableblock halign-left valign-top"></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Depends on the selector.type value</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">interceptors</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Space-separated list of interceptors</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">interceptors.*</p></td>
+<td class="tableblock halign-left valign-top"></td>
+<td class="tableblock halign-left valign-top"></td>
+</tr>
+</tbody>
+</table>
+<div class="paragraph">
+<p>Example for agent named a1:</p>
+</div>
+<div class="listingblock">
+<div class="content">
+<pre class="highlightjs highlight"><code class="language-properties hljs" data-lang="properties">a1.sources = r1
+a1.channels = c1
+a1.sources.r1.type = netcatudp
+a1.sources.r1.bind = 0.0.0.0
+a1.sources.r1.port = 6666
+a1.sources.r1.channels = c1</code></pre>
+</div>
+</div>
+</div>
+<div class="sect3">
+<h4 id="_sequence_generator_source"><a class="anchor" href="#_sequence_generator_source"></a>Sequence Generator Source</h4>
+<div class="paragraph">
+<p>A simple sequence generator that continuously generates events with a counter that starts from 0,
+increments by 1 and stops at totalEvents. Retries when it can&#8217;t send events to the channel. Useful
+mainly for testing. During retries it keeps the body of the retried messages the same as before so
+that the number of unique events - after de-duplication at destination - is expected to be
+equal to the specified <code>totalEvents</code>. Required properties are in <strong>bold</strong>.</p>
+</div>
+<table class="tableblock frame-all grid-all stretch">
+<colgroup>
+<col style="width: 33.3333%;">
+<col style="width: 33.3333%;">
+<col style="width: 33.3334%;">
+</colgroup>
+<thead>
+<tr>
+<th class="tableblock halign-left valign-top">Property Name</th>
+<th class="tableblock halign-left valign-top">Default</th>
+<th class="tableblock halign-left valign-top">Description</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock"><strong>channels</strong></p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock"><strong>type</strong></p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">The component type name, needs to be <code>seq</code></p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">selector.type</p></td>
+<td class="tableblock halign-left valign-top"></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">replicating or multiplexing</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">selector.*</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">replicating</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Depends on the selector.type value</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">interceptors</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Space-separated list of interceptors</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">interceptors.*</p></td>
+<td class="tableblock halign-left valign-top"></td>
+<td class="tableblock halign-left valign-top"></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">batchSize</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">1</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Number of events to attempt to process per request loop.</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">totalEvents</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Long.MAX_VALUE</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Number of unique events sent by the source.</p></td>
+</tr>
+</tbody>
+</table>
+<div class="paragraph">
+<p>Example for agent named a1:</p>
+</div>
+<div class="listingblock">
+<div class="content">
+<pre class="highlightjs highlight"><code class="language-properties hljs" data-lang="properties">a1.sources = r1
+a1.channels = c1
+a1.sources.r1.type = seq
+a1.sources.r1.channels = c1</code></pre>
+</div>
+</div>
+</div>
+<div class="sect3">
+<h4 id="_syslog_sources"><a class="anchor" href="#_syslog_sources"></a>Syslog Sources</h4>
+<div class="paragraph">
+<p>Reads syslog data and generate Flume events. The UDP source treats an entire
+message as a single event. The TCP sources create a new event for each string
+of characters separated by a newline ('n').</p>
+</div>
+<div class="paragraph">
+<p>Required properties are in <strong>bold</strong>.</p>
+</div>
+<div class="sect4">
+<h5 id="_syslog_tcp_source"><a class="anchor" href="#_syslog_tcp_source"></a>Syslog TCP Source</h5>
+<div class="paragraph">
+<p>The original, tried-and-true syslog TCP source.</p>
+</div>
+<table class="tableblock frame-all grid-all stretch">
+<colgroup>
+<col style="width: 33.3333%;">
+<col style="width: 33.3333%;">
+<col style="width: 33.3334%;">
+</colgroup>
+<thead>
+<tr>
+<th class="tableblock halign-left valign-top">Property Name</th>
+<th class="tableblock halign-left valign-top">Default</th>
+<th class="tableblock halign-left valign-top">Description</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock"><strong>channels</strong></p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock"><strong>type</strong></p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">The component type name, needs to be <code>syslogtcp</code></p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock"><strong>host</strong></p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Host name or IP address to bind to</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock"><strong>port</strong></p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Port # to bind to</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">eventSize</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">2500</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Maximum size of a single event line, in bytes</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">keepFields</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">none</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Setting this to 'all' will preserve the Priority,
+Timestamp and Hostname in the body of the event.
+A spaced separated list of fields to include
+is allowed as well. Currently, the following
+fields can be included: priority, version,
+timestamp, hostname. The values 'true' and 'false'
+have been deprecated in favor of 'all' and 'none'.</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">clientIPHeader</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">If specified, the IP address of the client will be stored in
+the header of each event using the header name specified here.
+This allows for interceptors and channel selectors to customize
+routing logic based on the IP address of the client.
+Do not use the standard Syslog header names here (like <span id="host">host</span>_)
+because the event header will be overridden in that case.</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">clientHostnameHeader</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">If specified, the host name of the client will be stored in
+the header of each event using the header name specified here.
+This allows for interceptors and channel selectors to customize
+routing logic based on the host name of the client.
+Retrieving the host name may involve a name service reverse
+lookup which may affect the performance.
+Do not use the standard Syslog header names here (like <span id="host">host</span>_)
+because the event header will be overridden in that case.</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">selector.type</p></td>
+<td class="tableblock halign-left valign-top"></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">replicating or multiplexing</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">selector.*</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">replicating</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Depends on the selector.type value</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">interceptors</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Space-separated list of interceptors</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">interceptors.*</p></td>
+<td class="tableblock halign-left valign-top"></td>
+<td class="tableblock halign-left valign-top"></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">ssl</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">false</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Set this to true to enable SSL encryption. If SSL is enabled,
+you must also specify a "keystore" and a "keystore-password",
+either through component level parameters (see below)
+or as global SSL parameters (see <a href="#ssltls-support">SSL/TLS support</a> section).</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">keystore</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">This is the path to a Java keystore file.
+If not specified here, then the global keystore will be used
+(if defined, otherwise configuration error).</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">keystore-password</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">The password for the Java keystore.
+If not specified here, then the global keystore password will be used
+(if defined, otherwise configuration error).</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">keystore-type</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">JKS</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">The type of the Java keystore. This can be "JKS" or "PKCS12".
+If not specified here, then the global keystore type will be used
+(if defined, otherwise the default is JKS).</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">exclude-protocols</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">SSLv3</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Space-separated list of SSL/TLS protocols to exclude.
+SSLv3 will always be excluded in addition to the protocols specified.</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">include-protocols</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Space-separated list of SSL/TLS protocols to include.
+The enabled protocols will be the included protocols without the excluded protocols.
+If included-protocols is empty, it includes every supported protocols.</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">exclude-cipher-suites</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Space-separated list of cipher suites to exclude.</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">include-cipher-suites</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Space-separated list of cipher suites to include.
+The enabled cipher suites will be the included cipher suites without the excluded cipher suites.
+If included-cipher-suites is empty, it includes every supported cipher suites.</p></td>
+</tr>
+</tbody>
+</table>
+<div class="paragraph">
+<p>For example, a syslog TCP source for agent named a1:</p>
+</div>
+<div class="listingblock">
+<div class="content">
+<pre class="highlightjs highlight"><code class="language-properties hljs" data-lang="properties">a1.sources = r1
+a1.channels = c1
+a1.sources.r1.type = syslogtcp
+a1.sources.r1.port = 5140
+a1.sources.r1.host = localhost
+a1.sources.r1.channels = c1</code></pre>
+</div>
+</div>
+</div>
+<div class="sect4">
+<h5 id="_multiport_syslog_tcp_source"><a class="anchor" href="#_multiport_syslog_tcp_source"></a>Multiport Syslog TCP Source</h5>
+<div class="paragraph">
+<p>This is a newer, faster, multi-port capable version of the Syslog TCP source.
+Note that the <code>ports</code> configuration setting has replaced <code>port</code>.
+Multi-port capability means that it can listen on many ports at once in an
+efficient manner. This source uses the Apache Mina library to do that.
+Provides support for RFC-3164 and many common RFC-5424 formatted messages.
+Also provides the capability to configure the character set used on a per-port
+basis.</p>
+</div>
+<table class="tableblock frame-all grid-all stretch">
+<colgroup>
+<col style="width: 33.3333%;">
+<col style="width: 33.3333%;">
+<col style="width: 33.3334%;">
+</colgroup>
+<thead>
+<tr>
+<th class="tableblock halign-left valign-top">Property Name</th>
+<th class="tableblock halign-left valign-top">Default</th>
+<th class="tableblock halign-left valign-top">Description</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock"><strong>channels</strong></p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock"><strong>type</strong></p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">The component type name, needs to be <code>multiport_syslogtcp</code></p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock"><strong>host</strong></p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Host name or IP address to bind to.</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock"><strong>ports</strong></p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Space-separated list (one or more) of ports to bind to.</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">eventSize</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">2500</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Maximum size of a single event line, in bytes.</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">keepFields</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">none</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Setting this to 'all' will preserve the
+Priority, Timestamp and Hostname in the body of the event.
+A spaced separated list of fields to include
+is allowed as well. Currently, the following
+fields can be included: priority, version,
+timestamp, hostname. The values 'true' and 'false'
+have been deprecated in favor of 'all' and 'none'.</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">portHeader</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">If specified, the port number will be stored in the header of each event using the header name specified here. This allows for interceptors and channel selectors to customize routing logic based on the incoming port.</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">clientIPHeader</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">If specified, the IP address of the client will be stored in
+the header of each event using the header name specified here.
+This allows for interceptors and channel selectors to customize
+routing logic based on the IP address of the client.
+Do not use the standard Syslog header names here (like <span id="host">host</span>_)
+because the event header will be overridden in that case.</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">clientHostnameHeader</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">If specified, the host name of the client will be stored in
+the header of each event using the header name specified here.
+This allows for interceptors and channel selectors to customize
+routing logic based on the host name of the client.
+Retrieving the host name may involve a name service reverse
+lookup which may affect the performance.
+Do not use the standard Syslog header names here (like <span id="host">host</span>_)
+because the event header will be overridden in that case.</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">charset.default</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">UTF-8</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Default character set used while parsing syslog events into strings.</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">charset.port.&lt;port&gt;</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Character set is configurable on a per-port basis.</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">batchSize</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">100</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Maximum number of events to attempt to process per request loop. Using the default is usually fine.</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">readBufferSize</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">1024</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Size of the internal Mina read buffer. Provided for performance tuning. Using the default is usually fine.</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">numProcessors</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">(auto-detected)</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Number of processors available on the system for use while processing messages. Default is to auto-detect # of CPUs using the Java Runtime API. Mina will spawn 2 request-processing threads per detected CPU, which is often reasonable.</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">selector.type</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">replicating</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">replicating, multiplexing, or custom</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">selector.*</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Depends on the <code>selector.type</code> value</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">interceptors</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Space-separated list of interceptors.</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">interceptors.*</p></td>
+<td class="tableblock halign-left valign-top"></td>
+<td class="tableblock halign-left valign-top"></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">ssl</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">false</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Set this to true to enable SSL encryption. If SSL is enabled,
+you must also specify a "keystore" and a "keystore-password",
+either through component level parameters (see below)
+or as global SSL parameters (see <a href="#ssltls-support">SSL/TLS support</a> section).</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">keystore</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">This is the path to a Java keystore file.
+If not specified here, then the global keystore will be used
+(if defined, otherwise configuration error).</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">keystore-password</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">The password for the Java keystore.
+If not specified here, then the global keystore password will be used
+(if defined, otherwise configuration error).</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">keystore-type</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">JKS</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">The type of the Java keystore. This can be "JKS" or "PKCS12".
+If not specified here, then the global keystore type will be used
+(if defined, otherwise the default is JKS).</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">exclude-protocols</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">SSLv3</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Space-separated list of SSL/TLS protocols to exclude.
+SSLv3 will always be excluded in addition to the protocols specified.</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">include-protocols</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Space-separated list of SSL/TLS protocols to include.
+The enabled protocols will be the included protocols without the excluded protocols.
+If included-protocols is empty, it includes every supported protocols.</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">exclude-cipher-suites</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Space-separated list of cipher suites to exclude.</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">include-cipher-suites</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Space-separated list of cipher suites to include.
+The enabled cipher suites will be the included cipher suites without the excluded cipher suites.
+If included-cipher-suites is empty, it includes every supported cipher suites.</p></td>
+</tr>
+</tbody>
+</table>
+<div class="paragraph">
+<p>For example, a multiport syslog TCP source for agent named a1:</p>
+</div>
+<div class="listingblock">
+<div class="content">
+<pre class="highlightjs highlight"><code class="language-properties hljs" data-lang="properties">a1.sources = r1
+a1.channels = c1
+a1.sources.r1.type = multiport_syslogtcp
+a1.sources.r1.channels = c1
+a1.sources.r1.host = 0.0.0.0
+a1.sources.r1.ports = 10001 10002 10003
+a1.sources.r1.portHeader = port</code></pre>
+</div>
+</div>
+</div>
+<div class="sect4">
+<h5 id="_syslog_udp_source"><a class="anchor" href="#_syslog_udp_source"></a>Syslog UDP Source</h5>
+<table class="tableblock frame-all grid-all stretch">
+<colgroup>
+<col style="width: 33.3333%;">
+<col style="width: 33.3333%;">
+<col style="width: 33.3334%;">
+</colgroup>
+<thead>
+<tr>
+<th class="tableblock halign-left valign-top">Property Name</th>
+<th class="tableblock halign-left valign-top">Default</th>
+<th class="tableblock halign-left valign-top">Description</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock"><strong>channels</strong></p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock"><strong>type</strong></p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">The component type name, needs to be <code>syslogudp</code></p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock"><strong>host</strong></p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Host name or IP address to bind to</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock"><strong>port</strong></p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Port # to bind to</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">keepFields</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">false</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Setting this to true will preserve the Priority,
+Timestamp and Hostname in the body of the event.</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">clientIPHeader</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">If specified, the IP address of the client will be stored in
+the header of each event using the header name specified here.
+This allows for interceptors and channel selectors to customize
+routing logic based on the IP address of the client.
+Do not use the standard Syslog header names here (like <span id="host">host</span>_)
+because the event header will be overridden in that case.</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">clientHostnameHeader</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">If specified, the host name of the client will be stored in
+the header of each event using the header name specified here.
+This allows for interceptors and channel selectors to customize
+routing logic based on the host name of the client.
+Retrieving the host name may involve a name service reverse
+lookup which may affect the performance.
+Do not use the standard Syslog header names here (like <span id="host">host</span>_)
+because the event header will be overridden in that case.</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">selector.type</p></td>
+<td class="tableblock halign-left valign-top"></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">replicating or multiplexing</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">selector.*</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">replicating</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Depends on the selector.type value</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">interceptors</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Space-separated list of interceptors</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">interceptors.*</p></td>
+<td class="tableblock halign-left valign-top"></td>
+<td class="tableblock halign-left valign-top"></td>
+</tr>
+</tbody>
+</table>
+<div class="paragraph">
+<p>For example, a syslog UDP source for agent named a1:</p>
+</div>
+<div class="listingblock">
+<div class="content">
+<pre class="highlightjs highlight"><code class="language-properties hljs" data-lang="properties">a1.sources = r1
+a1.channels = c1
+a1.sources.r1.type = syslogudp
+a1.sources.r1.port = 5140
+a1.sources.r1.host = localhost
+a1.sources.r1.channels = c1</code></pre>
+</div>
+</div>
+</div>
+</div>
+<div class="sect3">
+<h4 id="_http_source"><a class="anchor" href="#_http_source"></a>HTTP Source</h4>
+<div class="paragraph">
+<p>A source which accepts Flume Events by HTTP POST and GET. GET should be used
+for experimentation only. HTTP requests are converted into flume events by
+a pluggable "handler" which must implement the HTTPSourceHandler interface.
+This handler takes a HttpServletRequest and returns a list of
+flume events. All events handled from one Http request are committed to the channel
+in one transaction, thus allowing for increased efficiency on channels like
+the file channel. If the handler throws an exception, this source will
+return a HTTP status of 400. If the channel is full, or the source is unable to
+append events to the channel, the source will return a HTTP 503 - Temporarily
+unavailable status.</p>
+</div>
+<div class="paragraph">
+<p>All events sent in one post request are considered to be one batch and
+inserted into the channel in one transaction.</p>
+</div>
+<div class="paragraph">
+<p>This source is based on Jetty 9.4 and offers the ability to set additional
+Jetty-specific parameters which will be passed directly to the Jetty components.</p>
+</div>
+<table class="tableblock frame-all grid-all stretch">
+<colgroup>
+<col style="width: 33.3333%;">
+<col style="width: 33.3333%;">
+<col style="width: 33.3334%;">
+</colgroup>
+<thead>
+<tr>
+<th class="tableblock halign-left valign-top">Property Name</th>
+<th class="tableblock halign-left valign-top">Default</th>
+<th class="tableblock halign-left valign-top">Description</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock"><strong>type</strong></p></td>
+<td class="tableblock halign-left valign-top"></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">The component type name, needs to be <code>http</code></p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock"><strong>port</strong></p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">The port the source should bind to.</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">bind</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">0.0.0.0</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">The hostname or IP address to listen on</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">handler</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock"><code>org.apache.flume.source.http.JSONHandler</code></p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">The FQCN of the handler class.</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">handler.*</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Config parameters for the handler</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">selector.type</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">replicating</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">replicating or multiplexing</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">selector.*</p></td>
+<td class="tableblock halign-left valign-top"></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Depends on the selector.type value</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">interceptors</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Space-separated list of interceptors</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">interceptors.*</p></td>
+<td class="tableblock halign-left valign-top"></td>
+<td class="tableblock halign-left valign-top"></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">ssl</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">false</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Set the property true, to enable SSL. <em>HTTP Source does not support SSLv3.</em></p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">exclude-protocols</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">SSLv3</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Space-separated list of SSL/TLS protocols to exclude.
+SSLv3 will always be excluded in addition to the protocols specified.</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">include-protocols</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Space-separated list of SSL/TLS protocols to include.
+The enabled protocols will be the included protocols without the excluded protocols.
+If included-protocols is empty, it includes every supported protocols.</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">exclude-cipher-suites</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Space-separated list of cipher suites to exclude.</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">include-cipher-suites</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Space-separated list of cipher suites to include.
+The enabled cipher suites will be the included cipher suites without the excluded cipher suites.</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">keystore</p></td>
+<td class="tableblock halign-left valign-top"></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Location of the keystore including keystore file name.
+If SSL is enabled but the keystore is not specified here,
+then the global keystore will be used
+(if defined, otherwise configuration error).</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">keystore-password</p></td>
+<td class="tableblock halign-left valign-top"></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Keystore password.
+If SSL is enabled but the keystore password is not specified here,
+then the global keystore password will be used
+(if defined, otherwise configuration error).</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">keystore-type</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">JKS</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Keystore type. This can be "JKS" or "PKCS12".</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">QueuedThreadPool.*</p></td>
+<td class="tableblock halign-left valign-top"></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Jetty specific settings to be set on org.eclipse.jetty.util.thread.QueuedThreadPool.
+N.B. QueuedThreadPool will only be used if at least one property of this class is set.</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">HttpConfiguration.*</p></td>
+<td class="tableblock halign-left valign-top"></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Jetty specific settings to be set on org.eclipse.jetty.server.HttpConfiguration</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">SslContextFactory.*</p></td>
+<td class="tableblock halign-left valign-top"></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Jetty specific settings to be set on org.eclipse.jetty.util.ssl.SslContextFactory (only
+applicable when <em>ssl</em> is set to true).</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">ServerConnector.*</p></td>
+<td class="tableblock halign-left valign-top"></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Jetty specific settings to be set on org.eclipse.jetty.server.ServerConnector</p></td>
+</tr>
+</tbody>
+</table>
+<div class="paragraph">
+<p>Deprecated Properties</p>
+</div>
+<table class="tableblock frame-all grid-all stretch">
+<colgroup>
+<col style="width: 33.3333%;">
+<col style="width: 33.3333%;">
+<col style="width: 33.3334%;">
+</colgroup>
+<thead>
+<tr>
+<th class="tableblock halign-left valign-top">Property Name</th>
+<th class="tableblock halign-left valign-top">Default</th>
+<th class="tableblock halign-left valign-top">Description</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">keystorePassword</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Use <em>keystore-password</em>. Deprecated value will be overwritten with the new one.</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">excludeProtocols</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">SSLv3</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Use <em>exclude-protocols</em>. Deprecated value will be overwritten with the new one.</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">enableSSL</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">false</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Use <em>ssl</em>. Deprecated value will be overwritten with the new one.</p></td>
+</tr>
+</tbody>
+</table>
+<div class="paragraph">
+<p>N.B. Jetty-specific settings are set using the setter-methods on the objects listed above. For full details see the Javadoc for these classes
+(<a href="https://www.eclipse.org/jetty/javadoc/jetty-9/org/eclipse/jetty/util/thread/QueuedThreadPool.html">QueuedThreadPool</a>,
+<a href="https://www.eclipse.org/jetty/javadoc/jetty-9/org/eclipse/jetty/server/HttpConfiguration.html">HttpConfiguration</a>,
+<a href="https://www.eclipse.org/jetty/javadoc/jetty-9/org/eclipse/jetty/util/ssl/SslContextFactory.html">SslContextFactory</a> and
+<a href="https://www.eclipse.org/jetty/javadoc/jetty-9/org/eclipse/jetty/server/ServerConnector.html">ServerConnector</a>).</p>
+</div>
+<div class="paragraph">
+<p>When using Jetty-specific setings, named properites above will take precedence (for example excludeProtocols will take
+precedence over SslContextFactory.ExcludeProtocols). All properties will be inital lower case.</p>
+</div>
+<div class="paragraph">
+<p>An example http source for agent named a1:</p>
+</div>
+<div class="listingblock">
+<div class="content">
+<pre class="highlightjs highlight"><code class="language-properties hljs" data-lang="properties">a1.sources = r1
+a1.channels = c1
+a1.sources.r1.type = http
+a1.sources.r1.port = 5140
+a1.sources.r1.channels = c1
+a1.sources.r1.handler = org.example.rest.RestHandler
+a1.sources.r1.handler.nickname = random props
+a1.sources.r1.HttpConfiguration.sendServerVersion = false
+a1.sources.r1.ServerConnector.idleTimeout = 300</code></pre>
+</div>
+</div>
+<div class="sect4">
+<h5 id="_jsonhandler"><a class="anchor" href="#_jsonhandler"></a>JSONHandler</h5>
+<div class="paragraph">
+<p>A handler is provided out of the box which can handle events represented in
+JSON format, and supports UTF-8, UTF-16 and UTF-32 character sets. The handler
+accepts an array of events (even if there is only one event, the event has to be
+sent in an array) and converts them to a Flume event based on the
+encoding specified in the request. If no encoding is specified, UTF-8 is assumed.
+The JSON handler supports UTF-8, UTF-16 and UTF-32.
+Events are represented as follows.</p>
+</div>
+<div class="listingblock">
+<div class="content">
+<pre class="highlightjs highlight"><code class="language-javascript hljs" data-lang="javascript">[{
+  "headers" : {
+             "timestamp" : "434324343",
+             "host" : "random_host.example.com"
+             },
+  "body" : "random_body"
+  },
+  {
+  "headers" : {
+             "namenode" : "namenode.example.com",
+             "datanode" : "random_datanode.example.com"
+             },
+  "body" : "really_random_body"
+  }]</code></pre>
+</div>
+</div>
+<div class="paragraph">
+<p>To set the charset, the request must have content type specified as
+<code>application/json; charset=UTF-8</code> (replace UTF-8 with UTF-16 or UTF-32 as
+required).</p>
+</div>
+<div class="paragraph">
+<p>One way to create an event in the format expected by this handler is to
+use JSONEvent provided in the Flume SDK and use Google Gson to create the JSON
+string using the Gson#fromJson(Object, Type)
+method. The type token to pass as the 2nd argument of this method
+for list of events can be created by:</p>
+</div>
+<div class="listingblock">
+<div class="content">
+<pre class="highlightjs highlight"><code class="language-java hljs" data-lang="java">Type type = new TypeToken&lt;List&lt;JSONEvent&gt;&gt;() {}.getType();</code></pre>
+</div>
+</div>
+</div>
+<div class="sect4">
+<h5 id="_blobhandler"><a class="anchor" href="#_blobhandler"></a>BlobHandler</h5>
+<div class="paragraph">
+<p>By default HTTPSource splits JSON input into Flume events. As an alternative, BlobHandler is a handler for HTTPSource that returns an event that contains the request parameters as well as the Binary Large Object (BLOB) uploaded with this request. For example a PDF or JPG file. Note that this approach is not suitable for very large objects because it buffers up the entire BLOB in RAM.</p>
+</div>
+<table class="tableblock frame-all grid-all stretch">
+<colgroup>
+<col style="width: 33.3333%;">
+<col style="width: 33.3333%;">
+<col style="width: 33.3334%;">
+</colgroup>
+<thead>
+<tr>
+<th class="tableblock halign-left valign-top">Property Name</th>
+<th class="tableblock halign-left valign-top">Default</th>
+<th class="tableblock halign-left valign-top">Description</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock"><strong>handler</strong></p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">The FQCN of this class: <code>org.apache.flume.sink.solr.morphline.BlobHandler</code></p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">handler.maxBlobLength</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">100000000</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">The maximum number of bytes to read and buffer for a given request</p></td>
+</tr>
+</tbody>
+</table>
+</div>
+</div>
+<div class="sect3">
+<h4 id="_stress_source"><a class="anchor" href="#_stress_source"></a>Stress Source</h4>
+<div class="paragraph">
+<p>StressSource is an internal load-generating source implementation which is very useful for
+stress tests. It allows User to configure the size of Event payload, with empty headers.
+User can configure total number of events to be sent as well maximum number of Successful
+Event to be delivered.</p>
+</div>
+<div class="paragraph">
+<p>Required properties are in <strong>bold</strong>.</p>
+</div>
+<table class="tableblock frame-all grid-all stretch">
+<colgroup>
+<col style="width: 33.3333%;">
+<col style="width: 33.3333%;">
+<col style="width: 33.3334%;">
+</colgroup>
+<thead>
+<tr>
+<th class="tableblock halign-left valign-top">Property Name</th>
+<th class="tableblock halign-left valign-top">Default</th>
+<th class="tableblock halign-left valign-top">Description</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock"><strong>type</strong></p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">The component type name, needs to be <code>org.apache.flume.source.StressSource</code></p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">size</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">500</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Payload size of each Event. Unit:**byte**</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">maxTotalEvents</p></td>
+<td class="tableblock halign-left valign-top"><div class="content"><div class="dlist">
+<dl>
+<dt class="hdlist1"><code>-1</code></dt>
+</dl>
+</div></div></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Maximum number of Events to be sent</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">maxSuccessfulEvents</p></td>
+<td class="tableblock halign-left valign-top"><div class="content"><div class="dlist">
+<dl>
+<dt class="hdlist1"><code>-1</code></dt>
+</dl>
+</div></div></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Maximum number of Events successfully sent</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">batchSize</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">1</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Number of Events to be sent in one batch</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">maxEventsPerSecond</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">0</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">When set to an integer greater than zero, enforces a rate limiter onto the source.</p></td>
+</tr>
+</tbody>
+</table>
+<div class="paragraph">
+<p>Example for agent named <strong>a1</strong>:</p>
+</div>
+<div class="listingblock">
+<div class="content">
+<pre class="highlightjs highlight"><code class="language-properties hljs" data-lang="properties">a1.sources = stresssource-1
+a1.channels = memoryChannel-1
+a1.sources.stresssource-1.type = org.apache.flume.source.StressSource
+a1.sources.stresssource-1.size = 10240
+a1.sources.stresssource-1.maxTotalEvents = 1000000
+a1.sources.stresssource-1.channels = memoryChannel-1</code></pre>
+</div>
+</div>
+</div>
+<div class="sect3">
+<h4 id="_legacy_sources"><a class="anchor" href="#_legacy_sources"></a>Legacy Sources</h4>
+<div class="paragraph">
+<p>The legacy sources allow a Flume 1.x agent to receive events from Flume 0.9.4
+agents. It accepts events in the Flume 0.9.4 format, converts them to the Flume
+1.0 format, and stores them in the connected channel. The 0.9.4 event
+properties like timestamp, pri, host, nanos, etc get converted to 1.x event
+header attributes. The legacy source supports both Avro and Thrift RPC
+connections. To use this bridge between two Flume versions, you need to start a
+Flume 1.x agent with the avroLegacy or thriftLegacy source. The 0.9.4 agent
+should have the agent Sink pointing to the host/port of the 1.x agent.</p>
+</div>
+<div class="admonitionblock note">
+<table>
+<tr>
+<td class="icon">
+<i class="fa icon-note" title="Note"></i>
+</td>
+<td class="content">
+<div class="paragraph">
+<p>The reliability semantics of Flume 1.x are different from that of
+Flume 0.9.x. The E2E or DFO mode of a Flume 0.9.x agent will not be
+supported by the legacy source. The only supported 0.9.x mode is the
+best effort, though the reliability setting of the 1.x flow will be
+applicable to the events once they are saved into the Flume 1.x
+channel by the legacy source.</p>
+</div>
+</td>
+</tr>
+</table>
+</div>
+<div class="paragraph">
+<p>Required properties are in <strong>bold</strong>.</p>
+</div>
+<div class="sect4">
+<h5 id="_avro_legacy_source"><a class="anchor" href="#_avro_legacy_source"></a>Avro Legacy Source</h5>
+<table class="tableblock frame-all grid-all stretch">
+<colgroup>
+<col style="width: 33.3333%;">
+<col style="width: 33.3333%;">
+<col style="width: 33.3334%;">
+</colgroup>
+<thead>
+<tr>
+<th class="tableblock halign-left valign-top">Property Name</th>
+<th class="tableblock halign-left valign-top">Default</th>
+<th class="tableblock halign-left valign-top">Description</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock"><strong>channels</strong></p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock"><strong>type</strong></p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">The component type name, needs to be <code>org.apache.flume.source.avroLegacy.AvroLegacySource</code></p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock"><strong>host</strong></p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">The hostname or IP address to bind to</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock"><strong>port</strong></p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">The port # to listen on</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">selector.type</p></td>
+<td class="tableblock halign-left valign-top"></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">replicating or multiplexing</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">selector.*</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">replicating</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Depends on the selector.type value</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">interceptors</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Space-separated list of interceptors</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">interceptors.*</p></td>
+<td class="tableblock halign-left valign-top"></td>
+<td class="tableblock halign-left valign-top"></td>
+</tr>
+</tbody>
+</table>
+<div class="paragraph">
+<p>Example for agent named a1:</p>
+</div>
+<div class="listingblock">
+<div class="content">
+<pre class="highlightjs highlight"><code class="language-properties hljs" data-lang="properties">a1.sources = r1
+a1.channels = c1
+a1.sources.r1.type = org.apache.flume.source.avroLegacy.AvroLegacySource
+a1.sources.r1.host = 0.0.0.0
+a1.sources.r1.bind = 6666
+a1.sources.r1.channels = c1</code></pre>
+</div>
+</div>
+</div>
+<div class="sect4">
+<h5 id="_thrift_legacy_source"><a class="anchor" href="#_thrift_legacy_source"></a>Thrift Legacy Source</h5>
+<table class="tableblock frame-all grid-all stretch">
+<colgroup>
+<col style="width: 33.3333%;">
+<col style="width: 33.3333%;">
+<col style="width: 33.3334%;">
+</colgroup>
+<thead>
+<tr>
+<th class="tableblock halign-left valign-top">Property Name</th>
+<th class="tableblock halign-left valign-top">Default</th>
+<th class="tableblock halign-left valign-top">Description</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock"><strong>channels</strong></p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock"><strong>type</strong></p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">The component type name, needs to be <code>org.apache.flume.source.thriftLegacy.ThriftLegacySource</code></p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock"><strong>host</strong></p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">The hostname or IP address to bind to</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock"><strong>port</strong></p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">The port # to listen on</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">selector.type</p></td>
+<td class="tableblock halign-left valign-top"></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">replicating or multiplexing</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">selector.*</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">replicating</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Depends on the selector.type value</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">interceptors</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Space-separated list of interceptors</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">interceptors.*</p></td>
+<td class="tableblock halign-left valign-top"></td>
+<td class="tableblock halign-left valign-top"></td>
+</tr>
+</tbody>
+</table>
+<div class="paragraph">
+<p>Example for agent named a1:</p>
+</div>
+<div class="listingblock">
+<div class="content">
+<pre class="highlightjs highlight"><code class="language-properties hljs" data-lang="properties">a1.sources = r1
+a1.channels = c1
+a1.sources.r1.type = org.apache.flume.source.thriftLegacy.ThriftLegacySource
+a1.sources.r1.host = 0.0.0.0
+a1.sources.r1.bind = 6666
+a1.sources.r1.channels = c1</code></pre>
+</div>
+</div>
+</div>
+</div>
+<div class="sect3">
+<h4 id="_custom_source"><a class="anchor" href="#_custom_source"></a>Custom Source</h4>
+<div class="paragraph">
+<p>A custom source is your own implementation of the Source interface. A custom
+source&#8217;s class and its dependencies must be included in the agent&#8217;s classpath
+when starting the Flume agent. The type of the custom source is its FQCN.</p>
+</div>
+<table class="tableblock frame-all grid-all stretch">
+<colgroup>
+<col style="width: 33.3333%;">
+<col style="width: 33.3333%;">
+<col style="width: 33.3334%;">
+</colgroup>
+<thead>
+<tr>
+<th class="tableblock halign-left valign-top">Property Name</th>
+<th class="tableblock halign-left valign-top">Default</th>
+<th class="tableblock halign-left valign-top">Description</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock"><strong>channels</strong></p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock"><strong>type</strong></p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">The component type name, needs to be your FQCN</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">selector.type</p></td>
+<td class="tableblock halign-left valign-top"></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock"><code>replicating</code> or <code>multiplexing</code></p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">selector.*</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">replicating</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Depends on the selector.type value</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">interceptors</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Space-separated list of interceptors</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">interceptors.*</p></td>
+<td class="tableblock halign-left valign-top"></td>
+<td class="tableblock halign-left valign-top"></td>
+</tr>
+</tbody>
+</table>
+<div class="paragraph">
+<p>Example for agent named a1:</p>
+</div>
+<div class="listingblock">
+<div class="content">
+<pre class="highlightjs highlight"><code class="language-properties hljs" data-lang="properties">a1.sources = r1
+a1.channels = c1
+a1.sources.r1.type = org.example.MySource
+a1.sources.r1.channels = c1</code></pre>
+</div>
+</div>
+</div>
+<div class="sect3">
+<h4 id="_scribe_source"><a class="anchor" href="#_scribe_source"></a>Scribe Source</h4>
+<div class="paragraph">
+<p>Scribe is another type of ingest system. To adopt existing Scribe ingest system,
+Flume should use ScribeSource based on Thrift with compatible transfering protocol.
+For deployment of Scribe please follow the guide from Facebook.
+Required properties are in <strong>bold</strong>.</p>
+</div>
+<table class="tableblock frame-all grid-all stretch">
+<colgroup>
+<col style="width: 33.3333%;">
+<col style="width: 33.3333%;">
+<col style="width: 33.3334%;">
+</colgroup>
+<thead>
+<tr>
+<th class="tableblock halign-left valign-top">Property Name</th>
+<th class="tableblock halign-left valign-top">Default</th>
+<th class="tableblock halign-left valign-top">Description</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock"><strong>type</strong></p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">The component type name, needs to be <code>org.apache.flume.source.scribe.ScribeSource</code></p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">port</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">1499</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Port that Scribe should be connected</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">maxReadBufferBytes</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">16384000</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Thrift Default FrameBuffer Size</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">workerThreads</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">5</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Handing threads number in Thrift</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">selector.type</p></td>
+<td class="tableblock halign-left valign-top"></td>
+<td class="tableblock halign-left valign-top"></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">selector.*</p></td>
+<td class="tableblock halign-left valign-top"></td>
+<td class="tableblock halign-left valign-top"></td>
+</tr>
+</tbody>
+</table>
+<div class="paragraph">
+<p>Example for agent named a1:</p>
+</div>
+<div class="listingblock">
+<div class="content">
+<pre class="highlightjs highlight"><code class="language-properties hljs" data-lang="properties">a1.sources = r1
+a1.channels = c1
+a1.sources.r1.type = org.apache.flume.source.scribe.ScribeSource
+a1.sources.r1.port = 1463
+a1.sources.r1.workerThreads = 5
+a1.sources.r1.channels = c1</code></pre>
+</div>
+</div>
+</div>
+</div>
+<div class="sect2">
+<h3 id="_flume_sinks"><a class="anchor" href="#_flume_sinks"></a>Flume Sinks</h3>
+<div class="sect3">
+<h4 id="_hdfs_sink"><a class="anchor" href="#_hdfs_sink"></a>HDFS Sink</h4>
+<div class="paragraph">
+<p>This sink writes events into the Hadoop Distributed File System (HDFS). It
+currently supports creating text and sequence files. It supports compression in
+both file types. The files can be rolled (close current file and create a new
+one) periodically based on the elapsed time or size of data or number of events.
+It also buckets/partitions data by attributes like timestamp or machine
+where the event originated. The HDFS directory path may contain formatting
+escape sequences that will replaced by the HDFS sink to generate a
+directory/file name to store the events. Using this sink requires hadoop to be
+installed so that Flume can use the Hadoop jars to communicate with the HDFS
+cluster. Note that a version of Hadoop that supports the sync() call is
+required.</p>
+</div>
+<div class="paragraph">
+<p>The following are the escape sequences supported:</p>
+</div>
+<table class="tableblock frame-all grid-all stretch">
+<colgroup>
+<col style="width: 50%;">
+<col style="width: 50%;">
+</colgroup>
+<thead>
+<tr>
+<th class="tableblock halign-left valign-top">Alias</th>
+<th class="tableblock halign-left valign-top">Description</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">%{host}</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Substitute value of event header named "host". Arbitrary header names are supported.</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">%t</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Unix time in milliseconds</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">%a</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">locale&#8217;s short weekday name (Mon, Tue, &#8230;&#8203;)</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">%A</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">locale&#8217;s full weekday name (Monday, Tuesday, &#8230;&#8203;)</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">%b</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">locale&#8217;s short month name (Jan, Feb, &#8230;&#8203;)</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">%B</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">locale&#8217;s long month name (January, February, &#8230;&#8203;)</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">%c</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">locale&#8217;s date and time (Thu Mar 3 23:05:25 2005)</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">%d</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">day of month (01)</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">%e</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">day of month without padding (1)</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">%D</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">date; same as %m/%d/%y</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">%H</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">hour (00..23)</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">%I</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">hour (01..12)</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">%j</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">day of year (001..366)</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">%k</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">hour ( 0..23)</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">%m</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">month (01..12)</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">%n</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">month without padding (1..12)</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">%M</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">minute (00..59)</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">%p</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">locale&#8217;s equivalent of am or pm</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">%s</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">seconds since 1970-01-01 00:00:00 UTC</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">%S</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">second (00..59)</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">%y</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">last two digits of year (00..99)</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">%Y</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">year (2010)</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">%z</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#43;hhmm numeric timezone (for example, -0400)</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">%[localhost]</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Substitute the hostname of the host where the agent is running</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">%[IP]</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Substitute the IP address of the host where the agent is running</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">%[FQDN]</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Substitute the canonical hostname of the host where the agent is running</p></td>
+</tr>
+</tbody>
+</table>
+<div class="paragraph">
+<p>Note: The escape strings %[localhost], %[IP] and %[FQDN] all rely on Java&#8217;s ability to obtain the
+hostname, which may fail in some networking environments.</p>
+</div>
+<div class="paragraph">
+<p>The file in use will have the name mangled to include ".tmp" at the end. Once
+the file is closed, this extension is removed. This allows excluding partially
+complete files in the directory.
+Required properties are in <strong>bold</strong>.</p>
+</div>
+<div class="admonitionblock note">
+<table>
+<tr>
+<td class="icon">
+<i class="fa icon-note" title="Note"></i>
+</td>
+<td class="content">
+<div class="paragraph">
+<p>For all of the time related escape sequences, a header with the key
+"timestamp" must exist among the headers of the event (unless <code>hdfs.useLocalTimeStamp</code> is set to <code>true</code>). One way to add
+this automatically is to use the TimestampInterceptor.</p>
+</div>
+</td>
+</tr>
+</table>
+</div>
+<table class="tableblock frame-all grid-all stretch">
+<colgroup>
+<col style="width: 33.3333%;">
+<col style="width: 33.3333%;">
+<col style="width: 33.3334%;">
+</colgroup>
+<thead>
+<tr>
+<th class="tableblock halign-left valign-top">Name</th>
+<th class="tableblock halign-left valign-top">Default</th>
+<th class="tableblock halign-left valign-top">Description</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock"><strong>channel</strong></p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock"><strong>type</strong></p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">The component type name, needs to be <code>hdfs</code></p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock"><strong>hdfs.path</strong></p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">HDFS directory path (eg hdfs://namenode/flume/webdata/)</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">hdfs.filePrefix</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">FlumeData</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Name prefixed to files created by Flume in hdfs directory</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">hdfs.fileSuffix</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Suffix to append to file (eg <code>.avro</code> - <em>NOTE: period is not automatically added</em>)</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">hdfs.inUsePrefix</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Prefix that is used for temporal files that flume actively writes into</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">hdfs.inUseSuffix</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock"><code>.tmp</code></p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Suffix that is used for temporal files that flume actively writes into</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">hdfs.emptyInUseSuffix</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">false</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">If <code>false</code> an <code>hdfs.inUseSuffix</code> is used while writing the output. After closing the output <code>hdfs.inUseSuffix</code> is removed from the output file name. If <code>true</code> the <code>hdfs.inUseSuffix</code> parameter is ignored an empty string is used instead.</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">hdfs.rollInterval</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">30</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Number of seconds to wait before rolling current file
+(0 = never roll based on time interval)</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">hdfs.rollSize</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">1024</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">File size to trigger roll, in bytes (0: never roll based on file size)</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">hdfs.rollCount</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">10</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Number of events written to file before it rolled
+(0 = never roll based on number of events)</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">hdfs.idleTimeout</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">0</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Timeout after which inactive files get closed
+(0 = disable automatic closing of idle files)</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">hdfs.batchSize</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">100</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">number of events written to file before it is flushed to HDFS</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">hdfs.codeC</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Compression codec. one of following : gzip, bzip2, lzo, lzop, snappy</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">hdfs.fileType</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">SequenceFile</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">File format: currently <code>SequenceFile</code>, <code>DataStream</code> or <code>CompressedStream</code>
+(1)DataStream will not compress output file and please don&#8217;t set codeC
+(2)CompressedStream requires set hdfs.codeC with an available codeC</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">hdfs.maxOpenFiles</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">5000</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Allow only this number of open files. If this number is exceeded, the oldest file is closed.</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">hdfs.minBlockReplicas</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Specify minimum number of replicas per HDFS block. If not specified, it comes from the default Hadoop config in the classpath.</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">hdfs.writeFormat</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Writable</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Format for sequence file records. One of <code>Text</code> or <code>Writable</code>. Set to <code>Text</code> before creating data files with Flume, otherwise those files cannot be read by either Apache Impala (incubating) or Apache Hive.</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">hdfs.threadsPoolSize</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">10</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Number of threads per HDFS sink for HDFS IO ops (open, write, etc.)</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">hdfs.rollTimerPoolSize</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">1</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Number of threads per HDFS sink for scheduling timed file rolling</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">hdfs.kerberosPrincipal</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Kerberos user principal for accessing secure HDFS</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">hdfs.kerberosKeytab</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Kerberos keytab for accessing secure HDFS</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">hdfs.proxyUser</p></td>
+<td class="tableblock halign-left valign-top"></td>
+<td class="tableblock halign-left valign-top"></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">hdfs.round</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">false</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Should the timestamp be rounded down (if true, affects all time based escape sequences except %t)</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">hdfs.roundValue</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">1</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Rounded down to the highest multiple of this (in the unit configured using <code>hdfs.roundUnit</code>), less than current time.</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">hdfs.roundUnit</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">second</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">The unit of the round down value - <code>second</code>, <code>minute</code> or <code>hour</code>.</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">hdfs.timeZone</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Local Time</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Name of the timezone that should be used for resolving the directory path, e.g. America/Los_Angeles.</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">hdfs.useLocalTimeStamp</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">false</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Use the local time (instead of the timestamp from the event header) while replacing the escape sequences.</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">hdfs.closeTries</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">0</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Number of times the sink must try renaming a file, after initiating a close attempt. If set to 1, this sink will not re-try a failed rename
+(due to, for example, NameNode or DataNode failure), and may leave the file in an open state with a .tmp extension.
+If set to 0, the sink will try to rename the file until the file is eventually renamed (there is no limit on the number of times it would try).
+The file may still remain open if the close call fails but the data will be intact and in this case, the file will be closed only after a Flume restart.</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">hdfs.retryInterval</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">180</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Time in seconds between consecutive attempts to close a file. Each close call costs multiple RPC round-trips to the Namenode,
+so setting this too low can cause a lot of load on the name node. If set to 0 or less, the sink will not
+attempt to close the file if the first attempt fails, and may leave the file open or with a ".tmp" extension.</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">serializer</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock"><code>TEXT</code></p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Other possible options include <code>avro_event</code> or the
+fully-qualified class name of an implementation of the
+<code>EventSerializer.Builder</code> interface.</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">serializer.*</p></td>
+<td class="tableblock halign-left valign-top"></td>
+<td class="tableblock halign-left valign-top"></td>
+</tr>
+</tbody>
+</table>
+<div class="paragraph">
+<p>Deprecated Properties</p>
+</div>
+<table class="tableblock frame-all grid-all stretch">
+<colgroup>
+<col style="width: 33.3333%;">
+<col style="width: 33.3333%;">
+<col style="width: 33.3334%;">
+</colgroup>
+<thead>
+<tr>
+<th class="tableblock halign-left valign-top">Name</th>
+<th class="tableblock halign-left valign-top">Default</th>
+<th class="tableblock halign-left valign-top">Description</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">hdfs.callTimeout</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">30000</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Number of milliseconds allowed for HDFS operations, such as open, write, flush, close.
+This number should be increased if many HDFS timeout operations are occurring.</p></td>
+</tr>
+</tbody>
+</table>
+<div class="paragraph">
+<p>Example for agent named a1:</p>
+</div>
+<div class="listingblock">
+<div class="content">
+<pre class="highlightjs highlight"><code class="language-properties hljs" data-lang="properties">a1.channels = c1
+a1.sinks = k1
+a1.sinks.k1.type = hdfs
+a1.sinks.k1.channel = c1
+a1.sinks.k1.hdfs.path = /flume/events/%Y-%m-%d/%H%M/%S
+a1.sinks.k1.hdfs.filePrefix = events-
+a1.sinks.k1.hdfs.round = true
+a1.sinks.k1.hdfs.roundValue = 10
+a1.sinks.k1.hdfs.roundUnit = minute</code></pre>
+</div>
+</div>
+<div class="paragraph">
+<p>The above configuration will round down the timestamp to the last 10th minute. For example, an event with
+timestamp 11:54:34 AM, June 12, 2012 will cause the hdfs path to become <code>/flume/events/2012-06-12/1150/00</code>.</p>
+</div>
+</div>
+<div class="sect3">
+<h4 id="_hive_sink"><a class="anchor" href="#_hive_sink"></a>Hive Sink</h4>
+<div class="paragraph">
+<p>This sink streams events containing delimited text or JSON data directly into a Hive table or partition.
+Events are written using Hive transactions. As soon as a set of events are committed to Hive, they become
+immediately visible to Hive queries. Partitions to which flume will stream to can either be pre-created
+or, optionally, Flume can create them if they are missing. Fields from incoming event data are mapped to
+corresponding columns in the Hive table.</p>
+</div>
+<table class="tableblock frame-all grid-all stretch">
+<colgroup>
+<col style="width: 33.3333%;">
+<col style="width: 33.3333%;">
+<col style="width: 33.3334%;">
+</colgroup>
+<thead>
+<tr>
+<th class="tableblock halign-left valign-top">Name</th>
+<th class="tableblock halign-left valign-top">Default</th>
+<th class="tableblock halign-left valign-top">Description</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock"><strong>channel</strong></p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock"><strong>type</strong></p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">The component type name, needs to be <code>hive</code></p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock"><strong>hive.metastore</strong></p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Hive metastore URI (eg thrift://a.b.com:9083 )</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock"><strong>hive.database</strong></p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Hive database name</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock"><strong>hive.table</strong></p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Hive table name</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">hive.partition</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Comma separate list of partition values identifying the partition to write to. May contain escape
+sequences. E.g: If the table is partitioned by (continent: string, country :string, time : string)
+then 'Asia,India,2014-02-26-01-21' will indicate continent=Asia,country=India,time=2014-02-26-01-21</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">hive.txnsPerBatchAsk</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">100</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Hive grants a <em>batch of transactions</em> instead of single transactions to streaming clients like Flume.
+This setting configures the number of desired transactions per Transaction Batch. Data from all
+transactions in a single batch end up in a single file. Flume will write a maximum of batchSize events
+in each transaction in the batch. This setting in conjunction with batchSize provides control over the
+size of each file. Note that eventually Hive will transparently compact these files into larger files.</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">heartBeatInterval</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">240</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">(In seconds) Interval between consecutive heartbeats sent to Hive to keep unused transactions from expiring.
+Set this value to 0 to disable heartbeats.</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">autoCreatePartitions</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">true</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Flume will automatically create the necessary Hive partitions to stream to</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">batchSize</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">15000</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Max number of events written to Hive in a single Hive transaction</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">maxOpenConnections</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">500</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Allow only this number of open connections. If this number is exceeded, the least recently used connection is closed.</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">callTimeout</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">10000</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">(In milliseconds) Timeout for Hive &amp; HDFS I/O operations, such as openTxn, write, commit, abort.</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock"><strong>serializer</strong></p></td>
+<td class="tableblock halign-left valign-top"></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Serializer is responsible for parsing out field from the event and mapping them to columns in the hive table.
+Choice of serializer depends upon the format of the data in the event. Supported serializers: DELIMITED and JSON</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">roundUnit</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">minute</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">The unit of the round down value - <code>second</code>, <code>minute</code> or <code>hour</code>.</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">roundValue</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">1</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Rounded down to the highest multiple of this (in the unit configured using hive.roundUnit), less than current time</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">timeZone</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Local Time</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Name of the timezone that should be used for resolving the escape sequences in partition, e.g. America/Los_Angeles.</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">useLocalTimeStamp</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">false</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Use the local time (instead of the timestamp from the event header) while replacing the escape sequences.</p></td>
+</tr>
+</tbody>
+</table>
+<div class="paragraph">
+<p>Following serializers are provided for Hive sink:</p>
+</div>
+<div class="paragraph">
+<p><strong>JSON</strong>: Handles UTF8 encoded Json (strict syntax) events and requires no configration. Object names
+in the JSON are mapped directly to columns with the same name in the Hive table.
+Internally uses org.apache.hive.hcatalog.data.JsonSerDe but is independent of the Serde of the Hive table.
+This serializer requires HCatalog to be installed.</p>
+</div>
+<div class="paragraph">
+<p><strong>DELIMITED</strong>: Handles simple delimited textual events.
+Internally uses LazySimpleSerde but is independent of the Serde of the Hive table.</p>
+</div>
+<table class="tableblock frame-all grid-all stretch">
+<colgroup>
+<col style="width: 33.3333%;">
+<col style="width: 33.3333%;">
+<col style="width: 33.3334%;">
+</colgroup>
+<thead>
+<tr>
+<th class="tableblock halign-left valign-top">Name</th>
+<th class="tableblock halign-left valign-top">Default</th>
+<th class="tableblock halign-left valign-top">Description</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">serializer.delimiter</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">,</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">(Type: string) The field delimiter in the incoming data. To use special
+characters, surround them with double quotes like "\t"</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock"><strong>serializer.fieldnames</strong></p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">The mapping from input fields to columns in hive table. Specified as a
+comma separated list (no spaces) of hive table columns names, identifying
+the input fields in order of their occurrence. To skip fields leave the
+column name unspecified. Eg. 'time,,ip,message' indicates the 1st, 3rd
+and 4th fields in input map to time, ip and message columns in the hive table.</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">serializer.serdeSeparator</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Ctrl-A</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">(Type: character) Customizes the separator used by underlying serde. There
+can be a gain in efficiency if the fields in serializer.fieldnames are in
+same order as table columns, the serializer.delimiter is same as the
+serializer.serdeSeparator and number of fields in serializer.fieldnames
+is less than or equal to number of table columns, as the fields in incoming
+event body do not need to be reordered to match order of table columns.
+Use single quotes for special characters like '\t'.
+Ensure input fields do not contain this character. NOTE: If serializer.delimiter
+is a single character, preferably set this to the same character</p></td>
+</tr>
+</tbody>
+</table>
+<div class="paragraph">
+<p>The following are the escape sequences supported:</p>
+</div>
+<table class="tableblock frame-all grid-all stretch">
+<colgroup>
+<col style="width: 50%;">
+<col style="width: 50%;">
+</colgroup>
+<thead>
+<tr>
+<th class="tableblock halign-left valign-top">Alias</th>
+<th class="tableblock halign-left valign-top">Description</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">%{host}</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Substitute value of event header named "host". Arbitrary header names are supported.</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">%t</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Unix time in milliseconds</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">%a</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">locale&#8217;s short weekday name (Mon, Tue, &#8230;&#8203;)</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">%A</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">locale&#8217;s full weekday name (Monday, Tuesday, &#8230;&#8203;)</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">%b</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">locale&#8217;s short month name (Jan, Feb, &#8230;&#8203;)</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">%B</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">locale&#8217;s long month name (January, February, &#8230;&#8203;)</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">%c</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">locale&#8217;s date and time (Thu Mar 3 23:05:25 2005)</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">%d</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">day of month (01)</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">%D</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">date; same as %m/%d/%y</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">%H</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">hour (00..23)</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">%I</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">hour (01..12)</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">%j</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">day of year (001..366)</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">%k</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">hour ( 0..23)</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">%m</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">month (01..12)</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">%M</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">minute (00..59)</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">%p</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">locale&#8217;s equivalent of am or pm</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">%s</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">seconds since 1970-01-01 00:00:00 UTC</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">%S</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">second (00..59)</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">%y</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">last two digits of year (00..99)</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">%Y</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">year (2010)</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">%z</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#43;hhmm numeric timezone (for example, -0400)</p></td>
+</tr>
+</tbody>
+</table>
+<div class="admonitionblock note">
+<table>
+<tr>
+<td class="icon">
+<i class="fa icon-note" title="Note"></i>
+</td>
+<td class="content">
+<div class="paragraph">
+<p>For all of the time related escape sequences, a header with the key
+"timestamp" must exist among the headers of the event (unless <code>useLocalTimeStamp</code> is set to <code>true</code>). One way to add
+this automatically is to use the TimestampInterceptor.</p>
+</div>
+</td>
+</tr>
+</table>
+</div>
+<div class="paragraph">
+<p>Example Hive table :</p>
+</div>
+<div class="listingblock">
+<div class="content">
+<pre class="highlightjs highlight"><code class="language-properties hljs" data-lang="properties">create table weblogs ( id int , msg string )
+    partitioned by (continent string, country string, time string)
+    clustered by (id) into 5 buckets
+    stored as orc;</code></pre>
+</div>
+</div>
+<div class="paragraph">
+<p>Example for agent named a1:</p>
+</div>
+<div class="listingblock">
+<div class="content">
+<pre class="highlightjs highlight"><code class="language-properties hljs" data-lang="properties">a1.channels = c1
+a1.channels.c1.type = memory
+a1.sinks = k1
+a1.sinks.k1.type = hive
+a1.sinks.k1.channel = c1
+a1.sinks.k1.hive.metastore = thrift://127.0.0.1:9083
+a1.sinks.k1.hive.database = logsdb
+a1.sinks.k1.hive.table = weblogs
+a1.sinks.k1.hive.partition = asia,%{country},%Y-%m-%d-%H-%M
+a1.sinks.k1.useLocalTimeStamp = false
+a1.sinks.k1.round = true
+a1.sinks.k1.roundValue = 10
+a1.sinks.k1.roundUnit = minute
+a1.sinks.k1.serializer = DELIMITED
+a1.sinks.k1.serializer.delimiter = "\t"
+a1.sinks.k1.serializer.serdeSeparator = '\t'
+a1.sinks.k1.serializer.fieldnames =id,,msg</code></pre>
+</div>
+</div>
+<div class="paragraph">
+<p>The above configuration will round down the timestamp to the last 10th minute. For example, an event with
+timestamp header set to 11:54:34 AM, June 12, 2012 and 'country' header set to 'india' will evaluate to the
+partition (continent='asia',country='india',time='2012-06-12-11-50'. The serializer is configured to
+accept tab separated input containing three fields and to skip the second field.</p>
+</div>
+</div>
+<div class="sect3">
+<h4 id="_logger_sink"><a class="anchor" href="#_logger_sink"></a>Logger Sink</h4>
+<div class="paragraph">
+<p>Logs event at INFO level. Typically useful for testing/debugging purpose. Required properties are
+in <strong>bold</strong>. This sink is the only exception which doesn&#8217;t require the extra configuration
+explained in the <a href="#logging-raw-data">Logging raw data</a> section.</p>
+</div>
+<table class="tableblock frame-all grid-all stretch">
+<colgroup>
+<col style="width: 33.3333%;">
+<col style="width: 33.3333%;">
+<col style="width: 33.3334%;">
+</colgroup>
+<thead>
+<tr>
+<th class="tableblock halign-left valign-top">Property Name</th>
+<th class="tableblock halign-left valign-top">Default</th>
+<th class="tableblock halign-left valign-top">Description</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock"><strong>channel</strong></p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock"><strong>type</strong></p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">The component type name, needs to be <code>logger</code></p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">maxBytesToLog</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">16</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Maximum number of bytes of the Event body to log</p></td>
+</tr>
+</tbody>
+</table>
+<div class="paragraph">
+<p>Example for agent named a1:</p>
+</div>
+<div class="listingblock">
+<div class="content">
+<pre class="highlightjs highlight"><code class="language-properties hljs" data-lang="properties">a1.channels = c1
+a1.sinks = k1
+a1.sinks.k1.type = logger
+a1.sinks.k1.channel = c1</code></pre>
+</div>
+</div>
+</div>
+<div class="sect3">
+<h4 id="_avro_sink"><a class="anchor" href="#_avro_sink"></a>Avro Sink</h4>
+<div class="paragraph">
+<p>This sink forms one half of Flume&#8217;s tiered collection support. Flume events
+sent to this sink are turned into Avro events and sent to the configured
+hostname / port pair. The events are taken from the configured Channel in
+batches of the configured batch size.
+Required properties are in <strong>bold</strong>.</p>
+</div>
+<table class="tableblock frame-all grid-all stretch">
+<colgroup>
+<col style="width: 33.3333%;">
+<col style="width: 33.3333%;">
+<col style="width: 33.3334%;">
+</colgroup>
+<thead>
+<tr>
+<th class="tableblock halign-left valign-top">Property Name</th>
+<th class="tableblock halign-left valign-top">Default</th>
+<th class="tableblock halign-left valign-top">Description</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock"><strong>channel</strong></p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock"><strong>type</strong></p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">The component type name, needs to be <code>avro</code>.</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock"><strong>hostname</strong></p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">The hostname or IP address to bind to.</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock"><strong>port</strong></p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">The port # to listen on.</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">batch-size</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">100</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">number of event to batch together for send.</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">connect-timeout</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">20000</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Amount of time (ms) to allow for the first (handshake) request.</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">request-timeout</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">20000</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Amount of time (ms) to allow for requests after the first.</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">reset-connection-interval</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">none</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Amount of time (s) before the connection to the next hop is reset. This will force the Avro Sink to reconnect to the next hop. This will allow the sink to connect to hosts behind a hardware load-balancer when news hosts are added without having to restart the agent.</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">compression-type</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">none</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">This can be "none" or "deflate". The compression-type must match the compression-type of matching AvroSource</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">compression-level</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">6</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">The level of compression to compress event. 0 = no compression and 1-9 is compression. The higher the number the more compression</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">ssl</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">false</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Set to true to enable SSL for this AvroSink. When configuring SSL, you can optionally set a "truststore", "truststore-password", "truststore-type", and specify whether to "trust-all-certs".</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">trust-all-certs</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">false</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">If this is set to true, SSL server certificates for remote servers (Avro Sources) will not be checked. This should NOT be used in production because it makes it easier for an attacker to execute a man-in-the-middle attack and "listen in" on the encrypted connection.</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">truststore</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">The path to a custom Java truststore file. Flume uses the certificate authority information in this file to determine whether the remote Avro Source&#8217;s SSL authentication credentials should be trusted. If not specified, then the global keystore will be used. If the global keystore not specified either, then the default Java JSSE certificate authority files (typically "jssecacerts" or "cacerts" in the Oracle JRE) wil [...]
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">truststore-password</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">The password for the truststore. If not specified, then the global keystore password will be used (if defined).</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">truststore-type</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">JKS</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">The type of the Java truststore. This can be "JKS" or other supported Java truststore type. If not specified, then the global keystore type will be used (if defined, otherwise the defautl is JKS).</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">exclude-protocols</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">SSLv3</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Space-separated list of SSL/TLS protocols to exclude. SSLv3 will always be excluded in addition to the protocols specified.</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">maxIoWorkers</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">2 * the number of available processors in the machine</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">The maximum number of I/O worker threads. This is configured on the NettyAvroRpcClient NioClientSocketChannelFactory.</p></td>
+</tr>
+</tbody>
+</table>
+<div class="paragraph">
+<p>Example for agent named a1:</p>
+</div>
+<div class="listingblock">
+<div class="content">
+<pre class="highlightjs highlight"><code class="language-properties hljs" data-lang="properties">a1.channels = c1
+a1.sinks = k1
+a1.sinks.k1.type = avro
+a1.sinks.k1.channel = c1
+a1.sinks.k1.hostname = 10.10.10.10
+a1.sinks.k1.port = 4545</code></pre>
+</div>
+</div>
+</div>
+<div class="sect3">
+<h4 id="_thrift_sink"><a class="anchor" href="#_thrift_sink"></a>Thrift Sink</h4>
+<div class="paragraph">
+<p>This sink forms one half of Flume&#8217;s tiered collection support. Flume events
+sent to this sink are turned into Thrift events and sent to the configured
+hostname / port pair. The events are taken from the configured Channel in
+batches of the configured batch size.</p>
+</div>
+<div class="paragraph">
+<p>Thrift sink can be configured to start in secure mode by enabling kerberos authentication.
+To communicate with a Thrift source started in secure mode, the Thrift sink should also
+operate in secure mode. client-principal and client-keytab are the properties used by the
+Thrift sink to authenticate to the kerberos KDC. The server-principal represents the
+principal of the Thrift source this sink is configured to connect to in secure mode.
+Required properties are in <strong>bold</strong>.</p>
+</div>
+<table class="tableblock frame-all grid-all stretch">
+<colgroup>
+<col style="width: 33.3333%;">
+<col style="width: 33.3333%;">
+<col style="width: 33.3334%;">
+</colgroup>
+<thead>
+<tr>
+<th class="tableblock halign-left valign-top">Property Name</th>
+<th class="tableblock halign-left valign-top">Default</th>
+<th class="tableblock halign-left valign-top">Description</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock"><strong>channel</strong></p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock"><strong>type</strong></p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">The component type name, needs to be <code>thrift</code>.</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock"><strong>hostname</strong></p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">The hostname or IP address to bind to.</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock"><strong>port</strong></p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">The port # to listen on.</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">batch-size</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">100</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">number of event to batch together for send.</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">connect-timeout</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">20000</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Amount of time (ms) to allow for the first (handshake) request.</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">request-timeout</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">20000</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Amount of time (ms) to allow for requests after the first.</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">connection-reset-interval</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">none</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Amount of time (s) before the connection to the next hop is reset. This will force the Thrift Sink to reconnect to the next hop. This will allow the sink to connect to hosts behind a hardware load-balancer when news hosts are added without having to restart the agent.</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">ssl</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">false</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Set to true to enable SSL for this ThriftSink. When configuring SSL, you can optionally set a "truststore", "truststore-password" and "truststore-type"</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">truststore</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">The path to a custom Java truststore file. Flume uses the certificate authority information in this file to determine whether the remote Thrift Source&#8217;s SSL authentication credentials should be trusted. If not specified, then the global keystore will be used. If the global keystore not specified either, then the default Java JSSE certificate authority files (typically "jssecacerts" or "cacerts" in the Oracle JRE) w [...]
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">truststore-password</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">The password for the truststore. If not specified, then the global keystore password will be used (if defined).</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">truststore-type</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">JKS</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">The type of the Java truststore. This can be "JKS" or other supported Java truststore type. If not specified, then the global keystore type will be used (if defined, otherwise the defautl is JKS).</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">exclude-protocols</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">SSLv3</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Space-separated list of SSL/TLS protocols to exclude</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">kerberos</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">false</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Set to true to enable kerberos authentication. In kerberos mode, client-principal, client-keytab and server-principal are required for successful authentication and communication to a kerberos enabled Thrift Source.</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">client-principal</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">—-</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">The kerberos principal used by the Thrift Sink to authenticate to the kerberos KDC.</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">client-keytab</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">—-</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">The keytab location used by the Thrift Sink in combination with the client-principal to authenticate to the kerberos KDC.</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">server-principal</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">The kerberos principal of the Thrift Source to which the Thrift Sink is configured to connect to.</p></td>
+</tr>
+</tbody>
+</table>
+<div class="paragraph">
+<p>Example for agent named a1:</p>
+</div>
+<div class="listingblock">
+<div class="content">
+<pre class="highlightjs highlight"><code class="language-properties hljs" data-lang="properties">a1.channels = c1
+a1.sinks = k1
+a1.sinks.k1.type = thrift
+a1.sinks.k1.channel = c1
+a1.sinks.k1.hostname = 10.10.10.10
+a1.sinks.k1.port = 4545</code></pre>
+</div>
+</div>
+</div>
+<div class="sect3">
+<h4 id="_irc_sink"><a class="anchor" href="#_irc_sink"></a>IRC Sink</h4>
+<div class="paragraph">
+<p>The IRC sink takes messages from attached channel and relays those to
+configured IRC destinations.
+Required properties are in <strong>bold</strong>.</p>
+</div>
+<table class="tableblock frame-all grid-all stretch">
+<colgroup>
+<col style="width: 33.3333%;">
+<col style="width: 33.3333%;">
+<col style="width: 33.3334%;">
+</colgroup>
+<thead>
+<tr>
+<th class="tableblock halign-left valign-top">Property Name</th>
+<th class="tableblock halign-left valign-top">Default</th>
+<th class="tableblock halign-left valign-top">Description</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock"><strong>channel</strong></p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock"><strong>type</strong></p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">The component type name, needs to be <code>irc</code></p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock"><strong>hostname</strong></p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">The hostname or IP address to connect to</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">port</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">6667</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">The port number of remote host to connect</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock"><strong>nick</strong></p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Nick name</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">user</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">User name</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">password</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">User password</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock"><strong>chan</strong></p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">channel</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">name</p></td>
+<td class="tableblock halign-left valign-top"></td>
+<td class="tableblock halign-left valign-top"></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">splitlines</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">(boolean)</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">splitchars</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">n</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">line separator (if you were to enter the default value
+into the config file, then you would need to escape the
+backslash, like this: "\n")</p></td>
+</tr>
+</tbody>
+</table>
+<div class="paragraph">
+<p>Example for agent named a1:</p>
+</div>
+<div class="listingblock">
+<div class="content">
+<pre class="highlightjs highlight"><code class="language-properties hljs" data-lang="properties">a1.channels = c1
+a1.sinks = k1
+a1.sinks.k1.type = irc
+a1.sinks.k1.channel = c1
+a1.sinks.k1.hostname = irc.yourdomain.com
+a1.sinks.k1.nick = flume
+a1.sinks.k1.chan = #flume</code></pre>
+</div>
+</div>
+</div>
+<div class="sect3">
+<h4 id="_file_roll_sink"><a class="anchor" href="#_file_roll_sink"></a>File Roll Sink</h4>
+<div class="paragraph">
+<p>Stores events on the local filesystem.
+Required properties are in <strong>bold</strong>.</p>
+</div>
+<table class="tableblock frame-all grid-all stretch">
+<colgroup>
+<col style="width: 33.3333%;">
+<col style="width: 33.3333%;">
+<col style="width: 33.3334%;">
+</colgroup>
+<thead>
+<tr>
+<th class="tableblock halign-left valign-top">Property Name</th>
+<th class="tableblock halign-left valign-top">Default</th>
+<th class="tableblock halign-left valign-top">Description</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock"><strong>channel</strong></p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock"><strong>type</strong></p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">The component type name, needs to be <code>file_roll</code>.</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock"><strong>sink.directory</strong></p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">The directory where files will be stored</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">sink.pathManager</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">DEFAULT</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">The PathManager implementation to use.</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">sink.pathManager.extension</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">The file extension if the default PathManager is used.</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">sink.pathManager.prefix</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">A character string to add to the beginning of the file name if the default PathManager is used</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">sink.rollInterval</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">30</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Roll the file every 30 seconds. Specifying 0 will disable rolling and cause all events to be written to a single file.</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">sink.serializer</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">TEXT</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Other possible options include <code>avro_event</code> or the FQCN of an implementation of EventSerializer.Builder interface.</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">sink.batchSize</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">100</p></td>
+<td class="tableblock halign-left valign-top"></td>
+</tr>
+</tbody>
+</table>
+<div class="paragraph">
+<p>Example for agent named a1:</p>
+</div>
+<div class="listingblock">
+<div class="content">
+<pre class="highlightjs highlight"><code class="language-properties hljs" data-lang="properties">a1.channels = c1
+a1.sinks = k1
+a1.sinks.k1.type = file_roll
+a1.sinks.k1.channel = c1
+a1.sinks.k1.sink.directory = /var/log/flume</code></pre>
+</div>
+</div>
+</div>
+<div class="sect3">
+<h4 id="_null_sink"><a class="anchor" href="#_null_sink"></a>Null Sink</h4>
+<div class="paragraph">
+<p>Discards all events it receives from the channel.
+Required properties are in <strong>bold</strong>.</p>
+</div>
+<table class="tableblock frame-all grid-all stretch">
+<colgroup>
+<col style="width: 33.3333%;">
+<col style="width: 33.3333%;">
+<col style="width: 33.3334%;">
+</colgroup>
+<thead>
+<tr>
+<th class="tableblock halign-left valign-top">Property Name</th>
+<th class="tableblock halign-left valign-top">Default</th>
+<th class="tableblock halign-left valign-top">Description</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock"><strong>channel</strong></p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock"><strong>type</strong></p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">The component type name, needs to be <code>null</code>.</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">batchSize</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">100</p></td>
+<td class="tableblock halign-left valign-top"></td>
+</tr>
+</tbody>
+</table>
+<div class="paragraph">
+<p>Example for agent named a1:</p>
+</div>
+<div class="listingblock">
+<div class="content">
+<pre class="highlightjs highlight"><code class="language-properties hljs" data-lang="properties">a1.channels = c1
+a1.sinks = k1
+a1.sinks.k1.type = null
+a1.sinks.k1.channel = c1</code></pre>
+</div>
+</div>
+</div>
+<div class="sect3">
+<h4 id="_hbasesinks"><a class="anchor" href="#_hbasesinks"></a>HBaseSinks</h4>
+<div class="sect4">
+<h5 id="_hbasesink"><a class="anchor" href="#_hbasesink"></a>HBaseSink</h5>
+<div class="paragraph">
+<p>This sink writes data to HBase. The Hbase configuration is picked up from the first
+hbase-site.xml encountered in the classpath. A class implementing HbaseEventSerializer
+which is specified by the configuration is used to convert the events into
+HBase puts and/or increments. These puts and increments are then written
+to HBase. This sink provides the same consistency guarantees as HBase,
+which is currently row-wise atomicity. In the event of Hbase failing to
+write certain events, the sink will replay all events in that transaction.</p>
+</div>
+<div class="paragraph">
+<p>The HBaseSink supports writing data to secure HBase. To write to secure HBase, the user
+the agent is running as must have write permissions to the table the sink is configured
+to write to. The principal and keytab to use to authenticate against the KDC can be specified
+in the configuration. The hbase-site.xml in the Flume agent&#8217;s classpath
+must have authentication set to <code>kerberos</code> (For details on how to do this, please refer to
+HBase documentation).</p>
+</div>
+<div class="paragraph">
+<p>For convenience, two serializers are provided with Flume. The
+SimpleHbaseEventSerializer (org.apache.flume.sink.hbase.SimpleHbaseEventSerializer)
+writes the event body
+as-is to HBase, and optionally increments a column in Hbase. This is primarily
+an example implementation. The RegexHbaseEventSerializer
+(org.apache.flume.sink.hbase.RegexHbaseEventSerializer) breaks the event body
+based on the given regex and writes each part into different columns.</p>
+</div>
+<div class="paragraph">
+<p>The type is the FQCN: org.apache.flume.sink.hbase.HBaseSink.</p>
+</div>
+<div class="paragraph">
+<p>Required properties are in <strong>bold</strong>.</p>
+</div>
+<table class="tableblock frame-all grid-all stretch">
+<colgroup>
+<col style="width: 33.3333%;">
+<col style="width: 33.3333%;">
+<col style="width: 33.3334%;">
+</colgroup>
+<thead>
+<tr>
+<th class="tableblock halign-left valign-top">Property Name</th>
+<th class="tableblock halign-left valign-top">Default</th>
+<th class="tableblock halign-left valign-top">Description</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock"><strong>channel</strong></p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock"><strong>type</strong></p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">The component type name, needs to be <code>hbase</code></p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock"><strong>table</strong></p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">The name of the table in Hbase to write to.</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock"><strong>columnFamily</strong></p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">The column family in Hbase to write to.</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">zookeeperQuorum</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">The quorum spec. This is the value for the property <code>hbase.zookeeper.quorum</code> in hbase-site.xml</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">znodeParent</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">/hbase</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">The base path for the znode for the -ROOT- region. Value of <code>zookeeper.znode.parent</code> in hbase-site.xml</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">batchSize</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">100</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Number of events to be written per txn.</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">coalesceIncrements</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">false</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Should the sink coalesce multiple increments to a cell per batch. This might give
+better performance if there are multiple increments to a limited number of cells.</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">serializer</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">org.apache.flume.sink.hbase.SimpleHbaseEventSerializer</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Default increment column = "iCol", payload column = "pCol".</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">serializer.*</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Properties to be passed to the serializer.</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">kerberosPrincipal</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Kerberos user principal for accessing secure HBase</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">kerberosKeytab</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Kerberos keytab for accessing secure HBase</p></td>
+</tr>
+</tbody>
+</table>
+<div class="paragraph">
+<p>Example for agent named a1:</p>
+</div>
+<div class="listingblock">
+<div class="content">
+<pre class="highlightjs highlight"><code class="language-properties hljs" data-lang="properties">a1.channels = c1
+a1.sinks = k1
+a1.sinks.k1.type = hbase
+a1.sinks.k1.table = foo_table
+a1.sinks.k1.columnFamily = bar_cf
+a1.sinks.k1.serializer = org.apache.flume.sink.hbase.RegexHbaseEventSerializer
+a1.sinks.k1.channel = c1</code></pre>
+</div>
+</div>
+</div>
+<div class="sect4">
+<h5 id="_hbase2sink"><a class="anchor" href="#_hbase2sink"></a>HBase2Sink</h5>
+<div class="paragraph">
+<p>HBase2Sink is the equivalent of HBaseSink for HBase version 2.
+The provided functionality and the configuration parameters are the same as in case of HBaseSink (except the hbase2 tag in the sink type and the package/class names).</p>
+</div>
+<div class="paragraph">
+<p>The type is the FQCN: org.apache.flume.sink.hbase2.HBase2Sink.</p>
+</div>
+<div class="paragraph">
+<p>Required properties are in <strong>bold</strong>.</p>
+</div>
+<table class="tableblock frame-all grid-all stretch">
+<colgroup>
+<col style="width: 33.3333%;">
+<col style="width: 33.3333%;">
+<col style="width: 33.3334%;">
+</colgroup>
+<thead>
+<tr>
+<th class="tableblock halign-left valign-top">Property Name</th>
+<th class="tableblock halign-left valign-top">Default</th>
+<th class="tableblock halign-left valign-top">Description</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock"><strong>channel</strong></p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock"><strong>type</strong></p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">The component type name, needs to be <code>hbase2</code></p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock"><strong>table</strong></p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">The name of the table in HBase to write to.</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock"><strong>columnFamily</strong></p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">The column family in HBase to write to.</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">zookeeperQuorum</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">The quorum spec. This is the value for the property <code>hbase.zookeeper.quorum</code> in hbase-site.xml</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">znodeParent</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">/hbase</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">The base path for the znode for the -ROOT- region. Value of <code>zookeeper.znode.parent</code> in hbase-site.xml</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">batchSize</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">100</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Number of events to be written per txn.</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">coalesceIncrements</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">false</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Should the sink coalesce multiple increments to a cell per batch. This might give
+better performance if there are multiple increments to a limited number of cells.</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">serializer</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">org.apache.flume.sink.hbase2.SimpleHBase2EventSerializer</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Default increment column = "iCol", payload column = "pCol".</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">serializer.*</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Properties to be passed to the serializer.</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">kerberosPrincipal</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Kerberos user principal for accessing secure HBase</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">kerberosKeytab</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Kerberos keytab for accessing secure HBase</p></td>
+</tr>
+</tbody>
+</table>
+<div class="paragraph">
+<p>Example for agent named a1:</p>
+</div>
+<div class="listingblock">
+<div class="content">
+<pre class="highlightjs highlight"><code class="language-properties hljs" data-lang="properties">a1.channels = c1
+a1.sinks = k1
+a1.sinks.k1.type = hbase2
+a1.sinks.k1.table = foo_table
+a1.sinks.k1.columnFamily = bar_cf
+a1.sinks.k1.serializer = org.apache.flume.sink.hbase2.RegexHBase2EventSerializer
+a1.sinks.k1.channel = c1</code></pre>
+</div>
+</div>
+</div>
+<div class="sect4">
+<h5 id="_asynchbasesink"><a class="anchor" href="#_asynchbasesink"></a>AsyncHBaseSink</h5>
+<div class="paragraph">
+<p>This sink writes data to HBase using an asynchronous model. A class implementing
+AsyncHbaseEventSerializer which is specified by the configuration is used to convert the events into
+HBase puts and/or increments. These puts and increments are then written
+to HBase. This sink uses the <a href="https://github.com/OpenTSDB/asynchbase">Asynchbase API</a> to write to
+HBase. This sink provides the same consistency guarantees as HBase,
+which is currently row-wise atomicity. In the event of Hbase failing to
+write certain events, the sink will replay all events in that transaction.
+AsyncHBaseSink can only be used with HBase 1.x. The async client library used by AsyncHBaseSink is not available for HBase 2.
+The type is the FQCN: org.apache.flume.sink.hbase.AsyncHBaseSink.
+Required properties are in <strong>bold</strong>.</p>
+</div>
+<table class="tableblock frame-all grid-all stretch">
+<colgroup>
+<col style="width: 33.3333%;">
+<col style="width: 33.3333%;">
+<col style="width: 33.3334%;">
+</colgroup>
+<thead>
+<tr>
+<th class="tableblock halign-left valign-top">Property Name</th>
+<th class="tableblock halign-left valign-top">Default</th>
+<th class="tableblock halign-left valign-top">Description</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock"><strong>channel</strong></p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock"><strong>type</strong></p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">The component type name, needs to be <code>asynchbase</code></p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock"><strong>table</strong></p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">The name of the table in Hbase to write to.</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">zookeeperQuorum</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">The quorum spec. This is the value for the property <code>hbase.zookeeper.quorum</code> in hbase-site.xml</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">znodeParent</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">/hbase</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">The base path for the znode for the -ROOT- region. Value of <code>zookeeper.znode.parent</code> in hbase-site.xml</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock"><strong>columnFamily</strong></p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">The column family in Hbase to write to.</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">batchSize</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">100</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Number of events to be written per txn.</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">coalesceIncrements</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">false</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Should the sink coalesce multiple increments to a cell per batch. This might give
+better performance if there are multiple increments to a limited number of cells.</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">timeout</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">60000</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">The length of time (in milliseconds) the sink waits for acks from hbase for
+all events in a transaction.</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">serializer</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">org.apache.flume.sink.hbase.SimpleAsyncHbaseEventSerializer</p></td>
+<td class="tableblock halign-left valign-top"></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">serializer.*</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Properties to be passed to the serializer.</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">async.*</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Properties to be passed to asyncHbase library.
+These properties have precedence over the old <code>zookeeperQuorum</code> and <code>znodeParent</code> values.
+You can find the list of the available properties at
+<a href="https://opentsdb.github.io/asynchbase/docs/build/html/configuration.html#properties">the documentation page of AsyncHBase</a>.</p></td>
+</tr>
+</tbody>
+</table>
+<div class="paragraph">
+<p>Note that this sink takes the Zookeeper Quorum and parent znode information in
+the configuration. Zookeeper Quorum and parent node configuration may be
+specified in the flume configuration file. Alternatively, these configuration
+values are taken from the first hbase-site.xml file in the classpath.</p>
+</div>
+<div class="paragraph">
+<p>If these are not provided in the configuration, then the sink
+will read this information from the first hbase-site.xml file in the classpath.</p>
+</div>
+<div class="paragraph">
+<p>Example for agent named a1:</p>
+</div>
+<div class="listingblock">
+<div class="content">
+<pre class="highlightjs highlight"><code class="language-properties hljs" data-lang="properties">a1.channels = c1
+a1.sinks = k1
+a1.sinks.k1.type = asynchbase
+a1.sinks.k1.table = foo_table
+a1.sinks.k1.columnFamily = bar_cf
+a1.sinks.k1.serializer = org.apache.flume.sink.hbase.SimpleAsyncHbaseEventSerializer
+a1.sinks.k1.channel = c1</code></pre>
+</div>
+</div>
+</div>
+</div>
+<div class="sect3">
+<h4 id="_morphlinesolrsink"><a class="anchor" href="#_morphlinesolrsink"></a>MorphlineSolrSink</h4>
+<div class="paragraph">
+<p>This sink extracts data from Flume events, transforms it, and loads it in near-real-time into Apache Solr servers, which in turn serve queries to end users or search applications.</p>
+</div>
+<div class="paragraph">
+<p>This sink is well suited for use cases that stream raw data into HDFS (via the HdfsSink) and simultaneously extract, transform and load the same data into Solr (via MorphlineSolrSink). In particular, this sink can process arbitrary heterogeneous raw data from disparate data sources and turn it into a data model that is useful to Search applications.</p>
+</div>
+<div class="paragraph">
+<p>The ETL functionality is customizable using a <a href="https://cloudera.github.io/cdk/docs/current/cdk-morphlines/index.html">morphline configuration file</a> that defines a chain of transformation commands that pipe event records from one command to another.</p>
+</div>
+<div class="paragraph">
+<p>Morphlines can be seen as an evolution of Unix pipelines where the data model is generalized to work with streams of generic records, including arbitrary binary payloads. A morphline command is a bit like a Flume Interceptor. Morphlines can be embedded into Hadoop components such as Flume.</p>
+</div>
+<div class="paragraph">
+<p>Commands to parse and transform a set of standard data formats such as log files, Avro, CSV, Text, HTML, XML, PDF, Word, Excel, etc. are provided out of the box, and additional custom commands and parsers for additional data formats can be added as morphline plugins. Any kind of data format can be indexed and any Solr documents for any kind of Solr schema can be generated, and any custom ETL logic can be registered and executed.</p>
+</div>
+<div class="paragraph">
+<p>Morphlines manipulate continuous streams of records. The data model can be described as follows: A record is a set of named fields where each field has an ordered list of one or more values. A value can be any Java Object. That is, a record is essentially a hash table where each hash table entry contains a String key and a list of Java Objects as values. (The implementation uses Guava&#8217;s <code>ArrayListMultimap</code>, which is a <code>ListMultimap</code>). Note that a field can  [...]
+</div>
+<div class="paragraph">
+<p>This sink fills the body of the Flume event into the <code>_attachment_body</code> field of the morphline record, as well as copies the headers of the Flume event into record fields of the same name. The commands can then act on this data.</p>
+</div>
+<div class="paragraph">
+<p>Routing to a SolrCloud cluster is supported to improve scalability. Indexing load can be spread across a large number of MorphlineSolrSinks for improved scalability. Indexing load can be replicated across multiple MorphlineSolrSinks for high availability, for example using Flume features such as Load balancing Sink Processor. MorphlineInterceptor can also help to implement dynamic routing to multiple Solr collections (e.g. for multi-tenancy).</p>
+</div>
+<div class="paragraph">
+<p>The morphline and solr jars required for your environment must be placed in the lib directory of the Apache Flume installation.</p>
+</div>
+<div class="paragraph">
+<p>The type is the FQCN: org.apache.flume.sink.solr.morphline.MorphlineSolrSink</p>
+</div>
+<div class="paragraph">
+<p>Required properties are in <strong>bold</strong>.</p>
+</div>
+<table class="tableblock frame-all grid-all stretch">
+<colgroup>
+<col style="width: 33.3333%;">
+<col style="width: 33.3333%;">
+<col style="width: 33.3334%;">
+</colgroup>
+<thead>
+<tr>
+<th class="tableblock halign-left valign-top">Property Name</th>
+<th class="tableblock halign-left valign-top">Default</th>
+<th class="tableblock halign-left valign-top">Description</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock"><strong>channel</strong></p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock"><strong>type</strong></p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">The component type name, needs to be <code>org.apache.flume.sink.solr.morphline.MorphlineSolrSink</code></p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock"><strong>morphlineFile</strong></p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">The relative or absolute path on the local file system to the morphline configuration file. Example: <code>/etc/flume-ng/conf/morphline.conf</code></p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">morphlineId</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">null</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Optional name used to identify a morphline if there are multiple morphlines in a morphline config file</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">batchSize</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">1000</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">The maximum number of events to take per flume transaction.</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">batchDurationMillis</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">1000</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">The maximum duration per flume transaction (ms). The transaction commits after this duration or when batchSize is exceeded, whichever comes first.</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">handlerClass</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">org.apache.flume.sink.solr.morphline.MorphlineHandlerImpl</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">The FQCN of a class implementing org.apache.flume.sink.solr.morphline.MorphlineHandler</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">isProductionMode</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">false</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">This flag should be enabled for mission critical, large-scale online production systems that need to make progress without downtime when unrecoverable exceptions occur. Corrupt or malformed parser input data, parser bugs, and errors related to unknown Solr schema fields produce unrecoverable exceptions.</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">recoverableExceptionClasses</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">org.apache.solr.client.solrj.SolrServerException</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Comma separated list of recoverable exceptions that tend to be transient, in which case the corresponding task can be retried. Examples include network connection errors, timeouts, etc. When the production mode flag is set to true, the recoverable exceptions configured using this parameter will not be ignored and hence will lead to retries.</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">isIgnoringRecoverableExceptions</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">false</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">This flag should be enabled, if an unrecoverable exception is accidentally misclassified as recoverable. This enables the sink to make progress and avoid retrying an event forever.</p></td>
+</tr>
+</tbody>
+</table>
+<div class="paragraph">
+<p>Example for agent named a1:</p>
+</div>
+<div class="listingblock">
+<div class="content">
+<pre class="highlightjs highlight"><code class="language-properties hljs" data-lang="properties">a1.channels = c1
+a1.sinks = k1
+a1.sinks.k1.type = org.apache.flume.sink.solr.morphline.MorphlineSolrSink
+a1.sinks.k1.channel = c1
+a1.sinks.k1.morphlineFile = /etc/flume-ng/conf/morphline.conf
+# a1.sinks.k1.morphlineId = morphline1
+# a1.sinks.k1.batchSize = 1000
+# a1.sinks.k1.batchDurationMillis = 1000</code></pre>
+</div>
+</div>
+</div>
+<div class="sect3">
+<h4 id="_kite_dataset_sink"><a class="anchor" href="#_kite_dataset_sink"></a>Kite Dataset Sink</h4>
+<div class="paragraph">
+<p>Experimental sink that writes events to a <a href="http://kitesdk.org/docs/current/guide/">Kite Dataset</a>.
+This sink will deserialize the body of each incoming event and store the
+resulting record in a Kite Dataset. It determines target Dataset by loading a
+dataset by URI.</p>
+</div>
+<div class="paragraph">
+<p>The only supported serialization is avro, and the record schema must be passed
+in the event headers, using either <code>flume.avro.schema.literal</code> with the JSON
+schema representation or <code>flume.avro.schema.url</code> with a URL where the schema
+may be found (<code>hdfs:/&#8230;&#8203;</code> URIs are supported). This is compatible with the
+Log4jAppender flume client and the spooling directory source&#8217;s Avro
+deserializer using <code>deserializer.schemaType = LITERAL</code>.</p>
+</div>
+<div class="paragraph">
+<p>Note 1: The <code>flume.avro.schema.hash</code> header is <strong>not supported</strong>.
+Note 2: In some cases, file rolling may occur slightly after the roll interval
+has been exceeded. However, this delay will not exceed 5 seconds. In most
+cases, the delay is neglegible.</p>
+</div>
+<table class="tableblock frame-all grid-all stretch">
+<colgroup>
+<col style="width: 33.3333%;">
+<col style="width: 33.3333%;">
+<col style="width: 33.3334%;">
+</colgroup>
+<thead>
+<tr>
+<th class="tableblock halign-left valign-top">Property Name</th>
+<th class="tableblock halign-left valign-top">Default</th>
+<th class="tableblock halign-left valign-top">Description</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock"><strong>channel</strong></p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock"><strong>type</strong></p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Must be org.apache.flume.sink.kite.DatasetSink</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock"><strong>kite.dataset.uri</strong></p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">URI of the dataset to open</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">kite.repo.uri</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">URI of the repository to open
+(deprecated; use kite.dataset.uri instead)</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">kite.dataset.namespace</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Namespace of the Dataset where records will be written
+(deprecated; use kite.dataset.uri instead)</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">kite.dataset.name</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Name of the Dataset where records will be written
+(deprecated; use kite.dataset.uri instead)</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">kite.batchSize</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">100</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Number of records to process in each batch</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">kite.rollInterval</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">30</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Maximum wait time (seconds) before data files are released</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">kite.flushable.commitOnBatch</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">true</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">If <code>true</code>, the Flume transaction will be commited and the
+writer will be flushed on each batch of <code>kite.batchSize</code>
+records. This setting only applies to flushable datasets. When
+<code>true</code>, it&#8217;s possible for temp files with commited data to be
+left in the dataset directory. These files need to be recovered
+by hand for the data to be visible to DatasetReaders.</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">kite.syncable.syncOnBatch</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">true</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Controls whether the sink will also sync data when committing
+the transaction. This setting only applies to syncable datasets.
+Syncing gaurentees that data will be written on stable storage
+on the remote system while flushing only gaurentees that data
+has left Flume&#8217;s client buffers. When the
+<code>kite.flushable.commitOnBatch</code> property is set to <code>false</code>,
+this property must also be set to <code>false</code>.</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">kite.entityParser</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">avro</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Parser that turns Flume <code>Events</code> into Kite entities.
+Valid values are <code>avro</code> and the fully-qualified class name
+of an implementation of the <code>EntityParser.Builder</code> interface.</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">kite.failurePolicy</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">retry</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Policy that handles non-recoverable errors such as a missing
+<code>Schema</code> in the <code>Event</code> header. The default value, <code>retry</code>,
+will fail the current batch and try again which matches the old
+behavior. Other valid values are <code>save</code>, which will write the
+raw <code>Event</code> to the <code>kite.error.dataset.uri</code> dataset, and the
+fully-qualified class name of an implementation of the
+<code>FailurePolicy.Builder</code> interface.</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">kite.error.dataset.uri</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">URI of the dataset where failed events are saved when
+<code>kite.failurePolicy</code> is set to <code>save</code>. <strong>Required</strong> when
+the <code>kite.failurePolicy</code> is set to <code>save</code>.</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">auth.kerberosPrincipal</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Kerberos user principal for secure authentication to HDFS</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">auth.kerberosKeytab</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Kerberos keytab location (local FS) for the principal</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">auth.proxyUser</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">The effective user for HDFS actions, if different from
+the kerberos principal</p></td>
+</tr>
+</tbody>
+</table>
+</div>
+<div class="sect3">
+<h4 id="_kafka_sink"><a class="anchor" href="#_kafka_sink"></a>Kafka Sink</h4>
+<div class="paragraph">
+<p>This is a Flume Sink implementation that can publish data to a
+<a href="https://kafka.apache.org/">Kafka</a> topic. One of the objective is to integrate Flume
+with Kafka so that pull based processing systems can process the data coming
+through various Flume sources.</p>
+</div>
+<div class="paragraph">
+<p>This currently supports Kafka server releases 0.10.1.0 or higher. Testing was done up to 2.0.1 that was the highest avilable version at the time of the release.</p>
+</div>
+<div class="paragraph">
+<p>Required properties are marked in bold font.</p>
+</div>
+<table class="tableblock frame-all grid-all stretch">
+<colgroup>
+<col style="width: 33.3333%;">
+<col style="width: 33.3333%;">
+<col style="width: 33.3334%;">
+</colgroup>
+<thead>
+<tr>
+<th class="tableblock halign-left valign-top">Property Name</th>
+<th class="tableblock halign-left valign-top">Default</th>
+<th class="tableblock halign-left valign-top">Description</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock"><strong>type</strong></p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Must be set to <code>org.apache.flume.sink.kafka.KafkaSink</code></p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock"><strong>kafka.bootstrap.servers</strong></p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">List of brokers Kafka-Sink will connect to, to get the list of topic partitions
+This can be a partial list of brokers, but we recommend at least two for HA.
+The format is comma separated list of hostname:port</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">kafka.topic</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">default-flume-topic</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">The topic in Kafka to which the messages will be published. If this parameter is configured,
+messages will be published to this topic.
+If the event header contains a "topic" field, the event will be published to that topic
+overriding the topic configured here.
+Arbitrary header substitution is supported, eg. %{header} is replaced with value of event header named "header".
+(If using the substitution, it is recommended to set "auto.create.topics.enable" property of Kafka broker to true.)</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">flumeBatchSize</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">100</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">How many messages to process in one batch. Larger batches improve throughput while adding latency.</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">kafka.producer.acks</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">1</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">How many replicas must acknowledge a message before its considered successfully written.
+Accepted values are 0 (Never wait for acknowledgement), 1 (wait for leader only), -1 (wait for all replicas)
+Set this to -1 to avoid data loss in some cases of leader failure.</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">useFlumeEventFormat</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">false</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">By default events are put as bytes onto the Kafka topic directly from the event body. Set to
+true to store events as the Flume Avro binary format. Used in conjunction with the same property
+on the KafkaSource or with the parseAsFlumeEvent property on the Kafka Channel this will preserve
+any Flume headers for the producing side.</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">defaultPartitionId</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Specifies a Kafka partition ID (integer) for all events in this channel to be sent to, unless
+overriden by <code>partitionIdHeader</code>. By default, if this property is not set, events will be
+distributed by the Kafka Producer&#8217;s partitioner - including by <code>key</code> if specified (or by a
+partitioner specified by <code>kafka.partitioner.class</code>).</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">partitionIdHeader</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">When set, the sink will take the value of the field named using the value of this property
+from the event header and send the message to the specified partition of the topic. If the
+value represents an invalid partition, an EventDeliveryException will be thrown. If the header value
+is present then this setting overrides <code>defaultPartitionId</code>.</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">allowTopicOverride</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">true</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">When set, the sink will allow a message to be produced into a topic specified by the <code>topicHeader</code> property (if provided).</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">topicHeader</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">topic</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">When set in conjunction with <code>allowTopicOverride</code> will produce a message into the value of the header named using the value of this property.
+Care should be taken when using in conjunction with the Kafka Source <code>topicHeader</code> property to avoid creating a loopback.</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">timestampHeader</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">The header containing the caller provided timestamp value to use. If not provided the current time will
+be used.</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">header.NAME</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Used to identify which headers from the Flume Event should be passed to Kafka. The value of NAME should match
+the Flume header name and the value should be the name of the header to be used as the Kafka header name.</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">kafka.producer.security.protocol</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">PLAINTEXT</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Set to SASL_PLAINTEXT, SASL_SSL or SSL if writing to Kafka using some level of security. See below for additional info on secure setup.</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock"><em>more producer security props</em></p></td>
+<td class="tableblock halign-left valign-top"></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">If using SASL_PLAINTEXT, SASL_SSL or SSL refer to <a href="https://kafka.apache.org/documentation.html#security">Kafka security</a> for additional
+properties that need to be set on producer.</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Other Kafka Producer Properties</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">These properties are used to configure the Kafka Producer. Any producer property supported
+by Kafka can be used. The only requirement is to prepend the property name with the prefix
+<code>kafka.producer</code>.
+For example: kafka.producer.linger.ms</p></td>
+</tr>
+</tbody>
+</table>
+<div class="admonitionblock note">
+<table>
+<tr>
+<td class="icon">
+<i class="fa icon-note" title="Note"></i>
+</td>
+<td class="content">
+<div class="paragraph">
+<p>Kafka Sink uses the <code>topic</code> and <code>key</code> properties from the FlumeEvent headers to send events to Kafka.
+If <code>topic</code> exists in the headers, the event will be sent to that specific topic, overriding the topic configured for the Sink.
+If <code>key</code> exists in the headers, the key will used by Kafka to partition the data between the topic partitions. Events with same key
+will be sent to the same partition. If the key is null, events will be sent to random partitions.</p>
+</div>
+</td>
+</tr>
+</table>
+</div>
+<div class="paragraph">
+<p>The Kafka sink also provides defaults for the key.serializer(org.apache.kafka.common.serialization.StringSerializer)
+and value.serializer(org.apache.kafka.common.serialization.ByteArraySerializer). Modification of these parameters is not recommended.</p>
+</div>
+<div class="paragraph">
+<p>Deprecated Properties</p>
+</div>
+<table class="tableblock frame-all grid-all stretch">
+<colgroup>
+<col style="width: 33.3333%;">
+<col style="width: 33.3333%;">
+<col style="width: 33.3334%;">
+</colgroup>
+<thead>
+<tr>
+<th class="tableblock halign-left valign-top">Property Name</th>
+<th class="tableblock halign-left valign-top">Default</th>
+<th class="tableblock halign-left valign-top">Description</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">brokerList</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Use kafka.bootstrap.servers</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">topic</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">default-flume-topic</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Use kafka.topic</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">batchSize</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">100</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Use kafka.flumeBatchSize</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">requiredAcks</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">1</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Use kafka.producer.acks</p></td>
+</tr>
+</tbody>
+</table>
+<div class="paragraph">
+<p>An example configuration of a Kafka sink is given below. Properties starting
+with the prefix <code>kafka.producer</code> the Kafka producer. The properties that are passed when creating the Kafka
+producer are not limited to the properties given in this example.
+Also it is possible to include your custom properties here and access them inside
+the preprocessor through the Flume Context object passed in as a method
+argument.</p>
+</div>
+<div class="listingblock">
+<div class="content">
+<pre class="highlightjs highlight"><code class="language-properties hljs" data-lang="properties">a1.sinks.k1.channel = c1
+a1.sinks.k1.type = org.apache.flume.sink.kafka.KafkaSink
+a1.sinks.k1.kafka.topic = mytopic
+a1.sinks.k1.kafka.bootstrap.servers = localhost:9092
+a1.sinks.k1.kafka.flumeBatchSize = 20
+a1.sinks.k1.kafka.producer.acks = 1
+a1.sinks.k1.kafka.producer.linger.ms = 1
+a1.sinks.k1.kafka.producer.compression.type = snappy</code></pre>
+</div>
+</div>
+<div class="paragraph">
+<p><strong>Security and Kafka Sink:</strong></p>
+</div>
+<div class="paragraph">
+<p>Secure authentication as well as data encryption is supported on the communication channel between Flume and Kafka.
+For secure authentication SASL/GSSAPI (Kerberos V5) or SSL (even though the parameter is named SSL, the actual protocol is a TLS implementation) can be used from Kafka version 0.9.0.</p>
+</div>
+<div class="paragraph">
+<p>As of now data encryption is solely provided by SSL/TLS.</p>
+</div>
+<div class="paragraph">
+<p>Setting <code>kafka.producer.security.protocol</code> to any of the following value means:</p>
+</div>
+<div class="ulist">
+<ul>
+<li>
+<p><strong>SASL_PLAINTEXT</strong> - Kerberos or plaintext authentication with no data encryption</p>
+</li>
+<li>
+<p><strong>SASL_SSL</strong> - Kerberos or plaintext authentication with data encryption</p>
+</li>
+<li>
+<p><strong>SSL</strong> - TLS based encryption with optional authentication.</p>
+</li>
+</ul>
+</div>
+<div class="admonitionblock warning">
+<table>
+<tr>
+<td class="icon">
+<i class="fa icon-warning" title="Warning"></i>
+</td>
+<td class="content">
+<div class="paragraph">
+<p>There is a performance degradation when SSL is enabled,
+the magnitude of which depends on the CPU type and the JVM implementation.
+Reference: <a href="https://kafka.apache.org/documentation#security_overview">Kafka security overview</a>
+and the jira for tracking this issue:
+<a href="https://issues.apache.org/jira/browse/KAFKA-2561">KAFKA-2561</a></p>
+</div>
+</td>
+</tr>
+</table>
+</div>
+<div class="paragraph">
+<p><strong>TLS and Kafka Sink:</strong></p>
+</div>
+<div class="paragraph">
+<p>Please read the steps described in <a href="https://kafka.apache.org/documentation#security_configclients">Configuring Kafka Clients SSL</a>
+to learn about additional configuration settings for fine tuning for example any of the following:
+security provider, cipher suites, enabled protocols, truststore or keystore types.</p>
+</div>
+<div class="paragraph">
+<p>Example configuration with server side authentication and data encryption.</p>
+</div>
+<div class="listingblock">
+<div class="content">
+<pre class="highlightjs highlight"><code class="language-properties hljs" data-lang="properties">a1.sinks.sink1.type = org.apache.flume.sink.kafka.KafkaSink
+a1.sinks.sink1.kafka.bootstrap.servers = kafka-1:9093,kafka-2:9093,kafka-3:9093
+a1.sinks.sink1.kafka.topic = mytopic
+a1.sinks.sink1.kafka.producer.security.protocol = SSL
+# optional, the global truststore can be used alternatively
+a1.sinks.sink1.kafka.producer.ssl.truststore.location = /path/to/truststore.jks
+a1.sinks.sink1.kafka.producer.ssl.truststore.password = &lt;password to access the truststore&gt;</code></pre>
+</div>
+</div>
+<div class="paragraph">
+<p>Specyfing the truststore is optional here, the global truststore can be used instead.
+For more details about the global SSL setup, see the <a href="#ssltls-support">SSL/TLS support</a> section.</p>
+</div>
+<div class="paragraph">
+<p>Note: By default the property <code>ssl.endpoint.identification.algorithm</code>
+is not defined, so hostname verification is not performed.
+In order to enable hostname verification, set the following properties</p>
+</div>
+<div class="listingblock">
+<div class="content">
+<pre class="highlightjs highlight"><code class="language-properties hljs" data-lang="properties">a1.sinks.sink1.kafka.producer.ssl.endpoint.identification.algorithm = HTTPS</code></pre>
+</div>
+</div>
+<div class="paragraph">
+<p>Once enabled, clients will verify the server&#8217;s fully qualified domain name (FQDN)
+against one of the following two fields:</p>
+</div>
+<div class="olist arabic">
+<ol class="arabic">
+<li>
+<p>Common Name (CN) <a href="https://tools.ietf.org/html/rfc6125#section-2.3" class="bare">https://tools.ietf.org/html/rfc6125#section-2.3</a></p>
+</li>
+<li>
+<p>Subject Alternative Name (SAN) <a href="https://tools.ietf.org/html/rfc5280#section-4.2.1.6" class="bare">https://tools.ietf.org/html/rfc5280#section-4.2.1.6</a></p>
+</li>
+</ol>
+</div>
+<div class="paragraph">
+<p>If client side authentication is also required then additionally the following needs to be added to Flume agent
+configuration or the global SSL setup can be used (see <a href="#ssltls-support">SSL/TLS support</a> section).
+Each Flume agent has to have its client certificate which has to be trusted by Kafka brokers either
+individually or by their signature chain. Common example is to sign each client certificate by a single Root CA
+which in turn is trusted by Kafka brokers.</p>
+</div>
+<div class="listingblock">
+<div class="content">
+<pre class="highlightjs highlight"><code class="language-properties hljs" data-lang="properties"># optional, the global keystore can be used alternatively
+a1.sinks.sink1.kafka.producer.ssl.keystore.location = /path/to/client.keystore.jks
+a1.sinks.sink1.kafka.producer.ssl.keystore.password = &lt;password to access the keystore&gt;</code></pre>
+</div>
+</div>
+<div class="paragraph">
+<p>If keystore and key use different password protection then <code>ssl.key.password</code> property will
+provide the required additional secret for producer keystore:</p>
+</div>
+<div class="listingblock">
+<div class="content">
+<pre class="highlightjs highlight"><code class="language-properties hljs" data-lang="properties">a1.sinks.sink1.kafka.producer.ssl.key.password = &lt;password to access the key&gt;</code></pre>
+</div>
+</div>
+<div class="paragraph">
+<p><strong>Kerberos and Kafka Sink:</strong></p>
+</div>
+<div class="paragraph">
+<p>To use Kafka sink with a Kafka cluster secured with Kerberos, set the <code>producer.security.protocol</code> property noted above for producer.
+The Kerberos keytab and principal to be used with Kafka brokers is specified in a JAAS file&#8217;s "KafkaClient" section. "Client" section describes the Zookeeper connection if needed.
+See <a href="https://kafka.apache.org/documentation.html#security_sasl_clientconfig">Kafka doc</a>
+for information on the JAAS file contents. The location of this JAAS file and optionally the system wide kerberos configuration can be specified via JAVA_OPTS in flume-env.sh:</p>
+</div>
+<div class="listingblock">
+<div class="content">
+<pre class="highlightjs highlight"><code class="language-properties hljs" data-lang="properties">JAVA_OPTS="$JAVA_OPTS -Djava.security.krb5.conf=/path/to/krb5.conf"
+JAVA_OPTS="$JAVA_OPTS -Djava.security.auth.login.config=/path/to/flume_jaas.conf"</code></pre>
+</div>
+</div>
+<div class="paragraph">
+<p>Example secure configuration using SASL_PLAINTEXT:</p>
+</div>
+<div class="listingblock">
+<div class="content">
+<pre class="highlightjs highlight"><code class="language-properties hljs" data-lang="properties">a1.sinks.sink1.type = org.apache.flume.sink.kafka.KafkaSink
+a1.sinks.sink1.kafka.bootstrap.servers = kafka-1:9093,kafka-2:9093,kafka-3:9093
+a1.sinks.sink1.kafka.topic = mytopic
+a1.sinks.sink1.kafka.producer.security.protocol = SASL_PLAINTEXT
+a1.sinks.sink1.kafka.producer.sasl.mechanism = GSSAPI
+a1.sinks.sink1.kafka.producer.sasl.kerberos.service.name = kafka</code></pre>
+</div>
+</div>
+<div class="paragraph">
+<p>Example secure configuration using SASL_SSL:</p>
+</div>
+<div class="listingblock">
+<div class="content">
+<pre class="highlightjs highlight"><code class="language-properties hljs" data-lang="properties">a1.sinks.sink1.type = org.apache.flume.sink.kafka.KafkaSink
+a1.sinks.sink1.kafka.bootstrap.servers = kafka-1:9093,kafka-2:9093,kafka-3:9093
+a1.sinks.sink1.kafka.topic = mytopic
+a1.sinks.sink1.kafka.producer.security.protocol = SASL_SSL
+a1.sinks.sink1.kafka.producer.sasl.mechanism = GSSAPI
+a1.sinks.sink1.kafka.producer.sasl.kerberos.service.name = kafka
+# optional, the global truststore can be used alternatively
+a1.sinks.sink1.kafka.producer.ssl.truststore.location = /path/to/truststore.jks
+a1.sinks.sink1.kafka.producer.ssl.truststore.password = &lt;password to access the truststore&gt;</code></pre>
+</div>
+</div>
+<div class="paragraph">
+<p>Sample JAAS file. For reference of its content please see client config sections of the desired authentication mechanism (GSSAPI/PLAIN)
+in Kafka documentation of <a href="https://kafka.apache.org/documentation#security_sasl_clientconfig">SASL configuration</a>.
+Unlike the Kafka Source or Kafka Channel a "Client" section is not required, unless it is needed by other connecting components. Also please make sure
+that the operating system user of the Flume processes has read privileges on the jaas and keytab files.</p>
+</div>
+<div class="listingblock">
+<div class="content">
+<pre class="highlightjs highlight"><code class="language-javascript hljs" data-lang="javascript">KafkaClient {
+  com.sun.security.auth.module.Krb5LoginModule required
+  useKeyTab=true
+  storeKey=true
+  keyTab="/path/to/keytabs/flume.keytab"
+  principal="flume/flumehost1.example.com@YOURKERBEROSREALM";
+};</code></pre>
+</div>
+</div>
+</div>
+<div class="sect3">
+<h4 id="_http_sink"><a class="anchor" href="#_http_sink"></a>HTTP Sink</h4>
+<div class="paragraph">
+<p>Behaviour of this sink is that it will take events from the channel, and
+send those events to a remote service using an HTTP POST request. The event
+content is sent as the POST body.</p>
+</div>
+<div class="paragraph">
+<p>Error handling behaviour of this sink depends on the HTTP response returned
+by the target server. The sink backoff/ready status is configurable, as is the
+transaction commit/rollback result and whether the event contributes to the
+successful event drain count.</p>
+</div>
+<div class="paragraph">
+<p>Any malformed HTTP response returned by the server where the status code is
+not readable will result in a backoff signal and the event is not consumed
+from the channel.</p>
+</div>
+<div class="paragraph">
+<p>Required properties are in <strong>bold</strong>.</p>
+</div>
+<table class="tableblock frame-all grid-all stretch">
+<colgroup>
+<col style="width: 33.3333%;">
+<col style="width: 33.3333%;">
+<col style="width: 33.3334%;">
+</colgroup>
+<thead>
+<tr>
+<th class="tableblock halign-left valign-top">Property Name</th>
+<th class="tableblock halign-left valign-top">Default</th>
+<th class="tableblock halign-left valign-top">Description</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock"><strong>channel</strong></p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock"><strong>type</strong></p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">The component type name, needs to be <code>http</code>.</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock"><strong>endpoint</strong></p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">The fully qualified URL endpoint to POST to</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">connectTimeout</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">5000</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">The socket connection timeout in milliseconds</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">requestTimeout</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">5000</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">The maximum request processing time in milliseconds</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">contentTypeHeader</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">text/plain</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">The HTTP Content-Type header</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">acceptHeader</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">text/plain</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">The HTTP Accept header value</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">defaultBackoff</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">true</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Whether to backoff by default on receiving all HTTP status codes</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">defaultRollback</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">true</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Whether to rollback by default on receiving all HTTP status codes</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">defaultIncrementMetrics</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">false</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Whether to increment metrics by default on receiving all HTTP status codes</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">backoff.CODE</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Configures a specific backoff for an individual (i.e. 200) code or a group (i.e. 2XX) code</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">rollback.CODE</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Configures a specific rollback for an individual (i.e. 200) code or a group (i.e. 2XX) code</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">incrementMetrics.CODE</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Configures a specific metrics increment for an individual (i.e. 200) code or a group (i.e. 2XX) code</p></td>
+</tr>
+</tbody>
+</table>
+<div class="paragraph">
+<p>Note that the most specific HTTP status code match is used for the backoff,
+rollback and incrementMetrics configuration options. If there are configuration
+values for both 2XX and 200 status codes, then 200 HTTP codes will use the 200
+value, and all other HTTP codes in the 201-299 range will use the 2XX value.</p>
+</div>
+<div class="paragraph">
+<p>Any empty or null events are consumed without any request being made to the
+HTTP endpoint.</p>
+</div>
+<div class="paragraph">
+<p>Example for agent named a1:</p>
+</div>
+<div class="listingblock">
+<div class="content">
+<pre class="highlightjs highlight"><code class="language-properties hljs" data-lang="properties">a1.channels = c1
+a1.sinks = k1
+a1.sinks.k1.type = http
+a1.sinks.k1.channel = c1
+a1.sinks.k1.endpoint = http://localhost:8080/someuri
+a1.sinks.k1.connectTimeout = 2000
+a1.sinks.k1.requestTimeout = 2000
+a1.sinks.k1.acceptHeader = application/json
+a1.sinks.k1.contentTypeHeader = application/json
+a1.sinks.k1.defaultBackoff = true
+a1.sinks.k1.defaultRollback = true
+a1.sinks.k1.defaultIncrementMetrics = false
+a1.sinks.k1.backoff.4XX = false
+a1.sinks.k1.rollback.4XX = false
+a1.sinks.k1.incrementMetrics.4XX = true
+a1.sinks.k1.backoff.200 = false
+a1.sinks.k1.rollback.200 = false
+a1.sinks.k1.incrementMetrics.200 = true</code></pre>
+</div>
+</div>
+</div>
+<div class="sect3">
+<h4 id="_custom_sink"><a class="anchor" href="#_custom_sink"></a>Custom Sink</h4>
+<div class="paragraph">
+<p>A custom sink is your own implementation of the Sink interface. A custom
+sink&#8217;s class and its dependencies must be included in the agent&#8217;s classpath
+when starting the Flume agent. The type of the custom sink is its FQCN.
+Required properties are in <strong>bold</strong>.</p>
+</div>
+<table class="tableblock frame-all grid-all stretch">
+<colgroup>
+<col style="width: 33.3333%;">
+<col style="width: 33.3333%;">
+<col style="width: 33.3334%;">
+</colgroup>
+<thead>
+<tr>
+<th class="tableblock halign-left valign-top">Property Name</th>
+<th class="tableblock halign-left valign-top">Default</th>
+<th class="tableblock halign-left valign-top">Description</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock"><strong>channel</strong></p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock"><strong>type</strong></p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">The component type name, needs to be your FQCN</p></td>
+</tr>
+</tbody>
+</table>
+<div class="paragraph">
+<p>Example for agent named a1:</p>
+</div>
+<div class="listingblock">
+<div class="content">
+<pre class="highlightjs highlight"><code class="language-properties hljs" data-lang="properties">a1.channels = c1
+a1.sinks = k1
+a1.sinks.k1.type = org.example.MySink
+a1.sinks.k1.channel = c1</code></pre>
+</div>
+</div>
+</div>
+</div>
+<div class="sect2">
+<h3 id="_flume_channels"><a class="anchor" href="#_flume_channels"></a>Flume Channels</h3>
+<div class="paragraph">
+<p>Channels are the repositories where the events are staged on a agent.
+Source adds the events and Sink removes it.</p>
+</div>
+<div class="sect3">
+<h4 id="_memory_channel"><a class="anchor" href="#_memory_channel"></a>Memory Channel</h4>
+<div class="paragraph">
+<p>The events are stored in an in-memory queue with configurable max size. It&#8217;s
+ideal for flows that need higher throughput and are prepared to lose the staged
+data in the event of a agent failures.
+Required properties are in <strong>bold</strong>.</p>
+</div>
+<table class="tableblock frame-all grid-all stretch">
+<colgroup>
+<col style="width: 33.3333%;">
+<col style="width: 33.3333%;">
+<col style="width: 33.3334%;">
+</colgroup>
+<thead>
+<tr>
+<th class="tableblock halign-left valign-top">Property Name</th>
+<th class="tableblock halign-left valign-top">Default</th>
+<th class="tableblock halign-left valign-top">Description</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock"><strong>type</strong></p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">The component type name, needs to be <code>memory</code></p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">capacity</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">100</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">The maximum number of events stored in the channel</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">transactionCapacity</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">100</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">The maximum number of events the channel will take from a source or give to a
+sink per transaction</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">keep-alive</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">3</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Timeout in seconds for adding or removing an event</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">byteCapacityBufferPercentage</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">20</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Defines the percent of buffer between byteCapacity and the estimated total size
+of all events in the channel, to account for data in headers. See below.</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">byteCapacity</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">see description</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Maximum total <strong>bytes</strong> of memory allowed as a sum of all events in this channel.
+The implementation only counts the Event <code>body</code>, which is the reason for
+providing the <code>byteCapacityBufferPercentage</code> configuration parameter as well.
+Defaults to a computed value equal to 80% of the maximum memory available to
+the JVM (i.e. 80% of the -Xmx value passed on the command line).
+Note that if you have multiple memory channels on a single JVM, and they happen
+to hold the same physical events (i.e. if you are using a replicating channel
+selector from a single source) then those event sizes may be double-counted for
+channel byteCapacity purposes.
+Setting this value to <code>0</code> will cause this value to fall back to a hard
+internal limit of about 200 GB.</p></td>
+</tr>
+</tbody>
+</table>
+<div class="paragraph">
+<p>Example for agent named a1:</p>
+</div>
+<div class="listingblock">
+<div class="content">
+<pre class="highlightjs highlight"><code class="language-properties hljs" data-lang="properties">a1.channels = c1
+a1.channels.c1.type = memory
+a1.channels.c1.capacity = 10000
+a1.channels.c1.transactionCapacity = 10000
+a1.channels.c1.byteCapacityBufferPercentage = 20
+a1.channels.c1.byteCapacity = 800000</code></pre>
+</div>
+</div>
+</div>
+<div class="sect3">
+<h4 id="_jdbc_channel"><a class="anchor" href="#_jdbc_channel"></a>JDBC Channel</h4>
+<div class="paragraph">
+<p>The events are stored in a persistent storage that&#8217;s backed by a database.
+The JDBC channel currently supports embedded Derby. This is a durable channel
+that&#8217;s ideal for flows where recoverability is important.
+Required properties are in <strong>bold</strong>.</p>
+</div>
+<table class="tableblock frame-all grid-all stretch">
+<colgroup>
+<col style="width: 33.3333%;">
+<col style="width: 33.3333%;">
+<col style="width: 33.3334%;">
+</colgroup>
+<thead>
+<tr>
+<th class="tableblock halign-left valign-top">Property Name</th>
+<th class="tableblock halign-left valign-top">Default</th>
+<th class="tableblock halign-left valign-top">Description</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock"><strong>type</strong></p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">The component type name, needs to be <code>jdbc</code></p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">db.type</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">DERBY</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Database vendor, needs to be DERBY.</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">driver.class</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">org.apache.derby.jdbc.EmbeddedDriver</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Class for vendor&#8217;s JDBC driver</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">driver.url</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">(constructed from other properties)</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">JDBC connection URL</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">db.username</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">"sa"</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">User id for db connection</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">db.password</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">password for db connection</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">connection.properties.file</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">JDBC Connection property file path</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">create.schema</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">true</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">If true, then creates db schema if not there</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">create.index</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">true</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Create indexes to speed up lookups</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">create.foreignkey</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">true</p></td>
+<td class="tableblock halign-left valign-top"></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">transaction.isolation</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">"READ_COMMITTED"</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Isolation level for db session READ_UNCOMMITTED,
+READ_COMMITTED, SERIALIZABLE, REPEATABLE_READ</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">maximum.connections</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">10</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Max connections allowed to db</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">maximum.capacity</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">0 (unlimited)</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Max number of events in the channel</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">sysprop.*</p></td>
+<td class="tableblock halign-left valign-top"></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">DB Vendor specific properties</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">sysprop.user.home</p></td>
+<td class="tableblock halign-left valign-top"></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Home path to store embedded Derby database</p></td>
+</tr>
+</tbody>
+</table>
+<div class="paragraph">
+<p>Example for agent named a1:</p>
+</div>
+<div class="listingblock">
+<div class="content">
+<pre class="highlightjs highlight"><code class="language-properties hljs" data-lang="properties">a1.channels = c1
+a1.channels.c1.type = jdbc</code></pre>
+</div>
+</div>
+</div>
+<div class="sect3">
+<h4 id="_kafka_channel"><a class="anchor" href="#_kafka_channel"></a>Kafka Channel</h4>
+<div class="paragraph">
+<p>The events are stored in a Kafka cluster (must be installed separately). Kafka provides high availability and
+replication, so in case an agent or a kafka broker crashes, the events are immediately available to other sinks</p>
+</div>
+<div class="paragraph">
+<p>The Kafka channel can be used for multiple scenarios:</p>
+</div>
+<div class="olist arabic">
+<ol class="arabic">
+<li>
+<p>With Flume source and sink - it provides a reliable and highly available channel for events</p>
+</li>
+<li>
+<p>With Flume source and interceptor but no sink - it allows writing Flume events into a Kafka topic, for use by other apps</p>
+</li>
+<li>
+<p>With Flume sink, but no source - it is a low-latency, fault tolerant way to send events from Kafka to Flume sinks such as HDFS, HBase or Solr</p>
+</li>
+</ol>
+</div>
+<div class="paragraph">
+<p>This currently supports Kafka server releases 0.10.1.0 or higher. Testing was done up to 2.0.1 that was the highest avilable version at the time of the release.</p>
+</div>
+<div class="paragraph">
+<p>The configuration parameters are organized as such:</p>
+</div>
+<div class="olist arabic">
+<ol class="arabic">
+<li>
+<p>Configuration values related to the channel generically are applied at the channel config level, eg: a1.channel.k1.type =</p>
+</li>
+<li>
+<p>Configuration values related to Kafka or how the Channel operates are prefixed with "kafka.", (this are analgous to CommonClient Configs) eg: a1.channels.k1.kafka.topic and a1.channels.k1.kafka.bootstrap.servers. This is not dissimilar to how the hdfs sink operates</p>
+</li>
+<li>
+<p>Properties specific to the producer/consumer are prefixed by kafka.producer or kafka.consumer</p>
+</li>
+<li>
+<p>Where possible, the Kafka paramter names are used, eg: bootstrap.servers and acks</p>
+</li>
+</ol>
+</div>
+<div class="paragraph">
+<p>This version of flume is backwards-compatible with previous versions, however deprecated properties are indicated in the table below and a warning message
+is logged on startup when they are present in the configuration file.</p>
+</div>
+<div class="paragraph">
+<p>Required properties are in <strong>bold</strong>.</p>
+</div>
+<table class="tableblock frame-all grid-all stretch">
+<colgroup>
+<col style="width: 33.3333%;">
+<col style="width: 33.3333%;">
+<col style="width: 33.3334%;">
+</colgroup>
+<thead>
+<tr>
+<th class="tableblock halign-left valign-top">Property Name</th>
+<th class="tableblock halign-left valign-top">Default</th>
+<th class="tableblock halign-left valign-top">Description</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock"><strong>type</strong></p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">The component type name, needs to be <code>org.apache.flume.channel.kafka.KafkaChannel</code></p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock"><strong>kafka.bootstrap.servers</strong></p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">List of brokers in the Kafka cluster used by the channel
+This can be a partial list of brokers, but we recommend at least two for HA.
+The format is comma separated list of hostname:port</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">kafka.topic</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">flume-channel</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Kafka topic which the channel will use</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">kafka.consumer.group.id</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">flume</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Consumer group ID the channel uses to register with Kafka.
+Multiple channels must use the same topic and group to ensure that when one agent fails another can get the data
+Note that having non-channel consumers with the same ID can lead to data loss.</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">parseAsFlumeEvent</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">true</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Expecting Avro datums with FlumeEvent schema in the channel.
+This should be true if Flume source is writing to the channel and false if other producers are
+writing into the topic that the channel is using. Flume source messages to Kafka can be parsed outside of Flume by using
+org.apache.flume.source.avro.AvroFlumeEvent provided by the flume-ng-sdk artifact</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">pollTimeout</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">500</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">The amount of time(in milliseconds) to wait in the "poll()" call of the consumer.
+<a href="https://kafka.apache.org/090/javadoc/org/apache/kafka/clients/consumer/KafkaConsumer.html#poll(long" class="bare">https://kafka.apache.org/090/javadoc/org/apache/kafka/clients/consumer/KafkaConsumer.html#poll(long</a>)</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">defaultPartitionId</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Specifies a Kafka partition ID (integer) for all events in this channel to be sent to, unless
+overriden by <code>partitionIdHeader</code>. By default, if this property is not set, events will be
+distributed by the Kafka Producer&#8217;s partitioner - including by <code>key</code> if specified (or by a
+partitioner specified by <code>kafka.partitioner.class</code>).</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">partitionIdHeader</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">When set, the producer will take the value of the field named using the value of this property
+from the event header and send the message to the specified partition of the topic. If the
+value represents an invalid partition the event will not be accepted into the channel. If the header value
+is present then this setting overrides <code>defaultPartitionId</code>.</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">kafka.consumer.auto.offset.reset</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">latest</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">What to do when there is no initial offset in Kafka or if the current offset does not exist any more on the server
+(e.g. because that data has been deleted):
+earliest: automatically reset the offset to the earliest offset
+latest: automatically reset the offset to the latest offset
+none: throw exception to the consumer if no previous offset is found for the consumer&#8217;s group
+anything else: throw exception to the consumer.</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">kafka.producer.security.protocol</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">PLAINTEXT</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Set to SASL_PLAINTEXT, SASL_SSL or SSL if writing to Kafka using some level of security. See below for additional info on secure setup.</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">kafka.consumer.security.protocol</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">PLAINTEXT</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Same as kafka.producer.security.protocol but for reading/consuming from Kafka.</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock"><em>more producer/consumer security props</em></p></td>
+<td class="tableblock halign-left valign-top"></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">If using SASL_PLAINTEXT, SASL_SSL or SSL refer to <a href="https://kafka.apache.org/documentation.html#security">Kafka security</a> for additional
+properties that need to be set on producer/consumer.</p></td>
+</tr>
+</tbody>
+</table>
+<div class="paragraph">
+<p>Deprecated Properties</p>
+</div>
+<table class="tableblock frame-all grid-all stretch">
+<colgroup>
+<col style="width: 33.3333%;">
+<col style="width: 33.3333%;">
+<col style="width: 33.3334%;">
+</colgroup>
+<thead>
+<tr>
+<th class="tableblock halign-left valign-top">Property Name</th>
+<th class="tableblock halign-left valign-top">Default</th>
+<th class="tableblock halign-left valign-top">Description</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">brokerList</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">List of brokers in the Kafka cluster used by the channel
+This can be a partial list of brokers, but we recommend at least two for HA.
+The format is comma separated list of hostname:port</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">topic</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">flume-channel</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Use kafka.topic</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">groupId</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">flume</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Use kafka.consumer.group.id</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">readSmallestOffset</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">false</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Use kafka.consumer.auto.offset.reset</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">migrateZookeeperOffsets</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">true</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">When no Kafka stored offset is found, look up the offsets in Zookeeper and commit them to Kafka.
+This should be true to support seamless Kafka client migration from older versions of Flume. Once migrated this can be set
+to false, though that should generally not be required. If no Zookeeper offset is found the kafka.consumer.auto.offset.reset
+configuration defines how offsets are handled.</p></td>
+</tr>
+</tbody>
+</table>
+<div class="admonitionblock note">
+<table>
+<tr>
+<td class="icon">
+<i class="fa icon-note" title="Note"></i>
+</td>
+<td class="content">
+<div class="paragraph">
+<p>Due to the way the channel is load balanced, there may be duplicate events when the agent first starts up</p>
+</div>
+</td>
+</tr>
+</table>
+</div>
+<div class="paragraph">
+<p>Example for agent named a1:</p>
+</div>
+<div class="listingblock">
+<div class="content">
+<pre class="highlightjs highlight"><code class="language-properties hljs" data-lang="properties">a1.channels.channel1.type = org.apache.flume.channel.kafka.KafkaChannel
+a1.channels.channel1.kafka.bootstrap.servers = kafka-1:9092,kafka-2:9092,kafka-3:9092
+a1.channels.channel1.kafka.topic = channel1
+a1.channels.channel1.kafka.consumer.group.id = flume-consumer</code></pre>
+</div>
+</div>
+<div class="paragraph">
+<p><strong>Security and Kafka Channel:</strong></p>
+</div>
+<div class="paragraph">
+<p>Secure authentication as well as data encryption is supported on the communication channel between Flume and Kafka.
+For secure authentication SASL/GSSAPI (Kerberos V5) or SSL (even though the parameter is named SSL, the actual protocol is a TLS implementation) can be used from Kafka version 0.9.0.</p>
+</div>
+<div class="paragraph">
+<p>As of now data encryption is solely provided by SSL/TLS.</p>
+</div>
+<div class="paragraph">
+<p>Setting <code>kafka.producer|consumer.security.protocol</code> to any of the following value means:</p>
+</div>
+<div class="ulist">
+<ul>
+<li>
+<p><strong>SASL_PLAINTEXT</strong> - Kerberos or plaintext authentication with no data encryption</p>
+</li>
+<li>
+<p><strong>SASL_SSL</strong> - Kerberos or plaintext authentication with data encryption</p>
+</li>
+<li>
+<p><strong>SSL</strong> - TLS based encryption with optional authentication.</p>
+</li>
+</ul>
+</div>
+<div class="admonitionblock warning">
+<table>
+<tr>
+<td class="icon">
+<i class="fa icon-warning" title="Warning"></i>
+</td>
+<td class="content">
+<div class="paragraph">
+<p>There is a performance degradation when SSL is enabled,
+the magnitude of which depends on the CPU type and the JVM implementation.
+Reference: <a href="https://kafka.apache.org/documentation#security_overview">Kafka security overview</a>
+and the jira for tracking this issue:
+<a href="https://issues.apache.org/jira/browse/KAFKA-2561">KAFKA-2561</a></p>
+</div>
+</td>
+</tr>
+</table>
+</div>
+<div class="paragraph">
+<p><strong>TLS and Kafka Channel:</strong></p>
+</div>
+<div class="paragraph">
+<p>Please read the steps described in <a href="https://kafka.apache.org/documentation#security_configclients">Configuring Kafka Clients SSL</a>
+to learn about additional configuration settings for fine tuning for example any of the following:
+security provider, cipher suites, enabled protocols, truststore or keystore types.</p>
+</div>
+<div class="paragraph">
+<p>Example configuration with server side authentication and data encryption.</p>
+</div>
+<div class="listingblock">
+<div class="content">
+<pre class="highlightjs highlight"><code class="language-properties hljs" data-lang="properties">a1.channels.channel1.type = org.apache.flume.channel.kafka.KafkaChannel
+a1.channels.channel1.kafka.bootstrap.servers = kafka-1:9093,kafka-2:9093,kafka-3:9093
+a1.channels.channel1.kafka.topic = channel1
+a1.channels.channel1.kafka.consumer.group.id = flume-consumer
+a1.channels.channel1.kafka.producer.security.protocol = SSL
+# optional, the global truststore can be used alternatively
+a1.channels.channel1.kafka.producer.ssl.truststore.location = /path/to/truststore.jks
+a1.channels.channel1.kafka.producer.ssl.truststore.password = &lt;password to access the truststore&gt;
+a1.channels.channel1.kafka.consumer.security.protocol = SSL
+# optional, the global truststore can be used alternatively
+a1.channels.channel1.kafka.consumer.ssl.truststore.location = /path/to/truststore.jks
+a1.channels.channel1.kafka.consumer.ssl.truststore.password = &lt;password to access the truststore&gt;</code></pre>
+</div>
+</div>
+<div class="paragraph">
+<p>Specyfing the truststore is optional here, the global truststore can be used instead.
+For more details about the global SSL setup, see the <a href="#ssltls-support">SSL/TLS support</a> section.</p>
+</div>
+<div class="paragraph">
+<p>Note: By default the property <code>ssl.endpoint.identification.algorithm</code>
+is not defined, so hostname verification is not performed.
+In order to enable hostname verification, set the following properties</p>
+</div>
+<div class="listingblock">
+<div class="content">
+<pre class="highlightjs highlight"><code class="language-properties hljs" data-lang="properties">a1.channels.channel1.kafka.producer.ssl.endpoint.identification.algorithm = HTTPS
+a1.channels.channel1.kafka.consumer.ssl.endpoint.identification.algorithm = HTTPS</code></pre>
+</div>
+</div>
+<div class="paragraph">
+<p>Once enabled, clients will verify the server&#8217;s fully qualified domain name (FQDN)
+against one of the following two fields:</p>
+</div>
+<div class="olist arabic">
+<ol class="arabic">
+<li>
+<p>Common Name (CN) <a href="https://tools.ietf.org/html/rfc6125#section-2.3" class="bare">https://tools.ietf.org/html/rfc6125#section-2.3</a></p>
+</li>
+<li>
+<p>Subject Alternative Name (SAN) <a href="https://tools.ietf.org/html/rfc5280#section-4.2.1.6" class="bare">https://tools.ietf.org/html/rfc5280#section-4.2.1.6</a></p>
+</li>
+</ol>
+</div>
+<div class="paragraph">
+<p>If client side authentication is also required then additionally the following needs to be added to Flume agent
+configuration or the global SSL setup can be used (see <a href="#ssltls-support">SSL/TLS support</a> section).
+Each Flume agent has to have its client certificate which has to be trusted by Kafka brokers either
+individually or by their signature chain. Common example is to sign each client certificate by a single Root CA
+which in turn is trusted by Kafka brokers.</p>
+</div>
+<div class="listingblock">
+<div class="content">
+<pre class="highlightjs highlight"><code class="language-properties hljs" data-lang="properties"># optional, the global keystore can be used alternatively
+a1.channels.channel1.kafka.producer.ssl.keystore.location = /path/to/client.keystore.jks
+a1.channels.channel1.kafka.producer.ssl.keystore.password = &lt;password to access the keystore&gt;
+# optional, the global keystore can be used alternatively
+a1.channels.channel1.kafka.consumer.ssl.keystore.location = /path/to/client.keystore.jks
+a1.channels.channel1.kafka.consumer.ssl.keystore.password = &lt;password to access the keystore&gt;</code></pre>
+</div>
+</div>
+<div class="paragraph">
+<p>If keystore and key use different password protection then <code>ssl.key.password</code> property will
+provide the required additional secret for both consumer and producer keystores:</p>
+</div>
+<div class="listingblock">
+<div class="content">
+<pre class="highlightjs highlight"><code class="language-properties hljs" data-lang="properties">a1.channels.channel1.kafka.producer.ssl.key.password = &lt;password to access the key&gt;
+a1.channels.channel1.kafka.consumer.ssl.key.password = &lt;password to access the key&gt;</code></pre>
+</div>
+</div>
+<div class="paragraph">
+<p><strong>Kerberos and Kafka Channel:</strong></p>
+</div>
+<div class="paragraph">
+<p>To use Kafka channel with a Kafka cluster secured with Kerberos, set the <code>producer/consumer.security.protocol</code> properties noted above for producer and/or consumer.
+The Kerberos keytab and principal to be used with Kafka brokers is specified in a JAAS file&#8217;s "KafkaClient" section. "Client" section describes the Zookeeper connection if needed.
+See <a href="https://kafka.apache.org/documentation.html#security_sasl_clientconfig">Kafka doc</a>
+for information on the JAAS file contents. The location of this JAAS file and optionally the system wide kerberos configuration can be specified via JAVA_OPTS in flume-env.sh:</p>
+</div>
+<div class="listingblock">
+<div class="content">
+<pre class="highlightjs highlight"><code class="language-properties hljs" data-lang="properties">JAVA_OPTS="$JAVA_OPTS -Djava.security.krb5.conf=/path/to/krb5.conf"
+JAVA_OPTS="$JAVA_OPTS -Djava.security.auth.login.config=/path/to/flume_jaas.conf"</code></pre>
+</div>
+</div>
+<div class="paragraph">
+<p>Example secure configuration using SASL_PLAINTEXT:</p>
+</div>
+<div class="listingblock">
+<div class="content">
+<pre class="highlightjs highlight"><code class="language-properties hljs" data-lang="properties">a1.channels.channel1.type = org.apache.flume.channel.kafka.KafkaChannel
+a1.channels.channel1.kafka.bootstrap.servers = kafka-1:9093,kafka-2:9093,kafka-3:9093
+a1.channels.channel1.kafka.topic = channel1
+a1.channels.channel1.kafka.consumer.group.id = flume-consumer
+a1.channels.channel1.kafka.producer.security.protocol = SASL_PLAINTEXT
+a1.channels.channel1.kafka.producer.sasl.mechanism = GSSAPI
+a1.channels.channel1.kafka.producer.sasl.kerberos.service.name = kafka
+a1.channels.channel1.kafka.consumer.security.protocol = SASL_PLAINTEXT
+a1.channels.channel1.kafka.consumer.sasl.mechanism = GSSAPI
+a1.channels.channel1.kafka.consumer.sasl.kerberos.service.name = kafka</code></pre>
+</div>
+</div>
+<div class="paragraph">
+<p>Example secure configuration using SASL_SSL:</p>
+</div>
+<div class="listingblock">
+<div class="content">
+<pre class="highlightjs highlight"><code class="language-properties hljs" data-lang="properties">a1.channels.channel1.type = org.apache.flume.channel.kafka.KafkaChannel
+a1.channels.channel1.kafka.bootstrap.servers = kafka-1:9093,kafka-2:9093,kafka-3:9093
+a1.channels.channel1.kafka.topic = channel1
+a1.channels.channel1.kafka.consumer.group.id = flume-consumer
+a1.channels.channel1.kafka.producer.security.protocol = SASL_SSL
+a1.channels.channel1.kafka.producer.sasl.mechanism = GSSAPI
+a1.channels.channel1.kafka.producer.sasl.kerberos.service.name = kafka
+# optional, the global truststore can be used alternatively
+a1.channels.channel1.kafka.producer.ssl.truststore.location = /path/to/truststore.jks
+a1.channels.channel1.kafka.producer.ssl.truststore.password = &lt;password to access the truststore&gt;
+a1.channels.channel1.kafka.consumer.security.protocol = SASL_SSL
+a1.channels.channel1.kafka.consumer.sasl.mechanism = GSSAPI
+a1.channels.channel1.kafka.consumer.sasl.kerberos.service.name = kafka
+# optional, the global truststore can be used alternatively
+a1.channels.channel1.kafka.consumer.ssl.truststore.location = /path/to/truststore.jks
+a1.channels.channel1.kafka.consumer.ssl.truststore.password = &lt;password to access the truststore&gt;</code></pre>
+</div>
+</div>
+<div class="paragraph">
+<p>Sample JAAS file. For reference of its content please see client config sections of the desired authentication mechanism (GSSAPI/PLAIN)
+in Kafka documentation of <a href="https://kafka.apache.org/documentation#security_sasl_clientconfig">SASL configuration</a>.
+Since the Kafka Source may also connect to Zookeeper for offset migration, the "Client" section was also added to this example.
+This won&#8217;t be needed unless you require offset migration, or you require this section for other secure components.
+Also please make sure that the operating system user of the Flume processes has read privileges on the jaas and keytab files.</p>
+</div>
+<div class="listingblock">
+<div class="content">
+<pre class="highlightjs highlight"><code class="language-javascript hljs" data-lang="javascript">Client {
+  com.sun.security.auth.module.Krb5LoginModule required
+  useKeyTab=true
+  storeKey=true
+  keyTab="/path/to/keytabs/flume.keytab"
+  principal="flume/flumehost1.example.com@YOURKERBEROSREALM";
+};
+
+KafkaClient {
+  com.sun.security.auth.module.Krb5LoginModule required
+  useKeyTab=true
+  storeKey=true
+  keyTab="/path/to/keytabs/flume.keytab"
+  principal="flume/flumehost1.example.com@YOURKERBEROSREALM";
+};</code></pre>
+</div>
+</div>
+</div>
+<div class="sect3">
+<h4 id="_file_channel"><a class="anchor" href="#_file_channel"></a>File Channel</h4>
+<div class="paragraph">
+<p>Required properties are in <strong>bold</strong>.</p>
+</div>
+<table class="tableblock frame-all grid-all stretch">
+<colgroup>
+<col style="width: 33.3333%;">
+<col style="width: 33.3333%;">
+<col style="width: 33.3334%;">
+</colgroup>
+<thead>
+<tr>
+<th class="tableblock halign-left valign-top">Property Name Default</th>
+<th class="tableblock halign-left valign-top">Description</th>
+<th class="tableblock halign-left valign-top"></th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock"><strong>type</strong></p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">The component type name, needs to be <code>file</code>.</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">checkpointDir</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">~/.flume/file-channel/checkpoint</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">The directory where checkpoint file will be stored</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">useDualCheckpoints</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">false</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Backup the checkpoint. If this is set to <code>true</code>, <code>backupCheckpointDir</code> <strong>must</strong> be set</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">backupCheckpointDir</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">The directory where the checkpoint is backed up to. This directory <strong>must not</strong> be the same as the data directories or the checkpoint directory</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">dataDirs</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">~/.flume/file-channel/data</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Comma separated list of directories for storing log files. Using multiple directories on separate disks can improve file channel peformance</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">transactionCapacity</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">10000</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">The maximum size of transaction supported by the channel</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">checkpointInterval</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">30000</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Amount of time (in millis) between checkpoints</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">maxFileSize</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">2146435071</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Max size (in bytes) of a single log file</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">minimumRequiredSpace</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">524288000</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Minimum Required free space (in bytes). To avoid data corruption, File Channel stops accepting take/put requests when free space drops below this value</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">capacity</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">1000000</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Maximum capacity of the channel</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">keep-alive</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">3</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Amount of time (in sec) to wait for a put operation</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">use-log-replay-v1</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">false</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Expert: Use old replay logic</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">use-fast-replay</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">false</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Expert: Replay without using queue</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">checkpointOnClose</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">true</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Controls if a checkpoint is created when the channel is closed. Creating a checkpoint on close speeds up subsequent startup of the file channel by avoiding replay.</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">encryption.activeKey</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Key name used to encrypt new data</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">encryption.cipherProvider</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Cipher provider type, supported types: AESCTRNOPADDING</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">encryption.keyProvider</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Key provider type, supported types: JCEKSFILE</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">encryption.keyProvider.keyStoreFile</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Path to the keystore file</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">encrpytion.keyProvider.keyStorePasswordFile</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Path to the keystore password file</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">encryption.keyProvider.keys</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">List of all keys (e.g. history of the activeKey setting)</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">encyption.keyProvider.keys.*.passwordFile</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Path to the optional key password file</p></td>
+</tr>
+</tbody>
+</table>
+<div class="admonitionblock note">
+<table>
+<tr>
+<td class="icon">
+<i class="fa icon-note" title="Note"></i>
+</td>
+<td class="content">
+<div class="paragraph">
+<p>By default the File Channel uses paths for checkpoint and data
+directories that are within the user home as specified above.
+As a result if you have more than one File Channel instances
+active within the agent, only one will be able to lock the
+directories and cause the other channel initialization to fail.
+It is therefore necessary that you provide explicit paths to
+all the configured channels, preferably on different disks.
+Furthermore, as file channel will sync to disk after every commit,
+coupling it with a sink/source that batches events together may
+be necessary to provide good performance where multiple disks are
+not available for checkpoint and data directories.</p>
+</div>
+</td>
+</tr>
+</table>
+</div>
+<div class="paragraph">
+<p>Example for agent named a1:</p>
+</div>
+<div class="listingblock">
+<div class="content">
+<pre class="highlightjs highlight"><code class="language-properties hljs" data-lang="properties">a1.channels = c1
+a1.channels.c1.type = file
+a1.channels.c1.checkpointDir = /mnt/flume/checkpoint
+a1.channels.c1.dataDirs = /mnt/flume/data</code></pre>
+</div>
+</div>
+<div class="paragraph">
+<p><strong>Encryption</strong></p>
+</div>
+<div class="paragraph">
+<p>Below is a few sample configurations:</p>
+</div>
+<div class="paragraph">
+<p>Generating a key with a password seperate from the key store password:</p>
+</div>
+<div class="listingblock">
+<div class="content">
+<pre class="highlightjs highlight"><code class="language-bash hljs" data-lang="bash">keytool -genseckey -alias key-0 -keypass keyPassword -keyalg AES \
+  -keysize 128 -validity 9000 -keystore test.keystore \
+  -storetype jceks -storepass keyStorePassword</code></pre>
+</div>
+</div>
+<div class="paragraph">
+<p>Generating a key with the password the same as the key store password:</p>
+</div>
+<div class="listingblock">
+<div class="content">
+<pre class="highlightjs highlight"><code class="language-bash hljs" data-lang="bash">keytool -genseckey -alias key-1 -keyalg AES -keysize 128 -validity 9000 \
+  -keystore src/test/resources/test.keystore -storetype jceks \
+  -storepass keyStorePassword</code></pre>
+</div>
+</div>
+<div class="listingblock">
+<div class="content">
+<pre class="highlightjs highlight"><code class="language-properties hljs" data-lang="properties">a1.channels.c1.encryption.activeKey = key-0
+a1.channels.c1.encryption.cipherProvider = AESCTRNOPADDING
+a1.channels.c1.encryption.keyProvider = key-provider-0
+a1.channels.c1.encryption.keyProvider = JCEKSFILE
+a1.channels.c1.encryption.keyProvider.keyStoreFile = /path/to/my.keystore
+a1.channels.c1.encryption.keyProvider.keyStorePasswordFile = /path/to/my.keystore.password
+a1.channels.c1.encryption.keyProvider.keys = key-0</code></pre>
+</div>
+</div>
+<div class="paragraph">
+<p>Let&#8217;s say you have aged key-0 out and new files should be encrypted with key-1:</p>
+</div>
+<div class="listingblock">
+<div class="content">
+<pre class="highlightjs highlight"><code class="language-properties hljs" data-lang="properties">a1.channels.c1.encryption.activeKey = key-1
+a1.channels.c1.encryption.cipherProvider = AESCTRNOPADDING
+a1.channels.c1.encryption.keyProvider = JCEKSFILE
+a1.channels.c1.encryption.keyProvider.keyStoreFile = /path/to/my.keystore
+a1.channels.c1.encryption.keyProvider.keyStorePasswordFile = /path/to/my.keystore.password
+a1.channels.c1.encryption.keyProvider.keys = key-0 key-1</code></pre>
+</div>
+</div>
+<div class="paragraph">
+<p>The same scenerio as above, however key-0 has its own password:</p>
+</div>
+<div class="listingblock">
+<div class="content">
+<pre class="highlightjs highlight"><code class="language-properties hljs" data-lang="properties">a1.channels.c1.encryption.activeKey = key-1
+a1.channels.c1.encryption.cipherProvider = AESCTRNOPADDING
+a1.channels.c1.encryption.keyProvider = JCEKSFILE
+a1.channels.c1.encryption.keyProvider.keyStoreFile = /path/to/my.keystore
+a1.channels.c1.encryption.keyProvider.keyStorePasswordFile = /path/to/my.keystore.password
+a1.channels.c1.encryption.keyProvider.keys = key-0 key-1
+a1.channels.c1.encryption.keyProvider.keys.key-0.passwordFile = /path/to/key-0.password</code></pre>
+</div>
+</div>
+</div>
+<div class="sect3">
+<h4 id="_spillable_memory_channel"><a class="anchor" href="#_spillable_memory_channel"></a>Spillable Memory Channel</h4>
+<div class="paragraph">
+<p>The events are stored in an in-memory queue and on disk. The in-memory queue serves as the primary store and the disk as overflow.
+The disk store is managed using an embedded File channel. When the in-memory queue is full, additional incoming events are stored in
+the file channel. This channel is ideal for flows that need high throughput of memory channel during normal operation, but at the
+same time need the larger capacity of the file channel for better tolerance of intermittent sink side outages or drop in drain rates.
+The throughput will reduce approximately to file channel speeds during such abnormal situations. In case of an agent crash or restart,
+only the events stored on disk are recovered when the agent comes online. <strong>This channel is currently experimental and
+not recommended for use in production.</strong></p>
+</div>
+<div class="paragraph">
+<p>Required properties are in <strong>bold</strong>. Please refer to file channel for additional required properties.</p>
+</div>
+<table class="tableblock frame-all grid-all stretch">
+<colgroup>
+<col style="width: 33.3333%;">
+<col style="width: 33.3333%;">
+<col style="width: 33.3334%;">
+</colgroup>
+<thead>
+<tr>
+<th class="tableblock halign-left valign-top">Property Name</th>
+<th class="tableblock halign-left valign-top">Default</th>
+<th class="tableblock halign-left valign-top">Description</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock"><strong>type</strong></p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">&#8201;&#8212;&#8201;</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">The component type name, needs to be <code>SPILLABLEMEMORY</code></p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">memoryCapacity</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">10000</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Maximum number of events stored in memory queue. To disable use of in-memory queue, set this to zero.</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">overflowCapacity</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">100000000</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Maximum number of events stored in overflow disk (i.e File channel). To disable use of overflow, set this to zero.</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">overflowTimeout</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">3</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">The number of seconds to wait before enabling disk overflow when memory fills up.</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">byteCapacityBufferPercentage</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">20</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Defines the percent of buffer between byteCapacity and the estimated total size
+of all events in the channel, to account for data in headers. See below.</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">byteCapacity</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">see description</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Maximum <strong>bytes</strong> of memory allowed as a sum of all events in the memory queue.
+The implementation only counts the Event <code>body</code>, which is the reason for
+providing the <code>byteCapacityBufferPercentage</code> configuration parameter as well.
+Defaults to a computed value equal to 80% of the maximum memory available to
+the JVM (i.e. 80% of the -Xmx value passed on the command line).
+Note that if you have multiple memory channels on a single JVM, and they happen
+to hold the same physical events (i.e. if you are using a replicating channel
+selector from a single source) then those event sizes may be double-counted for
+channel byteCapacity purposes.
+Setting this value to <code>0</code> will cause this value to fall back to a hard
+internal limit of about 200 GB.</p></td>
... 18382 lines suppressed ...