You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@logging.apache.org by pk...@apache.org on 2026/07/10 14:09:53 UTC
(logging-flume) 01/35: Create Flume web site
This is an automated email from the ASF dual-hosted git repository.
ppkarwasz pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/logging-flume.git
commit 6dc752778fdfac5e6808e2b7733c5acbbd15dd4e
Author: Ralph Goers <rg...@apache.org>
AuthorDate: Sun Sep 20 12:21:21 2020 -0700
Create Flume web site
---
source/sphinx/FlumeDeveloperGuide.rst | 868 ++++
source/sphinx/FlumeUserGuide.rst | 5592 ++++++++++++++++++++++++
source/sphinx/_templates/extranav.html | 23 +
source/sphinx/_themes/flume/globaltoc.html | 2 +
source/sphinx/_themes/flume/layout.html | 55 +
source/sphinx/_themes/flume/localtoc.html | 4 +
source/sphinx/_themes/flume/static/flume.css_t | 346 ++
source/sphinx/_themes/flume/theme.conf | 35 +
source/sphinx/conf.py | 142 +
source/sphinx/contents.rsx | 38 +
source/sphinx/documentation.rst | 32 +
source/sphinx/download.rst | 35 +
source/sphinx/getinvolved.rst | 15 +
source/sphinx/images/DevGuide_image00.png | Bin 0 -> 11849 bytes
source/sphinx/images/DevGuide_image01.png | Bin 0 -> 17014 bytes
source/sphinx/images/UserGuide_image00.png | Bin 0 -> 14069 bytes
source/sphinx/images/UserGuide_image01.png | Bin 0 -> 23462 bytes
source/sphinx/images/UserGuide_image02.png | Bin 0 -> 37785 bytes
source/sphinx/images/UserGuide_image03.png | Bin 0 -> 13666 bytes
source/sphinx/images/feather-small.png | Bin 0 -> 31327 bytes
source/sphinx/images/flume-logo.png | Bin 0 -> 33341 bytes
source/sphinx/index.rst | 557 +++
source/sphinx/license.rst | 136 +
source/sphinx/mailinglists.rst | 14 +
source/sphinx/releases/1.0.0.rst | 140 +
source/sphinx/releases/1.1.0.rst | 72 +
source/sphinx/releases/1.2.0.rst | 207 +
source/sphinx/releases/1.3.0.rst | 195 +
source/sphinx/releases/1.3.1.rst | 41 +
source/sphinx/releases/1.4.0.rst | 227 +
source/sphinx/releases/1.5.0.1.rst | 24 +
source/sphinx/releases/1.5.0.rst | 152 +
source/sphinx/releases/1.5.2.rst | 30 +
source/sphinx/releases/1.6.0.rst | 139 +
source/sphinx/releases/1.7.0.rst | 131 +
source/sphinx/releases/1.8.0.rst | 80 +
source/sphinx/releases/1.9.0.rst | 97 +
source/sphinx/releases/index.rst | 31 +
source/sphinx/source.rst | 102 +
source/sphinx/team.rst | 48 +
40 files changed, 9610 insertions(+)
diff --git a/source/sphinx/FlumeDeveloperGuide.rst b/source/sphinx/FlumeDeveloperGuide.rst
new file mode 100644
index 00000000..5e98bd3e
--- /dev/null
+++ b/source/sphinx/FlumeDeveloperGuide.rst
@@ -0,0 +1,868 @@
+.. Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+
+
+======================================
+Flume 1.9.0 Developer Guide
+======================================
+
+Introduction
+============
+
+Overview
+--------
+
+Apache Flume is a distributed, reliable, and available system for
+efficiently collecting, aggregating and moving large amounts of log
+data from many different sources to a centralized data store.
+
+Apache Flume is a top-level project at the Apache Software Foundation.
+There are currently two release code lines available, versions 0.9.x and 1.x.
+This documentation applies to the 1.x codeline.
+For the 0.9.x codeline, please see the `Flume 0.9.x Developer Guide
+<http://archive.cloudera.com/cdh/3/flume/DeveloperGuide/>`_.
+
+Architecture
+------------
+
+Data flow model
+~~~~~~~~~~~~~~~
+
+An ``Event`` is a unit of data that flows through a Flume agent. The ``Event``
+flows from ``Source`` to ``Channel`` to ``Sink``, and is represented by an
+implementation of the ``Event`` interface. An ``Event`` carries a payload (byte
+array) that is accompanied by an optional set of headers (string attributes).
+A Flume agent is a process (JVM) that hosts the components that allow
+``Event``\ s to flow from an external source to a external destination.
+
+.. figure:: images/DevGuide_image00.png
+ :align: center
+ :alt: Agent component diagram
+
+A ``Source`` consumes ``Event``\ s having a specific format, and those
+``Event``\ s are delivered to the ``Source`` by an external source like a web
+server. For example, an ``AvroSource`` can be used to receive Avro ``Event``\ s
+from clients or from other Flume agents in the flow. When a ``Source`` receives
+an ``Event``, it stores it into one or more ``Channel``\ s. The ``Channel`` is
+a passive store that holds the ``Event`` until that ``Event`` is consumed by a
+``Sink``. One type of ``Channel`` available in Flume is the ``FileChannel``
+which uses the local filesystem as its backing store. A ``Sink`` is responsible
+for removing an ``Event`` from the ``Channel`` and putting it into an external
+repository like HDFS (in the case of an ``HDFSEventSink``) or forwarding it to
+the ``Source`` at the next hop of the flow. The ``Source`` and ``Sink`` within
+the given agent run asynchronously with the ``Event``\ s staged in the
+``Channel``.
+
+Reliability
+~~~~~~~~~~~
+
+An ``Event`` is staged in a Flume agent's ``Channel``. Then it's the
+``Sink``\ 's responsibility to deliver the ``Event`` to the next agent or
+terminal repository (like HDFS) in the flow. The ``Sink`` removes an ``Event``
+from the ``Channel`` only after the ``Event`` is stored into the ``Channel`` of
+the next agent or stored in the terminal repository. This is how the single-hop
+message delivery semantics in Flume provide end-to-end reliability of the flow.
+Flume uses a transactional approach to guarantee the reliable delivery of the
+``Event``\ s. The ``Source``\ s and ``Sink``\ s encapsulate the
+storage/retrieval of the ``Event``\ s in a ``Transaction`` provided by the
+``Channel``. This ensures that the set of ``Event``\ s are reliably passed from
+point to point in the flow. In the case of a multi-hop flow, the ``Sink`` from
+the previous hop and the ``Source`` of the next hop both have their
+``Transaction``\ s open to ensure that the ``Event`` data is safely stored in
+the ``Channel`` of the next hop.
+
+Building Flume
+--------------
+
+Getting the source
+~~~~~~~~~~~~~~~~~~
+
+Check-out the code using Git. Click here for
+`the git repository root <https://git-wip-us.apache.org/repos/asf/flume.git>`_.
+
+The Flume 1.x development happens under the branch "trunk" so this command line
+can be used:
+
+ git clone https://git-wip-us.apache.org/repos/asf/flume.git
+
+
+Compile/test Flume
+~~~~~~~~~~~~~~~~~~
+
+The Flume build is mavenized. You can compile Flume using the standard Maven
+commands:
+
+#. Compile only: ``mvn clean compile``
+#. Compile and run unit tests: ``mvn clean test``
+#. Run individual test(s): ``mvn clean test -Dtest=<Test1>,<Test2>,... -DfailIfNoTests=false``
+#. Create tarball package: ``mvn clean install``
+#. Create tarball package (skip unit tests): ``mvn clean install -DskipTests``
+
+Please note that Flume builds requires that the Google Protocol Buffers compiler
+be in the path. You can download and install it by following the instructions
+`here <https://developers.google.com/protocol-buffers/>`_.
+
+Updating Protocol Buffer Version
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+File channel has a dependency on Protocol Buffer. When updating the version of Protocol Buffer
+used by Flume, it is necessary to regenerate the data access classes using the protoc compiler
+that is part of Protocol Buffer as follows.
+
+#. Install the desired version of Protocol Buffer on your local machine
+#. Update version of Protocol Buffer in pom.xml
+#. Generate new Protocol Buffer data access classes in Flume: ``cd flume-ng-channels/flume-file-channel; mvn -P compile-proto clean package -DskipTests``
+#. Add Apache license header to any of the generated files that are missing it
+#. Rebuild and test Flume: ``cd ../..; mvn clean install``
+
+Developing custom components
+----------------------------
+
+Client
+~~~~~~
+
+The client operates at the point of origin of events and delivers them to a
+Flume agent. Clients typically operate in the process space of the application
+they are consuming data from. Flume currently supports Avro, log4j, syslog,
+and Http POST (with a JSON body) as ways to transfer data from a external
+source. Additionally, there’s an ``ExecSource`` that can consume the output of a
+local process as input to Flume.
+
+It’s quite possible to have a use case where these existing options are not
+sufficient. In this case you can build a custom mechanism to send data to
+Flume. There are two ways of achieving this. The first option is to create a
+custom client that communicates with one of Flume’s existing ``Source``\ s like
+``AvroSource`` or ``SyslogTcpSource``. Here the client should convert its data
+into messages understood by these Flume ``Source``\ s. The other option is to
+write a custom Flume ``Source`` that directly talks with your existing client
+application using some IPC or RPC protocol, and then converts the client data
+into Flume ``Event``\ s to be sent downstream. Note that all events stored
+within the ``Channel`` of a Flume agent must exist as Flume ``Event``\ s.
+
+
+Client SDK
+''''''''''
+
+Though Flume contains a number of built-in mechanisms (i.e. ``Source``\ s) to
+ingest data, often one wants the ability to communicate with Flume directly from
+a custom application. The Flume Client SDK is a library that enables
+applications to connect to Flume and send data into Flume’s data flow over RPC.
+
+
+RPC client interface
+''''''''''''''''''''
+
+An implementation of Flume's RpcClient interface encapsulates the RPC mechanism
+supported by Flume. The user's application can simply call the Flume Client
+SDK's ``append(Event)`` or ``appendBatch(List<Event>)`` to send data and not
+worry about the underlying message exchange details. The user can provide the
+required ``Event`` arg by either directly implementing the ``Event`` interface,
+by using a convenience implementation such as the SimpleEvent class, or by using
+``EventBuilder``\ 's overloaded ``withBody()`` static helper methods.
+
+
+RPC clients - Avro and Thrift
+'''''''''''''''''''''''''''''
+
+As of Flume 1.4.0, Avro is the default RPC protocol. The
+``NettyAvroRpcClient`` and ``ThriftRpcClient`` implement the ``RpcClient``
+interface. The client needs to create this object with the host and port of
+the target Flume agent, and can then use the ``RpcClient`` to send data into
+the agent. The following example shows how to use the Flume Client SDK API
+within a user's data-generating application:
+
+.. code-block:: java
+
+ import org.apache.flume.Event;
+ import org.apache.flume.EventDeliveryException;
+ import org.apache.flume.api.RpcClient;
+ import org.apache.flume.api.RpcClientFactory;
+ import org.apache.flume.event.EventBuilder;
+ import java.nio.charset.Charset;
+
+ public class MyApp {
+ public static void main(String[] args) {
+ MyRpcClientFacade client = new MyRpcClientFacade();
+ // Initialize client with the remote Flume agent's host and port
+ client.init("host.example.org", 41414);
+
+ // Send 10 events to the remote Flume agent. That agent should be
+ // configured to listen with an AvroSource.
+ String sampleData = "Hello Flume!";
+ for (int i = 0; i < 10; i++) {
+ client.sendDataToFlume(sampleData);
+ }
+
+ client.cleanUp();
+ }
+ }
+
+ class MyRpcClientFacade {
+ private RpcClient client;
+ private String hostname;
+ private int port;
+
+ public void init(String hostname, int port) {
+ // Setup the RPC connection
+ this.hostname = hostname;
+ this.port = port;
+ this.client = RpcClientFactory.getDefaultInstance(hostname, port);
+ // Use the following method to create a thrift client (instead of the above line):
+ // this.client = RpcClientFactory.getThriftInstance(hostname, port);
+ }
+
+ public void sendDataToFlume(String data) {
+ // Create a Flume Event object that encapsulates the sample data
+ Event event = EventBuilder.withBody(data, Charset.forName("UTF-8"));
+
+ // Send the event
+ try {
+ client.append(event);
+ } catch (EventDeliveryException e) {
+ // clean up and recreate the client
+ client.close();
+ client = null;
+ client = RpcClientFactory.getDefaultInstance(hostname, port);
+ // Use the following method to create a thrift client (instead of the above line):
+ // this.client = RpcClientFactory.getThriftInstance(hostname, port);
+ }
+ }
+
+ public void cleanUp() {
+ // Close the RPC connection
+ client.close();
+ }
+
+ }
+
+The remote Flume agent needs to have an ``AvroSource`` (or a
+``ThriftSource`` if you are using a Thrift client) listening on some port.
+Below is an example Flume agent configuration that's waiting for a connection
+from MyApp:
+
+.. code-block:: properties
+
+ a1.channels = c1
+ a1.sources = r1
+ a1.sinks = k1
+
+ a1.channels.c1.type = memory
+
+ a1.sources.r1.channels = c1
+ a1.sources.r1.type = avro
+ # For using a thrift source set the following instead of the above line.
+ # a1.source.r1.type = thrift
+ a1.sources.r1.bind = 0.0.0.0
+ a1.sources.r1.port = 41414
+
+ a1.sinks.k1.channel = c1
+ a1.sinks.k1.type = logger
+
+For more flexibility, the default Flume client implementations
+(``NettyAvroRpcClient`` and ``ThriftRpcClient``) can be configured with these
+properties:
+
+.. code-block:: properties
+
+ client.type = default (for avro) or thrift (for thrift)
+
+ hosts = h1 # default client accepts only 1 host
+ # (additional hosts will be ignored)
+
+ hosts.h1 = host1.example.org:41414 # host and port must both be specified
+ # (neither has a default)
+
+ batch-size = 100 # Must be >=1 (default: 100)
+
+ connect-timeout = 20000 # Must be >=1000 (default: 20000)
+
+ request-timeout = 20000 # Must be >=1000 (default: 20000)
+
+Secure RPC client - Thrift
+''''''''''''''''''''''''''
+
+As of Flume 1.6.0, Thrift source and sink supports kerberos based authentication.
+The client needs to use the getThriftInstance method of ``SecureRpcClientFactory``
+to get hold of a ``SecureThriftRpcClient``. ``SecureThriftRpcClient`` extends
+``ThriftRpcClient`` which implements the ``RpcClient`` interface. The kerberos
+authentication module resides in flume-ng-auth module which is
+required in classpath, when using the ``SecureRpcClientFactory``. Both the client
+principal and the client keytab should be passed in as parameters through the
+properties and they reflect the credentials of the client to authenticate
+against the kerberos KDC. In addition, the server principal of the destination
+Thrift source to which this client is connecting to, should also be provided.
+The following example shows how to use the ``SecureRpcClientFactory``
+within a user's data-generating application:
+
+.. code-block:: java
+
+ import org.apache.flume.Event;
+ import org.apache.flume.EventDeliveryException;
+ import org.apache.flume.event.EventBuilder;
+ import org.apache.flume.api.SecureRpcClientFactory;
+ import org.apache.flume.api.RpcClientConfigurationConstants;
+ import org.apache.flume.api.RpcClient;
+ import java.nio.charset.Charset;
+ import java.util.Properties;
+
+ public class MyApp {
+ public static void main(String[] args) {
+ MySecureRpcClientFacade client = new MySecureRpcClientFacade();
+ // Initialize client with the remote Flume agent's host, port
+ Properties props = new Properties();
+ props.setProperty(RpcClientConfigurationConstants.CONFIG_CLIENT_TYPE, "thrift");
+ props.setProperty("hosts", "h1");
+ props.setProperty("hosts.h1", "client.example.org"+":"+ String.valueOf(41414));
+
+ // Initialize client with the kerberos authentication related properties
+ props.setProperty("kerberos", "true");
+ props.setProperty("client-principal", "flumeclient/client.example.org@EXAMPLE.ORG");
+ props.setProperty("client-keytab", "/tmp/flumeclient.keytab");
+ props.setProperty("server-principal", "flume/server.example.org@EXAMPLE.ORG");
+ client.init(props);
+
+ // Send 10 events to the remote Flume agent. That agent should be
+ // configured to listen with an AvroSource.
+ String sampleData = "Hello Flume!";
+ for (int i = 0; i < 10; i++) {
+ client.sendDataToFlume(sampleData);
+ }
+
+ client.cleanUp();
+ }
+ }
+
+ class MySecureRpcClientFacade {
+ private RpcClient client;
+ private Properties properties;
+
+ public void init(Properties properties) {
+ // Setup the RPC connection
+ this.properties = properties;
+ // Create the ThriftSecureRpcClient instance by using SecureRpcClientFactory
+ this.client = SecureRpcClientFactory.getThriftInstance(properties);
+ }
+
+ public void sendDataToFlume(String data) {
+ // Create a Flume Event object that encapsulates the sample data
+ Event event = EventBuilder.withBody(data, Charset.forName("UTF-8"));
+
+ // Send the event
+ try {
+ client.append(event);
+ } catch (EventDeliveryException e) {
+ // clean up and recreate the client
+ client.close();
+ client = null;
+ client = SecureRpcClientFactory.getThriftInstance(properties);
+ }
+ }
+
+ public void cleanUp() {
+ // Close the RPC connection
+ client.close();
+ }
+ }
+
+The remote ``ThriftSource`` should be started in kerberos mode.
+Below is an example Flume agent configuration that's waiting for a connection
+from MyApp:
+
+.. code-block:: properties
+
+ a1.channels = c1
+ a1.sources = r1
+ a1.sinks = k1
+
+ a1.channels.c1.type = memory
+
+ a1.sources.r1.channels = c1
+ a1.sources.r1.type = thrift
+ a1.sources.r1.bind = 0.0.0.0
+ a1.sources.r1.port = 41414
+ a1.sources.r1.kerberos = true
+ a1.sources.r1.agent-principal = flume/server.example.org@EXAMPLE.ORG
+ a1.sources.r1.agent-keytab = /tmp/flume.keytab
+
+
+ a1.sinks.k1.channel = c1
+ a1.sinks.k1.type = logger
+
+Failover Client
+'''''''''''''''
+
+This class wraps the default Avro RPC client to provide failover handling
+capability to clients. This takes a whitespace-separated list of <host>:<port>
+representing the Flume agents that make-up a failover group. The Failover RPC
+Client currently does not support thrift. If there’s a
+communication error with the currently selected host (i.e. agent) agent,
+then the failover client automatically fails-over to the next host in the list.
+For example:
+
+.. code-block:: java
+
+ // Setup properties for the failover
+ Properties props = new Properties();
+ props.put("client.type", "default_failover");
+
+ // List of hosts (space-separated list of user-chosen host aliases)
+ props.put("hosts", "h1 h2 h3");
+
+ // host/port pair for each host alias
+ String host1 = "host1.example.org:41414";
+ String host2 = "host2.example.org:41414";
+ String host3 = "host3.example.org:41414";
+ props.put("hosts.h1", host1);
+ props.put("hosts.h2", host2);
+ props.put("hosts.h3", host3);
+
+ // create the client with failover properties
+ RpcClient client = RpcClientFactory.getInstance(props);
+
+For more flexibility, the failover Flume client implementation
+(``FailoverRpcClient``) can be configured with these properties:
+
+.. code-block:: properties
+
+ client.type = default_failover
+
+ hosts = h1 h2 h3 # at least one is required, but 2 or
+ # more makes better sense
+
+ hosts.h1 = host1.example.org:41414
+
+ hosts.h2 = host2.example.org:41414
+
+ hosts.h3 = host3.example.org:41414
+
+ max-attempts = 3 # Must be >=0 (default: number of hosts
+ # specified, 3 in this case). A '0'
+ # value doesn't make much sense because
+ # it will just cause an append call to
+ # immmediately fail. A '1' value means
+ # that the failover client will try only
+ # once to send the Event, and if it
+ # fails then there will be no failover
+ # to a second client, so this value
+ # causes the failover client to
+ # degenerate into just a default client.
+ # It makes sense to set this value to at
+ # least the number of hosts that you
+ # specified.
+
+ batch-size = 100 # Must be >=1 (default: 100)
+
+ connect-timeout = 20000 # Must be >=1000 (default: 20000)
+
+ request-timeout = 20000 # Must be >=1000 (default: 20000)
+
+LoadBalancing RPC client
+''''''''''''''''''''''''
+
+The Flume Client SDK also supports an RpcClient which load-balances among
+multiple hosts. This type of client takes a whitespace-separated list of
+<host>:<port> representing the Flume agents that make-up a load-balancing group.
+This client can be configured with a load balancing strategy that either
+randomly selects one of the configured hosts, or selects a host in a round-robin
+fashion. You can also specify your own custom class that implements the
+``LoadBalancingRpcClient$HostSelector`` interface so that a custom selection
+order is used. In that case, the FQCN of the custom class needs to be specified
+as the value of the ``host-selector`` property. The LoadBalancing RPC Client
+currently does not support thrift.
+
+If ``backoff`` is enabled then the client will temporarily blacklist
+hosts that fail, causing them to be excluded from being selected as a failover
+host until a given timeout. When the timeout elapses, if the host is still
+unresponsive then this is considered a sequential failure, and the timeout is
+increased exponentially to avoid potentially getting stuck in long waits on
+unresponsive hosts.
+
+The maximum backoff time can be configured by setting ``maxBackoff`` (in
+milliseconds). The maxBackoff default is 30 seconds (specified in the
+``OrderSelector`` class that's the superclass of both load balancing
+strategies). The backoff timeout will increase exponentially with each
+sequential failure up to the maximum possible backoff timeout.
+The maximum possible backoff is limited to 65536 seconds (about 18.2 hours).
+For example:
+
+.. code-block:: java
+
+ // Setup properties for the load balancing
+ Properties props = new Properties();
+ props.put("client.type", "default_loadbalance");
+
+ // List of hosts (space-separated list of user-chosen host aliases)
+ props.put("hosts", "h1 h2 h3");
+
+ // host/port pair for each host alias
+ String host1 = "host1.example.org:41414";
+ String host2 = "host2.example.org:41414";
+ String host3 = "host3.example.org:41414";
+ props.put("hosts.h1", host1);
+ props.put("hosts.h2", host2);
+ props.put("hosts.h3", host3);
+
+ props.put("host-selector", "random"); // For random host selection
+ // props.put("host-selector", "round_robin"); // For round-robin host
+ // // selection
+ props.put("backoff", "true"); // Disabled by default.
+
+ props.put("maxBackoff", "10000"); // Defaults 0, which effectively
+ // becomes 30000 ms
+
+ // Create the client with load balancing properties
+ RpcClient client = RpcClientFactory.getInstance(props);
+
+For more flexibility, the load-balancing Flume client implementation
+(``LoadBalancingRpcClient``) can be configured with these properties:
+
+.. code-block:: properties
+
+ client.type = default_loadbalance
+
+ hosts = h1 h2 h3 # At least 2 hosts are required
+
+ hosts.h1 = host1.example.org:41414
+
+ hosts.h2 = host2.example.org:41414
+
+ hosts.h3 = host3.example.org:41414
+
+ backoff = false # Specifies whether the client should
+ # back-off from (i.e. temporarily
+ # blacklist) a failed host
+ # (default: false).
+
+ maxBackoff = 0 # Max timeout in millis that a will
+ # remain inactive due to a previous
+ # failure with that host (default: 0,
+ # which effectively becomes 30000)
+
+ host-selector = round_robin # The host selection strategy used
+ # when load-balancing among hosts
+ # (default: round_robin).
+ # Other values are include "random"
+ # or the FQCN of a custom class
+ # that implements
+ # LoadBalancingRpcClient$HostSelector
+
+ batch-size = 100 # Must be >=1 (default: 100)
+
+ connect-timeout = 20000 # Must be >=1000 (default: 20000)
+
+ request-timeout = 20000 # Must be >=1000 (default: 20000)
+
+Embedded agent
+~~~~~~~~~~~~~~
+
+Flume has an embedded agent api which allows users to embed an agent in their
+application. This agent is meant to be lightweight and as such not all
+sources, sinks, and channels are allowed. Specifically the source used
+is a special embedded source and events should be send to the source
+via the put, putAll methods on the EmbeddedAgent object. Only File Channel
+and Memory Channel are allowed as channels while Avro Sink is the only
+supported sink. Interceptors are also supported by the embedded agent.
+
+Note: The embedded agent has a dependency on hadoop-core.jar.
+
+Configuration of an Embedded Agent is similar to configuration of a
+full Agent. The following is an exhaustive list of configration options:
+
+Required properties are in **bold**.
+
+===================== ================ ======================================================================
+Property Name Default Description
+===================== ================ ======================================================================
+source.type embedded The only available source is the embedded source.
+**channel.type** -- Either ``memory`` or ``file`` which correspond
+ to MemoryChannel and FileChannel respectively.
+channel.* -- Configuration options for the channel type requested,
+ see MemoryChannel or FileChannel user guide for an exhaustive list.
+**sinks** -- List of sink names
+**sink.type** -- Property name must match a name in the list of sinks.
+ Value must be ``avro``
+sink.* -- Configuration options for the sink.
+ See AvroSink user guide for an exhaustive list,
+ however note AvroSink requires at least hostname and port.
+**processor.type** -- Either ``failover`` or ``load_balance`` which correspond
+ to FailoverSinksProcessor and LoadBalancingSinkProcessor respectively.
+processor.* -- Configuration options for the sink processor selected.
+ See FailoverSinksProcessor and LoadBalancingSinkProcessor
+ user guide for an exhaustive list.
+source.interceptors -- Space-separated list of interceptors
+source.interceptors.* -- Configuration options for individual interceptors
+ specified in the source.interceptors property
+===================== ================ ======================================================================
+
+Below is an example of how to use the agent:
+
+.. code-block:: java
+
+ Map<String, String> properties = new HashMap<String, String>();
+ properties.put("channel.type", "memory");
+ properties.put("channel.capacity", "200");
+ properties.put("sinks", "sink1 sink2");
+ properties.put("sink1.type", "avro");
+ properties.put("sink2.type", "avro");
+ properties.put("sink1.hostname", "collector1.apache.org");
+ properties.put("sink1.port", "5564");
+ properties.put("sink2.hostname", "collector2.apache.org");
+ properties.put("sink2.port", "5565");
+ properties.put("processor.type", "load_balance");
+ properties.put("source.interceptors", "i1");
+ properties.put("source.interceptors.i1.type", "static");
+ properties.put("source.interceptors.i1.key", "key1");
+ properties.put("source.interceptors.i1.value", "value1");
+
+ EmbeddedAgent agent = new EmbeddedAgent("myagent");
+
+ agent.configure(properties);
+ agent.start();
+
+ List<Event> events = Lists.newArrayList();
+
+ events.add(event);
+ events.add(event);
+ events.add(event);
+ events.add(event);
+
+ agent.putAll(events);
+
+ ...
+
+ agent.stop();
+
+
+Transaction interface
+~~~~~~~~~~~~~~~~~~~~~
+
+The ``Transaction`` interface is the basis of reliability for Flume. All the
+major components (ie. ``Source``\ s, ``Sink``\ s and ``Channel``\ s) must use a
+Flume ``Transaction``.
+
+.. figure:: images/DevGuide_image01.png
+ :align: center
+ :alt: Transaction sequence diagram
+
+A ``Transaction`` is implemented within a ``Channel`` implementation. Each
+``Source`` and ``Sink`` that is connected to a ``Channel`` must obtain a
+``Transaction`` object. The ``Source``\ s use a ``ChannelProcessor``
+to manage the ``Transaction``\ s, the ``Sink``\ s manage them explicitly via
+their configured ``Channel``. The operation to stage an
+``Event`` (put it into a ``Channel``) or extract an ``Event`` (take it out of a
+``Channel``) is done inside an active ``Transaction``. For example:
+
+.. code-block:: java
+
+ Channel ch = new MemoryChannel();
+ Transaction txn = ch.getTransaction();
+ txn.begin();
+ try {
+ // This try clause includes whatever Channel operations you want to do
+
+ Event eventToStage = EventBuilder.withBody("Hello Flume!",
+ Charset.forName("UTF-8"));
+ ch.put(eventToStage);
+ // Event takenEvent = ch.take();
+ // ...
+ txn.commit();
+ } catch (Throwable t) {
+ txn.rollback();
+
+ // Log exception, handle individual exceptions as needed
+
+ // re-throw all Errors
+ if (t instanceof Error) {
+ throw (Error)t;
+ }
+ } finally {
+ txn.close();
+ }
+
+Here we get hold of a ``Transaction`` from a ``Channel``. After ``begin()``
+returns, the ``Transaction`` is now active/open and the ``Event`` is then put
+into the ``Channel``. If the put is successful, then the ``Transaction`` is
+committed and closed.
+
+Sink
+~~~~
+
+The purpose of a ``Sink`` to extract ``Event``\ s from the ``Channel`` and
+forward them to the next Flume Agent in the flow or store them in an external
+repository. A ``Sink`` is associated with exactly one ``Channel``\ s, as
+configured in the Flume properties file. There’s one ``SinkRunner`` instance
+associated with every configured ``Sink``, and when the Flume framework calls
+``SinkRunner.start()``, a new thread is created to drive the ``Sink`` (using
+``SinkRunner.PollingRunner`` as the thread's ``Runnable``). This thread manages
+the ``Sink``\ ’s lifecycle. The ``Sink`` needs to implement the ``start()`` and
+``stop()`` methods that are part of the ``LifecycleAware`` interface. The
+``Sink.start()`` method should initialize the ``Sink`` and bring it to a state
+where it can forward the ``Event``\ s to its next destination. The
+``Sink.process()`` method should do the core processing of extracting the
+``Event`` from the ``Channel`` and forwarding it. The ``Sink.stop()`` method
+should do the necessary cleanup (e.g. releasing resources). The ``Sink``
+implementation also needs to implement the ``Configurable`` interface for
+processing its own configuration settings. For example:
+
+.. code-block:: java
+
+ public class MySink extends AbstractSink implements Configurable {
+ private String myProp;
+
+ @Override
+ public void configure(Context context) {
+ String myProp = context.getString("myProp", "defaultValue");
+
+ // Process the myProp value (e.g. validation)
+
+ // Store myProp for later retrieval by process() method
+ this.myProp = myProp;
+ }
+
+ @Override
+ public void start() {
+ // Initialize the connection to the external repository (e.g. HDFS) that
+ // this Sink will forward Events to ..
+ }
+
+ @Override
+ public void stop () {
+ // Disconnect from the external respository and do any
+ // additional cleanup (e.g. releasing resources or nulling-out
+ // field values) ..
+ }
+
+ @Override
+ public Status process() throws EventDeliveryException {
+ Status status = null;
+
+ // Start transaction
+ Channel ch = getChannel();
+ Transaction txn = ch.getTransaction();
+ txn.begin();
+ try {
+ // This try clause includes whatever Channel operations you want to do
+
+ Event event = ch.take();
+
+ // Send the Event to the external repository.
+ // storeSomeData(e);
+
+ txn.commit();
+ status = Status.READY;
+ } catch (Throwable t) {
+ txn.rollback();
+
+ // Log exception, handle individual exceptions as needed
+
+ status = Status.BACKOFF;
+
+ // re-throw all Errors
+ if (t instanceof Error) {
+ throw (Error)t;
+ }
+ }
+ return status;
+ }
+ }
+
+Source
+~~~~~~
+
+The purpose of a ``Source`` is to receive data from an external client and store
+it into the configured ``Channel``\ s. A ``Source`` can get an instance of its own
+``ChannelProcessor`` to process an ``Event``, commited within a ``Channel``
+local transaction, in serial. In the case of an exception, required
+``Channel``\ s will propagate the exception, all ``Channel``\ s will rollback their
+transaction, but events processed previously on other ``Channel``\ s will remain
+committed.
+
+Similar to the ``SinkRunner.PollingRunner`` ``Runnable``, there’s
+a ``PollingRunner`` ``Runnable`` that executes on a thread created when the
+Flume framework calls ``PollableSourceRunner.start()``. Each configured
+``PollableSource`` is associated with its own thread that runs a
+``PollingRunner``. This thread manages the ``PollableSource``\ ’s lifecycle,
+such as starting and stopping. A ``PollableSource`` implementation must
+implement the ``start()`` and ``stop()`` methods that are declared in the
+``LifecycleAware`` interface. The runner of a ``PollableSource`` invokes that
+``Source``\ 's ``process()`` method. The ``process()`` method should check for
+new data and store it into the ``Channel`` as Flume ``Event``\ s.
+
+Note that there are actually two types of ``Source``\ s. The ``PollableSource``
+was already mentioned. The other is the ``EventDrivenSource``. The
+``EventDrivenSource``, unlike the ``PollableSource``, must have its own callback
+mechanism that captures the new data and stores it into the ``Channel``. The
+``EventDrivenSource``\ s are not each driven by their own thread like the
+``PollableSource``\ s are. Below is an example of a custom ``PollableSource``:
+
+.. code-block:: java
+
+ public class MySource extends AbstractSource implements Configurable, PollableSource {
+ private String myProp;
+
+ @Override
+ public void configure(Context context) {
+ String myProp = context.getString("myProp", "defaultValue");
+
+ // Process the myProp value (e.g. validation, convert to another type, ...)
+
+ // Store myProp for later retrieval by process() method
+ this.myProp = myProp;
+ }
+
+ @Override
+ public void start() {
+ // Initialize the connection to the external client
+ }
+
+ @Override
+ public void stop () {
+ // Disconnect from external client and do any additional cleanup
+ // (e.g. releasing resources or nulling-out field values) ..
+ }
+
+ @Override
+ public Status process() throws EventDeliveryException {
+ Status status = null;
+
+ try {
+ // This try clause includes whatever Channel/Event operations you want to do
+
+ // Receive new data
+ Event e = getSomeData();
+
+ // Store the Event into this Source's associated Channel(s)
+ getChannelProcessor().processEvent(e);
+
+ status = Status.READY;
+ } catch (Throwable t) {
+ // Log exception, handle individual exceptions as needed
+
+ status = Status.BACKOFF;
+
+ // re-throw all Errors
+ if (t instanceof Error) {
+ throw (Error)t;
+ }
+ } finally {
+ txn.close();
+ }
+ return status;
+ }
+ }
+
+Channel
+~~~~~~~
+
+TBD
diff --git a/source/sphinx/FlumeUserGuide.rst b/source/sphinx/FlumeUserGuide.rst
new file mode 100644
index 00000000..b740507f
--- /dev/null
+++ b/source/sphinx/FlumeUserGuide.rst
@@ -0,0 +1,5592 @@
+.. Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+
+
+===============================
+Flume 1.9.0 User Guide
+===============================
+
+Introduction
+============
+
+Overview
+--------
+
+Apache Flume is a distributed, reliable, and available system for efficiently
+collecting, aggregating and moving large amounts of log data from many
+different sources to a centralized data store.
+
+The use of Apache Flume is not only restricted to log data aggregation.
+Since data sources are customizable, Flume can be used to transport massive quantities
+of event data including but not limited to network traffic data, social-media-generated data,
+email messages and pretty much any data source possible.
+
+Apache Flume is a top level project at the Apache Software Foundation.
+
+System Requirements
+-------------------
+
+#. Java Runtime Environment - Java 1.8 or later
+#. Memory - Sufficient memory for configurations used by sources, channels or sinks
+#. Disk Space - Sufficient disk space for configurations used by channels or sinks
+#. Directory Permissions - Read/Write permissions for directories used by agent
+
+Architecture
+------------
+
+Data flow model
+~~~~~~~~~~~~~~~
+
+A Flume event is defined as a unit of data flow having a byte payload and an
+optional set of string attributes. A Flume agent is a (JVM) process that hosts
+the components through which events flow from an external source to the next
+destination (hop).
+
+.. figure:: images/UserGuide_image00.png
+ :align: center
+ :alt: Agent component diagram
+
+A Flume source consumes events delivered to it by an external source like a web
+server. The external source sends events to Flume in a format that is
+recognized by the target Flume source. For example, an Avro Flume source can be
+used to receive Avro events from Avro clients or other Flume agents in the flow
+that send events from an Avro sink. A similar flow can be defined using
+a Thrift Flume Source to receive events from a Thrift Sink or a Flume
+Thrift Rpc Client or Thrift clients written in any language generated from
+the Flume thrift protocol.When a Flume source receives an event, it
+stores it into one or more channels. The channel is a passive store that keeps
+the event until it's consumed by a Flume sink. The file channel is one example
+-- it is backed by the local filesystem. The sink removes the event
+from the channel and puts it into an external repository like HDFS (via Flume
+HDFS sink) or forwards it to the Flume source of the next Flume agent (next
+hop) in the flow. The source and sink within the given agent run asynchronously
+with the events staged in the channel.
+
+Complex flows
+~~~~~~~~~~~~~
+
+Flume allows a user to build multi-hop flows where events travel through
+multiple agents before reaching the final destination. It also allows fan-in
+and fan-out flows, contextual routing and backup routes (fail-over) for failed
+hops.
+
+Reliability
+~~~~~~~~~~~
+
+The events are staged in a channel on each agent. The events are then delivered
+to the next agent or terminal repository (like HDFS) in the flow. The events
+are removed from a channel only after they are stored in the channel of next
+agent or in the terminal repository. This is a how the single-hop message
+delivery semantics in Flume provide end-to-end reliability of the flow.
+
+Flume uses a transactional approach to guarantee the reliable delivery of the
+events. The sources and sinks encapsulate in a transaction the
+storage/retrieval, respectively, of the events placed in or provided by a
+transaction provided by the channel. This ensures that the set of events are
+reliably passed from point to point in the flow. In the case of a multi-hop
+flow, the sink from the previous hop and the source from the next hop both have
+their transactions running to ensure that the data is safely stored in the
+channel of the next hop.
+
+Recoverability
+~~~~~~~~~~~~~~
+
+The events are staged in the channel, which manages recovery from failure.
+Flume supports a durable file channel which is backed by the local file system.
+There's also a memory channel which simply stores the events in an in-memory
+queue, which is faster but any events still left in the memory channel when an
+agent process dies can't be recovered.
+
+Setup
+=====
+
+Setting up an agent
+-------------------
+
+Flume agent configuration is stored in a local configuration file. This is a
+text file that follows the Java properties file format.
+Configurations for one or more agents can be specified in the same
+configuration file. The configuration file includes properties of each source,
+sink and channel in an agent and how they are wired together to form data
+flows.
+
+Configuring individual components
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Each component (source, sink or channel) in the flow has a name, type, and set
+of properties that are specific to the type and instantiation. For example, an
+Avro source needs a hostname (or IP address) and a port number to receive data
+from. A memory channel can have max queue size ("capacity"), and an HDFS sink
+needs to know the file system URI, path to create files, frequency of file
+rotation ("hdfs.rollInterval") etc. All such attributes of a component needs to
+be set in the properties file of the hosting Flume agent.
+
+Wiring the pieces together
+~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The agent needs to know what individual components to load and how they are
+connected in order to constitute the flow. This is done by listing the names of
+each of the sources, sinks and channels in the agent, and then specifying the
+connecting channel for each sink and source. For example, an agent flows events
+from an Avro source called avroWeb to HDFS sink hdfs-cluster1 via a file
+channel called file-channel. The configuration file will contain names of these
+components and file-channel as a shared channel for both avroWeb source and
+hdfs-cluster1 sink.
+
+Starting an agent
+~~~~~~~~~~~~~~~~~
+
+An agent is started using a shell script called flume-ng which is located in
+the bin directory of the Flume distribution. You need to specify the agent
+name, the config directory, and the config file on the command line::
+
+ $ bin/flume-ng agent -n $agent_name -c conf -f conf/flume-conf.properties.template
+
+Now the agent will start running source and sinks configured in the given
+properties file.
+
+A simple example
+~~~~~~~~~~~~~~~~
+Here, we give an example configuration file, describing a single-node Flume deployment.
+This configuration lets a user generate events and subsequently logs them to the console.
+
+.. code-block:: properties
+
+ # example.conf: A single-node Flume configuration
+
+ # Name the components on this agent
+ a1.sources = r1
+ a1.sinks = k1
+ a1.channels = c1
+
+ # Describe/configure the source
+ a1.sources.r1.type = netcat
+ a1.sources.r1.bind = localhost
+ a1.sources.r1.port = 44444
+
+ # Describe the sink
+ a1.sinks.k1.type = logger
+
+ # Use a channel which buffers events in memory
+ a1.channels.c1.type = memory
+ a1.channels.c1.capacity = 1000
+ a1.channels.c1.transactionCapacity = 100
+
+ # Bind the source and sink to the channel
+ a1.sources.r1.channels = c1
+ a1.sinks.k1.channel = c1
+
+This configuration defines a single agent named a1. a1 has a source that listens for data on port 44444, a channel
+that buffers event data in memory, and a sink that logs event data to the console. The configuration file names the
+various components, then describes their types and configuration parameters. A given configuration file might define
+several named agents; when a given Flume process is launched a flag is passed telling it which named agent to manifest.
+
+Given this configuration file, we can start Flume as follows::
+
+ $ bin/flume-ng agent --conf conf --conf-file example.conf --name a1 -Dflume.root.logger=INFO,console
+
+Note that in a full deployment we would typically include one more option: ``--conf=<conf-dir>``.
+The ``<conf-dir>`` directory would include a shell script *flume-env.sh* and potentially a log4j properties file.
+In this example, we pass a Java option to force Flume to log to the console and we go without a custom environment script.
+
+From a separate terminal, we can then telnet port 44444 and send Flume an event:
+
+.. code-block:: properties
+
+ $ telnet localhost 44444
+ Trying 127.0.0.1...
+ Connected to localhost.localdomain (127.0.0.1).
+ Escape character is '^]'.
+ Hello world! <ENTER>
+ OK
+
+The original Flume terminal will output the event in a log message.
+
+.. code-block:: properties
+
+ 12/06/19 15:32:19 INFO source.NetcatSource: Source starting
+ 12/06/19 15:32:19 INFO source.NetcatSource: Created serverSocket:sun.nio.ch.ServerSocketChannelImpl[/127.0.0.1:44444]
+ 12/06/19 15:32:34 INFO sink.LoggerSink: Event: { headers:{} body: 48 65 6C 6C 6F 20 77 6F 72 6C 64 21 0D Hello world!. }
+
+Congratulations - you've successfully configured and deployed a Flume agent! Subsequent sections cover agent configuration in much more detail.
+
+Using environment variables in configuration files
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+Flume has the ability to substitute environment variables in the configuration. For example::
+
+ a1.sources = r1
+ a1.sources.r1.type = netcat
+ a1.sources.r1.bind = 0.0.0.0
+ a1.sources.r1.port = ${NC_PORT}
+ a1.sources.r1.channels = c1
+
+NB: it currently works for values only, not for keys. (Ie. only on the "right side" of the `=` mark of the config lines.)
+
+This can be enabled via Java system properties on agent invocation by setting `propertiesImplementation = org.apache.flume.node.EnvVarResolverProperties`.
+
+For example::
+ $ NC_PORT=44444 bin/flume-ng agent --conf conf --conf-file example.conf --name a1 -Dflume.root.logger=INFO,console -DpropertiesImplementation=org.apache.flume.node.EnvVarResolverProperties
+
+Note the above is just an example, environment variables can be configured in other ways, including being set in `conf/flume-env.sh`.
+
+Logging raw data
+~~~~~~~~~~~~~~~~
+
+
+Logging the raw stream of data flowing through the ingest pipeline is not desired behaviour in
+many production environments because this may result in leaking sensitive data or security related
+configurations, such as secret keys, to Flume log files.
+By default, Flume will not log such information. On the other hand, if the data pipeline is broken,
+Flume will attempt to provide clues for debugging the problem.
+
+One way to debug problems with event pipelines is to set up an additional `Memory Channel`_
+connected to a `Logger Sink`_, which will output all event data to the Flume logs.
+In some situations, however, this approach is insufficient.
+
+In order to enable logging of event- and configuration-related data, some Java system properties
+must be set in addition to log4j properties.
+
+To enable configuration-related logging, set the Java system property
+``-Dorg.apache.flume.log.printconfig=true``. This can either be passed on the command line or by
+setting this in the ``JAVA_OPTS`` variable in *flume-env.sh*.
+
+To enable data logging, set the Java system property ``-Dorg.apache.flume.log.rawdata=true``
+in the same way described above. For most components, the log4j logging level must also be set to
+DEBUG or TRACE to make event-specific logging appear in the Flume logs.
+
+Here is an example of enabling both configuration logging and raw data logging while also
+setting the Log4j loglevel to DEBUG for console output::
+
+ $ bin/flume-ng agent --conf conf --conf-file example.conf --name a1 -Dflume.root.logger=DEBUG,console -Dorg.apache.flume.log.printconfig=true -Dorg.apache.flume.log.rawdata=true
+
+
+Zookeeper based Configuration
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Flume supports Agent configurations via Zookeeper. *This is an experimental feature.* The configuration file needs to be uploaded
+in the Zookeeper, under a configurable prefix. The configuration file is stored in Zookeeper Node data.
+Following is how the Zookeeper Node tree would look like for agents a1 and a2
+
+.. code-block:: properties
+
+ - /flume
+ |- /a1 [Agent config file]
+ |- /a2 [Agent config file]
+
+Once the configuration file is uploaded, start the agent with following options
+
+ $ bin/flume-ng agent --conf conf -z zkhost:2181,zkhost1:2181 -p /flume --name a1 -Dflume.root.logger=INFO,console
+
+================== ================ =========================================================================
+Argument Name Default Description
+================== ================ =========================================================================
+**z** -- Zookeeper connection string. Comma separated list of hostname:port
+**p** /flume Base Path in Zookeeper to store Agent configurations
+================== ================ =========================================================================
+
+Installing third-party plugins
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Flume has a fully plugin-based architecture. While Flume ships with many
+out-of-the-box sources, channels, sinks, serializers, and the like, many
+implementations exist which ship separately from Flume.
+
+While it has always been possible to include custom Flume components by
+adding their jars to the FLUME_CLASSPATH variable in the flume-env.sh file,
+Flume now supports a special directory called ``plugins.d`` which automatically
+picks up plugins that are packaged in a specific format. This allows for easier
+management of plugin packaging issues as well as simpler debugging and
+troubleshooting of several classes of issues, especially library dependency
+conflicts.
+
+The plugins.d directory
+'''''''''''''''''''''''
+
+The ``plugins.d`` directory is located at ``$FLUME_HOME/plugins.d``. At startup
+time, the ``flume-ng`` start script looks in the ``plugins.d`` directory for
+plugins that conform to the below format and includes them in proper paths when
+starting up ``java``.
+
+Directory layout for plugins
+''''''''''''''''''''''''''''
+
+Each plugin (subdirectory) within ``plugins.d`` can have up to three
+sub-directories:
+
+#. lib - the plugin's jar(s)
+#. libext - the plugin's dependency jar(s)
+#. native - any required native libraries, such as ``.so`` files
+
+Example of two plugins within the plugins.d directory:
+
+.. code-block:: none
+
+ plugins.d/
+ plugins.d/custom-source-1/
+ plugins.d/custom-source-1/lib/my-source.jar
+ plugins.d/custom-source-1/libext/spring-core-2.5.6.jar
+ plugins.d/custom-source-2/
+ plugins.d/custom-source-2/lib/custom.jar
+ plugins.d/custom-source-2/native/gettext.so
+
+Data ingestion
+--------------
+
+Flume supports a number of mechanisms to ingest data from external sources.
+
+RPC
+~~~
+
+An Avro client included in the Flume distribution can send a given file to
+Flume Avro source using avro RPC mechanism::
+
+ $ bin/flume-ng avro-client -H localhost -p 41414 -F /usr/logs/log.10
+
+The above command will send the contents of /usr/logs/log.10 to to the Flume
+source listening on that ports.
+
+Executing commands
+~~~~~~~~~~~~~~~~~~
+
+There's an exec source that executes a given command and consumes the output. A
+single 'line' of output ie. text followed by carriage return ('\\r') or line
+feed ('\\n') or both together.
+
+Network streams
+~~~~~~~~~~~~~~~
+
+Flume supports the following mechanisms to read data from popular log stream
+types, such as:
+
+#. Avro
+#. Thrift
+#. Syslog
+#. Netcat
+
+Setting multi-agent flow
+------------------------
+
+.. figure:: images/UserGuide_image03.png
+ :align: center
+ :alt: Two agents communicating over Avro RPC
+
+In order to flow the data across multiple agents or hops, the sink of the
+previous agent and source of the current hop need to be avro type with the sink
+pointing to the hostname (or IP address) and port of the source.
+
+Consolidation
+-------------
+
+A very common scenario in log collection is a large number of log producing
+clients sending data to a few consumer agents that are attached to the storage
+subsystem. For example, logs collected from hundreds of web servers sent to a
+dozen of agents that write to HDFS cluster.
+
+.. figure:: images/UserGuide_image02.png
+ :align: center
+ :alt: A fan-in flow using Avro RPC to consolidate events in one place
+
+This can be achieved in Flume by configuring a number of first tier agents with
+an avro sink, all pointing to an avro source of single agent (Again you could
+use the thrift sources/sinks/clients in such a scenario). This source
+on the second tier agent consolidates the received events into a single
+channel which is consumed by a sink to its final destination.
+
+Multiplexing the flow
+---------------------
+
+Flume supports multiplexing the event flow to one or more destinations. This is
+achieved by defining a flow multiplexer that can replicate or selectively route
+an event to one or more channels.
+
+.. figure:: images/UserGuide_image01.png
+ :align: center
+ :alt: A fan-out flow using a (multiplexing) channel selector
+
+The above example shows a source from agent "foo" fanning out the flow to three
+different channels. This fan out can be replicating or multiplexing. In case of
+replicating flow, each event is sent to all three channels. For the
+multiplexing case, an event is delivered to a subset of available channels when
+an event's attribute matches a preconfigured value. For example, if an event
+attribute called "txnType" is set to "customer", then it should go to channel1
+and channel3, if it's "vendor" then it should go to channel2, otherwise
+channel3. The mapping can be set in the agent's configuration file.
+
+Configuration
+=============
+
+As mentioned in the earlier section, Flume agent configuration is read from a
+file that resembles a Java property file format with hierarchical property
+settings.
+
+Defining the flow
+-----------------
+
+To define the flow within a single agent, you need to link the sources and
+sinks via a channel. You need to list the sources, sinks and channels for the
+given agent, and then point the source and sink to a channel. A source instance
+can specify multiple channels, but a sink instance can only specify one channel.
+The format is as follows:
+
+.. code-block:: properties
+
+ # list the sources, sinks and channels for the agent
+ <Agent>.sources = <Source>
+ <Agent>.sinks = <Sink>
+ <Agent>.channels = <Channel1> <Channel2>
+
+ # set channel for source
+ <Agent>.sources.<Source>.channels = <Channel1> <Channel2> ...
+
+ # set channel for sink
+ <Agent>.sinks.<Sink>.channel = <Channel1>
+
+For example, an agent named agent_foo is reading data from an external avro client and sending
+it to HDFS via a memory channel. The config file weblog.config could look like:
+
+.. code-block:: properties
+
+ # list the sources, sinks and channels for the agent
+ agent_foo.sources = avro-appserver-src-1
+ agent_foo.sinks = hdfs-sink-1
+ agent_foo.channels = mem-channel-1
+
+ # set channel for source
+ agent_foo.sources.avro-appserver-src-1.channels = mem-channel-1
+
+ # set channel for sink
+ agent_foo.sinks.hdfs-sink-1.channel = mem-channel-1
+
+This will make the events flow from avro-AppSrv-source to hdfs-Cluster1-sink
+through the memory channel mem-channel-1. When the agent is started with the
+weblog.config as its config file, it will instantiate that flow.
+
+Configuring individual components
+---------------------------------
+
+After defining the flow, you need to set properties of each source, sink and
+channel. This is done in the same hierarchical namespace fashion where you set
+the component type and other values for the properties specific to each
+component:
+
+.. code-block:: properties
+
+ # properties for sources
+ <Agent>.sources.<Source>.<someProperty> = <someValue>
+
+ # properties for channels
+ <Agent>.channel.<Channel>.<someProperty> = <someValue>
+
+ # properties for sinks
+ <Agent>.sources.<Sink>.<someProperty> = <someValue>
+
+The property "type" needs to be set for each component for Flume to understand
+what kind of object it needs to be. Each source, sink and channel type has its
+own set of properties required for it to function as intended. All those need
+to be set as needed. In the previous example, we have a flow from
+avro-AppSrv-source to hdfs-Cluster1-sink through the memory channel
+mem-channel-1. Here's an example that shows configuration of each of those
+components:
+
+.. code-block:: properties
+
+ agent_foo.sources = avro-AppSrv-source
+ agent_foo.sinks = hdfs-Cluster1-sink
+ agent_foo.channels = mem-channel-1
+
+ # set channel for sources, sinks
+
+ # properties of avro-AppSrv-source
+ agent_foo.sources.avro-AppSrv-source.type = avro
+ agent_foo.sources.avro-AppSrv-source.bind = localhost
+ agent_foo.sources.avro-AppSrv-source.port = 10000
+
+ # properties of mem-channel-1
+ agent_foo.channels.mem-channel-1.type = memory
+ agent_foo.channels.mem-channel-1.capacity = 1000
+ agent_foo.channels.mem-channel-1.transactionCapacity = 100
+
+ # properties of hdfs-Cluster1-sink
+ agent_foo.sinks.hdfs-Cluster1-sink.type = hdfs
+ agent_foo.sinks.hdfs-Cluster1-sink.hdfs.path = hdfs://namenode/flume/webdata
+
+ #...
+
+Adding multiple flows in an agent
+---------------------------------
+
+A single Flume agent can contain several independent flows. You can list
+multiple sources, sinks and channels in a config. These components can be
+linked to form multiple flows:
+
+.. code-block:: properties
+
+ # list the sources, sinks and channels for the agent
+ <Agent>.sources = <Source1> <Source2>
+ <Agent>.sinks = <Sink1> <Sink2>
+ <Agent>.channels = <Channel1> <Channel2>
+
+Then you can link the sources and sinks to their corresponding channels (for
+sources) of channel (for sinks) to setup two different flows. For example, if
+you need to setup two flows in an agent, one going from an external avro client
+to external HDFS and another from output of a tail to avro sink, then here's a
+config to do that:
+
+.. code-block:: properties
+
+ # list the sources, sinks and channels in the agent
+ agent_foo.sources = avro-AppSrv-source1 exec-tail-source2
+ agent_foo.sinks = hdfs-Cluster1-sink1 avro-forward-sink2
+ agent_foo.channels = mem-channel-1 file-channel-2
+
+ # flow #1 configuration
+ agent_foo.sources.avro-AppSrv-source1.channels = mem-channel-1
+ agent_foo.sinks.hdfs-Cluster1-sink1.channel = mem-channel-1
+
+ # flow #2 configuration
+ agent_foo.sources.exec-tail-source2.channels = file-channel-2
+ agent_foo.sinks.avro-forward-sink2.channel = file-channel-2
+
+Configuring a multi agent flow
+------------------------------
+
+To setup a multi-tier flow, you need to have an avro/thrift sink of first hop
+pointing to avro/thrift source of the next hop. This will result in the first
+Flume agent forwarding events to the next Flume agent. For example, if you are
+periodically sending files (1 file per event) using avro client to a local
+Flume agent, then this local agent can forward it to another agent that has the
+mounted for storage.
+
+Weblog agent config:
+
+.. code-block:: properties
+
+ # list sources, sinks and channels in the agent
+ agent_foo.sources = avro-AppSrv-source
+ agent_foo.sinks = avro-forward-sink
+ agent_foo.channels = file-channel
+
+ # define the flow
+ agent_foo.sources.avro-AppSrv-source.channels = file-channel
+ agent_foo.sinks.avro-forward-sink.channel = file-channel
+
+ # avro sink properties
+ agent_foo.sinks.avro-forward-sink.type = avro
+ agent_foo.sinks.avro-forward-sink.hostname = 10.1.1.100
+ agent_foo.sinks.avro-forward-sink.port = 10000
+
+ # configure other pieces
+ #...
+
+
+HDFS agent config:
+
+.. code-block:: properties
+
+ # list sources, sinks and channels in the agent
+ agent_foo.sources = avro-collection-source
+ agent_foo.sinks = hdfs-sink
+ agent_foo.channels = mem-channel
+
+ # define the flow
+ agent_foo.sources.avro-collection-source.channels = mem-channel
+ agent_foo.sinks.hdfs-sink.channel = mem-channel
+
+ # avro source properties
+ agent_foo.sources.avro-collection-source.type = avro
+ agent_foo.sources.avro-collection-source.bind = 10.1.1.100
+ agent_foo.sources.avro-collection-source.port = 10000
+
+ # configure other pieces
+ #...
+
+Here we link the avro-forward-sink from the weblog agent to the
+avro-collection-source of the hdfs agent. This will result in the events coming
+from the external appserver source eventually getting stored in HDFS.
+
+
+
+Fan out flow
+------------
+
+As discussed in previous section, Flume supports fanning out the flow from one
+source to multiple channels. There are two modes of fan out, replicating and
+multiplexing. In the replicating flow, the event is sent to all the configured
+channels. In case of multiplexing, the event is sent to only a subset of
+qualifying channels. To fan out the flow, one needs to specify a list of
+channels for a source and the policy for the fanning it out. This is done by
+adding a channel "selector" that can be replicating or multiplexing. Then
+further specify the selection rules if it's a multiplexer. If you don't specify
+a selector, then by default it's replicating:
+
+.. code-block:: properties
+
+ # List the sources, sinks and channels for the agent
+ <Agent>.sources = <Source1>
+ <Agent>.sinks = <Sink1> <Sink2>
+ <Agent>.channels = <Channel1> <Channel2>
+
+ # set list of channels for source (separated by space)
+ <Agent>.sources.<Source1>.channels = <Channel1> <Channel2>
+
+ # set channel for sinks
+ <Agent>.sinks.<Sink1>.channel = <Channel1>
+ <Agent>.sinks.<Sink2>.channel = <Channel2>
+
+ <Agent>.sources.<Source1>.selector.type = replicating
+
+The multiplexing select has a further set of properties to bifurcate the flow.
+This requires specifying a mapping of an event attribute to a set for channel.
+The selector checks for each configured attribute in the event header. If it
+matches the specified value, then that event is sent to all the channels mapped
+to that value. If there's no match, then the event is sent to set of channels
+configured as default:
+
+.. code-block:: properties
+
+ # Mapping for multiplexing selector
+ <Agent>.sources.<Source1>.selector.type = multiplexing
+ <Agent>.sources.<Source1>.selector.header = <someHeader>
+ <Agent>.sources.<Source1>.selector.mapping.<Value1> = <Channel1>
+ <Agent>.sources.<Source1>.selector.mapping.<Value2> = <Channel1> <Channel2>
+ <Agent>.sources.<Source1>.selector.mapping.<Value3> = <Channel2>
+ #...
+
+ <Agent>.sources.<Source1>.selector.default = <Channel2>
+
+The mapping allows overlapping the channels for each value.
+
+The following example has a single flow that multiplexed to two paths. The
+agent named agent_foo has a single avro source and two channels linked to two sinks:
+
+.. code-block:: properties
+
+ # list the sources, sinks and channels in the agent
+ agent_foo.sources = avro-AppSrv-source1
+ agent_foo.sinks = hdfs-Cluster1-sink1 avro-forward-sink2
+ agent_foo.channels = mem-channel-1 file-channel-2
+
+ # set channels for source
+ agent_foo.sources.avro-AppSrv-source1.channels = mem-channel-1 file-channel-2
+
+ # set channel for sinks
+ agent_foo.sinks.hdfs-Cluster1-sink1.channel = mem-channel-1
+ agent_foo.sinks.avro-forward-sink2.channel = file-channel-2
+
+ # channel selector configuration
+ agent_foo.sources.avro-AppSrv-source1.selector.type = multiplexing
+ agent_foo.sources.avro-AppSrv-source1.selector.header = State
+ agent_foo.sources.avro-AppSrv-source1.selector.mapping.CA = mem-channel-1
+ agent_foo.sources.avro-AppSrv-source1.selector.mapping.AZ = file-channel-2
+ agent_foo.sources.avro-AppSrv-source1.selector.mapping.NY = mem-channel-1 file-channel-2
+ agent_foo.sources.avro-AppSrv-source1.selector.default = mem-channel-1
+
+The selector checks for a header called "State". If the value is "CA" then its
+sent to mem-channel-1, if its "AZ" then it goes to file-channel-2 or if its
+"NY" then both. If the "State" header is not set or doesn't match any of the
+three, then it goes to mem-channel-1 which is designated as 'default'.
+
+The selector also supports optional channels. To specify optional channels for
+a header, the config parameter 'optional' is used in the following way:
+
+.. code-block:: properties
+
+ # channel selector configuration
+ agent_foo.sources.avro-AppSrv-source1.selector.type = multiplexing
+ agent_foo.sources.avro-AppSrv-source1.selector.header = State
+ agent_foo.sources.avro-AppSrv-source1.selector.mapping.CA = mem-channel-1
+ agent_foo.sources.avro-AppSrv-source1.selector.mapping.AZ = file-channel-2
+ agent_foo.sources.avro-AppSrv-source1.selector.mapping.NY = mem-channel-1 file-channel-2
+ agent_foo.sources.avro-AppSrv-source1.selector.optional.CA = mem-channel-1 file-channel-2
+ agent_foo.sources.avro-AppSrv-source1.selector.mapping.AZ = file-channel-2
+ agent_foo.sources.avro-AppSrv-source1.selector.default = mem-channel-1
+
+The selector will attempt to write to the required channels first and will fail
+the transaction if even one of these channels fails to consume the events. The
+transaction is reattempted on **all** of the channels. Once all required
+channels have consumed the events, then the selector will attempt to write to
+the optional channels. A failure by any of the optional channels to consume the
+event is simply ignored and not retried.
+
+If there is an overlap between the optional channels and required channels for a
+specific header, the channel is considered to be required, and a failure in the
+channel will cause the entire set of required channels to be retried. For
+instance, in the above example, for the header "CA" mem-channel-1 is considered
+to be a required channel even though it is marked both as required and optional,
+and a failure to write to this channel will cause that
+event to be retried on **all** channels configured for the selector.
+
+Note that if a header does not have any required channels, then the event will
+be written to the default channels and will be attempted to be written to the
+optional channels for that header. Specifying optional channels will still cause
+the event to be written to the default channels, if no required channels are
+specified. If no channels are designated as default and there are no required,
+the selector will attempt to write the events to the optional channels. Any
+failures are simply ignored in that case.
+
+
+SSL/TLS support
+---------------
+
+Several Flume components support the SSL/TLS protocols in order to communicate with other systems
+securely.
+
+=========================== ======================
+Component SSL server or client
+=========================== ======================
+Avro Source server
+Avro Sink client
+Thrift Source server
+Thrift Sink client
+Kafka Source client
+Kafka Channel client
+Kafka Sink client
+HTTP Source server
+JMS Source client
+Syslog TCP Source server
+Multiport Syslog TCP Source server
+=========================== ======================
+
+The SSL compatible components have several configuration parameters to set up SSL, like
+enable SSL flag, keystore / truststore parameters (location, password, type) and additional
+SSL parameters (eg. disabled protocols).
+
+Enabling SSL for a component is always specified at component level in the agent configuration file.
+So some components may be configured to use SSL while others not (even with the same component type).
+
+The keystore / truststore setup can be specified at component level or globally.
+
+In case of the component level setup, the keystore / truststore is configured in the agent
+configuration file through component specific parameters. The advantage of this method is that the
+components can use different keystores (if this would be needed). The disadvantage is that the
+keystore parameters must be copied for each component in the agent configuration file.
+The component level setup is optional, but if it is defined, it has higher precedence than
+the global parameters.
+
+With the global setup, it is enough to define the keystore / truststore parameters once
+and use the same settings for all components, which means less and more centralized configuration.
+
+The global setup can be configured either through system properties or through environment variables.
+
+================================== =============================== ============================================================================================
+System property Environment variable Description
+================================== =============================== ============================================================================================
+javax.net.ssl.keyStore FLUME_SSL_KEYSTORE_PATH Keystore location
+javax.net.ssl.keyStorePassword FLUME_SSL_KEYSTORE_PASSWORD Keystore password
+javax.net.ssl.keyStoreType FLUME_SSL_KEYSTORE_TYPE Keystore type (by default JKS)
+javax.net.ssl.trustStore FLUME_SSL_TRUSTSTORE_PATH Truststore location
+javax.net.ssl.trustStorePassword FLUME_SSL_TRUSTSTORE_PASSWORD Truststore password
+javax.net.ssl.trustStoreType FLUME_SSL_TRUSTSTORE_TYPE Truststore type (by default JKS)
+flume.ssl.include.protocols FLUME_SSL_INCLUDE_PROTOCOLS Protocols to include when calculating enabled protocols. A comma (,) separated list.
+ Excluded protocols will be excluded from this list if provided.
+flume.ssl.exclude.protocols FLUME_SSL_EXCLUDE_PROTOCOLS Protocols to exclude when calculating enabled protocols. A comma (,) separated list.
+flume.ssl.include.cipherSuites FLUME_SSL_INCLUDE_CIPHERSUITES Cipher suites to include when calculating enabled cipher suites. A comma (,) separated list.
+ Excluded cipher suites will be excluded from this list if provided.
+flume.ssl.exclude.cipherSuites FLUME_SSL_EXCLUDE_CIPHERSUITES Cipher suites to exclude when calculating enabled cipher suites. A comma (,) separated list.
+================================== =============================== ============================================================================================
+
+The SSL system properties can either be passed on the command line or by setting the ``JAVA_OPTS``
+environment variable in *conf/flume-env.sh*. (Although, using the command line is inadvisable because
+the commands including the passwords will be saved to the command history.)
+
+.. code-block:: properties
+
+ export JAVA_OPTS="$JAVA_OPTS -Djavax.net.ssl.keyStore=/path/to/keystore.jks"
+ export JAVA_OPTS="$JAVA_OPTS -Djavax.net.ssl.keyStorePassword=password"
+
+Flume uses the system properties defined in JSSE (Java Secure Socket Extension), so this is
+a standard way for setting up SSL. On the other hand, specifying passwords in system properties
+means that the passwords can be seen in the process list. For cases where it is not acceptable,
+it is also be possible to define the parameters in environment variables. Flume initializes
+the JSSE system properties from the corresponding environment variables internally in this case.
+
+The SSL environment variables can either be set in the shell environment before
+starting Flume or in *conf/flume-env.sh*. (Although, using the command line is inadvisable because
+the commands including the passwords will be saved to the command history.)
+
+.. code-block:: properties
+
+ export FLUME_SSL_KEYSTORE_PATH=/path/to/keystore.jks
+ export FLUME_SSL_KEYSTORE_PASSWORD=password
+
+**Please note:**
+
+* SSL must be enabled at component level. Specifying the global SSL parameters alone will not
+ have any effect.
+* If the global SSL parameters are specified at multiple levels, the priority is the
+ following (from higher to lower):
+
+ * component parameters in agent config
+ * system properties
+ * environment variables
+
+* If SSL is enabled for a component, but the SSL parameters are not specified in any of the ways
+ described above, then
+
+ * in case of keystores: configuration error
+ * in case of truststores: the default truststore will be used (``jssecacerts`` / ``cacerts`` in Oracle JDK)
+* The trustore password is optional in all cases. If not specified, then no integrity check will be
+ performed on the truststore when it is opened by the JDK.
+
+
+Source and sink batch sizes and channel transaction capacities
+--------------------------------------------------------------
+
+Sources and sinks can have a batch size parameter that determines the maximum number of events they
+process in one batch. This happens within a channel transaction that has an upper limit called
+transaction capacity. Batch size must be smaller than the channel's transaction capacity.
+There is an explicit check to prevent incompatible settings. This check happens
+whenever the configuration is read.
+
+Flume Sources
+-------------
+
+Avro Source
+~~~~~~~~~~~
+
+Listens on Avro port and receives events from external Avro client streams.
+When paired with the built-in Avro Sink on another (previous hop) Flume agent,
+it can create tiered collection topologies.
+Required properties are in **bold**.
+
+===================== ================ =======================================================
+Property Name Default Description
+===================== ================ =======================================================
+**channels** --
+**type** -- The component type name, needs to be ``avro``
+**bind** -- hostname or IP address to listen on
+**port** -- Port # to bind to
+threads -- Maximum number of worker threads to spawn
+selector.type
+selector.*
+interceptors -- Space-separated list of interceptors
+interceptors.*
+compression-type none This can be "none" or "deflate". The compression-type must match the compression-type of matching AvroSource
+ssl false Set this to true to enable SSL encryption. If SSL is enabled,
+ you must also specify a "keystore" and a "keystore-password",
+ either through component level parameters (see below)
+ or as global SSL parameters (see `SSL/TLS support`_ section).
+keystore -- This is the path to a Java keystore file.
+ If not specified here, then the global keystore will be used
+ (if defined, otherwise configuration error).
+keystore-password -- The password for the Java keystore.
+ If not specified here, then the global keystore password will be used
+ (if defined, otherwise configuration error).
+keystore-type JKS The type of the Java keystore. This can be "JKS" or "PKCS12".
+ If not specified here, then the global keystore type will be used
+ (if defined, otherwise the default is JKS).
+exclude-protocols SSLv3 Space-separated list of SSL/TLS protocols to exclude.
+ SSLv3 will always be excluded in addition to the protocols specified.
+include-protocols -- Space-separated list of SSL/TLS protocols to include.
+ The enabled protocols will be the included protocols without the excluded protocols.
+ If included-protocols is empty, it includes every supported protocols.
+exclude-cipher-suites -- Space-separated list of cipher suites to exclude.
+include-cipher-suites -- Space-separated list of cipher suites to include.
+ The enabled cipher suites will be the included cipher suites without the excluded cipher suites.
+ If included-cipher-suites is empty, it includes every supported cipher suites.
+ipFilter false Set this to true to enable ipFiltering for netty
+ipFilterRules -- Define N netty ipFilter pattern rules with this config.
+===================== ================ =======================================================
+
+Example for agent named a1:
+
+.. code-block:: properties
+
+ a1.sources = r1
+ a1.channels = c1
+ a1.sources.r1.type = avro
+ a1.sources.r1.channels = c1
+ a1.sources.r1.bind = 0.0.0.0
+ a1.sources.r1.port = 4141
+
+Example of ipFilterRules
+
+ipFilterRules defines N netty ipFilters separated by a comma a pattern rule must be in this format.
+
+<'allow' or deny>:<'ip' or 'name' for computer name>:<pattern>
+or
+allow/deny:ip/name:pattern
+
+example: ipFilterRules=allow:ip:127.*,allow:name:localhost,deny:ip:*
+
+Note that the first rule to match will apply as the example below shows from a client on the localhost
+
+This will Allow the client on localhost be deny clients from any other ip "allow:name:localhost,deny:ip:*"
+This will deny the client on localhost be allow clients from any other ip "deny:name:localhost,allow:ip:*"
+
+Thrift Source
+~~~~~~~~~~~~~
+
+Listens on Thrift port and receives events from external Thrift client streams.
+When paired with the built-in ThriftSink on another (previous hop) Flume agent,
+it can create tiered collection topologies.
+Thrift source can be configured to start in secure mode by enabling kerberos authentication.
+agent-principal and agent-keytab are the properties used by the
+Thrift source to authenticate to the kerberos KDC.
+Required properties are in **bold**.
+
+===================== =========== ==========================================================================================================================
+Property Name Default Description
+===================== =========== ==========================================================================================================================
+**channels** --
+**type** -- The component type name, needs to be ``thrift``
+**bind** -- hostname or IP address to listen on
+**port** -- Port # to bind to
+threads -- Maximum number of worker threads to spawn
+selector.type
+selector.*
+interceptors -- Space separated list of interceptors
+interceptors.*
+ssl false Set this to true to enable SSL encryption. If SSL is enabled,
+ you must also specify a "keystore" and a "keystore-password",
+ either through component level parameters (see below)
+ or as global SSL parameters (see `SSL/TLS support`_ section)
+keystore -- This is the path to a Java keystore file.
+ If not specified here, then the global keystore will be used
+ (if defined, otherwise configuration error).
+keystore-password -- The password for the Java keystore.
+ If not specified here, then the global keystore password will be used
+ (if defined, otherwise configuration error).
+keystore-type JKS The type of the Java keystore. This can be "JKS" or "PKCS12".
+ If not specified here, then the global keystore type will be used
+ (if defined, otherwise the default is JKS).
+exclude-protocols SSLv3 Space-separated list of SSL/TLS protocols to exclude.
+ SSLv3 will always be excluded in addition to the protocols specified.
+include-protocols -- Space-separated list of SSL/TLS protocols to include.
+ The enabled protocols will be the included protocols without the excluded protocols.
+ If included-protocols is empty, it includes every supported protocols.
+exclude-cipher-suites -- Space-separated list of cipher suites to exclude.
+include-cipher-suites -- Space-separated list of cipher suites to include.
+ The enabled cipher suites will be the included cipher suites without the excluded cipher suites.
+
+kerberos false Set to true to enable kerberos authentication. In kerberos mode, agent-principal and agent-keytab are required for successful authentication. The Thrift source in secure mode, will accept connections only from Thrift clients that have kerberos enabled and are successfully authenticated to the kerberos KDC.
+agent-principal -- The kerberos principal used by the Thrift Source to authenticate to the kerberos KDC.
+agent-keytab —- The keytab location used by the Thrift Source in combination with the agent-principal to authenticate to the kerberos KDC.
+===================== =========== ==========================================================================================================================
+
+Example for agent named a1:
+
+.. code-block:: properties
+
+ a1.sources = r1
+ a1.channels = c1
+ a1.sources.r1.type = thrift
+ a1.sources.r1.channels = c1
+ a1.sources.r1.bind = 0.0.0.0
+ a1.sources.r1.port = 4141
+
+Exec Source
+~~~~~~~~~~~
+
+Exec source runs a given Unix command on start-up and expects that process to
+continuously produce data on standard out (stderr is simply discarded, unless
+property logStdErr is set to true). If the process exits for any reason, the source also exits and
+will produce no further data. This means configurations such as ``cat [named pipe]``
+or ``tail -F [file]`` are going to produce the desired results where as ``date``
+will probably not - the former two commands produce streams of data where as the
+latter produces a single event and exits.
+
+Required properties are in **bold**.
+
+=============== =========== ==============================================================
+Property Name Default Description
+=============== =========== ==============================================================
+**channels** --
+**type** -- The component type name, needs to be ``exec``
+**command** -- The command to execute
+shell -- A shell invocation used to run the command. e.g. /bin/sh -c. Required only for commands relying on shell features like wildcards, back ticks, pipes etc.
+restartThrottle 10000 Amount of time (in millis) to wait before attempting a restart
+restart false Whether the executed cmd should be restarted if it dies
+logStdErr false Whether the command's stderr should be logged
+batchSize 20 The max number of lines to read and send to the channel at a time
+batchTimeout 3000 Amount of time (in milliseconds) to wait, if the buffer size was not reached, before data is pushed downstream
+selector.type replicating replicating or multiplexing
+selector.* Depends on the selector.type value
+interceptors -- Space-separated list of interceptors
+interceptors.*
+=============== =========== ==============================================================
+
+
+.. warning:: The problem with ExecSource and other asynchronous sources is that the
+ source can not guarantee that if there is a failure to put the event
+ into the Channel the client knows about it. In such cases, the data will
+ be lost. As a for instance, one of the most commonly requested features
+ is the ``tail -F [file]``-like use case where an application writes
+ to a log file on disk and Flume tails the file, sending each line as an
+ event. While this is possible, there's an obvious problem; what happens
+ if the channel fills up and Flume can't send an event? Flume has no way
+ of indicating to the application writing the log file that it needs to
+ retain the log or that the event hasn't been sent, for some reason. If
+ this doesn't make sense, you need only know this: Your application can
+ never guarantee data has been received when using a unidirectional
+ asynchronous interface such as ExecSource! As an extension of this
+ warning - and to be completely clear - there is absolutely zero guarantee
+ of event delivery when using this source. For stronger reliability
+ guarantees, consider the Spooling Directory Source, Taildir Source or direct integration
+ with Flume via the SDK.
+
+Example for agent named a1:
+
+.. code-block:: properties
+
+ a1.sources = r1
+ a1.channels = c1
+ a1.sources.r1.type = exec
+ a1.sources.r1.command = tail -F /var/log/secure
+ a1.sources.r1.channels = c1
+
+The 'shell' config is used to invoke the 'command' through a command shell (such as Bash
+or Powershell). The 'command' is passed as an argument to 'shell' for execution. This
+allows the 'command' to use features from the shell such as wildcards, back ticks, pipes,
+loops, conditionals etc. In the absence of the 'shell' config, the 'command' will be
+invoked directly. Common values for 'shell' : '/bin/sh -c', '/bin/ksh -c',
+'cmd /c', 'powershell -Command', etc.
+
+.. code-block:: properties
+
+ a1.sources.tailsource-1.type = exec
+ a1.sources.tailsource-1.shell = /bin/bash -c
+ a1.sources.tailsource-1.command = for i in /path/*.txt; do cat $i; done
+
+JMS Source
+~~~~~~~~~~
+
+JMS Source reads messages from a JMS destination such as a queue or topic. Being a JMS
+application it should work with any JMS provider but has only been tested with ActiveMQ.
+The JMS source provides configurable batch size, message selector, user/pass, and message
+to flume event converter. Note that the vendor provided JMS jars should be included in the
+Flume classpath using plugins.d directory (preferred), --classpath on command line, or
+via FLUME_CLASSPATH variable in flume-env.sh.
+
+Required properties are in **bold**.
+
+========================= =========== ==============================================================
+Property Name Default Description
+========================= =========== ==============================================================
+**channels** --
+**type** -- The component type name, needs to be ``jms``
+**initialContextFactory** -- Inital Context Factory, e.g: org.apache.activemq.jndi.ActiveMQInitialContextFactory
+**connectionFactory** -- The JNDI name the connection factory should appear as
+**providerURL** -- The JMS provider URL
+**destinationName** -- Destination name
+**destinationType** -- Destination type (queue or topic)
+messageSelector -- Message selector to use when creating the consumer
+userName -- Username for the destination/provider
+passwordFile -- File containing the password for the destination/provider
+batchSize 100 Number of messages to consume in one batch
+converter.type DEFAULT Class to use to convert messages to flume events. See below.
+converter.* -- Converter properties.
+converter.charset UTF-8 Default converter only. Charset to use when converting JMS TextMessages to byte arrays.
+createDurableSubscription false Whether to create durable subscription. Durable subscription can only be used with
+ destinationType topic. If true, "clientId" and "durableSubscriptionName"
+ have to be specified.
+clientId -- JMS client identifier set on Connection right after it is created.
+ Required for durable subscriptions.
+durableSubscriptionName -- Name used to identify the durable subscription. Required for durable subscriptions.
+========================= =========== ==============================================================
+
+
+JMS message converter
+'''''''''''''''''''''
+The JMS source allows pluggable converters, though it's likely the default converter will work
+for most purposes. The default converter is able to convert Bytes, Text, and Object messages
+to FlumeEvents. In all cases, the properties in the message are added as headers to the
+FlumeEvent.
+
+BytesMessage:
+ Bytes of message are copied to body of the FlumeEvent. Cannot convert more than 2GB
+ of data per message.
+
+TextMessage:
+ Text of message is converted to a byte array and copied to the body of the
+ FlumeEvent. The default converter uses UTF-8 by default but this is configurable.
+
+ObjectMessage:
+ Object is written out to a ByteArrayOutputStream wrapped in an ObjectOutputStream and
+ the resulting array is copied to the body of the FlumeEvent.
+
+
+Example for agent named a1:
+
+.. code-block:: properties
+
+ a1.sources = r1
+ a1.channels = c1
+ a1.sources.r1.type = jms
+ a1.sources.r1.channels = c1
+ a1.sources.r1.initialContextFactory = org.apache.activemq.jndi.ActiveMQInitialContextFactory
+ a1.sources.r1.connectionFactory = GenericConnectionFactory
+ a1.sources.r1.providerURL = tcp://mqserver:61616
+ a1.sources.r1.destinationName = BUSINESS_DATA
+ a1.sources.r1.destinationType = QUEUE
+
+
+SSL and JMS Source
+''''''''''''''''''
+
+JMS client implementations typically support to configure SSL/TLS via some Java system properties defined by JSSE
+(Java Secure Socket Extension). Specifying these system properties for Flume's JVM, JMS Source (or more precisely the
+JMS client implementation used by the JMS Source) can connect to the JMS server through SSL (of course only when the JMS
+server has also been set up to use SSL).
+It should work with any JMS provider and has been tested with ActiveMQ, IBM MQ and Oracle WebLogic.
+
+The following sections describe the SSL configuration steps needed on the Flume side only. You can find more detailed
+descriptions about the server side setup of the different JMS providers and also full working configuration examples on
+Flume Wiki.
+
+**SSL transport / server authentication:**
+
+If the JMS server uses self-signed certificate or its certificate is signed by a non-trusted CA (eg. the company's own
+CA), then a truststore (containing the right certificate) needs to be set up and passed to Flume. It can be done via
+the global SSL parameters. For more details about the global SSL setup, see the `SSL/TLS support`_ section.
+
+Some JMS providers require SSL specific JNDI Initial Context Factory and/or Provider URL settings when using SSL (eg.
+ActiveMQ uses ssl:// URL prefix instead of tcp://).
+In this case the source properties (``initialContextFactory`` and/or ``providerURL``) have to be adjusted in the agent
+config file.
+
+**Client certificate authentication (two-way SSL):**
+
+JMS Source can authenticate to the JMS server through client certificate authentication instead of the usual
+user/password login (when SSL is used and the JMS server is configured to accept this kind of authentication).
+
+The keystore containing Flume's key used for the authentication needs to be configured via the global SSL parameters
+again. For more details about the global SSL setup, see the `SSL/TLS support`_ section.
+
+The keystore should contain only one key (if multiple keys are present, then the first one will be used).
+The key password must be the same as the keystore password.
+
+In case of client certificate authentication, it is not needed to specify the ``userName`` / ``passwordFile`` properties
+for the JMS Source in the Flume agent config file.
+
+**Please note:**
+
+There are no component level configuration parameters for JMS Source unlike in case of other components.
+No enable SSL flag either.
+SSL setup is controlled by JNDI/Provider URL settings (ultimately the JMS server settings) and by the presence / absence
+of the truststore / keystore.
+
+
+Spooling Directory Source
+~~~~~~~~~~~~~~~~~~~~~~~~~
+This source lets you ingest data by placing files to be ingested into a
+"spooling" directory on disk.
+This source will watch the specified directory for new files, and will parse
+events out of new files as they appear.
+The event parsing logic is pluggable.
+After a given file has been fully read
+into the channel, completion by default is indicated by renaming the file or it can be deleted or the trackerDir is used
+to keep track of processed files.
+
+Unlike the Exec source, this source is reliable and will not miss data, even if
+Flume is restarted or killed. In exchange for this reliability, only immutable,
+uniquely-named files must be dropped into the spooling directory. Flume tries
+to detect these problem conditions and will fail loudly if they are violated:
+
+#. If a file is written to after being placed into the spooling directory,
+ Flume will print an error to its log file and stop processing.
+#. If a file name is reused at a later time, Flume will print an error to its
+ log file and stop processing.
+
+To avoid the above issues, it may be useful to add a unique identifier
+(such as a timestamp) to log file names when they are moved into the spooling
+directory.
+
+Despite the reliability guarantees of this source, there are still
+cases in which events may be duplicated if certain downstream failures occur.
+This is consistent with the guarantees offered by other Flume components.
+
+======================== ============== ==========================================================
+Property Name Default Description
+======================== ============== ==========================================================
+**channels** --
+**type** -- The component type name, needs to be ``spooldir``.
+**spoolDir** -- The directory from which to read files from.
+fileSuffix .COMPLETED Suffix to append to completely ingested files
+deletePolicy never When to delete completed files: ``never`` or ``immediate``
+fileHeader false Whether to add a header storing the absolute path filename.
+fileHeaderKey file Header key to use when appending absolute path filename to event header.
+basenameHeader false Whether to add a header storing the basename of the file.
+basenameHeaderKey basename Header Key to use when appending basename of file to event header.
+includePattern ^.*$ Regular expression specifying which files to include.
+ It can used together with ``ignorePattern``.
+ If a file matches both ``ignorePattern`` and ``includePattern`` regex,
+ the file is ignored.
+ignorePattern ^$ Regular expression specifying which files to ignore (skip).
+ It can used together with ``includePattern``.
+ If a file matches both ``ignorePattern`` and ``includePattern`` regex,
+ the file is ignored.
+trackerDir .flumespool Directory to store metadata related to processing of files.
+ If this path is not an absolute path, then it is interpreted as relative to the spoolDir.
+trackingPolicy rename The tracking policy defines how file processing is tracked. It can be "rename" or
+ "tracker_dir". This parameter is only effective if the deletePolicy is "never".
+ "rename" - After processing files they get renamed according to the fileSuffix parameter.
+ "tracker_dir" - Files are not renamed but a new empty file is created in the trackerDir.
+ The new tracker file name is derived from the ingested one plus the fileSuffix.
+consumeOrder oldest In which order files in the spooling directory will be consumed ``oldest``,
+ ``youngest`` and ``random``. In case of ``oldest`` and ``youngest``, the last modified
+ time of the files will be used to compare the files. In case of a tie, the file
+ with smallest lexicographical order will be consumed first. In case of ``random`` any
+ file will be picked randomly. When using ``oldest`` and ``youngest`` the whole
+ directory will be scanned to pick the oldest/youngest file, which might be slow if there
+ are a large number of files, while using ``random`` may cause old files to be consumed
+ very late if new files keep coming in the spooling directory.
+pollDelay 500 Delay (in milliseconds) used when polling for new files.
+recursiveDirectorySearch false Whether to monitor sub directories for new files to read.
+maxBackoff 4000 The maximum time (in millis) to wait between consecutive attempts to
+ write to the channel(s) if the channel is full. The source will start at
+ a low backoff and increase it exponentially each time the channel throws a
+ ChannelException, upto the value specified by this parameter.
+batchSize 100 Granularity at which to batch transfer to the channel
+inputCharset UTF-8 Character set used by deserializers that treat the input file as text.
+decodeErrorPolicy ``FAIL`` What to do when we see a non-decodable character in the input file.
+ ``FAIL``: Throw an exception and fail to parse the file.
+ ``REPLACE``: Replace the unparseable character with the "replacement character" char,
+ typically Unicode U+FFFD.
+ ``IGNORE``: Drop the unparseable character sequence.
+deserializer ``LINE`` Specify the deserializer used to parse the file into events.
+ Defaults to parsing each line as an event. The class specified must implement
+ ``EventDeserializer.Builder``.
+deserializer.* Varies per event deserializer.
+bufferMaxLines -- (Obselete) This option is now ignored.
+bufferMaxLineLength 5000 (Deprecated) Maximum length of a line in the commit buffer. Use deserializer.maxLineLength instead.
+selector.type replicating replicating or multiplexing
+selector.* Depends on the selector.type value
+interceptors -- Space-separated list of interceptors
+interceptors.*
+======================== ============== ==========================================================
+
+Example for an agent named agent-1:
+
+.. code-block:: properties
+
+ a1.channels = ch-1
+ a1.sources = src-1
+
+ a1.sources.src-1.type = spooldir
+ a1.sources.src-1.channels = ch-1
+ a1.sources.src-1.spoolDir = /var/log/apache/flumeSpool
+ a1.sources.src-1.fileHeader = true
+
+Event Deserializers
+'''''''''''''''''''
+
+The following event deserializers ship with Flume.
+
+LINE
+^^^^
+
+This deserializer generates one event per line of text input.
+
+============================== ============== ==========================================================
+Property Name Default Description
+============================== ============== ==========================================================
+deserializer.maxLineLength 2048 Maximum number of characters to include in a single event.
+ If a line exceeds this length, it is truncated, and the
+ remaining characters on the line will appear in a
+ subsequent event.
+deserializer.outputCharset UTF-8 Charset to use for encoding events put into the channel.
+============================== ============== ==========================================================
+
+AVRO
+^^^^
+
+This deserializer is able to read an Avro container file, and it generates
+one event per Avro record in the file.
+Each event is annotated with a header that indicates the schema used.
+The body of the event is the binary Avro record data, not
+including the schema or the rest of the container file elements.
+
+Note that if the spool directory source must retry putting one of these events
+onto a channel (for example, because the channel is full), then it will reset
+and retry from the most recent Avro container file sync point. To reduce
+potential event duplication in such a failure scenario, write sync markers more
+frequently in your Avro input files.
+
+============================== ============== ======================================================================
+Property Name Default Description
+============================== ============== ======================================================================
+deserializer.schemaType HASH How the schema is represented. By default, or when the value ``HASH``
+ is specified, the Avro schema is hashed and
+ the hash is stored in every event in the event header
+ "flume.avro.schema.hash". If ``LITERAL`` is specified, the JSON-encoded
+ schema itself is stored in every event in the event header
+ "flume.avro.schema.literal". Using ``LITERAL`` mode is relatively
+ inefficient compared to ``HASH`` mode.
+============================== ============== ======================================================================
+
+BlobDeserializer
+^^^^^^^^^^^^^^^^
+
+This deserializer reads a Binary Large Object (BLOB) per event, typically one BLOB per file. For example a PDF or JPG file. Note that this approach is not suitable for very large objects because the entire BLOB is buffered in RAM.
+
+========================== ================== =======================================================================
+Property Name Default Description
+========================== ================== =======================================================================
+**deserializer** -- The FQCN of this class: ``org.apache.flume.sink.solr.morphline.BlobDeserializer$Builder``
+deserializer.maxBlobLength 100000000 The maximum number of bytes to read and buffer for a given request
+========================== ================== =======================================================================
+
+Taildir Source
+~~~~~~~~~~~~~~
+.. note:: **This source is provided as a preview feature. It does not work on Windows.**
+
+Watch the specified files, and tail them in nearly real-time once detected new lines appended to the each files.
+If the new lines are being written, this source will retry reading them in wait for the completion of the write.
+
+This source is reliable and will not miss data even when the tailing files rotate.
+It periodically writes the last read position of each files on the given position file in JSON format.
+If Flume is stopped or down for some reason, it can restart tailing from the position written on the existing position file.
+
+In other use case, this source can also start tailing from the arbitrary position for each files using the given position file.
+When there is no position file on the specified path, it will start tailing from the first line of each files by default.
+
+Files will be consumed in order of their modification time. File with the oldest modification time will be consumed first.
+
+This source does not rename or delete or do any modifications to the file being tailed.
+Currently this source does not support tailing binary files. It reads text files line by line.
+
+=================================== ============================== ===================================================
+Property Name Default Description
+=================================== ============================== ===================================================
+**channels** --
+**type** -- The component type name, needs to be ``TAILDIR``.
+**filegroups** -- Space-separated list of file groups. Each file group indicates a set of files to be tailed.
+**filegroups.<filegroupName>** -- Absolute path of the file group. Regular expression (and not file system patterns) can be used for filename only.
+positionFile ~/.flume/taildir_position.json File in JSON format to record the inode, the absolute path and the last position of each tailing file.
+headers.<filegroupName>.<headerKey> -- Header value which is the set with header key. Multiple headers can be specified for one file group.
+byteOffsetHeader false Whether to add the byte offset of a tailed line to a header called 'byteoffset'.
+skipToEnd false Whether to skip the position to EOF in the case of files not written on the position file.
+idleTimeout 120000 Time (ms) to close inactive files. If the closed file is appended new lines to, this source will automatically re-open it.
+writePosInterval 3000 Interval time (ms) to write the last position of each file on the position file.
+batchSize 100 Max number of lines to read and send to the channel at a time. Using the default is usually fine.
+maxBatchCount Long.MAX_VALUE Controls the number of batches being read consecutively from the same file.
+ If the source is tailing multiple files and one of them is written at a fast rate,
+ it can prevent other files to be processed, because the busy file would be read in an endless loop.
+ In this case lower this value.
+backoffSleepIncrement 1000 The increment for time delay before reattempting to poll for new data, when the last attempt did not find any new data.
+maxBackoffSleep 5000 The max time delay between each reattempt to poll for new data, when the last attempt did not find any new data.
+cachePatternMatching true Listing directories and applying the filename regex pattern may be time consuming for directories
+ containing thousands of files. Caching the list of matching files can improve performance.
+ The order in which files are consumed will also be cached.
+ Requires that the file system keeps track of modification times with at least a 1-second granularity.
+fileHeader false Whether to add a header storing the absolute path filename.
+fileHeaderKey file Header key to use when appending absolute path filename to event header.
+=================================== ============================== ===================================================
+
+Example for agent named a1:
+
+.. code-block:: properties
+
+ a1.sources = r1
+ a1.channels = c1
+ a1.sources.r1.type = TAILDIR
+ a1.sources.r1.channels = c1
+ a1.sources.r1.positionFile = /var/log/flume/taildir_position.json
+ a1.sources.r1.filegroups = f1 f2
+ a1.sources.r1.filegroups.f1 = /var/log/test1/example.log
+ a1.sources.r1.headers.f1.headerKey1 = value1
+ a1.sources.r1.filegroups.f2 = /var/log/test2/.*log.*
+ a1.sources.r1.headers.f2.headerKey1 = value2
+ a1.sources.r1.headers.f2.headerKey2 = value2-2
+ a1.sources.r1.fileHeader = true
+ a1.sources.ri.maxBatchCount = 1000
+
+Twitter 1% firehose Source (experimental)
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+.. warning::
+ This source is highly experimental and may change between minor versions of Flume.
+ Use at your own risk.
+
+Experimental source that connects via Streaming API to the 1% sample twitter
+firehose, continously downloads tweets, converts them to Avro format and
+sends Avro events to a downstream Flume sink. Requires the consumer and
+access tokens and secrets of a Twitter developer account.
+Required properties are in **bold**.
+
+====================== =========== ===================================================
+Property Name Default Description
+====================== =========== ===================================================
+**channels** --
+**type** -- The component type name, needs to be ``org.apache.flume.source.twitter.TwitterSource``
+**consumerKey** -- OAuth consumer key
+**consumerSecret** -- OAuth consumer secret
+**accessToken** -- OAuth access token
+**accessTokenSecret** -- OAuth token secret
+maxBatchSize 1000 Maximum number of twitter messages to put in a single batch
+maxBatchDurationMillis 1000 Maximum number of milliseconds to wait before closing a batch
+====================== =========== ===================================================
+
+Example for agent named a1:
+
+.. code-block:: properties
+
+ a1.sources = r1
+ a1.channels = c1
+ a1.sources.r1.type = org.apache.flume.source.twitter.TwitterSource
+ a1.sources.r1.channels = c1
+ a1.sources.r1.consumerKey = YOUR_TWITTER_CONSUMER_KEY
+ a1.sources.r1.consumerSecret = YOUR_TWITTER_CONSUMER_SECRET
+ a1.sources.r1.accessToken = YOUR_TWITTER_ACCESS_TOKEN
+ a1.sources.r1.accessTokenSecret = YOUR_TWITTER_ACCESS_TOKEN_SECRET
+ a1.sources.r1.maxBatchSize = 10
+ a1.sources.r1.maxBatchDurationMillis = 200
+
+Kafka Source
+~~~~~~~~~~~~
+
+Kafka Source is an Apache Kafka consumer that reads messages from Kafka topics.
+If you have multiple Kafka sources running, you can configure them with the same Consumer Group
+so each will read a unique set of partitions for the topics. This currently supports Kafka server releases 0.10.1.0 or higher. Testing was done up to 2.0.1 that was the highest avilable version at the time of the release.
+
+================================== =========== ===================================================
+Property Name Default Description
+================================== =========== ===================================================
+**channels** --
+**type** -- The component type name, needs to be ``org.apache.flume.source.kafka.KafkaSource``
+**kafka.bootstrap.servers** -- List of brokers in the Kafka cluster used by the source
+kafka.consumer.group.id flume Unique identified of consumer group. Setting the same id in multiple sources or agents
+ indicates that they are part of the same consumer group
+**kafka.topics** -- Comma-separated list of topics the kafka consumer will read messages from.
+**kafka.topics.regex** -- Regex that defines set of topics the source is subscribed on. This property has higher priority
+ than ``kafka.topics`` and overrides ``kafka.topics`` if exists.
+batchSize 1000 Maximum number of messages written to Channel in one batch
+batchDurationMillis 1000 Maximum time (in ms) before a batch will be written to Channel
+ The batch will be written whenever the first of size and time will be reached.
+backoffSleepIncrement 1000 Initial and incremental wait time that is triggered when a Kafka Topic appears to be empty.
+ Wait period will reduce aggressive pinging of an empty Kafka Topic. One second is ideal for
+ ingestion use cases but a lower value may be required for low latency operations with
+ interceptors.
+maxBackoffSleep 5000 Maximum wait time that is triggered when a Kafka Topic appears to be empty. Five seconds is
+ ideal for ingestion use cases but a lower value may be required for low latency operations
+ with interceptors.
+useFlumeEventFormat false By default events are taken as bytes from the Kafka topic directly into the event body. Set to
+ true to read events as the Flume Avro binary format. Used in conjunction with the same property
+ on the KafkaSink or with the parseAsFlumeEvent property on the Kafka Channel this will preserve
+ any Flume headers sent on the producing side.
+setTopicHeader true When set to true, stores the topic of the retrieved message into a header, defined by the
+ ``topicHeader`` property.
+topicHeader topic Defines the name of the header in which to store the name of the topic the message was received
+ from, if the ``setTopicHeader`` property is set to ``true``. Care should be taken if combining
+ with the Kafka Sink ``topicHeader`` property so as to avoid sending the message back to the same
+ topic in a loop.
+kafka.consumer.security.protocol PLAINTEXT Set to SASL_PLAINTEXT, SASL_SSL or SSL if writing to Kafka using some level of security. See below for additional info on secure setup.
+*more consumer security props* If using SASL_PLAINTEXT, SASL_SSL or SSL refer to `Kafka security <http://kafka.apache.org/documentation.html#security>`_ for additional
+ properties that need to be set on consumer.
+Other Kafka Consumer Properties -- These properties are used to configure the Kafka Consumer. Any consumer property supported
+ by Kafka can be used. The only requirement is to prepend the property name with the prefix
+ ``kafka.consumer``.
+ For example: ``kafka.consumer.auto.offset.reset``
+================================== =========== ===================================================
+
+.. note:: The Kafka Source overrides two Kafka consumer parameters:
+ auto.commit.enable is set to "false" by the source and every batch is committed. Kafka source guarantees at least once
+ strategy of messages retrieval. The duplicates can be present when the source starts.
+ The Kafka Source also provides defaults for the key.deserializer(org.apache.kafka.common.serialization.StringSerializer)
+ and value.deserializer(org.apache.kafka.common.serialization.ByteArraySerializer). Modification of these parameters is not recommended.
+
+Deprecated Properties
+
+=============================== =================== ================================================================================================
+Property Name Default Description
+=============================== =================== ================================================================================================
+topic -- Use kafka.topics
+groupId flume Use kafka.consumer.group.id
+zookeeperConnect -- Is no longer supported by kafka consumer client since 0.9.x. Use kafka.bootstrap.servers
+ to establish connection with kafka cluster
+migrateZookeeperOffsets true When no Kafka stored offset is found, look up the offsets in Zookeeper and commit them to Kafka.
+ This should be true to support seamless Kafka client migration from older versions of Flume.
+ Once migrated this can be set to false, though that should generally not be required.
+ If no Zookeeper offset is found, the Kafka configuration kafka.consumer.auto.offset.reset
+ defines how offsets are handled.
+ Check `Kafka documentation <http://kafka.apache.org/documentation.html#newconsumerconfigs>`_
+ for details
+=============================== =================== ================================================================================================
+
+Example for topic subscription by comma-separated topic list.
+
+.. code-block:: properties
+
+ tier1.sources.source1.type = org.apache.flume.source.kafka.KafkaSource
+ tier1.sources.source1.channels = channel1
+ tier1.sources.source1.batchSize = 5000
+ tier1.sources.source1.batchDurationMillis = 2000
+ tier1.sources.source1.kafka.bootstrap.servers = localhost:9092
+ tier1.sources.source1.kafka.topics = test1, test2
+ tier1.sources.source1.kafka.consumer.group.id = custom.g.id
+
+Example for topic subscription by regex
+
+.. code-block:: properties
+
+ tier1.sources.source1.type = org.apache.flume.source.kafka.KafkaSource
+ tier1.sources.source1.channels = channel1
+ tier1.sources.source1.kafka.bootstrap.servers = localhost:9092
+ tier1.sources.source1.kafka.topics.regex = ^topic[0-9]$
+ # the default kafka.consumer.group.id=flume is used
+
+
+**Security and Kafka Source:**
+
+Secure authentication as well as data encryption is supported on the communication channel between Flume and Kafka.
+For secure authentication SASL/GSSAPI (Kerberos V5) or SSL (even though the parameter is named SSL, the actual protocol is a TLS implementation) can be used from Kafka version 0.9.0.
+
+As of now data encryption is solely provided by SSL/TLS.
+
+Setting ``kafka.consumer.security.protocol`` to any of the following value means:
+
+- **SASL_PLAINTEXT** - Kerberos or plaintext authentication with no data encryption
+- **SASL_SSL** - Kerberos or plaintext authentication with data encryption
+- **SSL** - TLS based encryption with optional authentication.
+
+.. warning::
+ There is a performance degradation when SSL is enabled,
+ the magnitude of which depends on the CPU type and the JVM implementation.
+ Reference: `Kafka security overview <http://kafka.apache.org/documentation#security_overview>`_
+ and the jira for tracking this issue:
+ `KAFKA-2561 <https://issues.apache.org/jira/browse/KAFKA-2561>`_
+
+
+**TLS and Kafka Source:**
+
+Please read the steps described in `Configuring Kafka Clients SSL <http://kafka.apache.org/documentation#security_configclients>`_
+to learn about additional configuration settings for fine tuning for example any of the following:
+security provider, cipher suites, enabled protocols, truststore or keystore types.
+
+Example configuration with server side authentication and data encryption.
+
+.. code-block:: properties
+
+ a1.sources.source1.type = org.apache.flume.source.kafka.KafkaSource
+ a1.sources.source1.kafka.bootstrap.servers = kafka-1:9093,kafka-2:9093,kafka-3:9093
+ a1.sources.source1.kafka.topics = mytopic
+ a1.sources.source1.kafka.consumer.group.id = flume-consumer
+ a1.sources.source1.kafka.consumer.security.protocol = SSL
+ # optional, the global truststore can be used alternatively
+ a1.sources.source1.kafka.consumer.ssl.truststore.location=/path/to/truststore.jks
+ a1.sources.source1.kafka.consumer.ssl.truststore.password=<password to access the truststore>
+
+Specyfing the truststore is optional here, the global truststore can be used instead.
+For more details about the global SSL setup, see the `SSL/TLS support`_ section.
+
+Note: By default the property ``ssl.endpoint.identification.algorithm``
+is not defined, so hostname verification is not performed.
+In order to enable hostname verification, set the following properties
+
+.. code-block:: properties
+
+ a1.sources.source1.kafka.consumer.ssl.endpoint.identification.algorithm=HTTPS
+
+Once enabled, clients will verify the server's fully qualified domain name (FQDN)
+against one of the following two fields:
+
+#) Common Name (CN) https://tools.ietf.org/html/rfc6125#section-2.3
+#) Subject Alternative Name (SAN) https://tools.ietf.org/html/rfc5280#section-4.2.1.6
+
+If client side authentication is also required then additionally the following needs to be added to Flume agent
+configuration or the global SSL setup can be used (see `SSL/TLS support`_ section).
+Each Flume agent has to have its client certificate which has to be trusted by Kafka brokers either
+individually or by their signature chain. Common example is to sign each client certificate by a single Root CA
+which in turn is trusted by Kafka brokers.
+
+.. code-block:: properties
+
+ # optional, the global keystore can be used alternatively
+ a1.sources.source1.kafka.consumer.ssl.keystore.location=/path/to/client.keystore.jks
+ a1.sources.source1.kafka.consumer.ssl.keystore.password=<password to access the keystore>
+
+If keystore and key use different password protection then ``ssl.key.password`` property will
+provide the required additional secret for both consumer keystores:
+
+.. code-block:: properties
+
+ a1.sources.source1.kafka.consumer.ssl.key.password=<password to access the key>
+
+
+**Kerberos and Kafka Source:**
+
+To use Kafka source with a Kafka cluster secured with Kerberos, set the ``consumer.security.protocol`` properties noted above for consumer.
+The Kerberos keytab and principal to be used with Kafka brokers is specified in a JAAS file's "KafkaClient" section. "Client" section describes the Zookeeper connection if needed.
+See `Kafka doc <http://kafka.apache.org/documentation.html#security_sasl_clientconfig>`_
+for information on the JAAS file contents. The location of this JAAS file and optionally the system wide kerberos configuration can be specified via JAVA_OPTS in flume-env.sh:
+
+.. code-block:: properties
+
+ JAVA_OPTS="$JAVA_OPTS -Djava.security.krb5.conf=/path/to/krb5.conf"
+ JAVA_OPTS="$JAVA_OPTS -Djava.security.auth.login.config=/path/to/flume_jaas.conf"
+
+Example secure configuration using SASL_PLAINTEXT:
+
+.. code-block:: properties
+
+ a1.sources.source1.type = org.apache.flume.source.kafka.KafkaSource
+ a1.sources.source1.kafka.bootstrap.servers = kafka-1:9093,kafka-2:9093,kafka-3:9093
+ a1.sources.source1.kafka.topics = mytopic
+ a1.sources.source1.kafka.consumer.group.id = flume-consumer
+ a1.sources.source1.kafka.consumer.security.protocol = SASL_PLAINTEXT
+ a1.sources.source1.kafka.consumer.sasl.mechanism = GSSAPI
+ a1.sources.source1.kafka.consumer.sasl.kerberos.service.name = kafka
+
+Example secure configuration using SASL_SSL:
+
+.. code-block:: properties
+
+ a1.sources.source1.type = org.apache.flume.source.kafka.KafkaSource
+ a1.sources.source1.kafka.bootstrap.servers = kafka-1:9093,kafka-2:9093,kafka-3:9093
+ a1.sources.source1.kafka.topics = mytopic
+ a1.sources.source1.kafka.consumer.group.id = flume-consumer
+ a1.sources.source1.kafka.consumer.security.protocol = SASL_SSL
+ a1.sources.source1.kafka.consumer.sasl.mechanism = GSSAPI
+ a1.sources.source1.kafka.consumer.sasl.kerberos.service.name = kafka
+ # optional, the global truststore can be used alternatively
+ a1.sources.source1.kafka.consumer.ssl.truststore.location=/path/to/truststore.jks
+ a1.sources.source1.kafka.consumer.ssl.truststore.password=<password to access the truststore>
+
+
+Sample JAAS file. For reference of its content please see client config sections of the desired authentication mechanism (GSSAPI/PLAIN)
+in Kafka documentation of `SASL configuration <http://kafka.apache.org/documentation#security_sasl_clientconfig>`_.
+Since the Kafka Source may also connect to Zookeeper for offset migration, the "Client" section was also added to this example.
+This won't be needed unless you require offset migration, or you require this section for other secure components.
+Also please make sure that the operating system user of the Flume processes has read privileges on the jaas and keytab files.
+
+.. code-block:: javascript
+
+ Client {
+ com.sun.security.auth.module.Krb5LoginModule required
+ useKeyTab=true
+ storeKey=true
+ keyTab="/path/to/keytabs/flume.keytab"
+ principal="flume/flumehost1.example.com@YOURKERBEROSREALM";
+ };
+
+ KafkaClient {
+ com.sun.security.auth.module.Krb5LoginModule required
+ useKeyTab=true
+ storeKey=true
+ keyTab="/path/to/keytabs/flume.keytab"
+ principal="flume/flumehost1.example.com@YOURKERBEROSREALM";
+ };
+
+
+NetCat TCP Source
+~~~~~~~~~~~~~~~~~
+
+A netcat-like source that listens on a given port and turns each line of text
+into an event. Acts like ``nc -k -l [host] [port]``. In other words,
+it opens a specified port and listens for data. The expectation is that the
+supplied data is newline separated text. Each line of text is turned into a
+Flume event and sent via the connected channel.
+
+Required properties are in **bold**.
+
+=============== =========== ===========================================
+Property Name Default Description
+=============== =========== ===========================================
+**channels** --
+**type** -- The component type name, needs to be ``netcat``
+**bind** -- Host name or IP address to bind to
+**port** -- Port # to bind to
+max-line-length 512 Max line length per event body (in bytes)
+ack-every-event true Respond with an "OK" for every event received
+selector.type replicating replicating or multiplexing
+selector.* Depends on the selector.type value
+interceptors -- Space-separated list of interceptors
+interceptors.*
+=============== =========== ===========================================
+
+Example for agent named a1:
+
+.. code-block:: properties
+
+ a1.sources = r1
+ a1.channels = c1
+ a1.sources.r1.type = netcat
+ a1.sources.r1.bind = 0.0.0.0
+ a1.sources.r1.port = 6666
+ a1.sources.r1.channels = c1
+
+NetCat UDP Source
+~~~~~~~~~~~~~~~~~
+
+As per the original Netcat (TCP) source, this source that listens on a given
+port and turns each line of text into an event and sent via the connected channel.
+Acts like ``nc -u -k -l [host] [port]``.
+
+Required properties are in **bold**.
+
+=================== =========== ===========================================
+Property Name Default Description
+=================== =========== ===========================================
+**channels** --
+**type** -- The component type name, needs to be ``netcatudp``
+**bind** -- Host name or IP address to bind to
+**port** -- Port # to bind to
+remoteAddressHeader --
+selector.type replicating replicating or multiplexing
+selector.* Depends on the selector.type value
+interceptors -- Space-separated list of interceptors
+interceptors.*
+=================== =========== ===========================================
+
+Example for agent named a1:
+
+.. code-block:: properties
+
+ a1.sources = r1
+ a1.channels = c1
+ a1.sources.r1.type = netcatudp
+ a1.sources.r1.bind = 0.0.0.0
+ a1.sources.r1.port = 6666
+ a1.sources.r1.channels = c1
+
+Sequence Generator Source
+~~~~~~~~~~~~~~~~~~~~~~~~~
+
+A simple sequence generator that continuously generates events with a counter that starts from 0,
+increments by 1 and stops at totalEvents. Retries when it can't send events to the channel. Useful
+mainly for testing. During retries it keeps the body of the retried messages the same as before so
+that the number of unique events - after de-duplication at destination - is expected to be
+equal to the specified ``totalEvents``. Required properties are in **bold**.
+
+============== =============== ========================================
+Property Name Default Description
+============== =============== ========================================
+**channels** --
+**type** -- The component type name, needs to be ``seq``
+selector.type replicating or multiplexing
+selector.* replicating Depends on the selector.type value
+interceptors -- Space-separated list of interceptors
+interceptors.*
+batchSize 1 Number of events to attempt to process per request loop.
+totalEvents Long.MAX_VALUE Number of unique events sent by the source.
+============== =============== ========================================
+
+Example for agent named a1:
+
+.. code-block:: properties
+
+ a1.sources = r1
+ a1.channels = c1
+ a1.sources.r1.type = seq
+ a1.sources.r1.channels = c1
+
+Syslog Sources
+~~~~~~~~~~~~~~
+
+Reads syslog data and generate Flume events. The UDP source treats an entire
+message as a single event. The TCP sources create a new event for each string
+of characters separated by a newline ('\n').
+
+Required properties are in **bold**.
+
+Syslog TCP Source
+'''''''''''''''''
+
+The original, tried-and-true syslog TCP source.
+
+===================== =========== =================================================================================================
+Property Name Default Description
+===================== =========== =================================================================================================
+**channels** --
+**type** -- The component type name, needs to be ``syslogtcp``
+**host** -- Host name or IP address to bind to
+**port** -- Port # to bind to
+eventSize 2500 Maximum size of a single event line, in bytes
+keepFields none Setting this to 'all' will preserve the Priority,
+ Timestamp and Hostname in the body of the event.
+ A spaced separated list of fields to include
+ is allowed as well. Currently, the following
+ fields can be included: priority, version,
+ timestamp, hostname. The values 'true' and 'false'
+ have been deprecated in favor of 'all' and 'none'.
+clientIPHeader -- If specified, the IP address of the client will be stored in
+ the header of each event using the header name specified here.
+ This allows for interceptors and channel selectors to customize
+ routing logic based on the IP address of the client.
+ Do not use the standard Syslog header names here (like _host_)
+ because the event header will be overridden in that case.
+clientHostnameHeader -- If specified, the host name of the client will be stored in
+ the header of each event using the header name specified here.
+ This allows for interceptors and channel selectors to customize
+ routing logic based on the host name of the client.
+ Retrieving the host name may involve a name service reverse
+ lookup which may affect the performance.
+ Do not use the standard Syslog header names here (like _host_)
+ because the event header will be overridden in that case.
+selector.type replicating or multiplexing
+selector.* replicating Depends on the selector.type value
+interceptors -- Space-separated list of interceptors
+interceptors.*
+ssl false Set this to true to enable SSL encryption. If SSL is enabled,
+ you must also specify a "keystore" and a "keystore-password",
+ either through component level parameters (see below)
+ or as global SSL parameters (see `SSL/TLS support`_ section).
+keystore -- This is the path to a Java keystore file.
+ If not specified here, then the global keystore will be used
+ (if defined, otherwise configuration error).
+keystore-password -- The password for the Java keystore.
+ If not specified here, then the global keystore password will be used
+ (if defined, otherwise configuration error).
+keystore-type JKS The type of the Java keystore. This can be "JKS" or "PKCS12".
+ If not specified here, then the global keystore type will be used
+ (if defined, otherwise the default is JKS).
+exclude-protocols SSLv3 Space-separated list of SSL/TLS protocols to exclude.
+ SSLv3 will always be excluded in addition to the protocols specified.
+include-protocols -- Space-separated list of SSL/TLS protocols to include.
+ The enabled protocols will be the included protocols without the excluded protocols.
+ If included-protocols is empty, it includes every supported protocols.
+exclude-cipher-suites -- Space-separated list of cipher suites to exclude.
+include-cipher-suites -- Space-separated list of cipher suites to include.
+ The enabled cipher suites will be the included cipher suites without the excluded cipher suites.
+ If included-cipher-suites is empty, it includes every supported cipher suites.
+===================== =========== =================================================================================================
+
+For example, a syslog TCP source for agent named a1:
+
+.. code-block:: properties
+
+ a1.sources = r1
+ a1.channels = c1
+ a1.sources.r1.type = syslogtcp
+ a1.sources.r1.port = 5140
+ a1.sources.r1.host = localhost
+ a1.sources.r1.channels = c1
+
+Multiport Syslog TCP Source
+'''''''''''''''''''''''''''
+
+This is a newer, faster, multi-port capable version of the Syslog TCP source.
+Note that the ``ports`` configuration setting has replaced ``port``.
+Multi-port capability means that it can listen on many ports at once in an
+efficient manner. This source uses the Apache Mina library to do that.
+Provides support for RFC-3164 and many common RFC-5424 formatted messages.
+Also provides the capability to configure the character set used on a per-port
+basis.
+
+===================== ================ ==============================================
+Property Name Default Description
+===================== ================ ==============================================
+**channels** --
+**type** -- The component type name, needs to be ``multiport_syslogtcp``
+**host** -- Host name or IP address to bind to.
+**ports** -- Space-separated list (one or more) of ports to bind to.
+eventSize 2500 Maximum size of a single event line, in bytes.
+keepFields none Setting this to 'all' will preserve the
+ Priority, Timestamp and Hostname in the body of the event.
+ A spaced separated list of fields to include
+ is allowed as well. Currently, the following
+ fields can be included: priority, version,
+ timestamp, hostname. The values 'true' and 'false'
+ have been deprecated in favor of 'all' and 'none'.
+portHeader -- If specified, the port number will be stored in the header of each event using the header name specified here. This allows for interceptors and channel selectors to customize routing logic based on the incoming port.
+clientIPHeader -- If specified, the IP address of the client will be stored in
+ the header of each event using the header name specified here.
+ This allows for interceptors and channel selectors to customize
+ routing logic based on the IP address of the client.
+ Do not use the standard Syslog header names here (like _host_)
+ because the event header will be overridden in that case.
+clientHostnameHeader -- If specified, the host name of the client will be stored in
+ the header of each event using the header name specified here.
+ This allows for interceptors and channel selectors to customize
+ routing logic based on the host name of the client.
+ Retrieving the host name may involve a name service reverse
+ lookup which may affect the performance.
+ Do not use the standard Syslog header names here (like _host_)
+ because the event header will be overridden in that case.
+charset.default UTF-8 Default character set used while parsing syslog events into strings.
+charset.port.<port> -- Character set is configurable on a per-port basis.
+batchSize 100 Maximum number of events to attempt to process per request loop. Using the default is usually fine.
+readBufferSize 1024 Size of the internal Mina read buffer. Provided for performance tuning. Using the default is usually fine.
+numProcessors (auto-detected) Number of processors available on the system for use while processing messages. Default is to auto-detect # of CPUs using the Java Runtime API. Mina will spawn 2 request-processing threads per detected CPU, which is often reasonable.
+selector.type replicating replicating, multiplexing, or custom
+selector.* -- Depends on the ``selector.type`` value
+interceptors -- Space-separated list of interceptors.
+interceptors.*
+ssl false Set this to true to enable SSL encryption. If SSL is enabled,
+ you must also specify a "keystore" and a "keystore-password",
+ either through component level parameters (see below)
+ or as global SSL parameters (see `SSL/TLS support`_ section).
+keystore -- This is the path to a Java keystore file.
+ If not specified here, then the global keystore will be used
+ (if defined, otherwise configuration error).
+keystore-password -- The password for the Java keystore.
+ If not specified here, then the global keystore password will be used
+ (if defined, otherwise configuration error).
+keystore-type JKS The type of the Java keystore. This can be "JKS" or "PKCS12".
+ If not specified here, then the global keystore type will be used
+ (if defined, otherwise the default is JKS).
+exclude-protocols SSLv3 Space-separated list of SSL/TLS protocols to exclude.
+ SSLv3 will always be excluded in addition to the protocols specified.
+include-protocols -- Space-separated list of SSL/TLS protocols to include.
+ The enabled protocols will be the included protocols without the excluded protocols.
+ If included-protocols is empty, it includes every supported protocols.
+exclude-cipher-suites -- Space-separated list of cipher suites to exclude.
+include-cipher-suites -- Space-separated list of cipher suites to include.
+ The enabled cipher suites will be the included cipher suites without the excluded cipher suites.
+ If included-cipher-suites is empty, it includes every supported cipher suites.
+===================== ================ ==============================================
+
+For example, a multiport syslog TCP source for agent named a1:
+
+.. code-block:: properties
+
+ a1.sources = r1
+ a1.channels = c1
+ a1.sources.r1.type = multiport_syslogtcp
+ a1.sources.r1.channels = c1
+ a1.sources.r1.host = 0.0.0.0
+ a1.sources.r1.ports = 10001 10002 10003
+ a1.sources.r1.portHeader = port
+
+Syslog UDP Source
+'''''''''''''''''
+
+==================== =========== ================================================================
+Property Name Default Description
+==================== =========== ================================================================
+**channels** --
+**type** -- The component type name, needs to be ``syslogudp``
+**host** -- Host name or IP address to bind to
+**port** -- Port # to bind to
+keepFields false Setting this to true will preserve the Priority,
+ Timestamp and Hostname in the body of the event.
+clientIPHeader -- If specified, the IP address of the client will be stored in
+ the header of each event using the header name specified here.
+ This allows for interceptors and channel selectors to customize
+ routing logic based on the IP address of the client.
+ Do not use the standard Syslog header names here (like _host_)
+ because the event header will be overridden in that case.
+clientHostnameHeader -- If specified, the host name of the client will be stored in
+ the header of each event using the header name specified here.
+ This allows for interceptors and channel selectors to customize
+ routing logic based on the host name of the client.
+ Retrieving the host name may involve a name service reverse
+ lookup which may affect the performance.
+ Do not use the standard Syslog header names here (like _host_)
+ because the event header will be overridden in that case.
+selector.type replicating or multiplexing
+selector.* replicating Depends on the selector.type value
+interceptors -- Space-separated list of interceptors
+interceptors.*
+==================== =========== ================================================================
+
+
+For example, a syslog UDP source for agent named a1:
+
+.. code-block:: properties
+
+ a1.sources = r1
+ a1.channels = c1
+ a1.sources.r1.type = syslogudp
+ a1.sources.r1.port = 5140
+ a1.sources.r1.host = localhost
+ a1.sources.r1.channels = c1
+
+HTTP Source
+~~~~~~~~~~~
+A source which accepts Flume Events by HTTP POST and GET. GET should be used
+for experimentation only. HTTP requests are converted into flume events by
+a pluggable "handler" which must implement the HTTPSourceHandler interface.
+This handler takes a HttpServletRequest and returns a list of
+flume events. All events handled from one Http request are committed to the channel
+in one transaction, thus allowing for increased efficiency on channels like
+the file channel. If the handler throws an exception, this source will
+return a HTTP status of 400. If the channel is full, or the source is unable to
+append events to the channel, the source will return a HTTP 503 - Temporarily
+unavailable status.
+
+All events sent in one post request are considered to be one batch and
+inserted into the channel in one transaction.
+
+This source is based on Jetty 9.4 and offers the ability to set additional
+Jetty-specific parameters which will be passed directly to the Jetty components.
+
+===================== ============================================ =====================================================================================
+Property Name Default Description
+===================== ============================================ =====================================================================================
+**type** The component type name, needs to be ``http``
+**port** -- The port the source should bind to.
+bind 0.0.0.0 The hostname or IP address to listen on
+handler ``org.apache.flume.source.http.JSONHandler`` The FQCN of the handler class.
+handler.* -- Config parameters for the handler
+selector.type replicating replicating or multiplexing
+selector.* Depends on the selector.type value
+interceptors -- Space-separated list of interceptors
+interceptors.*
+ssl false Set the property true, to enable SSL. *HTTP Source does not support SSLv3.*
+exclude-protocols SSLv3 Space-separated list of SSL/TLS protocols to exclude.
+ SSLv3 will always be excluded in addition to the protocols specified.
+include-protocols -- Space-separated list of SSL/TLS protocols to include.
+ The enabled protocols will be the included protocols without the excluded protocols.
+ If included-protocols is empty, it includes every supported protocols.
+exclude-cipher-suites -- Space-separated list of cipher suites to exclude.
+include-cipher-suites -- Space-separated list of cipher suites to include.
+ The enabled cipher suites will be the included cipher suites without the excluded cipher suites.
+keystore Location of the keystore including keystore file name.
+ If SSL is enabled but the keystore is not specified here,
+ then the global keystore will be used
+ (if defined, otherwise configuration error).
+keystore-password Keystore password.
+ If SSL is enabled but the keystore password is not specified here,
+ then the global keystore password will be used
+ (if defined, otherwise configuration error).
+keystore-type JKS Keystore type. This can be "JKS" or "PKCS12".
+QueuedThreadPool.* Jetty specific settings to be set on org.eclipse.jetty.util.thread.QueuedThreadPool.
+ N.B. QueuedThreadPool will only be used if at least one property of this class is set.
+HttpConfiguration.* Jetty specific settings to be set on org.eclipse.jetty.server.HttpConfiguration
+SslContextFactory.* Jetty specific settings to be set on org.eclipse.jetty.util.ssl.SslContextFactory (only
+ applicable when *ssl* is set to true).
+ServerConnector.* Jetty specific settings to be set on org.eclipse.jetty.server.ServerConnector
+===================== ============================================ =====================================================================================
+
+Deprecated Properties
+
+=============================== =================== =============================================================================================
+Property Name Default Description
+=============================== =================== =============================================================================================
+keystorePassword -- Use *keystore-password*. Deprecated value will be overwritten with the new one.
+excludeProtocols SSLv3 Use *exclude-protocols*. Deprecated value will be overwritten with the new one.
+enableSSL false Use *ssl*. Deprecated value will be overwritten with the new one.
+=============================== =================== =============================================================================================
+
+N.B. Jetty-specific settings are set using the setter-methods on the objects listed above. For full details see the Javadoc for these classes
+(`QueuedThreadPool <http://www.eclipse.org/jetty/javadoc/9.4.6.v20170531/org/eclipse/jetty/util/thread/QueuedThreadPool.html>`_,
+`HttpConfiguration <http://www.eclipse.org/jetty/javadoc/9.4.6.v20170531/org/eclipse/jetty/server/HttpConfiguration.html>`_,
+`SslContextFactory <http://www.eclipse.org/jetty/javadoc/9.4.6.v20170531/org/eclipse/jetty/util/ssl/SslContextFactory.html>`_ and
+`ServerConnector <http://www.eclipse.org/jetty/javadoc/9.4.6.v20170531/org/eclipse/jetty/server/ServerConnector.html>`_).
+
+When using Jetty-specific setings, named properites above will take precedence (for example excludeProtocols will take
+precedence over SslContextFactory.ExcludeProtocols). All properties will be inital lower case.
+
+An example http source for agent named a1:
+
+.. code-block:: properties
+
+ a1.sources = r1
+ a1.channels = c1
+ a1.sources.r1.type = http
+ a1.sources.r1.port = 5140
+ a1.sources.r1.channels = c1
+ a1.sources.r1.handler = org.example.rest.RestHandler
+ a1.sources.r1.handler.nickname = random props
+ a1.sources.r1.HttpConfiguration.sendServerVersion = false
+ a1.sources.r1.ServerConnector.idleTimeout = 300
+
+JSONHandler
+'''''''''''
+A handler is provided out of the box which can handle events represented in
+JSON format, and supports UTF-8, UTF-16 and UTF-32 character sets. The handler
+accepts an array of events (even if there is only one event, the event has to be
+sent in an array) and converts them to a Flume event based on the
+encoding specified in the request. If no encoding is specified, UTF-8 is assumed.
+The JSON handler supports UTF-8, UTF-16 and UTF-32.
+Events are represented as follows.
+
+.. code-block:: javascript
+
+ [{
+ "headers" : {
+ "timestamp" : "434324343",
+ "host" : "random_host.example.com"
+ },
+ "body" : "random_body"
+ },
+ {
+ "headers" : {
+ "namenode" : "namenode.example.com",
+ "datanode" : "random_datanode.example.com"
+ },
+ "body" : "really_random_body"
+ }]
+
+To set the charset, the request must have content type specified as
+``application/json; charset=UTF-8`` (replace UTF-8 with UTF-16 or UTF-32 as
+required).
+
+One way to create an event in the format expected by this handler is to
+use JSONEvent provided in the Flume SDK and use Google Gson to create the JSON
+string using the Gson#fromJson(Object, Type)
+method. The type token to pass as the 2nd argument of this method
+for list of events can be created by:
+
+.. code-block:: java
+
+ Type type = new TypeToken<List<JSONEvent>>() {}.getType();
+
+BlobHandler
+'''''''''''
+By default HTTPSource splits JSON input into Flume events. As an alternative, BlobHandler is a handler for HTTPSource that returns an event that contains the request parameters as well as the Binary Large Object (BLOB) uploaded with this request. For example a PDF or JPG file. Note that this approach is not suitable for very large objects because it buffers up the entire BLOB in RAM.
+
+===================== ================== ============================================================================
+Property Name Default Description
+===================== ================== ============================================================================
+**handler** -- The FQCN of this class: ``org.apache.flume.sink.solr.morphline.BlobHandler``
+handler.maxBlobLength 100000000 The maximum number of bytes to read and buffer for a given request
+===================== ================== ============================================================================
+
+Stress Source
+~~~~~~~~~~~~~
+
+StressSource is an internal load-generating source implementation which is very useful for
+stress tests. It allows User to configure the size of Event payload, with empty headers.
+User can configure total number of events to be sent as well maximum number of Successful
+Event to be delivered.
+
+Required properties are in **bold**.
+
+=================== =========== ===================================================
+Property Name Default Description
+=================== =========== ===================================================
+**type** -- The component type name, needs to be ``org.apache.flume.source.StressSource``
+size 500 Payload size of each Event. Unit:**byte**
+maxTotalEvents -1 Maximum number of Events to be sent
+maxSuccessfulEvents -1 Maximum number of Events successfully sent
+batchSize 1 Number of Events to be sent in one batch
+maxEventsPerSecond 0 When set to an integer greater than zero, enforces a rate limiter onto the source.
+=================== =========== ===================================================
+
+Example for agent named **a1**:
+
+.. code-block:: properties
+
+ a1.sources = stresssource-1
+ a1.channels = memoryChannel-1
+ a1.sources.stresssource-1.type = org.apache.flume.source.StressSource
+ a1.sources.stresssource-1.size = 10240
+ a1.sources.stresssource-1.maxTotalEvents = 1000000
+ a1.sources.stresssource-1.channels = memoryChannel-1
+
+Legacy Sources
+~~~~~~~~~~~~~~
+
+The legacy sources allow a Flume 1.x agent to receive events from Flume 0.9.4
+agents. It accepts events in the Flume 0.9.4 format, converts them to the Flume
+1.0 format, and stores them in the connected channel. The 0.9.4 event
+properties like timestamp, pri, host, nanos, etc get converted to 1.x event
+header attributes. The legacy source supports both Avro and Thrift RPC
+connections. To use this bridge between two Flume versions, you need to start a
+Flume 1.x agent with the avroLegacy or thriftLegacy source. The 0.9.4 agent
+should have the agent Sink pointing to the host/port of the 1.x agent.
+
+.. note:: The reliability semantics of Flume 1.x are different from that of
+ Flume 0.9.x. The E2E or DFO mode of a Flume 0.9.x agent will not be
+ supported by the legacy source. The only supported 0.9.x mode is the
+ best effort, though the reliability setting of the 1.x flow will be
+ applicable to the events once they are saved into the Flume 1.x
+ channel by the legacy source.
+
+Required properties are in **bold**.
+
+
+Avro Legacy Source
+''''''''''''''''''
+
+============== =========== ========================================================================================
+Property Name Default Description
+============== =========== ========================================================================================
+**channels** --
+**type** -- The component type name, needs to be ``org.apache.flume.source.avroLegacy.AvroLegacySource``
+**host** -- The hostname or IP address to bind to
+**port** -- The port # to listen on
+selector.type replicating or multiplexing
+selector.* replicating Depends on the selector.type value
+interceptors -- Space-separated list of interceptors
+interceptors.*
+============== =========== ========================================================================================
+
+Example for agent named a1:
+
+.. code-block:: properties
+
+ a1.sources = r1
+ a1.channels = c1
+ a1.sources.r1.type = org.apache.flume.source.avroLegacy.AvroLegacySource
+ a1.sources.r1.host = 0.0.0.0
+ a1.sources.r1.bind = 6666
+ a1.sources.r1.channels = c1
+
+Thrift Legacy Source
+''''''''''''''''''''
+
+============== =========== ======================================================================================
+Property Name Default Description
+============== =========== ======================================================================================
+**channels** --
+**type** -- The component type name, needs to be ``org.apache.flume.source.thriftLegacy.ThriftLegacySource``
+**host** -- The hostname or IP address to bind to
+**port** -- The port # to listen on
+selector.type replicating or multiplexing
+selector.* replicating Depends on the selector.type value
+interceptors -- Space-separated list of interceptors
+interceptors.*
+============== =========== ======================================================================================
+
+Example for agent named a1:
+
+.. code-block:: properties
+
+ a1.sources = r1
+ a1.channels = c1
+ a1.sources.r1.type = org.apache.flume.source.thriftLegacy.ThriftLegacySource
+ a1.sources.r1.host = 0.0.0.0
+ a1.sources.r1.bind = 6666
+ a1.sources.r1.channels = c1
+
+Custom Source
+~~~~~~~~~~~~~
+
+A custom source is your own implementation of the Source interface. A custom
+source's class and its dependencies must be included in the agent's classpath
+when starting the Flume agent. The type of the custom source is its FQCN.
+
+============== =========== ==============================================
+Property Name Default Description
+============== =========== ==============================================
+**channels** --
+**type** -- The component type name, needs to be your FQCN
+selector.type ``replicating`` or ``multiplexing``
+selector.* replicating Depends on the selector.type value
+interceptors -- Space-separated list of interceptors
+interceptors.*
+============== =========== ==============================================
+
+Example for agent named a1:
+
+.. code-block:: properties
+
+ a1.sources = r1
+ a1.channels = c1
+ a1.sources.r1.type = org.example.MySource
+ a1.sources.r1.channels = c1
+
+Scribe Source
+~~~~~~~~~~~~~
+
+Scribe is another type of ingest system. To adopt existing Scribe ingest system,
+Flume should use ScribeSource based on Thrift with compatible transfering protocol.
+For deployment of Scribe please follow the guide from Facebook.
+Required properties are in **bold**.
+
+==================== =========== ==============================================
+Property Name Default Description
+==================== =========== ==============================================
+**type** -- The component type name, needs to be ``org.apache.flume.source.scribe.ScribeSource``
+port 1499 Port that Scribe should be connected
+maxReadBufferBytes 16384000 Thrift Default FrameBuffer Size
+workerThreads 5 Handing threads number in Thrift
+selector.type
+selector.*
+==================== =========== ==============================================
+
+Example for agent named a1:
+
+.. code-block:: properties
+
+ a1.sources = r1
+ a1.channels = c1
+ a1.sources.r1.type = org.apache.flume.source.scribe.ScribeSource
+ a1.sources.r1.port = 1463
+ a1.sources.r1.workerThreads = 5
+ a1.sources.r1.channels = c1
+
+Flume Sinks
+-----------
+
+HDFS Sink
+~~~~~~~~~
+
+This sink writes events into the Hadoop Distributed File System (HDFS). It
+currently supports creating text and sequence files. It supports compression in
+both file types. The files can be rolled (close current file and create a new
+one) periodically based on the elapsed time or size of data or number of events.
+It also buckets/partitions data by attributes like timestamp or machine
+where the event originated. The HDFS directory path may contain formatting
+escape sequences that will replaced by the HDFS sink to generate a
+directory/file name to store the events. Using this sink requires hadoop to be
+installed so that Flume can use the Hadoop jars to communicate with the HDFS
+cluster. Note that a version of Hadoop that supports the sync() call is
+required.
+
+The following are the escape sequences supported:
+
+=============== =================================================
+Alias Description
+=============== =================================================
+%{host} Substitute value of event header named "host". Arbitrary header names are supported.
+%t Unix time in milliseconds
+%a locale's short weekday name (Mon, Tue, ...)
+%A locale's full weekday name (Monday, Tuesday, ...)
+%b locale's short month name (Jan, Feb, ...)
+%B locale's long month name (January, February, ...)
+%c locale's date and time (Thu Mar 3 23:05:25 2005)
+%d day of month (01)
+%e day of month without padding (1)
+%D date; same as %m/%d/%y
+%H hour (00..23)
+%I hour (01..12)
+%j day of year (001..366)
+%k hour ( 0..23)
+%m month (01..12)
+%n month without padding (1..12)
+%M minute (00..59)
+%p locale's equivalent of am or pm
+%s seconds since 1970-01-01 00:00:00 UTC
+%S second (00..59)
+%y last two digits of year (00..99)
+%Y year (2010)
+%z +hhmm numeric timezone (for example, -0400)
+%[localhost] Substitute the hostname of the host where the agent is running
+%[IP] Substitute the IP address of the host where the agent is running
+%[FQDN] Substitute the canonical hostname of the host where the agent is running
+=============== =================================================
+
+Note: The escape strings %[localhost], %[IP] and %[FQDN] all rely on Java's ability to obtain the
+hostname, which may fail in some networking environments.
+
+The file in use will have the name mangled to include ".tmp" at the end. Once
+the file is closed, this extension is removed. This allows excluding partially
+complete files in the directory.
+Required properties are in **bold**.
+
+.. note:: For all of the time related escape sequences, a header with the key
+ "timestamp" must exist among the headers of the event (unless ``hdfs.useLocalTimeStamp`` is set to ``true``). One way to add
+ this automatically is to use the TimestampInterceptor.
+
+====================== ============ ======================================================================
+Name Default Description
+====================== ============ ======================================================================
+**channel** --
+**type** -- The component type name, needs to be ``hdfs``
+**hdfs.path** -- HDFS directory path (eg hdfs://namenode/flume/webdata/)
+hdfs.filePrefix FlumeData Name prefixed to files created by Flume in hdfs directory
+hdfs.fileSuffix -- Suffix to append to file (eg ``.avro`` - *NOTE: period is not automatically added*)
+hdfs.inUsePrefix -- Prefix that is used for temporal files that flume actively writes into
+hdfs.inUseSuffix ``.tmp`` Suffix that is used for temporal files that flume actively writes into
+hdfs.emptyInUseSuffix false If ``false`` an ``hdfs.inUseSuffix`` is used while writing the output. After closing the output ``hdfs.inUseSuffix`` is removed from the output file name. If ``true`` the ``hdfs.inUseSuffix`` parameter is ignored an empty string is used instead.
+hdfs.rollInterval 30 Number of seconds to wait before rolling current file
+ (0 = never roll based on time interval)
+hdfs.rollSize 1024 File size to trigger roll, in bytes (0: never roll based on file size)
+hdfs.rollCount 10 Number of events written to file before it rolled
+ (0 = never roll based on number of events)
+hdfs.idleTimeout 0 Timeout after which inactive files get closed
+ (0 = disable automatic closing of idle files)
+hdfs.batchSize 100 number of events written to file before it is flushed to HDFS
+hdfs.codeC -- Compression codec. one of following : gzip, bzip2, lzo, lzop, snappy
+hdfs.fileType SequenceFile File format: currently ``SequenceFile``, ``DataStream`` or ``CompressedStream``
+ (1)DataStream will not compress output file and please don't set codeC
+ (2)CompressedStream requires set hdfs.codeC with an available codeC
+hdfs.maxOpenFiles 5000 Allow only this number of open files. If this number is exceeded, the oldest file is closed.
+hdfs.minBlockReplicas -- Specify minimum number of replicas per HDFS block. If not specified, it comes from the default Hadoop config in the classpath.
+hdfs.writeFormat Writable Format for sequence file records. One of ``Text`` or ``Writable``. Set to ``Text`` before creating data files with Flume, otherwise those files cannot be read by either Apache Impala (incubating) or Apache Hive.
+hdfs.threadsPoolSize 10 Number of threads per HDFS sink for HDFS IO ops (open, write, etc.)
+hdfs.rollTimerPoolSize 1 Number of threads per HDFS sink for scheduling timed file rolling
+hdfs.kerberosPrincipal -- Kerberos user principal for accessing secure HDFS
+hdfs.kerberosKeytab -- Kerberos keytab for accessing secure HDFS
+hdfs.proxyUser
+hdfs.round false Should the timestamp be rounded down (if true, affects all time based escape sequences except %t)
+hdfs.roundValue 1 Rounded down to the highest multiple of this (in the unit configured using ``hdfs.roundUnit``), less than current time.
+hdfs.roundUnit second The unit of the round down value - ``second``, ``minute`` or ``hour``.
+hdfs.timeZone Local Time Name of the timezone that should be used for resolving the directory path, e.g. America/Los_Angeles.
+hdfs.useLocalTimeStamp false Use the local time (instead of the timestamp from the event header) while replacing the escape sequences.
+hdfs.closeTries 0 Number of times the sink must try renaming a file, after initiating a close attempt. If set to 1, this sink will not re-try a failed rename
+ (due to, for example, NameNode or DataNode failure), and may leave the file in an open state with a .tmp extension.
+ If set to 0, the sink will try to rename the file until the file is eventually renamed (there is no limit on the number of times it would try).
+ The file may still remain open if the close call fails but the data will be intact and in this case, the file will be closed only after a Flume restart.
+hdfs.retryInterval 180 Time in seconds between consecutive attempts to close a file. Each close call costs multiple RPC round-trips to the Namenode,
+ so setting this too low can cause a lot of load on the name node. If set to 0 or less, the sink will not
+ attempt to close the file if the first attempt fails, and may leave the file open or with a ".tmp" extension.
+serializer ``TEXT`` Other possible options include ``avro_event`` or the
+ fully-qualified class name of an implementation of the
+ ``EventSerializer.Builder`` interface.
+serializer.*
+====================== ============ ======================================================================
+
+Deprecated Properties
+
+Name Default Description
+====================== ============ ======================================================================
+hdfs.callTimeout 30000 Number of milliseconds allowed for HDFS operations, such as open, write, flush, close. This number should be increased if many HDFS timeout operations are occurring.
+====================== ============ ======================================================================
+
+Example for agent named a1:
+
+.. code-block:: properties
+
+ a1.channels = c1
+ a1.sinks = k1
+ a1.sinks.k1.type = hdfs
+ a1.sinks.k1.channel = c1
+ a1.sinks.k1.hdfs.path = /flume/events/%y-%m-%d/%H%M/%S
+ a1.sinks.k1.hdfs.filePrefix = events-
+ a1.sinks.k1.hdfs.round = true
+ a1.sinks.k1.hdfs.roundValue = 10
+ a1.sinks.k1.hdfs.roundUnit = minute
+
+The above configuration will round down the timestamp to the last 10th minute. For example, an event with
+timestamp 11:54:34 AM, June 12, 2012 will cause the hdfs path to become ``/flume/events/2012-06-12/1150/00``.
+
+
+Hive Sink
+~~~~~~~~~
+
+This sink streams events containing delimited text or JSON data directly into a Hive table or partition.
+Events are written using Hive transactions. As soon as a set of events are committed to Hive, they become
+immediately visible to Hive queries. Partitions to which flume will stream to can either be pre-created
+or, optionally, Flume can create them if they are missing. Fields from incoming event data are mapped to
+corresponding columns in the Hive table.
+
+====================== ============ ======================================================================
+Name Default Description
+====================== ============ ======================================================================
+**channel** --
+**type** -- The component type name, needs to be ``hive``
+**hive.metastore** -- Hive metastore URI (eg thrift://a.b.com:9083 )
+**hive.database** -- Hive database name
+**hive.table** -- Hive table name
+hive.partition -- Comma separate list of partition values identifying the partition to write to. May contain escape
+ sequences. E.g: If the table is partitioned by (continent: string, country :string, time : string)
+ then 'Asia,India,2014-02-26-01-21' will indicate continent=Asia,country=India,time=2014-02-26-01-21
+hive.txnsPerBatchAsk 100 Hive grants a *batch of transactions* instead of single transactions to streaming clients like Flume.
+ This setting configures the number of desired transactions per Transaction Batch. Data from all
+ transactions in a single batch end up in a single file. Flume will write a maximum of batchSize events
+ in each transaction in the batch. This setting in conjunction with batchSize provides control over the
+ size of each file. Note that eventually Hive will transparently compact these files into larger files.
+heartBeatInterval 240 (In seconds) Interval between consecutive heartbeats sent to Hive to keep unused transactions from expiring.
+ Set this value to 0 to disable heartbeats.
+autoCreatePartitions true Flume will automatically create the necessary Hive partitions to stream to
+batchSize 15000 Max number of events written to Hive in a single Hive transaction
+maxOpenConnections 500 Allow only this number of open connections. If this number is exceeded, the least recently used connection is closed.
+callTimeout 10000 (In milliseconds) Timeout for Hive & HDFS I/O operations, such as openTxn, write, commit, abort.
+**serializer** Serializer is responsible for parsing out field from the event and mapping them to columns in the hive table.
+ Choice of serializer depends upon the format of the data in the event. Supported serializers: DELIMITED and JSON
+roundUnit minute The unit of the round down value - ``second``, ``minute`` or ``hour``.
+roundValue 1 Rounded down to the highest multiple of this (in the unit configured using hive.roundUnit), less than current time
+timeZone Local Time Name of the timezone that should be used for resolving the escape sequences in partition, e.g. America/Los_Angeles.
+useLocalTimeStamp false Use the local time (instead of the timestamp from the event header) while replacing the escape sequences.
+====================== ============ ======================================================================
+
+Following serializers are provided for Hive sink:
+
+**JSON**: Handles UTF8 encoded Json (strict syntax) events and requires no configration. Object names
+in the JSON are mapped directly to columns with the same name in the Hive table.
+Internally uses org.apache.hive.hcatalog.data.JsonSerDe but is independent of the Serde of the Hive table.
+This serializer requires HCatalog to be installed.
+
+**DELIMITED**: Handles simple delimited textual events.
+Internally uses LazySimpleSerde but is independent of the Serde of the Hive table.
+
+========================== ============ ======================================================================
+Name Default Description
+========================== ============ ======================================================================
+serializer.delimiter , (Type: string) The field delimiter in the incoming data. To use special
+ characters, surround them with double quotes like "\\t"
+**serializer.fieldnames** -- The mapping from input fields to columns in hive table. Specified as a
+ comma separated list (no spaces) of hive table columns names, identifying
+ the input fields in order of their occurrence. To skip fields leave the
+ column name unspecified. Eg. 'time,,ip,message' indicates the 1st, 3rd
+ and 4th fields in input map to time, ip and message columns in the hive table.
+serializer.serdeSeparator Ctrl-A (Type: character) Customizes the separator used by underlying serde. There
+ can be a gain in efficiency if the fields in serializer.fieldnames are in
+ same order as table columns, the serializer.delimiter is same as the
+ serializer.serdeSeparator and number of fields in serializer.fieldnames
+ is less than or equal to number of table columns, as the fields in incoming
+ event body do not need to be reordered to match order of table columns.
+ Use single quotes for special characters like '\\t'.
+ Ensure input fields do not contain this character. NOTE: If serializer.delimiter
+ is a single character, preferably set this to the same character
+========================== ============ ======================================================================
+
+
+The following are the escape sequences supported:
+
+========= =================================================
+Alias Description
+========= =================================================
+%{host} Substitute value of event header named "host". Arbitrary header names are supported.
+%t Unix time in milliseconds
+%a locale's short weekday name (Mon, Tue, ...)
+%A locale's full weekday name (Monday, Tuesday, ...)
+%b locale's short month name (Jan, Feb, ...)
+%B locale's long month name (January, February, ...)
+%c locale's date and time (Thu Mar 3 23:05:25 2005)
+%d day of month (01)
+%D date; same as %m/%d/%y
+%H hour (00..23)
+%I hour (01..12)
+%j day of year (001..366)
+%k hour ( 0..23)
+%m month (01..12)
+%M minute (00..59)
+%p locale's equivalent of am or pm
+%s seconds since 1970-01-01 00:00:00 UTC
+%S second (00..59)
+%y last two digits of year (00..99)
+%Y year (2010)
+%z +hhmm numeric timezone (for example, -0400)
+========= =================================================
+
+
+.. note:: For all of the time related escape sequences, a header with the key
+ "timestamp" must exist among the headers of the event (unless ``useLocalTimeStamp`` is set to ``true``). One way to add
+ this automatically is to use the TimestampInterceptor.
+
+Example Hive table :
+
+.. code-block:: properties
+
+ create table weblogs ( id int , msg string )
+ partitioned by (continent string, country string, time string)
+ clustered by (id) into 5 buckets
+ stored as orc;
+
+Example for agent named a1:
+
+.. code-block:: properties
+
+ a1.channels = c1
+ a1.channels.c1.type = memory
+ a1.sinks = k1
+ a1.sinks.k1.type = hive
+ a1.sinks.k1.channel = c1
+ a1.sinks.k1.hive.metastore = thrift://127.0.0.1:9083
+ a1.sinks.k1.hive.database = logsdb
+ a1.sinks.k1.hive.table = weblogs
+ a1.sinks.k1.hive.partition = asia,%{country},%y-%m-%d-%H-%M
+ a1.sinks.k1.useLocalTimeStamp = false
+ a1.sinks.k1.round = true
+ a1.sinks.k1.roundValue = 10
+ a1.sinks.k1.roundUnit = minute
+ a1.sinks.k1.serializer = DELIMITED
+ a1.sinks.k1.serializer.delimiter = "\t"
+ a1.sinks.k1.serializer.serdeSeparator = '\t'
+ a1.sinks.k1.serializer.fieldnames =id,,msg
+
+
+The above configuration will round down the timestamp to the last 10th minute. For example, an event with
+timestamp header set to 11:54:34 AM, June 12, 2012 and 'country' header set to 'india' will evaluate to the
+partition (continent='asia',country='india',time='2012-06-12-11-50'. The serializer is configured to
+accept tab separated input containing three fields and to skip the second field.
+
+
+Logger Sink
+~~~~~~~~~~~
+
+Logs event at INFO level. Typically useful for testing/debugging purpose. Required properties are
+in **bold**. This sink is the only exception which doesn't require the extra configuration
+explained in the `Logging raw data`_ section.
+
+============== ======= ===========================================
+Property Name Default Description
+============== ======= ===========================================
+**channel** --
+**type** -- The component type name, needs to be ``logger``
+maxBytesToLog 16 Maximum number of bytes of the Event body to log
+============== ======= ===========================================
+
+Example for agent named a1:
+
+.. code-block:: properties
+
+ a1.channels = c1
+ a1.sinks = k1
+ a1.sinks.k1.type = logger
+ a1.sinks.k1.channel = c1
+
+Avro Sink
+~~~~~~~~~
+
+This sink forms one half of Flume's tiered collection support. Flume events
+sent to this sink are turned into Avro events and sent to the configured
+hostname / port pair. The events are taken from the configured Channel in
+batches of the configured batch size.
+Required properties are in **bold**.
+
+========================== ===================================================== ===========================================================================================
+Property Name Default Description
+========================== ===================================================== ===========================================================================================
+**channel** --
+**type** -- The component type name, needs to be ``avro``.
+**hostname** -- The hostname or IP address to bind to.
+**port** -- The port # to listen on.
+batch-size 100 number of event to batch together for send.
+connect-timeout 20000 Amount of time (ms) to allow for the first (handshake) request.
+request-timeout 20000 Amount of time (ms) to allow for requests after the first.
+reset-connection-interval none Amount of time (s) before the connection to the next hop is reset. This will force the Avro Sink to reconnect to the next hop. This will allow the sink to connect to hosts behind a hardware load-balancer when news hosts are added without having to restart the agent.
+compression-type none This can be "none" or "deflate". The compression-type must match the compression-type of matching AvroSource
+compression-level 6 The level of compression to compress event. 0 = no compression and 1-9 is compression. The higher the number the more compression
+ssl false Set to true to enable SSL for this AvroSink. When configuring SSL, you can optionally set a "truststore", "truststore-password", "truststore-type", and specify whether to "trust-all-certs".
+trust-all-certs false If this is set to true, SSL server certificates for remote servers (Avro Sources) will not be checked. This should NOT be used in production because it makes it easier for an attacker to execute a man-in-the-middle attack and "listen in" on the encrypted connection.
+truststore -- The path to a custom Java truststore file. Flume uses the certificate authority information in this file to determine whether the remote Avro Source's SSL authentication credentials should be trusted. If not specified, then the global keystore will be used. If the global keystore not specified either, then the default Java JSSE certificate authority files (typically "jssecacerts" or "cacerts" in the Oracl [...]
+truststore-password -- The password for the truststore. If not specified, then the global keystore password will be used (if defined).
+truststore-type JKS The type of the Java truststore. This can be "JKS" or other supported Java truststore type. If not specified, then the global keystore type will be used (if defined, otherwise the defautl is JKS).
+exclude-protocols SSLv3 Space-separated list of SSL/TLS protocols to exclude. SSLv3 will always be excluded in addition to the protocols specified.
+maxIoWorkers 2 * the number of available processors in the machine The maximum number of I/O worker threads. This is configured on the NettyAvroRpcClient NioClientSocketChannelFactory.
+========================== ===================================================== ===========================================================================================
+
+Example for agent named a1:
+
+.. code-block:: properties
+
+ a1.channels = c1
+ a1.sinks = k1
+ a1.sinks.k1.type = avro
+ a1.sinks.k1.channel = c1
+ a1.sinks.k1.hostname = 10.10.10.10
+ a1.sinks.k1.port = 4545
+
+Thrift Sink
+~~~~~~~~~~~
+
+This sink forms one half of Flume's tiered collection support. Flume events
+sent to this sink are turned into Thrift events and sent to the configured
+hostname / port pair. The events are taken from the configured Channel in
+batches of the configured batch size.
+
+Thrift sink can be configured to start in secure mode by enabling kerberos authentication.
+To communicate with a Thrift source started in secure mode, the Thrift sink should also
+operate in secure mode. client-principal and client-keytab are the properties used by the
+Thrift sink to authenticate to the kerberos KDC. The server-principal represents the
+principal of the Thrift source this sink is configured to connect to in secure mode.
+Required properties are in **bold**.
+
+========================== ======= ==============================================
+Property Name Default Description
+========================== ======= ==============================================
+**channel** --
+**type** -- The component type name, needs to be ``thrift``.
+**hostname** -- The hostname or IP address to bind to.
+**port** -- The port # to listen on.
+batch-size 100 number of event to batch together for send.
+connect-timeout 20000 Amount of time (ms) to allow for the first (handshake) request.
+request-timeout 20000 Amount of time (ms) to allow for requests after the first.
+connection-reset-interval none Amount of time (s) before the connection to the next hop is reset. This will force the Thrift Sink to reconnect to the next hop. This will allow the sink to connect to hosts behind a hardware load-balancer when news hosts are added without having to restart the agent.
+ssl false Set to true to enable SSL for this ThriftSink. When configuring SSL, you can optionally set a "truststore", "truststore-password" and "truststore-type"
+truststore -- The path to a custom Java truststore file. Flume uses the certificate authority information in this file to determine whether the remote Thrift Source's SSL authentication credentials should be trusted. If not specified, then the global keystore will be used. If the global keystore not specified either, then the default Java JSSE certificate authority files (typically "jssecacerts" or "cacerts" in the Oracle JRE) will be used.
+truststore-password -- The password for the truststore. If not specified, then the global keystore password will be used (if defined).
+truststore-type JKS The type of the Java truststore. This can be "JKS" or other supported Java truststore type. If not specified, then the global keystore type will be used (if defined, otherwise the defautl is JKS).
+exclude-protocols SSLv3 Space-separated list of SSL/TLS protocols to exclude
+kerberos false Set to true to enable kerberos authentication. In kerberos mode, client-principal, client-keytab and server-principal are required for successful authentication and communication to a kerberos enabled Thrift Source.
+client-principal —- The kerberos principal used by the Thrift Sink to authenticate to the kerberos KDC.
+client-keytab —- The keytab location used by the Thrift Sink in combination with the client-principal to authenticate to the kerberos KDC.
+server-principal -- The kerberos principal of the Thrift Source to which the Thrift Sink is configured to connect to.
+========================== ======= ==============================================
+
+Example for agent named a1:
+
+.. code-block:: properties
+
+ a1.channels = c1
+ a1.sinks = k1
+ a1.sinks.k1.type = thrift
+ a1.sinks.k1.channel = c1
+ a1.sinks.k1.hostname = 10.10.10.10
+ a1.sinks.k1.port = 4545
+
+IRC Sink
+~~~~~~~~
+
+The IRC sink takes messages from attached channel and relays those to
+configured IRC destinations.
+Required properties are in **bold**.
+
+=============== ======= ========================================================
+Property Name Default Description
+=============== ======= ========================================================
+**channel** --
+**type** -- The component type name, needs to be ``irc``
+**hostname** -- The hostname or IP address to connect to
+port 6667 The port number of remote host to connect
+**nick** -- Nick name
+user -- User name
+password -- User password
+**chan** -- channel
+name
+splitlines -- (boolean)
+splitchars \n line separator (if you were to enter the default value
+ into the config file, then you would need to escape the
+ backslash, like this: "\\n")
+=============== ======= ========================================================
+
+Example for agent named a1:
+
+.. code-block:: properties
+
+ a1.channels = c1
+ a1.sinks = k1
+ a1.sinks.k1.type = irc
+ a1.sinks.k1.channel = c1
+ a1.sinks.k1.hostname = irc.yourdomain.com
+ a1.sinks.k1.nick = flume
+ a1.sinks.k1.chan = #flume
+
+File Roll Sink
+~~~~~~~~~~~~~~
+
+Stores events on the local filesystem.
+Required properties are in **bold**.
+
+========================== ======= ======================================================================================================================
+Property Name Default Description
+========================== ======= ======================================================================================================================
+**channel** --
+**type** -- The component type name, needs to be ``file_roll``.
+**sink.directory** -- The directory where files will be stored
+sink.pathManager DEFAULT The PathManager implementation to use.
+sink.pathManager.extension -- The file extension if the default PathManager is used.
+sink.pathManager.prefix -- A character string to add to the beginning of the file name if the default PathManager is used
+sink.rollInterval 30 Roll the file every 30 seconds. Specifying 0 will disable rolling and cause all events to be written to a single file.
+sink.serializer TEXT Other possible options include ``avro_event`` or the FQCN of an implementation of EventSerializer.Builder interface.
+sink.batchSize 100
+========================== ======= ======================================================================================================================
+
+Example for agent named a1:
+
+.. code-block:: properties
+
+ a1.channels = c1
+ a1.sinks = k1
+ a1.sinks.k1.type = file_roll
+ a1.sinks.k1.channel = c1
+ a1.sinks.k1.sink.directory = /var/log/flume
+
+Null Sink
+~~~~~~~~~
+
+Discards all events it receives from the channel.
+Required properties are in **bold**.
+
+============= ======= ==============================================
+Property Name Default Description
+============= ======= ==============================================
+**channel** --
+**type** -- The component type name, needs to be ``null``.
+batchSize 100
+============= ======= ==============================================
+
+Example for agent named a1:
+
+.. code-block:: properties
+
+ a1.channels = c1
+ a1.sinks = k1
+ a1.sinks.k1.type = null
+ a1.sinks.k1.channel = c1
+
+HBaseSinks
+~~~~~~~~~~
+
+HBaseSink
+'''''''''
+
+This sink writes data to HBase. The Hbase configuration is picked up from the first
+hbase-site.xml encountered in the classpath. A class implementing HbaseEventSerializer
+which is specified by the configuration is used to convert the events into
+HBase puts and/or increments. These puts and increments are then written
+to HBase. This sink provides the same consistency guarantees as HBase,
+which is currently row-wise atomicity. In the event of Hbase failing to
+write certain events, the sink will replay all events in that transaction.
+
+The HBaseSink supports writing data to secure HBase. To write to secure HBase, the user
+the agent is running as must have write permissions to the table the sink is configured
+to write to. The principal and keytab to use to authenticate against the KDC can be specified
+in the configuration. The hbase-site.xml in the Flume agent's classpath
+must have authentication set to ``kerberos`` (For details on how to do this, please refer to
+HBase documentation).
+
+For convenience, two serializers are provided with Flume. The
+SimpleHbaseEventSerializer (org.apache.flume.sink.hbase.SimpleHbaseEventSerializer)
+writes the event body
+as-is to HBase, and optionally increments a column in Hbase. This is primarily
+an example implementation. The RegexHbaseEventSerializer
+(org.apache.flume.sink.hbase.RegexHbaseEventSerializer) breaks the event body
+based on the given regex and writes each part into different columns.
+
+The type is the FQCN: org.apache.flume.sink.hbase.HBaseSink.
+
+Required properties are in **bold**.
+
+================== ====================================================== ==============================================================================
+Property Name Default Description
+================== ====================================================== ==============================================================================
+**channel** --
+**type** -- The component type name, needs to be ``hbase``
+**table** -- The name of the table in Hbase to write to.
+**columnFamily** -- The column family in Hbase to write to.
+zookeeperQuorum -- The quorum spec. This is the value for the property ``hbase.zookeeper.quorum`` in hbase-site.xml
+znodeParent /hbase The base path for the znode for the -ROOT- region. Value of ``zookeeper.znode.parent`` in hbase-site.xml
+batchSize 100 Number of events to be written per txn.
+coalesceIncrements false Should the sink coalesce multiple increments to a cell per batch. This might give
+ better performance if there are multiple increments to a limited number of cells.
+serializer org.apache.flume.sink.hbase.SimpleHbaseEventSerializer Default increment column = "iCol", payload column = "pCol".
+serializer.* -- Properties to be passed to the serializer.
+kerberosPrincipal -- Kerberos user principal for accessing secure HBase
+kerberosKeytab -- Kerberos keytab for accessing secure HBase
+================== ====================================================== ==============================================================================
+
+Example for agent named a1:
+
+.. code-block:: properties
+
+ a1.channels = c1
+ a1.sinks = k1
+ a1.sinks.k1.type = hbase
+ a1.sinks.k1.table = foo_table
+ a1.sinks.k1.columnFamily = bar_cf
+ a1.sinks.k1.serializer = org.apache.flume.sink.hbase.RegexHbaseEventSerializer
+ a1.sinks.k1.channel = c1
+
+HBase2Sink
+''''''''''
+
+HBase2Sink is the equivalent of HBaseSink for HBase version 2.
+The provided functionality and the configuration parameters are the same as in case of HBaseSink (except the hbase2 tag in the sink type and the package/class names).
+
+The type is the FQCN: org.apache.flume.sink.hbase2.HBase2Sink.
+
+Required properties are in **bold**.
+
+================== ======================================================== ==============================================================================
+Property Name Default Description
+================== ======================================================== ==============================================================================
+**channel** --
+**type** -- The component type name, needs to be ``hbase2``
+**table** -- The name of the table in HBase to write to.
+**columnFamily** -- The column family in HBase to write to.
+zookeeperQuorum -- The quorum spec. This is the value for the property ``hbase.zookeeper.quorum`` in hbase-site.xml
+znodeParent /hbase The base path for the znode for the -ROOT- region. Value of ``zookeeper.znode.parent`` in hbase-site.xml
+batchSize 100 Number of events to be written per txn.
+coalesceIncrements false Should the sink coalesce multiple increments to a cell per batch. This might give
+ better performance if there are multiple increments to a limited number of cells.
+serializer org.apache.flume.sink.hbase2.SimpleHBase2EventSerializer Default increment column = "iCol", payload column = "pCol".
+serializer.* -- Properties to be passed to the serializer.
+kerberosPrincipal -- Kerberos user principal for accessing secure HBase
+kerberosKeytab -- Kerberos keytab for accessing secure HBase
+================== ======================================================== ==============================================================================
+
+Example for agent named a1:
+
+.. code-block:: properties
+
+ a1.channels = c1
+ a1.sinks = k1
+ a1.sinks.k1.type = hbase2
+ a1.sinks.k1.table = foo_table
+ a1.sinks.k1.columnFamily = bar_cf
+ a1.sinks.k1.serializer = org.apache.flume.sink.hbase2.RegexHBase2EventSerializer
+ a1.sinks.k1.channel = c1
+
+AsyncHBaseSink
+''''''''''''''
+
+This sink writes data to HBase using an asynchronous model. A class implementing
+AsyncHbaseEventSerializer which is specified by the configuration is used to convert the events into
+HBase puts and/or increments. These puts and increments are then written
+to HBase. This sink uses the `Asynchbase API <https://github.com/OpenTSDB/asynchbase>`_ to write to
+HBase. This sink provides the same consistency guarantees as HBase,
+which is currently row-wise atomicity. In the event of Hbase failing to
+write certain events, the sink will replay all events in that transaction.
+AsyncHBaseSink can only be used with HBase 1.x. The async client library used by AsyncHBaseSink is not available for HBase 2.
+The type is the FQCN: org.apache.flume.sink.hbase.AsyncHBaseSink.
+Required properties are in **bold**.
+
+=================== ============================================================ ====================================================================================
+Property Name Default Description
+=================== ============================================================ ====================================================================================
+**channel** --
+**type** -- The component type name, needs to be ``asynchbase``
+**table** -- The name of the table in Hbase to write to.
+zookeeperQuorum -- The quorum spec. This is the value for the property ``hbase.zookeeper.quorum`` in hbase-site.xml
+znodeParent /hbase The base path for the znode for the -ROOT- region. Value of ``zookeeper.znode.parent`` in hbase-site.xml
+**columnFamily** -- The column family in Hbase to write to.
+batchSize 100 Number of events to be written per txn.
+coalesceIncrements false Should the sink coalesce multiple increments to a cell per batch. This might give
+ better performance if there are multiple increments to a limited number of cells.
+timeout 60000 The length of time (in milliseconds) the sink waits for acks from hbase for
+ all events in a transaction.
+serializer org.apache.flume.sink.hbase.SimpleAsyncHbaseEventSerializer
+serializer.* -- Properties to be passed to the serializer.
+async.* -- Properties to be passed to asyncHbase library.
+ These properties have precedence over the old ``zookeeperQuorum`` and ``znodeParent`` values.
+ You can find the list of the available properties at
+ `the documentation page of AsyncHBase <http://opentsdb.github.io/asynchbase/docs/build/html/configuration.html#properties>`_.
+=================== ============================================================ ====================================================================================
+
+Note that this sink takes the Zookeeper Quorum and parent znode information in
+the configuration. Zookeeper Quorum and parent node configuration may be
+specified in the flume configuration file. Alternatively, these configuration
+values are taken from the first hbase-site.xml file in the classpath.
+
+If these are not provided in the configuration, then the sink
+will read this information from the first hbase-site.xml file in the classpath.
+
+Example for agent named a1:
+
+.. code-block:: properties
+
+ a1.channels = c1
+ a1.sinks = k1
+ a1.sinks.k1.type = asynchbase
+ a1.sinks.k1.table = foo_table
+ a1.sinks.k1.columnFamily = bar_cf
+ a1.sinks.k1.serializer = org.apache.flume.sink.hbase.SimpleAsyncHbaseEventSerializer
+ a1.sinks.k1.channel = c1
+
+MorphlineSolrSink
+~~~~~~~~~~~~~~~~~
+
+This sink extracts data from Flume events, transforms it, and loads it in near-real-time into Apache Solr servers, which in turn serve queries to end users or search applications.
+
+This sink is well suited for use cases that stream raw data into HDFS (via the HdfsSink) and simultaneously extract, transform and load the same data into Solr (via MorphlineSolrSink). In particular, this sink can process arbitrary heterogeneous raw data from disparate data sources and turn it into a data model that is useful to Search applications.
+
+The ETL functionality is customizable using a `morphline configuration file <http://cloudera.github.io/cdk/docs/current/cdk-morphlines/index.html>`_ that defines a chain of transformation commands that pipe event records from one command to another.
+
+Morphlines can be seen as an evolution of Unix pipelines where the data model is generalized to work with streams of generic records, including arbitrary binary payloads. A morphline command is a bit like a Flume Interceptor. Morphlines can be embedded into Hadoop components such as Flume.
+
+Commands to parse and transform a set of standard data formats such as log files, Avro, CSV, Text, HTML, XML, PDF, Word, Excel, etc. are provided out of the box, and additional custom commands and parsers for additional data formats can be added as morphline plugins. Any kind of data format can be indexed and any Solr documents for any kind of Solr schema can be generated, and any custom ETL logic can be registered and executed.
+
+Morphlines manipulate continuous streams of records. The data model can be described as follows: A record is a set of named fields where each field has an ordered list of one or more values. A value can be any Java Object. That is, a record is essentially a hash table where each hash table entry contains a String key and a list of Java Objects as values. (The implementation uses Guava's ``ArrayListMultimap``, which is a ``ListMultimap``). Note that a field can have multiple values and an [...]
+
+This sink fills the body of the Flume event into the ``_attachment_body`` field of the morphline record, as well as copies the headers of the Flume event into record fields of the same name. The commands can then act on this data.
+
+Routing to a SolrCloud cluster is supported to improve scalability. Indexing load can be spread across a large number of MorphlineSolrSinks for improved scalability. Indexing load can be replicated across multiple MorphlineSolrSinks for high availability, for example using Flume features such as Load balancing Sink Processor. MorphlineInterceptor can also help to implement dynamic routing to multiple Solr collections (e.g. for multi-tenancy).
+
+The morphline and solr jars required for your environment must be placed in the lib directory of the Apache Flume installation.
+
+The type is the FQCN: org.apache.flume.sink.solr.morphline.MorphlineSolrSink
+
+Required properties are in **bold**.
+
+=============================== ======================================================================= ========================
+Property Name Default Description
+=============================== ======================================================================= ========================
+**channel** --
+**type** -- The component type name, needs to be ``org.apache.flume.sink.solr.morphline.MorphlineSolrSink``
+**morphlineFile** -- The relative or absolute path on the local file system to the morphline configuration file. Example: ``/etc/flume-ng/conf/morphline.conf``
+morphlineId null Optional name used to identify a morphline if there are multiple morphlines in a morphline config file
+batchSize 1000 The maximum number of events to take per flume transaction.
+batchDurationMillis 1000 The maximum duration per flume transaction (ms). The transaction commits after this duration or when batchSize is exceeded, whichever comes first.
+handlerClass org.apache.flume.sink.solr.morphline.MorphlineHandlerImpl The FQCN of a class implementing org.apache.flume.sink.solr.morphline.MorphlineHandler
+isProductionMode false This flag should be enabled for mission critical, large-scale online production systems that need to make progress without downtime when unrecoverable exceptions occur. Corrupt or malformed parser input data, parser bugs, and errors related to unknown Solr schema fields produce unrecoverable exceptions.
+recoverableExceptionClasses org.apache.solr.client.solrj.SolrServerException Comma separated list of recoverable exceptions that tend to be transient, in which case the corresponding task can be retried. Examples include network connection errors, timeouts, etc. When the production mode flag is set to true, the recoverable exceptions configured using this parameter will not be ignored and hence will lead to retries.
+isIgnoringRecoverableExceptions false This flag should be enabled, if an unrecoverable exception is accidentally misclassified as recoverable. This enables the sink to make progress and avoid retrying an event forever.
+=============================== ======================================================================= ========================
+
+Example for agent named a1:
+
+.. code-block:: properties
+
+ a1.channels = c1
+ a1.sinks = k1
+ a1.sinks.k1.type = org.apache.flume.sink.solr.morphline.MorphlineSolrSink
+ a1.sinks.k1.channel = c1
+ a1.sinks.k1.morphlineFile = /etc/flume-ng/conf/morphline.conf
+ # a1.sinks.k1.morphlineId = morphline1
+ # a1.sinks.k1.batchSize = 1000
+ # a1.sinks.k1.batchDurationMillis = 1000
+
+ElasticSearchSink
+~~~~~~~~~~~~~~~~~
+
+This sink writes data to an elasticsearch cluster. By default, events will be written so that the `Kibana <http://kibana.org>`_ graphical interface
+can display them - just as if `logstash <https://logstash.net>`_ wrote them.
+
+The elasticsearch and lucene-core jars required for your environment must be placed in the lib directory of the Apache Flume installation.
+Elasticsearch requires that the major version of the client JAR match that of the server and that both are running the same minor version
+of the JVM. SerializationExceptions will appear if this is incorrect. To
+select the required version first determine the version of elasticsearch and the JVM version the target cluster is running. Then select an elasticsearch client
+library which matches the major version. A 0.19.x client can talk to a 0.19.x cluster; 0.20.x can talk to 0.20.x and 0.90.x can talk to 0.90.x. Once the
+elasticsearch version has been determined then read the pom.xml file to determine the correct lucene-core JAR version to use. The Flume agent
+which is running the ElasticSearchSink should also match the JVM the target cluster is running down to the minor version.
+
+Events will be written to a new index every day. The name will be <indexName>-yyyy-MM-dd where <indexName> is the indexName parameter. The sink
+will start writing to a new index at midnight UTC.
+
+Events are serialized for elasticsearch by the ElasticSearchLogStashEventSerializer by default. This behaviour can be
+overridden with the serializer parameter. This parameter accepts implementations of org.apache.flume.sink.elasticsearch.ElasticSearchEventSerializer
+or org.apache.flume.sink.elasticsearch.ElasticSearchIndexRequestBuilderFactory. Implementing ElasticSearchEventSerializer is deprecated in favour of
+the more powerful ElasticSearchIndexRequestBuilderFactory.
+
+The type is the FQCN: org.apache.flume.sink.elasticsearch.ElasticSearchSink
+
+Required properties are in **bold**.
+
+================ ======================================================================== =======================================================================================================
+Property Name Default Description
+================ ======================================================================== =======================================================================================================
+**channel** --
+**type** -- The component type name, needs to be ``org.apache.flume.sink.elasticsearch.ElasticSearchSink``
+**hostNames** -- Comma separated list of hostname:port, if the port is not present the default port '9300' will be used
+indexName flume The name of the index which the date will be appended to. Example 'flume' -> 'flume-yyyy-MM-dd'
+ Arbitrary header substitution is supported, eg. %{header} replaces with value of named event header
+indexType logs The type to index the document to, defaults to 'log'
+ Arbitrary header substitution is supported, eg. %{header} replaces with value of named event header
+clusterName elasticsearch Name of the ElasticSearch cluster to connect to
+batchSize 100 Number of events to be written per txn.
+ttl -- TTL in days, when set will cause the expired documents to be deleted automatically,
+ if not set documents will never be automatically deleted. TTL is accepted both in the earlier form of
+ integer only e.g. a1.sinks.k1.ttl = 5 and also with a qualifier ms (millisecond), s (second), m (minute),
+ h (hour), d (day) and w (week). Example a1.sinks.k1.ttl = 5d will set TTL to 5 days. Follow
+ http://www.elasticsearch.org/guide/reference/mapping/ttl-field/ for more information.
+serializer org.apache.flume.sink.elasticsearch.ElasticSearchLogStashEventSerializer The ElasticSearchIndexRequestBuilderFactory or ElasticSearchEventSerializer to use. Implementations of
+ either class are accepted but ElasticSearchIndexRequestBuilderFactory is preferred.
+serializer.* -- Properties to be passed to the serializer.
+================ ======================================================================== =======================================================================================================
+
+.. note:: Header substitution is a handy to use the value of an event header to dynamically decide the indexName and indexType to use when storing the event.
+ Caution should be used in using this feature as the event submitter now has control of the indexName and indexType.
+ Furthermore, if the elasticsearch REST client is used then the event submitter has control of the URL path used.
+
+Example for agent named a1:
+
+.. code-block:: properties
+
+ a1.channels = c1
+ a1.sinks = k1
+ a1.sinks.k1.type = elasticsearch
+ a1.sinks.k1.hostNames = 127.0.0.1:9200,127.0.0.2:9300
+ a1.sinks.k1.indexName = foo_index
+ a1.sinks.k1.indexType = bar_type
+ a1.sinks.k1.clusterName = foobar_cluster
+ a1.sinks.k1.batchSize = 500
+ a1.sinks.k1.ttl = 5d
+ a1.sinks.k1.serializer = org.apache.flume.sink.elasticsearch.ElasticSearchDynamicSerializer
+ a1.sinks.k1.channel = c1
+
+Kite Dataset Sink
+~~~~~~~~~~~~~~~~~
+
+Experimental sink that writes events to a `Kite Dataset <http://kitesdk.org/docs/current/guide/>`_.
+This sink will deserialize the body of each incoming event and store the
+resulting record in a Kite Dataset. It determines target Dataset by loading a
+dataset by URI.
+
+The only supported serialization is avro, and the record schema must be passed
+in the event headers, using either ``flume.avro.schema.literal`` with the JSON
+schema representation or ``flume.avro.schema.url`` with a URL where the schema
+may be found (``hdfs:/...`` URIs are supported). This is compatible with the
+Log4jAppender flume client and the spooling directory source's Avro
+deserializer using ``deserializer.schemaType = LITERAL``.
+
+Note 1: The ``flume.avro.schema.hash`` header is **not supported**.
+Note 2: In some cases, file rolling may occur slightly after the roll interval
+has been exceeded. However, this delay will not exceed 5 seconds. In most
+cases, the delay is neglegible.
+
+============================ ======= ===========================================================
+Property Name Default Description
+============================ ======= ===========================================================
+**channel** --
+**type** -- Must be org.apache.flume.sink.kite.DatasetSink
+**kite.dataset.uri** -- URI of the dataset to open
+kite.repo.uri -- URI of the repository to open
+ (deprecated; use kite.dataset.uri instead)
+kite.dataset.namespace -- Namespace of the Dataset where records will be written
+ (deprecated; use kite.dataset.uri instead)
+kite.dataset.name -- Name of the Dataset where records will be written
+ (deprecated; use kite.dataset.uri instead)
+kite.batchSize 100 Number of records to process in each batch
+kite.rollInterval 30 Maximum wait time (seconds) before data files are released
+kite.flushable.commitOnBatch true If ``true``, the Flume transaction will be commited and the
+ writer will be flushed on each batch of ``kite.batchSize``
+ records. This setting only applies to flushable datasets. When
+ ``true``, it's possible for temp files with commited data to be
+ left in the dataset directory. These files need to be recovered
+ by hand for the data to be visible to DatasetReaders.
+kite.syncable.syncOnBatch true Controls whether the sink will also sync data when committing
+ the transaction. This setting only applies to syncable datasets.
+ Syncing gaurentees that data will be written on stable storage
+ on the remote system while flushing only gaurentees that data
+ has left Flume's client buffers. When the
+ ``kite.flushable.commitOnBatch`` property is set to ``false``,
+ this property must also be set to ``false``.
+kite.entityParser avro Parser that turns Flume ``Events`` into Kite entities.
+ Valid values are ``avro`` and the fully-qualified class name
+ of an implementation of the ``EntityParser.Builder`` interface.
+kite.failurePolicy retry Policy that handles non-recoverable errors such as a missing
+ ``Schema`` in the ``Event`` header. The default value, ``retry``,
+ will fail the current batch and try again which matches the old
+ behavior. Other valid values are ``save``, which will write the
+ raw ``Event`` to the ``kite.error.dataset.uri`` dataset, and the
+ fully-qualified class name of an implementation of the
+ ``FailurePolicy.Builder`` interface.
+kite.error.dataset.uri -- URI of the dataset where failed events are saved when
+ ``kite.failurePolicy`` is set to ``save``. **Required** when
+ the ``kite.failurePolicy`` is set to ``save``.
+auth.kerberosPrincipal -- Kerberos user principal for secure authentication to HDFS
+auth.kerberosKeytab -- Kerberos keytab location (local FS) for the principal
+auth.proxyUser -- The effective user for HDFS actions, if different from
+ the kerberos principal
+============================ ======= ===========================================================
+
+
+Kafka Sink
+~~~~~~~~~~
+This is a Flume Sink implementation that can publish data to a
+`Kafka <http://kafka.apache.org/>`_ topic. One of the objective is to integrate Flume
+with Kafka so that pull based processing systems can process the data coming
+through various Flume sources.
+
+This currently supports Kafka server releases 0.10.1.0 or higher. Testing was done up to 2.0.1 that was the highest avilable version at the time of the release.
+
+Required properties are marked in bold font.
+
+
+================================== =================== =============================================================================================
+Property Name Default Description
+================================== =================== =============================================================================================
+**type** -- Must be set to ``org.apache.flume.sink.kafka.KafkaSink``
+**kafka.bootstrap.servers** -- List of brokers Kafka-Sink will connect to, to get the list of topic partitions
+ This can be a partial list of brokers, but we recommend at least two for HA.
+ The format is comma separated list of hostname:port
+kafka.topic default-flume-topic The topic in Kafka to which the messages will be published. If this parameter is configured,
+ messages will be published to this topic.
+ If the event header contains a "topic" field, the event will be published to that topic
+ overriding the topic configured here.
+ Arbitrary header substitution is supported, eg. %{header} is replaced with value of event header named "header".
+ (If using the substitution, it is recommended to set "auto.create.topics.enable" property of Kafka broker to true.)
+flumeBatchSize 100 How many messages to process in one batch. Larger batches improve throughput while adding latency.
+kafka.producer.acks 1 How many replicas must acknowledge a message before its considered successfully written.
+ Accepted values are 0 (Never wait for acknowledgement), 1 (wait for leader only), -1 (wait for all replicas)
+ Set this to -1 to avoid data loss in some cases of leader failure.
+useFlumeEventFormat false By default events are put as bytes onto the Kafka topic directly from the event body. Set to
+ true to store events as the Flume Avro binary format. Used in conjunction with the same property
+ on the KafkaSource or with the parseAsFlumeEvent property on the Kafka Channel this will preserve
+ any Flume headers for the producing side.
+defaultPartitionId -- Specifies a Kafka partition ID (integer) for all events in this channel to be sent to, unless
+ overriden by ``partitionIdHeader``. By default, if this property is not set, events will be
+ distributed by the Kafka Producer's partitioner - including by ``key`` if specified (or by a
+ partitioner specified by ``kafka.partitioner.class``).
+partitionIdHeader -- When set, the sink will take the value of the field named using the value of this property
+ from the event header and send the message to the specified partition of the topic. If the
+ value represents an invalid partition, an EventDeliveryException will be thrown. If the header value
+ is present then this setting overrides ``defaultPartitionId``.
+allowTopicOverride true When set, the sink will allow a message to be produced into a topic specified by the ``topicHeader`` property (if provided).
+topicHeader topic When set in conjunction with ``allowTopicOverride`` will produce a message into the value of the header named using the value of this property.
+ Care should be taken when using in conjunction with the Kafka Source ``topicHeader`` property to avoid creating a loopback.
+kafka.producer.security.protocol PLAINTEXT Set to SASL_PLAINTEXT, SASL_SSL or SSL if writing to Kafka using some level of security. See below for additional info on secure setup.
+*more producer security props* If using SASL_PLAINTEXT, SASL_SSL or SSL refer to `Kafka security <http://kafka.apache.org/documentation.html#security>`_ for additional
+ properties that need to be set on producer.
+Other Kafka Producer Properties -- These properties are used to configure the Kafka Producer. Any producer property supported
+ by Kafka can be used. The only requirement is to prepend the property name with the prefix
+ ``kafka.producer``.
+ For example: kafka.producer.linger.ms
+================================== =================== =============================================================================================
+
+.. note:: Kafka Sink uses the ``topic`` and ``key`` properties from the FlumeEvent headers to send events to Kafka.
+ If ``topic`` exists in the headers, the event will be sent to that specific topic, overriding the topic configured for the Sink.
+ If ``key`` exists in the headers, the key will used by Kafka to partition the data between the topic partitions. Events with same key
+ will be sent to the same partition. If the key is null, events will be sent to random partitions.
+
+The Kafka sink also provides defaults for the key.serializer(org.apache.kafka.common.serialization.StringSerializer)
+and value.serializer(org.apache.kafka.common.serialization.ByteArraySerializer). Modification of these parameters is not recommended.
+
+Deprecated Properties
+
+=============================== =================== =============================================================================================
+Property Name Default Description
+=============================== =================== =============================================================================================
+brokerList -- Use kafka.bootstrap.servers
+topic default-flume-topic Use kafka.topic
+batchSize 100 Use kafka.flumeBatchSize
+requiredAcks 1 Use kafka.producer.acks
+
+=============================== =================== =============================================================================================
+
+An example configuration of a Kafka sink is given below. Properties starting
+with the prefix ``kafka.producer`` the Kafka producer. The properties that are passed when creating the Kafka
+producer are not limited to the properties given in this example.
+Also it is possible to include your custom properties here and access them inside
+the preprocessor through the Flume Context object passed in as a method
+argument.
+
+.. code-block:: properties
+
+ a1.sinks.k1.channel = c1
+ a1.sinks.k1.type = org.apache.flume.sink.kafka.KafkaSink
+ a1.sinks.k1.kafka.topic = mytopic
+ a1.sinks.k1.kafka.bootstrap.servers = localhost:9092
+ a1.sinks.k1.kafka.flumeBatchSize = 20
+ a1.sinks.k1.kafka.producer.acks = 1
+ a1.sinks.k1.kafka.producer.linger.ms = 1
+ a1.sinks.k1.kafka.producer.compression.type = snappy
+
+
+**Security and Kafka Sink:**
+
+Secure authentication as well as data encryption is supported on the communication channel between Flume and Kafka.
+For secure authentication SASL/GSSAPI (Kerberos V5) or SSL (even though the parameter is named SSL, the actual protocol is a TLS implementation) can be used from Kafka version 0.9.0.
+
+As of now data encryption is solely provided by SSL/TLS.
+
+Setting ``kafka.producer.security.protocol`` to any of the following value means:
+
+- **SASL_PLAINTEXT** - Kerberos or plaintext authentication with no data encryption
+- **SASL_SSL** - Kerberos or plaintext authentication with data encryption
+- **SSL** - TLS based encryption with optional authentication.
+
+.. warning::
+ There is a performance degradation when SSL is enabled,
+ the magnitude of which depends on the CPU type and the JVM implementation.
+ Reference: `Kafka security overview <http://kafka.apache.org/documentation#security_overview>`_
+ and the jira for tracking this issue:
+ `KAFKA-2561 <https://issues.apache.org/jira/browse/KAFKA-2561>`__
+
+
+**TLS and Kafka Sink:**
+
+Please read the steps described in `Configuring Kafka Clients SSL <http://kafka.apache.org/documentation#security_configclients>`_
+to learn about additional configuration settings for fine tuning for example any of the following:
+security provider, cipher suites, enabled protocols, truststore or keystore types.
+
+Example configuration with server side authentication and data encryption.
+
+.. code-block:: properties
+
+ a1.sinks.sink1.type = org.apache.flume.sink.kafka.KafkaSink
+ a1.sinks.sink1.kafka.bootstrap.servers = kafka-1:9093,kafka-2:9093,kafka-3:9093
+ a1.sinks.sink1.kafka.topic = mytopic
+ a1.sinks.sink1.kafka.producer.security.protocol = SSL
+ # optional, the global truststore can be used alternatively
+ a1.sinks.sink1.kafka.producer.ssl.truststore.location = /path/to/truststore.jks
+ a1.sinks.sink1.kafka.producer.ssl.truststore.password = <password to access the truststore>
+
+Specyfing the truststore is optional here, the global truststore can be used instead.
+For more details about the global SSL setup, see the `SSL/TLS support`_ section.
+
+Note: By default the property ``ssl.endpoint.identification.algorithm``
+is not defined, so hostname verification is not performed.
+In order to enable hostname verification, set the following properties
+
+.. code-block:: properties
+
+ a1.sinks.sink1.kafka.producer.ssl.endpoint.identification.algorithm = HTTPS
+
+Once enabled, clients will verify the server's fully qualified domain name (FQDN)
+against one of the following two fields:
+
+#) Common Name (CN) https://tools.ietf.org/html/rfc6125#section-2.3
+#) Subject Alternative Name (SAN) https://tools.ietf.org/html/rfc5280#section-4.2.1.6
+
+If client side authentication is also required then additionally the following needs to be added to Flume agent
+configuration or the global SSL setup can be used (see `SSL/TLS support`_ section).
+Each Flume agent has to have its client certificate which has to be trusted by Kafka brokers either
+individually or by their signature chain. Common example is to sign each client certificate by a single Root CA
+which in turn is trusted by Kafka brokers.
+
+.. code-block:: properties
+
+ # optional, the global keystore can be used alternatively
+ a1.sinks.sink1.kafka.producer.ssl.keystore.location = /path/to/client.keystore.jks
+ a1.sinks.sink1.kafka.producer.ssl.keystore.password = <password to access the keystore>
+
+If keystore and key use different password protection then ``ssl.key.password`` property will
+provide the required additional secret for producer keystore:
+
+.. code-block:: properties
+
+ a1.sinks.sink1.kafka.producer.ssl.key.password = <password to access the key>
+
+
+**Kerberos and Kafka Sink:**
+
+To use Kafka sink with a Kafka cluster secured with Kerberos, set the ``producer.security.protocol`` property noted above for producer.
+The Kerberos keytab and principal to be used with Kafka brokers is specified in a JAAS file's "KafkaClient" section. "Client" section describes the Zookeeper connection if needed.
+See `Kafka doc <http://kafka.apache.org/documentation.html#security_sasl_clientconfig>`_
+for information on the JAAS file contents. The location of this JAAS file and optionally the system wide kerberos configuration can be specified via JAVA_OPTS in flume-env.sh:
+
+.. code-block:: properties
+
+ JAVA_OPTS="$JAVA_OPTS -Djava.security.krb5.conf=/path/to/krb5.conf"
+ JAVA_OPTS="$JAVA_OPTS -Djava.security.auth.login.config=/path/to/flume_jaas.conf"
+
+Example secure configuration using SASL_PLAINTEXT:
+
+.. code-block:: properties
+
+ a1.sinks.sink1.type = org.apache.flume.sink.kafka.KafkaSink
+ a1.sinks.sink1.kafka.bootstrap.servers = kafka-1:9093,kafka-2:9093,kafka-3:9093
+ a1.sinks.sink1.kafka.topic = mytopic
+ a1.sinks.sink1.kafka.producer.security.protocol = SASL_PLAINTEXT
+ a1.sinks.sink1.kafka.producer.sasl.mechanism = GSSAPI
+ a1.sinks.sink1.kafka.producer.sasl.kerberos.service.name = kafka
+
+
+Example secure configuration using SASL_SSL:
+
+.. code-block:: properties
+
+ a1.sinks.sink1.type = org.apache.flume.sink.kafka.KafkaSink
+ a1.sinks.sink1.kafka.bootstrap.servers = kafka-1:9093,kafka-2:9093,kafka-3:9093
+ a1.sinks.sink1.kafka.topic = mytopic
+ a1.sinks.sink1.kafka.producer.security.protocol = SASL_SSL
+ a1.sinks.sink1.kafka.producer.sasl.mechanism = GSSAPI
+ a1.sinks.sink1.kafka.producer.sasl.kerberos.service.name = kafka
+ # optional, the global truststore can be used alternatively
+ a1.sinks.sink1.kafka.producer.ssl.truststore.location = /path/to/truststore.jks
+ a1.sinks.sink1.kafka.producer.ssl.truststore.password = <password to access the truststore>
+
+
+Sample JAAS file. For reference of its content please see client config sections of the desired authentication mechanism (GSSAPI/PLAIN)
+in Kafka documentation of `SASL configuration <http://kafka.apache.org/documentation#security_sasl_clientconfig>`_.
+Unlike the Kafka Source or Kafka Channel a "Client" section is not required, unless it is needed by other connecting components. Also please make sure
+that the operating system user of the Flume processes has read privileges on the jaas and keytab files.
+
+.. code-block:: javascript
+
+ KafkaClient {
+ com.sun.security.auth.module.Krb5LoginModule required
+ useKeyTab=true
+ storeKey=true
+ keyTab="/path/to/keytabs/flume.keytab"
+ principal="flume/flumehost1.example.com@YOURKERBEROSREALM";
+ };
+
+
+HTTP Sink
+~~~~~~~~~
+
+Behaviour of this sink is that it will take events from the channel, and
+send those events to a remote service using an HTTP POST request. The event
+content is sent as the POST body.
+
+Error handling behaviour of this sink depends on the HTTP response returned
+by the target server. The sink backoff/ready status is configurable, as is the
+transaction commit/rollback result and whether the event contributes to the
+successful event drain count.
+
+Any malformed HTTP response returned by the server where the status code is
+not readable will result in a backoff signal and the event is not consumed
+from the channel.
+
+Required properties are in **bold**.
+
+========================== ================= ===========================================================================================
+Property Name Default Description
+========================== ================= ===========================================================================================
+**channel** --
+**type** -- The component type name, needs to be ``http``.
+**endpoint** -- The fully qualified URL endpoint to POST to
+connectTimeout 5000 The socket connection timeout in milliseconds
+requestTimeout 5000 The maximum request processing time in milliseconds
+contentTypeHeader text/plain The HTTP Content-Type header
+acceptHeader text/plain The HTTP Accept header value
+defaultBackoff true Whether to backoff by default on receiving all HTTP status codes
+defaultRollback true Whether to rollback by default on receiving all HTTP status codes
+defaultIncrementMetrics false Whether to increment metrics by default on receiving all HTTP status codes
+backoff.CODE -- Configures a specific backoff for an individual (i.e. 200) code or a group (i.e. 2XX) code
+rollback.CODE -- Configures a specific rollback for an individual (i.e. 200) code or a group (i.e. 2XX) code
+incrementMetrics.CODE -- Configures a specific metrics increment for an individual (i.e. 200) code or a group (i.e. 2XX) code
+========================== ================= ===========================================================================================
+
+Note that the most specific HTTP status code match is used for the backoff,
+rollback and incrementMetrics configuration options. If there are configuration
+values for both 2XX and 200 status codes, then 200 HTTP codes will use the 200
+value, and all other HTTP codes in the 201-299 range will use the 2XX value.
+
+Any empty or null events are consumed without any request being made to the
+HTTP endpoint.
+
+Example for agent named a1:
+
+.. code-block:: properties
+
+ a1.channels = c1
+ a1.sinks = k1
+ a1.sinks.k1.type = http
+ a1.sinks.k1.channel = c1
+ a1.sinks.k1.endpoint = http://localhost:8080/someuri
+ a1.sinks.k1.connectTimeout = 2000
+ a1.sinks.k1.requestTimeout = 2000
+ a1.sinks.k1.acceptHeader = application/json
+ a1.sinks.k1.contentTypeHeader = application/json
+ a1.sinks.k1.defaultBackoff = true
+ a1.sinks.k1.defaultRollback = true
+ a1.sinks.k1.defaultIncrementMetrics = false
+ a1.sinks.k1.backoff.4XX = false
+ a1.sinks.k1.rollback.4XX = false
+ a1.sinks.k1.incrementMetrics.4XX = true
+ a1.sinks.k1.backoff.200 = false
+ a1.sinks.k1.rollback.200 = false
+ a1.sinks.k1.incrementMetrics.200 = true
+
+
+Custom Sink
+~~~~~~~~~~~
+
+A custom sink is your own implementation of the Sink interface. A custom
+sink's class and its dependencies must be included in the agent's classpath
+when starting the Flume agent. The type of the custom sink is its FQCN.
+Required properties are in **bold**.
+
+============= ======= ==============================================
+Property Name Default Description
+============= ======= ==============================================
+**channel** --
+**type** -- The component type name, needs to be your FQCN
+============= ======= ==============================================
+
+Example for agent named a1:
+
+.. code-block:: properties
+
+ a1.channels = c1
+ a1.sinks = k1
+ a1.sinks.k1.type = org.example.MySink
+ a1.sinks.k1.channel = c1
+
+Flume Channels
+--------------
+
+Channels are the repositories where the events are staged on a agent.
+Source adds the events and Sink removes it.
+
+Memory Channel
+~~~~~~~~~~~~~~
+
+The events are stored in an in-memory queue with configurable max size. It's
+ideal for flows that need higher throughput and are prepared to lose the staged
+data in the event of a agent failures.
+Required properties are in **bold**.
+
+============================ ================ ===============================================================================
+Property Name Default Description
+============================ ================ ===============================================================================
+**type** -- The component type name, needs to be ``memory``
+capacity 100 The maximum number of events stored in the channel
+transactionCapacity 100 The maximum number of events the channel will take from a source or give to a
+ sink per transaction
+keep-alive 3 Timeout in seconds for adding or removing an event
+byteCapacityBufferPercentage 20 Defines the percent of buffer between byteCapacity and the estimated total size
+ of all events in the channel, to account for data in headers. See below.
+byteCapacity see description Maximum total **bytes** of memory allowed as a sum of all events in this channel.
+ The implementation only counts the Event ``body``, which is the reason for
+ providing the ``byteCapacityBufferPercentage`` configuration parameter as well.
+ Defaults to a computed value equal to 80% of the maximum memory available to
+ the JVM (i.e. 80% of the -Xmx value passed on the command line).
+ Note that if you have multiple memory channels on a single JVM, and they happen
+ to hold the same physical events (i.e. if you are using a replicating channel
+ selector from a single source) then those event sizes may be double-counted for
+ channel byteCapacity purposes.
+ Setting this value to ``0`` will cause this value to fall back to a hard
+ internal limit of about 200 GB.
+============================ ================ ===============================================================================
+
+Example for agent named a1:
+
+.. code-block:: properties
+
+ a1.channels = c1
+ a1.channels.c1.type = memory
+ a1.channels.c1.capacity = 10000
+ a1.channels.c1.transactionCapacity = 10000
+ a1.channels.c1.byteCapacityBufferPercentage = 20
+ a1.channels.c1.byteCapacity = 800000
+
+
+JDBC Channel
+~~~~~~~~~~~~
+
+The events are stored in a persistent storage that's backed by a database.
+The JDBC channel currently supports embedded Derby. This is a durable channel
+that's ideal for flows where recoverability is important.
+Required properties are in **bold**.
+
+========================== ==================================== =================================================
+Property Name Default Description
+========================== ==================================== =================================================
+**type** -- The component type name, needs to be ``jdbc``
+db.type DERBY Database vendor, needs to be DERBY.
+driver.class org.apache.derby.jdbc.EmbeddedDriver Class for vendor's JDBC driver
+driver.url (constructed from other properties) JDBC connection URL
+db.username "sa" User id for db connection
+db.password -- password for db connection
+connection.properties.file -- JDBC Connection property file path
+create.schema true If true, then creates db schema if not there
+create.index true Create indexes to speed up lookups
+create.foreignkey true
+transaction.isolation "READ_COMMITTED" Isolation level for db session READ_UNCOMMITTED,
+ READ_COMMITTED, SERIALIZABLE, REPEATABLE_READ
+maximum.connections 10 Max connections allowed to db
+maximum.capacity 0 (unlimited) Max number of events in the channel
+sysprop.* DB Vendor specific properties
+sysprop.user.home Home path to store embedded Derby database
+========================== ==================================== =================================================
+
+Example for agent named a1:
+
+.. code-block:: properties
+
+ a1.channels = c1
+ a1.channels.c1.type = jdbc
+
+Kafka Channel
+~~~~~~~~~~~~~
+
+The events are stored in a Kafka cluster (must be installed separately). Kafka provides high availability and
+replication, so in case an agent or a kafka broker crashes, the events are immediately available to other sinks
+
+The Kafka channel can be used for multiple scenarios:
+
+#. With Flume source and sink - it provides a reliable and highly available channel for events
+#. With Flume source and interceptor but no sink - it allows writing Flume events into a Kafka topic, for use by other apps
+#. With Flume sink, but no source - it is a low-latency, fault tolerant way to send events from Kafka to Flume sinks such as HDFS, HBase or Solr
+
+This currently supports Kafka server releases 0.10.1.0 or higher. Testing was done up to 2.0.1 that was the highest avilable version at the time of the release.
+
+The configuration parameters are organized as such:
+
+#. Configuration values related to the channel generically are applied at the channel config level, eg: a1.channel.k1.type =
+#. Configuration values related to Kafka or how the Channel operates are prefixed with "kafka.", (this are analgous to CommonClient Configs) eg: a1.channels.k1.kafka.topic and a1.channels.k1.kafka.bootstrap.servers. This is not dissimilar to how the hdfs sink operates
+#. Properties specific to the producer/consumer are prefixed by kafka.producer or kafka.consumer
+#. Where possible, the Kafka paramter names are used, eg: bootstrap.servers and acks
+
+This version of flume is backwards-compatible with previous versions, however deprecated properties are indicated in the table below and a warning message
+is logged on startup when they are present in the configuration file.
+
+Required properties are in **bold**.
+
+======================================= ========================== ===============================================================================================================
+Property Name Default Description
+======================================= ========================== ===============================================================================================================
+**type** -- The component type name, needs to be ``org.apache.flume.channel.kafka.KafkaChannel``
+**kafka.bootstrap.servers** -- List of brokers in the Kafka cluster used by the channel
+ This can be a partial list of brokers, but we recommend at least two for HA.
+ The format is comma separated list of hostname:port
+kafka.topic flume-channel Kafka topic which the channel will use
+kafka.consumer.group.id flume Consumer group ID the channel uses to register with Kafka.
+ Multiple channels must use the same topic and group to ensure that when one agent fails another can get the data
+ Note that having non-channel consumers with the same ID can lead to data loss.
+
+parseAsFlumeEvent true Expecting Avro datums with FlumeEvent schema in the channel.
+ This should be true if Flume source is writing to the channel and false if other producers are
+ writing into the topic that the channel is using. Flume source messages to Kafka can be parsed outside of Flume by using
+ org.apache.flume.source.avro.AvroFlumeEvent provided by the flume-ng-sdk artifact
+pollTimeout 500 The amount of time(in milliseconds) to wait in the "poll()" call of the consumer.
+ https://kafka.apache.org/090/javadoc/org/apache/kafka/clients/consumer/KafkaConsumer.html#poll(long)
+defaultPartitionId -- Specifies a Kafka partition ID (integer) for all events in this channel to be sent to, unless
+ overriden by ``partitionIdHeader``. By default, if this property is not set, events will be
+ distributed by the Kafka Producer's partitioner - including by ``key`` if specified (or by a
+ partitioner specified by ``kafka.partitioner.class``).
+partitionIdHeader -- When set, the producer will take the value of the field named using the value of this property
+ from the event header and send the message to the specified partition of the topic. If the
+ value represents an invalid partition the event will not be accepted into the channel. If the header value
+ is present then this setting overrides ``defaultPartitionId``.
+kafka.consumer.auto.offset.reset latest What to do when there is no initial offset in Kafka or if the current offset does not exist any more on the server
+ (e.g. because that data has been deleted):
+ earliest: automatically reset the offset to the earliest offset
+ latest: automatically reset the offset to the latest offset
+ none: throw exception to the consumer if no previous offset is found for the consumer\'s group
+ anything else: throw exception to the consumer.
+kafka.producer.security.protocol PLAINTEXT Set to SASL_PLAINTEXT, SASL_SSL or SSL if writing to Kafka using some level of security. See below for additional info on secure setup.
+kafka.consumer.security.protocol PLAINTEXT Same as kafka.producer.security.protocol but for reading/consuming from Kafka.
+*more producer/consumer security props* If using SASL_PLAINTEXT, SASL_SSL or SSL refer to `Kafka security <http://kafka.apache.org/documentation.html#security>`_ for additional
+ properties that need to be set on producer/consumer.
+======================================= ========================== ===============================================================================================================
+
+Deprecated Properties
+
+================================ ========================== ============================================================================================================================
+Property Name Default Description
+================================ ========================== ============================================================================================================================
+brokerList -- List of brokers in the Kafka cluster used by the channel
+ This can be a partial list of brokers, but we recommend at least two for HA.
+ The format is comma separated list of hostname:port
+topic flume-channel Use kafka.topic
+groupId flume Use kafka.consumer.group.id
+readSmallestOffset false Use kafka.consumer.auto.offset.reset
+migrateZookeeperOffsets true When no Kafka stored offset is found, look up the offsets in Zookeeper and commit them to Kafka.
+ This should be true to support seamless Kafka client migration from older versions of Flume. Once migrated this can be set
+ to false, though that should generally not be required. If no Zookeeper offset is found the kafka.consumer.auto.offset.reset
+ configuration defines how offsets are handled.
+================================ ========================== ============================================================================================================================
+
+.. note:: Due to the way the channel is load balanced, there may be duplicate events when the agent first starts up
+
+Example for agent named a1:
+
+.. code-block:: properties
+
+ a1.channels.channel1.type = org.apache.flume.channel.kafka.KafkaChannel
+ a1.channels.channel1.kafka.bootstrap.servers = kafka-1:9092,kafka-2:9092,kafka-3:9092
+ a1.channels.channel1.kafka.topic = channel1
+ a1.channels.channel1.kafka.consumer.group.id = flume-consumer
+
+
+**Security and Kafka Channel:**
+
+Secure authentication as well as data encryption is supported on the communication channel between Flume and Kafka.
+For secure authentication SASL/GSSAPI (Kerberos V5) or SSL (even though the parameter is named SSL, the actual protocol is a TLS implementation) can be used from Kafka version 0.9.0.
+
+As of now data encryption is solely provided by SSL/TLS.
+
+Setting ``kafka.producer|consumer.security.protocol`` to any of the following value means:
+
+- **SASL_PLAINTEXT** - Kerberos or plaintext authentication with no data encryption
+- **SASL_SSL** - Kerberos or plaintext authentication with data encryption
+- **SSL** - TLS based encryption with optional authentication.
+
+.. warning::
+ There is a performance degradation when SSL is enabled,
+ the magnitude of which depends on the CPU type and the JVM implementation.
+ Reference: `Kafka security overview <http://kafka.apache.org/documentation#security_overview>`_
+ and the jira for tracking this issue:
+ `KAFKA-2561 <https://issues.apache.org/jira/browse/KAFKA-2561>`_
+
+
+**TLS and Kafka Channel:**
+
+Please read the steps described in `Configuring Kafka Clients SSL <http://kafka.apache.org/documentation#security_configclients>`_
+to learn about additional configuration settings for fine tuning for example any of the following:
+security provider, cipher suites, enabled protocols, truststore or keystore types.
+
+Example configuration with server side authentication and data encryption.
+
+.. code-block:: properties
+
+ a1.channels.channel1.type = org.apache.flume.channel.kafka.KafkaChannel
+ a1.channels.channel1.kafka.bootstrap.servers = kafka-1:9093,kafka-2:9093,kafka-3:9093
+ a1.channels.channel1.kafka.topic = channel1
+ a1.channels.channel1.kafka.consumer.group.id = flume-consumer
+ a1.channels.channel1.kafka.producer.security.protocol = SSL
+ # optional, the global truststore can be used alternatively
+ a1.channels.channel1.kafka.producer.ssl.truststore.location = /path/to/truststore.jks
+ a1.channels.channel1.kafka.producer.ssl.truststore.password = <password to access the truststore>
+ a1.channels.channel1.kafka.consumer.security.protocol = SSL
+ # optional, the global truststore can be used alternatively
+ a1.channels.channel1.kafka.consumer.ssl.truststore.location = /path/to/truststore.jks
+ a1.channels.channel1.kafka.consumer.ssl.truststore.password = <password to access the truststore>
+
+Specyfing the truststore is optional here, the global truststore can be used instead.
+For more details about the global SSL setup, see the `SSL/TLS support`_ section.
+
+Note: By default the property ``ssl.endpoint.identification.algorithm``
+is not defined, so hostname verification is not performed.
+In order to enable hostname verification, set the following properties
+
+.. code-block:: properties
+
+ a1.channels.channel1.kafka.producer.ssl.endpoint.identification.algorithm = HTTPS
+ a1.channels.channel1.kafka.consumer.ssl.endpoint.identification.algorithm = HTTPS
+
+Once enabled, clients will verify the server's fully qualified domain name (FQDN)
+against one of the following two fields:
+
+#) Common Name (CN) https://tools.ietf.org/html/rfc6125#section-2.3
+#) Subject Alternative Name (SAN) https://tools.ietf.org/html/rfc5280#section-4.2.1.6
+
+If client side authentication is also required then additionally the following needs to be added to Flume agent
+configuration or the global SSL setup can be used (see `SSL/TLS support`_ section).
+Each Flume agent has to have its client certificate which has to be trusted by Kafka brokers either
+individually or by their signature chain. Common example is to sign each client certificate by a single Root CA
+which in turn is trusted by Kafka brokers.
+
+.. code-block:: properties
+
+ # optional, the global keystore can be used alternatively
+ a1.channels.channel1.kafka.producer.ssl.keystore.location = /path/to/client.keystore.jks
+ a1.channels.channel1.kafka.producer.ssl.keystore.password = <password to access the keystore>
+ # optional, the global keystore can be used alternatively
+ a1.channels.channel1.kafka.consumer.ssl.keystore.location = /path/to/client.keystore.jks
+ a1.channels.channel1.kafka.consumer.ssl.keystore.password = <password to access the keystore>
+
+If keystore and key use different password protection then ``ssl.key.password`` property will
+provide the required additional secret for both consumer and producer keystores:
+
+.. code-block:: properties
+
+ a1.channels.channel1.kafka.producer.ssl.key.password = <password to access the key>
+ a1.channels.channel1.kafka.consumer.ssl.key.password = <password to access the key>
+
+
+**Kerberos and Kafka Channel:**
+
+To use Kafka channel with a Kafka cluster secured with Kerberos, set the ``producer/consumer.security.protocol`` properties noted above for producer and/or consumer.
+The Kerberos keytab and principal to be used with Kafka brokers is specified in a JAAS file's "KafkaClient" section. "Client" section describes the Zookeeper connection if needed.
+See `Kafka doc <http://kafka.apache.org/documentation.html#security_sasl_clientconfig>`_
+for information on the JAAS file contents. The location of this JAAS file and optionally the system wide kerberos configuration can be specified via JAVA_OPTS in flume-env.sh:
+
+.. code-block:: properties
+
+ JAVA_OPTS="$JAVA_OPTS -Djava.security.krb5.conf=/path/to/krb5.conf"
+ JAVA_OPTS="$JAVA_OPTS -Djava.security.auth.login.config=/path/to/flume_jaas.conf"
+
+Example secure configuration using SASL_PLAINTEXT:
+
+.. code-block:: properties
+
+ a1.channels.channel1.type = org.apache.flume.channel.kafka.KafkaChannel
+ a1.channels.channel1.kafka.bootstrap.servers = kafka-1:9093,kafka-2:9093,kafka-3:9093
+ a1.channels.channel1.kafka.topic = channel1
+ a1.channels.channel1.kafka.consumer.group.id = flume-consumer
+ a1.channels.channel1.kafka.producer.security.protocol = SASL_PLAINTEXT
+ a1.channels.channel1.kafka.producer.sasl.mechanism = GSSAPI
+ a1.channels.channel1.kafka.producer.sasl.kerberos.service.name = kafka
+ a1.channels.channel1.kafka.consumer.security.protocol = SASL_PLAINTEXT
+ a1.channels.channel1.kafka.consumer.sasl.mechanism = GSSAPI
+ a1.channels.channel1.kafka.consumer.sasl.kerberos.service.name = kafka
+
+Example secure configuration using SASL_SSL:
+
+.. code-block:: properties
+
+ a1.channels.channel1.type = org.apache.flume.channel.kafka.KafkaChannel
+ a1.channels.channel1.kafka.bootstrap.servers = kafka-1:9093,kafka-2:9093,kafka-3:9093
+ a1.channels.channel1.kafka.topic = channel1
+ a1.channels.channel1.kafka.consumer.group.id = flume-consumer
+ a1.channels.channel1.kafka.producer.security.protocol = SASL_SSL
+ a1.channels.channel1.kafka.producer.sasl.mechanism = GSSAPI
+ a1.channels.channel1.kafka.producer.sasl.kerberos.service.name = kafka
+ # optional, the global truststore can be used alternatively
+ a1.channels.channel1.kafka.producer.ssl.truststore.location = /path/to/truststore.jks
+ a1.channels.channel1.kafka.producer.ssl.truststore.password = <password to access the truststore>
+ a1.channels.channel1.kafka.consumer.security.protocol = SASL_SSL
+ a1.channels.channel1.kafka.consumer.sasl.mechanism = GSSAPI
+ a1.channels.channel1.kafka.consumer.sasl.kerberos.service.name = kafka
+ # optional, the global truststore can be used alternatively
+ a1.channels.channel1.kafka.consumer.ssl.truststore.location = /path/to/truststore.jks
+ a1.channels.channel1.kafka.consumer.ssl.truststore.password = <password to access the truststore>
+
+
+Sample JAAS file. For reference of its content please see client config sections of the desired authentication mechanism (GSSAPI/PLAIN)
+in Kafka documentation of `SASL configuration <http://kafka.apache.org/documentation#security_sasl_clientconfig>`_.
+Since the Kafka Source may also connect to Zookeeper for offset migration, the "Client" section was also added to this example.
+This won't be needed unless you require offset migration, or you require this section for other secure components.
+Also please make sure that the operating system user of the Flume processes has read privileges on the jaas and keytab files.
+
+.. code-block:: javascript
+
+ Client {
+ com.sun.security.auth.module.Krb5LoginModule required
+ useKeyTab=true
+ storeKey=true
+ keyTab="/path/to/keytabs/flume.keytab"
+ principal="flume/flumehost1.example.com@YOURKERBEROSREALM";
+ };
+
+ KafkaClient {
+ com.sun.security.auth.module.Krb5LoginModule required
+ useKeyTab=true
+ storeKey=true
+ keyTab="/path/to/keytabs/flume.keytab"
+ principal="flume/flumehost1.example.com@YOURKERBEROSREALM";
+ };
+
+
+File Channel
+~~~~~~~~~~~~
+
+Required properties are in **bold**.
+
+================================================ ================================ ========================================================
+Property Name Default Description
+================================================ ================================ ========================================================
+**type** -- The component type name, needs to be ``file``.
+checkpointDir ~/.flume/file-channel/checkpoint The directory where checkpoint file will be stored
+useDualCheckpoints false Backup the checkpoint. If this is set to ``true``, ``backupCheckpointDir`` **must** be set
+backupCheckpointDir -- The directory where the checkpoint is backed up to. This directory **must not** be the same as the data directories or the checkpoint directory
+dataDirs ~/.flume/file-channel/data Comma separated list of directories for storing log files. Using multiple directories on separate disks can improve file channel peformance
+transactionCapacity 10000 The maximum size of transaction supported by the channel
+checkpointInterval 30000 Amount of time (in millis) between checkpoints
+maxFileSize 2146435071 Max size (in bytes) of a single log file
+minimumRequiredSpace 524288000 Minimum Required free space (in bytes). To avoid data corruption, File Channel stops accepting take/put requests when free space drops below this value
+capacity 1000000 Maximum capacity of the channel
+keep-alive 3 Amount of time (in sec) to wait for a put operation
+use-log-replay-v1 false Expert: Use old replay logic
+use-fast-replay false Expert: Replay without using queue
+checkpointOnClose true Controls if a checkpoint is created when the channel is closed. Creating a checkpoint on close speeds up subsequent startup of the file channel by avoiding replay.
+encryption.activeKey -- Key name used to encrypt new data
+encryption.cipherProvider -- Cipher provider type, supported types: AESCTRNOPADDING
+encryption.keyProvider -- Key provider type, supported types: JCEKSFILE
+encryption.keyProvider.keyStoreFile -- Path to the keystore file
+encrpytion.keyProvider.keyStorePasswordFile -- Path to the keystore password file
+encryption.keyProvider.keys -- List of all keys (e.g. history of the activeKey setting)
+encyption.keyProvider.keys.*.passwordFile -- Path to the optional key password file
+================================================ ================================ ========================================================
+
+.. note:: By default the File Channel uses paths for checkpoint and data
+ directories that are within the user home as specified above.
+ As a result if you have more than one File Channel instances
+ active within the agent, only one will be able to lock the
+ directories and cause the other channel initialization to fail.
+ It is therefore necessary that you provide explicit paths to
+ all the configured channels, preferably on different disks.
+ Furthermore, as file channel will sync to disk after every commit,
+ coupling it with a sink/source that batches events together may
+ be necessary to provide good performance where multiple disks are
+ not available for checkpoint and data directories.
+
+Example for agent named a1:
+
+.. code-block:: properties
+
+ a1.channels = c1
+ a1.channels.c1.type = file
+ a1.channels.c1.checkpointDir = /mnt/flume/checkpoint
+ a1.channels.c1.dataDirs = /mnt/flume/data
+
+**Encryption**
+
+Below is a few sample configurations:
+
+Generating a key with a password seperate from the key store password:
+
+.. code-block:: bash
+
+ keytool -genseckey -alias key-0 -keypass keyPassword -keyalg AES \
+ -keysize 128 -validity 9000 -keystore test.keystore \
+ -storetype jceks -storepass keyStorePassword
+
+Generating a key with the password the same as the key store password:
+
+.. code-block:: bash
+
+ keytool -genseckey -alias key-1 -keyalg AES -keysize 128 -validity 9000 \
+ -keystore src/test/resources/test.keystore -storetype jceks \
+ -storepass keyStorePassword
+
+
+.. code-block:: properties
+
+ a1.channels.c1.encryption.activeKey = key-0
+ a1.channels.c1.encryption.cipherProvider = AESCTRNOPADDING
+ a1.channels.c1.encryption.keyProvider = key-provider-0
+ a1.channels.c1.encryption.keyProvider = JCEKSFILE
+ a1.channels.c1.encryption.keyProvider.keyStoreFile = /path/to/my.keystore
+ a1.channels.c1.encryption.keyProvider.keyStorePasswordFile = /path/to/my.keystore.password
+ a1.channels.c1.encryption.keyProvider.keys = key-0
+
+Let's say you have aged key-0 out and new files should be encrypted with key-1:
+
+.. code-block:: properties
+
+ a1.channels.c1.encryption.activeKey = key-1
+ a1.channels.c1.encryption.cipherProvider = AESCTRNOPADDING
+ a1.channels.c1.encryption.keyProvider = JCEKSFILE
+ a1.channels.c1.encryption.keyProvider.keyStoreFile = /path/to/my.keystore
+ a1.channels.c1.encryption.keyProvider.keyStorePasswordFile = /path/to/my.keystore.password
+ a1.channels.c1.encryption.keyProvider.keys = key-0 key-1
+
+The same scenerio as above, however key-0 has its own password:
+
+.. code-block:: properties
+
+ a1.channels.c1.encryption.activeKey = key-1
+ a1.channels.c1.encryption.cipherProvider = AESCTRNOPADDING
+ a1.channels.c1.encryption.keyProvider = JCEKSFILE
+ a1.channels.c1.encryption.keyProvider.keyStoreFile = /path/to/my.keystore
+ a1.channels.c1.encryption.keyProvider.keyStorePasswordFile = /path/to/my.keystore.password
+ a1.channels.c1.encryption.keyProvider.keys = key-0 key-1
+ a1.channels.c1.encryption.keyProvider.keys.key-0.passwordFile = /path/to/key-0.password
+
+
+Spillable Memory Channel
+~~~~~~~~~~~~~~~~~~~~~~~~
+
+The events are stored in an in-memory queue and on disk. The in-memory queue serves as the primary store and the disk as overflow.
+The disk store is managed using an embedded File channel. When the in-memory queue is full, additional incoming events are stored in
+the file channel. This channel is ideal for flows that need high throughput of memory channel during normal operation, but at the
+same time need the larger capacity of the file channel for better tolerance of intermittent sink side outages or drop in drain rates.
+The throughput will reduce approximately to file channel speeds during such abnormal situations. In case of an agent crash or restart,
+only the events stored on disk are recovered when the agent comes online. **This channel is currently experimental and
+not recommended for use in production.**
+
+Required properties are in **bold**. Please refer to file channel for additional required properties.
+
+============================ ================ =============================================================================================
+Property Name Default Description
+============================ ================ =============================================================================================
+**type** -- The component type name, needs to be ``SPILLABLEMEMORY``
+memoryCapacity 10000 Maximum number of events stored in memory queue. To disable use of in-memory queue, set this to zero.
+overflowCapacity 100000000 Maximum number of events stored in overflow disk (i.e File channel). To disable use of overflow, set this to zero.
+overflowTimeout 3 The number of seconds to wait before enabling disk overflow when memory fills up.
+byteCapacityBufferPercentage 20 Defines the percent of buffer between byteCapacity and the estimated total size
+ of all events in the channel, to account for data in headers. See below.
+byteCapacity see description Maximum **bytes** of memory allowed as a sum of all events in the memory queue.
+ The implementation only counts the Event ``body``, which is the reason for
+ providing the ``byteCapacityBufferPercentage`` configuration parameter as well.
+ Defaults to a computed value equal to 80% of the maximum memory available to
+ the JVM (i.e. 80% of the -Xmx value passed on the command line).
+ Note that if you have multiple memory channels on a single JVM, and they happen
+ to hold the same physical events (i.e. if you are using a replicating channel
+ selector from a single source) then those event sizes may be double-counted for
+ channel byteCapacity purposes.
+ Setting this value to ``0`` will cause this value to fall back to a hard
+ internal limit of about 200 GB.
+avgEventSize 500 Estimated average size of events, in bytes, going into the channel
+<file channel properties> see file channel Any file channel property with the exception of 'keep-alive' and 'capacity' can be used.
+ The keep-alive of file channel is managed by Spillable Memory Channel. Use 'overflowCapacity'
+ to set the File channel's capacity.
+============================ ================ =============================================================================================
+
+In-memory queue is considered full if either memoryCapacity or byteCapacity limit is reached.
+
+Example for agent named a1:
+
+.. code-block:: properties
+
+ a1.channels = c1
+ a1.channels.c1.type = SPILLABLEMEMORY
+ a1.channels.c1.memoryCapacity = 10000
+ a1.channels.c1.overflowCapacity = 1000000
+ a1.channels.c1.byteCapacity = 800000
+ a1.channels.c1.checkpointDir = /mnt/flume/checkpoint
+ a1.channels.c1.dataDirs = /mnt/flume/data
+
+To disable the use of the in-memory queue and function like a file channel:
+
+.. code-block:: properties
+
+ a1.channels = c1
+ a1.channels.c1.type = SPILLABLEMEMORY
+ a1.channels.c1.memoryCapacity = 0
+ a1.channels.c1.overflowCapacity = 1000000
+ a1.channels.c1.checkpointDir = /mnt/flume/checkpoint
+ a1.channels.c1.dataDirs = /mnt/flume/data
+
+
+To disable the use of overflow disk and function purely as a in-memory channel:
+
+.. code-block:: properties
+
+ a1.channels = c1
+ a1.channels.c1.type = SPILLABLEMEMORY
+ a1.channels.c1.memoryCapacity = 100000
+ a1.channels.c1.overflowCapacity = 0
+
+
+Pseudo Transaction Channel
+~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+.. warning:: The Pseudo Transaction Channel is only for unit testing purposes
+ and is NOT meant for production use.
+
+Required properties are in **bold**.
+
+============= ======= ====================================================================================
+Property Name Default Description
+============= ======= ====================================================================================
+**type** -- The component type name, needs to be ``org.apache.flume.channel.PseudoTxnMemoryChannel``
+capacity 50 The max number of events stored in the channel
+keep-alive 3 Timeout in seconds for adding or removing an event
+============= ======= ====================================================================================
+
+
+Custom Channel
+~~~~~~~~~~~~~~
+
+A custom channel is your own implementation of the Channel interface. A
+custom channel's class and its dependencies must be included in the agent's
+classpath when starting the Flume agent. The type of the custom channel is
+its FQCN.
+Required properties are in **bold**.
+
+============= ======= =================================================================
+Property Name Default Description
+============= ======= =================================================================
+**type** -- The component type name, needs to be a FQCN
+============= ======= =================================================================
+
+Example for agent named a1:
+
+.. code-block:: properties
+
+ a1.channels = c1
+ a1.channels.c1.type = org.example.MyChannel
+
+Flume Channel Selectors
+-----------------------
+
+If the type is not specified, then defaults to "replicating".
+
+Replicating Channel Selector (default)
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Required properties are in **bold**.
+
+================== =========== ====================================================
+Property Name Default Description
+================== =========== ====================================================
+selector.type replicating The component type name, needs to be ``replicating``
+selector.optional -- Set of channels to be marked as ``optional``
+================== =========== ====================================================
+
+Example for agent named a1 and it's source called r1:
+
+.. code-block:: properties
+
+ a1.sources = r1
+ a1.channels = c1 c2 c3
+ a1.sources.r1.selector.type = replicating
+ a1.sources.r1.channels = c1 c2 c3
+ a1.sources.r1.selector.optional = c3
+
+In the above configuration, c3 is an optional channel. Failure to write to c3 is
+simply ignored. Since c1 and c2 are not marked optional, failure to write to
+those channels will cause the transaction to fail.
+
+
+Multiplexing Channel Selector
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Required properties are in **bold**.
+
+================== ===================== =================================================
+Property Name Default Description
+================== ===================== =================================================
+selector.type replicating The component type name, needs to be ``multiplexing``
+selector.header flume.selector.header
+selector.default --
+selector.mapping.* --
+================== ===================== =================================================
+
+Example for agent named a1 and it's source called r1:
+
+.. code-block:: properties
+
+ a1.sources = r1
+ a1.channels = c1 c2 c3 c4
+ a1.sources.r1.selector.type = multiplexing
+ a1.sources.r1.selector.header = state
+ a1.sources.r1.selector.mapping.CZ = c1
+ a1.sources.r1.selector.mapping.US = c2 c3
+ a1.sources.r1.selector.default = c4
+
+Custom Channel Selector
+~~~~~~~~~~~~~~~~~~~~~~~
+
+A custom channel selector is your own implementation of the ChannelSelector
+interface. A custom channel selector's class and its dependencies must be
+included in the agent's classpath when starting the Flume agent. The type of
+the custom channel selector is its FQCN.
+
+============= ======= ==============================================
+Property Name Default Description
+============= ======= ==============================================
+selector.type -- The component type name, needs to be your FQCN
+============= ======= ==============================================
+
+Example for agent named a1 and its source called r1:
+
+.. code-block:: properties
+
+ a1.sources = r1
+ a1.channels = c1
+ a1.sources.r1.selector.type = org.example.MyChannelSelector
+
+Flume Sink Processors
+---------------------
+
+Sink groups allow users to group multiple sinks into one entity.
+Sink processors can be used to provide load balancing capabilities over all
+sinks inside the group or to achieve fail over from one sink to another in
+case of temporal failure.
+
+Required properties are in **bold**.
+
+=================== =========== =================================================================================
+Property Name Default Description
+=================== =========== =================================================================================
+**sinks** -- Space-separated list of sinks that are participating in the group
+**processor.type** ``default`` The component type name, needs to be ``default``, ``failover`` or ``load_balance``
+=================== =========== =================================================================================
+
+
+Example for agent named a1:
+
+.. code-block:: properties
+
+ a1.sinkgroups = g1
+ a1.sinkgroups.g1.sinks = k1 k2
+ a1.sinkgroups.g1.processor.type = load_balance
+
+Default Sink Processor
+~~~~~~~~~~~~~~~~~~~~~~
+
+Default sink processor accepts only a single sink. User is not forced
+to create processor (sink group) for single sinks. Instead user can follow
+the source - channel - sink pattern that was explained above in this user
+guide.
+
+Failover Sink Processor
+~~~~~~~~~~~~~~~~~~~~~~~
+
+Failover Sink Processor maintains a prioritized list of sinks, guaranteeing
+that so long as one is available events will be processed (delivered).
+
+The failover mechanism works by relegating failed sinks to a pool where
+they are assigned a cool down period, increasing with sequential failures
+before they are retried. Once a sink successfully sends an event, it is
+restored to the live pool. The Sinks have a priority associated with them,
+larger the number, higher the priority. If a Sink fails while sending a Event
+the next Sink with highest priority shall be tried next for sending Events.
+For example, a sink with priority 100 is activated before the Sink with priority
+80. If no priority is specified, thr priority is determined based on the order in which
+the Sinks are specified in configuration.
+
+To configure, set a sink groups processor to ``failover`` and set
+priorities for all individual sinks. All specified priorities must
+be unique. Furthermore, upper limit to failover time can be set
+(in milliseconds) using ``maxpenalty`` property.
+
+Required properties are in **bold**.
+
+================================= =========== ===================================================================================
+Property Name Default Description
+================================= =========== ===================================================================================
+**sinks** -- Space-separated list of sinks that are participating in the group
+**processor.type** ``default`` The component type name, needs to be ``failover``
+**processor.priority.<sinkName>** -- Priority value. <sinkName> must be one of the sink instances associated with the current sink group
+ A higher priority value Sink gets activated earlier. A larger absolute value indicates higher priority
+processor.maxpenalty 30000 The maximum backoff period for the failed Sink (in millis)
+================================= =========== ===================================================================================
+
+Example for agent named a1:
+
+.. code-block:: properties
+
+ a1.sinkgroups = g1
+ a1.sinkgroups.g1.sinks = k1 k2
+ a1.sinkgroups.g1.processor.type = failover
+ a1.sinkgroups.g1.processor.priority.k1 = 5
+ a1.sinkgroups.g1.processor.priority.k2 = 10
+ a1.sinkgroups.g1.processor.maxpenalty = 10000
+
+
+Load balancing Sink Processor
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Load balancing sink processor provides the ability to load-balance flow over
+multiple sinks. It maintains an indexed list of active sinks on which the
+load must be distributed. Implementation supports distributing load using
+either via ``round_robin`` or ``random`` selection mechanisms.
+The choice of selection mechanism defaults to ``round_robin`` type,
+but can be overridden via configuration. Custom selection mechanisms are
+supported via custom classes that inherits from ``AbstractSinkSelector``.
+
+When invoked, this selector picks the next sink using its configured selection
+mechanism and invokes it. For ``round_robin`` and ``random`` In case the selected sink
+fails to deliver the event, the processor picks the next available sink via
+its configured selection mechanism. This implementation does not blacklist
+the failing sink and instead continues to optimistically attempt every
+available sink. If all sinks invocations result in failure, the selector
+propagates the failure to the sink runner.
+
+If ``backoff`` is enabled, the sink processor will blacklist
+sinks that fail, removing them for selection for a given timeout. When the
+timeout ends, if the sink is still unresponsive timeout is increased
+exponentially to avoid potentially getting stuck in long waits on unresponsive
+sinks. With this disabled, in round-robin all the failed sinks load will be
+passed to the next sink in line and thus not evenly balanced
+
+
+
+Required properties are in **bold**.
+
+============================= =============== ==========================================================================
+Property Name Default Description
+============================= =============== ==========================================================================
+**processor.sinks** -- Space-separated list of sinks that are participating in the group
+**processor.type** ``default`` The component type name, needs to be ``load_balance``
+processor.backoff false Should failed sinks be backed off exponentially.
+processor.selector ``round_robin`` Selection mechanism. Must be either ``round_robin``, ``random``
+ or FQCN of custom class that inherits from ``AbstractSinkSelector``
+processor.selector.maxTimeOut 30000 Used by backoff selectors to limit exponential backoff (in milliseconds)
+============================= =============== ==========================================================================
+
+Example for agent named a1:
+
+.. code-block:: properties
+
+ a1.sinkgroups = g1
+ a1.sinkgroups.g1.sinks = k1 k2
+ a1.sinkgroups.g1.processor.type = load_balance
+ a1.sinkgroups.g1.processor.backoff = true
+ a1.sinkgroups.g1.processor.selector = random
+
+
+Custom Sink Processor
+~~~~~~~~~~~~~~~~~~~~~
+
+Custom sink processors are not supported at the moment.
+
+Event Serializers
+-----------------
+
+The ``file_roll`` sink and the ``hdfs`` sink both support the
+``EventSerializer`` interface. Details of the EventSerializers that ship with
+Flume are provided below.
+
+Body Text Serializer
+~~~~~~~~~~~~~~~~~~~~
+
+Alias: ``text``. This interceptor writes the body of the event to an output
+stream without any transformation or modification. The event headers are
+ignored. Configuration options are as follows:
+
+========================= ================ ===========================================================================
+Property Name Default Description
+========================= ================ ===========================================================================
+appendNewline true Whether a newline will be appended to each event at write time. The default
+ of true assumes that events do not contain newlines, for legacy reasons.
+========================= ================ ===========================================================================
+
+Example for agent named a1:
+
+.. code-block:: properties
+
+ a1.sinks = k1
+ a1.sinks.k1.type = file_roll
+ a1.sinks.k1.channel = c1
+ a1.sinks.k1.sink.directory = /var/log/flume
+ a1.sinks.k1.sink.serializer = text
+ a1.sinks.k1.sink.serializer.appendNewline = false
+
+"Flume Event" Avro Event Serializer
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Alias: ``avro_event``.
+
+This interceptor serializes Flume events into an Avro container file. The schema used is the same schema used for
+Flume events in the Avro RPC mechanism.
+
+This serializer inherits from the ``AbstractAvroEventSerializer`` class.
+
+Configuration options are as follows:
+
+========================== ================ ===========================================================================
+Property Name Default Description
+========================== ================ ===========================================================================
+syncIntervalBytes 2048000 Avro sync interval, in approximate bytes.
+compressionCodec null Avro compression codec. For supported codecs, see Avro's CodecFactory docs.
+========================== ================ ===========================================================================
+
+Example for agent named a1:
+
+.. code-block:: properties
+
+ a1.sinks.k1.type = hdfs
+ a1.sinks.k1.channel = c1
+ a1.sinks.k1.hdfs.path = /flume/events/%y-%m-%d/%H%M/%S
+ a1.sinks.k1.serializer = avro_event
+ a1.sinks.k1.serializer.compressionCodec = snappy
+
+Avro Event Serializer
+~~~~~~~~~~~~~~~~~~~~~
+
+Alias: This serializer does not have an alias, and must be specified using the fully-qualified class name class name.
+
+This serializes Flume events into an Avro container file like the "Flume Event" Avro Event Serializer, however the
+record schema is configurable. The record schema may be specified either as a Flume configuration property or passed in an event header.
+
+To pass the record schema as part of the Flume configuration, use the property ``schemaURL`` as listed below.
+
+To pass the record schema in an event header, specify either the event header ``flume.avro.schema.literal``
+containing a JSON-format representation of the schema or ``flume.avro.schema.url`` with a URL where
+the schema may be found (``hdfs:/...`` URIs are supported).
+
+This serializer inherits from the ``AbstractAvroEventSerializer`` class.
+
+Configuration options are as follows:
+
+========================== ================ ===========================================================================
+Property Name Default Description
+========================== ================ ===========================================================================
+syncIntervalBytes 2048000 Avro sync interval, in approximate bytes.
+compressionCodec null Avro compression codec. For supported codecs, see Avro's CodecFactory docs.
+schemaURL null Avro schema URL. Schemas specified in the header ovverride this option.
+========================== ================ ===========================================================================
+
+Example for agent named a1:
+
+.. code-block:: properties
+
+ a1.sinks.k1.type = hdfs
+ a1.sinks.k1.channel = c1
+ a1.sinks.k1.hdfs.path = /flume/events/%y-%m-%d/%H%M/%S
+ a1.sinks.k1.serializer = org.apache.flume.sink.hdfs.AvroEventSerializer$Builder
+ a1.sinks.k1.serializer.compressionCodec = snappy
+ a1.sinks.k1.serializer.schemaURL = hdfs://namenode/path/to/schema.avsc
+
+
+Flume Interceptors
+------------------
+
+Flume has the capability to modify/drop events in-flight. This is done with the help of interceptors. Interceptors
+are classes that implement ``org.apache.flume.interceptor.Interceptor`` interface. An interceptor can
+modify or even drop events based on any criteria chosen by the developer of the interceptor. Flume supports
+chaining of interceptors. This is made possible through by specifying the list of interceptor builder class names
+in the configuration. Interceptors are specified as a whitespace separated list in the source configuration.
+The order in which the interceptors are specified is the order in which they are invoked.
+The list of events returned by one interceptor is passed to the next interceptor in the chain. Interceptors
+can modify or drop events. If an interceptor needs to drop events, it just does not return that event in
+the list that it returns. If it is to drop all events, then it simply returns an empty list. Interceptors
+are named components, here is an example of how they are created through configuration:
+
+.. code-block:: properties
+
+ a1.sources = r1
+ a1.sinks = k1
+ a1.channels = c1
+ a1.sources.r1.interceptors = i1 i2
+ a1.sources.r1.interceptors.i1.type = org.apache.flume.interceptor.HostInterceptor$Builder
+ a1.sources.r1.interceptors.i1.preserveExisting = false
+ a1.sources.r1.interceptors.i1.hostHeader = hostname
+ a1.sources.r1.interceptors.i2.type = org.apache.flume.interceptor.TimestampInterceptor$Builder
+ a1.sinks.k1.filePrefix = FlumeData.%{CollectorHost}.%Y-%m-%d
+ a1.sinks.k1.channel = c1
+
+Note that the interceptor builders are passed to the type config parameter. The interceptors are themselves
+configurable and can be passed configuration values just like they are passed to any other configurable component.
+In the above example, events are passed to the HostInterceptor first and the events returned by the HostInterceptor
+are then passed along to the TimestampInterceptor. You can specify either the fully qualified class name (FQCN)
+or the alias ``timestamp``. If you have multiple collectors writing to the same HDFS path, then you could also use
+the HostInterceptor.
+
+Timestamp Interceptor
+~~~~~~~~~~~~~~~~~~~~~
+
+This interceptor inserts into the event headers, the time in millis at which it processes the event. This interceptor
+inserts a header with key ``timestamp`` (or as specified by the ``header`` property) whose value is the relevant timestamp.
+This interceptor can preserve an existing timestamp if it is already present in the configuration.
+
+================ ========= ========================================================================
+Property Name Default Description
+================ ========= ========================================================================
+**type** -- The component type name, has to be ``timestamp`` or the FQCN
+headerName timestamp The name of the header in which to place the generated timestamp.
+preserveExisting false If the timestamp already exists, should it be preserved - true or false
+================ ========= ========================================================================
+
+Example for agent named a1:
+
+.. code-block:: properties
+
+ a1.sources = r1
+ a1.channels = c1
+ a1.sources.r1.channels = c1
+ a1.sources.r1.type = seq
+ a1.sources.r1.interceptors = i1
+ a1.sources.r1.interceptors.i1.type = timestamp
+
+Host Interceptor
+~~~~~~~~~~~~~~~~
+
+This interceptor inserts the hostname or IP address of the host that this agent is running on. It inserts a header
+with key ``host`` or a configured key whose value is the hostname or IP address of the host, based on configuration.
+
+================ ======= ========================================================================
+Property Name Default Description
+================ ======= ========================================================================
+**type** -- The component type name, has to be ``host``
+preserveExisting false If the host header already exists, should it be preserved - true or false
+useIP true Use the IP Address if true, else use hostname.
+hostHeader host The header key to be used.
+================ ======= ========================================================================
+
+Example for agent named a1:
+
+.. code-block:: properties
+
+ a1.sources = r1
+ a1.channels = c1
+ a1.sources.r1.interceptors = i1
+ a1.sources.r1.interceptors.i1.type = host
+
+Static Interceptor
+~~~~~~~~~~~~~~~~~~
+
+Static interceptor allows user to append a static header with static value to all events.
+
+The current implementation does not allow specifying multiple headers at one time. Instead user might chain
+multiple static interceptors each defining one static header.
+
+================ ======= ========================================================================
+Property Name Default Description
+================ ======= ========================================================================
+**type** -- The component type name, has to be ``static``
+preserveExisting true If configured header already exists, should it be preserved - true or false
+key key Name of header that should be created
+value value Static value that should be created
+================ ======= ========================================================================
+
+Example for agent named a1:
+
+.. code-block:: properties
+
+ a1.sources = r1
+ a1.channels = c1
+ a1.sources.r1.channels = c1
+ a1.sources.r1.type = seq
+ a1.sources.r1.interceptors = i1
+ a1.sources.r1.interceptors.i1.type = static
+ a1.sources.r1.interceptors.i1.key = datacenter
+ a1.sources.r1.interceptors.i1.value = NEW_YORK
+
+
+Remove Header Interceptor
+~~~~~~~~~~~~~~~~~~~~~~~~~
+
+This interceptor manipulates Flume event headers, by removing one or many headers. It can remove a statically defined header, headers based on a regular expression or headers in a list. If none of these is defined, or if no header matches the criteria, the Flume events are not modified.
+
+Note that if only one header needs to be removed, specifying it by name provides performance benefits over the other 2 methods.
+
+===================== =========== ===============================================================
+Property Name Default Description
+===================== =========== ===============================================================
+**type** -- The component type name has to be ``remove_header``
+withName -- Name of the header to remove
+fromList -- List of headers to remove, separated with the separator specified by ``fromListSeparator``
+fromListSeparator \\s*,\\s* Regular expression used to separate multiple header names in the list specified by ``fromList``. Default is a comma surrounded by any number of whitespace characters
+matching -- All the headers which names match this regular expression are removed
+===================== =========== ===============================================================
+
+
+UUID Interceptor
+~~~~~~~~~~~~~~~~
+
+This interceptor sets a universally unique identifier on all events that are intercepted. An example UUID is ``b5755073-77a9-43c1-8fad-b7a586fc1b97``, which represents a 128-bit value.
+
+Consider using UUIDInterceptor to automatically assign a UUID to an event if no application level unique key for the event is available. It can be important to assign UUIDs to events as soon as they enter the Flume network; that is, in the first Flume Source of the flow. This enables subsequent deduplication of events in the face of replication and redelivery in a Flume network that is designed for high availability and high performance. If an application level key is available, this is [...]
+
+================ ======= ========================================================================
+Property Name Default Description
+================ ======= ========================================================================
+**type** -- The component type name has to be ``org.apache.flume.sink.solr.morphline.UUIDInterceptor$Builder``
+headerName id The name of the Flume header to modify
+preserveExisting true If the UUID header already exists, should it be preserved - true or false
+prefix "" The prefix string constant to prepend to each generated UUID
+================ ======= ========================================================================
+
+Morphline Interceptor
+~~~~~~~~~~~~~~~~~~~~~
+
+This interceptor filters the events through a `morphline configuration file <http://cloudera.github.io/cdk/docs/current/cdk-morphlines/index.html>`_ that defines a chain of transformation commands that pipe records from one command to another.
+For example the morphline can ignore certain events or alter or insert certain event headers via regular expression based pattern matching, or it can auto-detect and set a MIME type via Apache Tika on events that are intercepted. For example, this kind of packet sniffing can be used for content based dynamic routing in a Flume topology.
+MorphlineInterceptor can also help to implement dynamic routing to multiple Apache Solr collections (e.g. for multi-tenancy).
+
+Currently, there is a restriction in that the morphline of an interceptor must not generate more than one output record for each input event. This interceptor is not intended for heavy duty ETL processing - if you need this consider moving ETL processing from the Flume Source to a Flume Sink, e.g. to a MorphlineSolrSink.
+
+Required properties are in **bold**.
+
+================= ======= ========================================================================
+Property Name Default Description
+================= ======= ========================================================================
+**type** -- The component type name has to be ``org.apache.flume.sink.solr.morphline.MorphlineInterceptor$Builder``
+**morphlineFile** -- The relative or absolute path on the local file system to the morphline configuration file. Example: ``/etc/flume-ng/conf/morphline.conf``
+morphlineId null Optional name used to identify a morphline if there are multiple morphlines in a morphline config file
+================= ======= ========================================================================
+
+Sample flume.conf file:
+
+.. code-block:: properties
+
+ a1.sources.avroSrc.interceptors = morphlineinterceptor
+ a1.sources.avroSrc.interceptors.morphlineinterceptor.type = org.apache.flume.sink.solr.morphline.MorphlineInterceptor$Builder
+ a1.sources.avroSrc.interceptors.morphlineinterceptor.morphlineFile = /etc/flume-ng/conf/morphline.conf
+ a1.sources.avroSrc.interceptors.morphlineinterceptor.morphlineId = morphline1
+
+Search and Replace Interceptor
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+This interceptor provides simple string-based search-and-replace functionality
+based on Java regular expressions. Backtracking / group capture is also available.
+This interceptor uses the same rules as in the Java Matcher.replaceAll() method.
+
+================ ======= ========================================================================
+Property Name Default Description
+================ ======= ========================================================================
+**type** -- The component type name has to be ``search_replace``
+searchPattern -- The pattern to search for and replace.
+replaceString -- The replacement string.
+charset UTF-8 The charset of the event body. Assumed by default to be UTF-8.
+================ ======= ========================================================================
+
+Example configuration:
+
+.. code-block:: properties
+
+ a1.sources.avroSrc.interceptors = search-replace
+ a1.sources.avroSrc.interceptors.search-replace.type = search_replace
+
+ # Remove leading alphanumeric characters in an event body.
+ a1.sources.avroSrc.interceptors.search-replace.searchPattern = ^[A-Za-z0-9_]+
+ a1.sources.avroSrc.interceptors.search-replace.replaceString =
+
+Another example:
+
+.. code-block:: properties
+
+ a1.sources.avroSrc.interceptors = search-replace
+ a1.sources.avroSrc.interceptors.search-replace.type = search_replace
+
+ # Use grouping operators to reorder and munge words on a line.
+ a1.sources.avroSrc.interceptors.search-replace.searchPattern = The quick brown ([a-z]+) jumped over the lazy ([a-z]+)
+ a1.sources.avroSrc.interceptors.search-replace.replaceString = The hungry $2 ate the careless $1
+
+
+Regex Filtering Interceptor
+~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+This interceptor filters events selectively by interpreting the event body as text and matching the text against a configured regular expression.
+The supplied regular expression can be used to include events or exclude events.
+
+================ ======= ========================================================================
+Property Name Default Description
+================ ======= ========================================================================
+**type** -- The component type name has to be ``regex_filter``
+regex ".*" Regular expression for matching against events
+excludeEvents false If true, regex determines events to exclude, otherwise regex determines
+ events to include.
+================ ======= ========================================================================
+
+Regex Extractor Interceptor
+~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+This interceptor extracts regex match groups using a specified regular expression and appends the match groups as headers on the event.
+It also supports pluggable serializers for formatting the match groups before adding them as event headers.
+
+================================ ========== =================================================================================================
+Property Name Default Description
+================================ ========== =================================================================================================
+**type** -- The component type name has to be ``regex_extractor``
+**regex** -- Regular expression for matching against events
+**serializers** -- Space-separated list of serializers for mapping matches to header names and serializing their
+ values. (See example below)
+ Flume provides built-in support for the following serializers:
+ ``org.apache.flume.interceptor.RegexExtractorInterceptorPassThroughSerializer``
+ ``org.apache.flume.interceptor.RegexExtractorInterceptorMillisSerializer``
+serializers.<s1>.type default Must be ``default`` (org.apache.flume.interceptor.RegexExtractorInterceptorPassThroughSerializer),
+ ``org.apache.flume.interceptor.RegexExtractorInterceptorMillisSerializer``,
+ or the FQCN of a custom class that implements ``org.apache.flume.interceptor.RegexExtractorInterceptorSerializer``
+serializers.<s1>.\ **name** --
+serializers.* -- Serializer-specific properties
+================================ ========== =================================================================================================
+
+The serializers are used to map the matches to a header name and a formatted header value; by default, you only need to specify
+the header name and the default ``org.apache.flume.interceptor.RegexExtractorInterceptorPassThroughSerializer`` will be used.
+This serializer simply maps the matches to the specified header name and passes the value through as it was extracted by the regex.
+You can plug custom serializer implementations into the extractor using the fully qualified class name (FQCN) to format the matches
+in anyway you like.
+
+Example 1:
+~~~~~~~~~~
+
+If the Flume event body contained ``1:2:3.4foobar5`` and the following configuration was used
+
+.. code-block:: properties
+
+ a1.sources.r1.interceptors.i1.regex = (\\d):(\\d):(\\d)
+ a1.sources.r1.interceptors.i1.serializers = s1 s2 s3
+ a1.sources.r1.interceptors.i1.serializers.s1.name = one
+ a1.sources.r1.interceptors.i1.serializers.s2.name = two
+ a1.sources.r1.interceptors.i1.serializers.s3.name = three
+
+The extracted event will contain the same body but the following headers will have been added ``one=>1, two=>2, three=>3``
+
+Example 2:
+~~~~~~~~~~
+
+If the Flume event body contained ``2012-10-18 18:47:57,614 some log line`` and the following configuration was used
+
+.. code-block:: properties
+
+ a1.sources.r1.interceptors.i1.regex = ^(?:\\n)?(\\d\\d\\d\\d-\\d\\d-\\d\\d\\s\\d\\d:\\d\\d)
+ a1.sources.r1.interceptors.i1.serializers = s1
+ a1.sources.r1.interceptors.i1.serializers.s1.type = org.apache.flume.interceptor.RegexExtractorInterceptorMillisSerializer
+ a1.sources.r1.interceptors.i1.serializers.s1.name = timestamp
+ a1.sources.r1.interceptors.i1.serializers.s1.pattern = yyyy-MM-dd HH:mm
+
+the extracted event will contain the same body but the following headers will have been added ``timestamp=>1350611220000``
+
+Flume Properties
+----------------
+
+========================= ======= ====================================================================
+Property Name Default Description
+========================= ======= ====================================================================
+flume.called.from.service -- If this property is specified then the Flume agent will continue
+ polling for the config file even if the config file is not found
+ at the expected location. Otherwise, the Flume agent will terminate
+ if the config doesn't exist at the expected location. No property
+ value is needed when setting this property (eg, just specifying
+ -Dflume.called.from.service is enough)
+========================= ======= ====================================================================
+
+Property: flume.called.from.service
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Flume periodically polls, every 30 seconds, for changes to the specified
+config file. A Flume agent loads a new configuration from the config file if
+either an existing file is polled for the first time, or if an existing
+file's modification date has changed since the last time it was polled.
+Renaming or moving a file does not change its modification time. When a
+Flume agent polls a non-existent file then one of two things happens: 1.
+When the agent polls a non-existent config file for the first time, then the
+agent behaves according to the flume.called.from.service property. If the
+property is set, then the agent will continue polling (always at the same
+period -- every 30 seconds). If the property is not set, then the agent
+immediately terminates. ...OR... 2. When the agent polls a non-existent
+config file and this is not the first time the file is polled, then the
+agent makes no config changes for this polling period. The agent continues
+polling rather than terminating.
+
+Configuration Filters
+=====================
+
+Flume provides a tool for injecting sensitive or generated data into the configuration
+in the form of configuration filters. A configuration key can be set as the value of configuration properties
+and it will be replaced by the configuration filter with the value it represents.
+
+Common usage of config filters
+------------------------------
+
+The format is similar to the Java Expression Language, however
+it is currently not a fully working EL expression parser, just a format that looks like it.
+
+.. code-block:: properties
+
+ <agent_name>.configfilters = <filter_name>
+ <agent_name>.configfilters.<filter_name>.type = <filter_type>
+
+ <agent_name>.sources.<source_name>.parameter = ${<filter_name>['<key_for_sensitive_or_generated_data>']}
+ <agent_name>.sinks.<sink_name>.parameter = ${<filter_name>['<key_for_sensitive_or_generated_data>']}
+ <agent_name>.<component_type>.<component_name>.parameter = ${<filter_name>['<key_for_sensitive_or_generated_data>']}
+ #or
+ <agent_name>.<component_type>.<component_name>.parameter = ${<filter_name>["<key_for_sensitive_or_generated_data>"]}
+ #or
+ <agent_name>.<component_type>.<component_name>.parameter = ${<filter_name>[<key_for_sensitive_or_generated_data>]}
+ #or
+ <agent_name>.<component_type>.<component_name>.parameter = some_constant_data${<filter_name>[<key_for_sensitive_or_generated_data>]}
+
+
+Environment Variable Config Filter
+----------------------------------
+
+================================ ========== ==============================================
+Property Name Default Description
+================================ ========== ==============================================
+**type** -- The component type name has to be ``env``
+================================ ========== ==============================================
+
+Example
+~~~~~~~
+To hide a password in the configuration set its value as in the following example.
+
+.. code-block:: properties
+
+ a1.sources = r1
+ a1.channels = c1
+ a1.configfilters = f1
+
+ a1.configfilters.f1.type = env
+
+ a1.sources.r1.channels = c1
+ a1.sources.r1.type = http
+ a1.sources.r1.keystorePassword = ${f1['my_keystore_password']} #will get the value Secret123
+
+Here the ``a1.sources.r1.keystorePassword`` configuration property will get the value of the ``my_keystore_password``
+environment variable. One way to set the environment variable is to run flume agent like this:
+
+``$ my_keystore_password=Secret123 bin/flume-ng agent --conf conf --conf-file example.conf ...``
+
+
+
+External Process Config Filter
+------------------------------
+
+================================ ========== ==============================================
+Property Name Default Description
+================================ ========== ==============================================
+**type** -- The component type name has to be ``external``
+**command** -- The command that will be executed to get the value for the given key. The command will be called like: ``<command> <key>`` And expected to return a single line value with exit code ``0``.
+charset UTF-8 The characterset of the returned string.
+================================ ========== ==============================================
+
+Example
+~~~~~~~
+To hide a password in the configuration set its value as in the following example.
+
+.. code-block:: properties
+
+ a1.sources = r1
+ a1.channels = c1
+ a1.configfilters = f1
+
+ a1.configfilters.f1.type = external
+ a1.configfilters.f1.command = /usr/bin/passwordResolver.sh
+ a1.configfilters.f1.charset = UTF-8
+
+ a1.sources.r1.channels = c1
+ a1.sources.r1.type = http
+ a1.sources.r1.keystorePassword = ${f1['my_keystore_password']} #will get the value Secret123
+
+In this example flume will run the following command to get the value
+
+``$ /usr/bin/passwordResolver.sh my_keystore_password``
+
+The ``passwordResolver.sh`` will return ``Secret123`` with an exit code ``0``.
+
+Example 2
+~~~~~~~~~
+To generate a part of the directory for rolling file sink set its value as in the following example.
+
+.. code-block:: properties
+
+ a1.sources = r1
+ a1.channels = c1
+ a1.configfilters = f1
+
+ a1.configfilters.f1.type = external
+ a1.configfilters.f1.command = /usr/bin/generateUniqId.sh
+ a1.configfilters.f1.charset = UTF-8
+
+ a1.sinks = k1
+ a1.sinks.k1.type = file_roll
+ a1.sinks.k1.channel = c1
+ a1.sinks.k1.sink.directory = /var/log/flume/agent_${f1['agent_name']} # will be /var/log/flume/agent_1234
+
+In this example flume will run the following command to get the value
+
+``$ /usr/bin/generateUniqId.sh agent_name``
+
+The ``generateUniqId.sh`` will return ``1234`` with an exit code ``0``.
+
+
+
+Hadoop Credential Store Config Filter
+-------------------------------------
+
+A hadoop-common library needed on the classpath for this feature (2.6+ version).
+If hadoop is installed the agent adds it to the classpath automatically
+
+=============================================== ========== ==============================================
+Property Name Default Description
+=============================================== ========== ==============================================
+**type** -- The component type name has to be ``hadoop``
+**credential.provider.path** -- The provider path. See hadoop documentation _here: https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/CredentialProviderAPI.html#Configuring_the_Provider_Path
+credstore.java-keystore-provider.password-file -- The name of the password file if a file is used to store the password. The file must e on the classpath.
+ Provider password can be set with the HADOOP_CREDSTORE_PASSWORD environment variable or left empty.
+=============================================== ========== ==============================================
+
+Example
+~~~~~~~
+To hide a password in the configuration set its value as in the following example.
+
+.. code-block:: properties
+
+ a1.sources = r1
+ a1.channels = c1
+ a1.configfilters = f1
+
+ a1.configfilters.f1.type = hadoop
+ a1.configfilters.f1.credential.provider.path = jceks://file/<path_to_jceks file>
+
+ a1.sources.r1.channels = c1
+ a1.sources.r1.type = http
+ a1.sources.r1.keystorePassword = ${f1['my_keystore_password']} #will get the value from the credential store
+
+
+
+Log4J Appender
+==============
+
+Appends Log4j events to a flume agent's avro source. A client using this
+appender must have the flume-ng-sdk in the classpath (eg,
+flume-ng-sdk-1.9.0.jar).
+Required properties are in **bold**.
+
+===================== ======= ==================================================================================
+Property Name Default Description
+===================== ======= ==================================================================================
+**Hostname** -- The hostname on which a remote Flume agent is running with an
+ avro source.
+**Port** -- The port at which the remote Flume agent's avro source is
+ listening.
+UnsafeMode false If true, the appender will not throw exceptions on failure to
+ send the events.
+AvroReflectionEnabled false Use Avro Reflection to serialize Log4j events. (Do not use when users log strings)
+AvroSchemaUrl -- A URL from which the Avro schema can be retrieved.
+===================== ======= ==================================================================================
+
+Sample log4j.properties file:
+
+.. code-block:: properties
+
+ #...
+ log4j.appender.flume = org.apache.flume.clients.log4jappender.Log4jAppender
+ log4j.appender.flume.Hostname = example.com
+ log4j.appender.flume.Port = 41414
+ log4j.appender.flume.UnsafeMode = true
+
+ # configure a class's logger to output to the flume appender
+ log4j.logger.org.example.MyClass = DEBUG,flume
+ #...
+
+By default each event is converted to a string by calling ``toString()``,
+or by using the Log4j layout, if specified.
+
+If the event is an instance of
+``org.apache.avro.generic.GenericRecord``, ``org.apache.avro.specific.SpecificRecord``,
+or if the property ``AvroReflectionEnabled`` is set to ``true`` then the event will be
+serialized using Avro serialization.
+
+Serializing every event with its Avro schema is inefficient, so it is good practice to
+provide a schema URL from which the schema can be retrieved by the downstream sink,
+typically the HDFS sink. If ``AvroSchemaUrl`` is not specified,
+then the schema will be included as a Flume header.
+
+Sample log4j.properties file configured to use Avro serialization:
+
+.. code-block:: properties
+
+ #...
+ log4j.appender.flume = org.apache.flume.clients.log4jappender.Log4jAppender
+ log4j.appender.flume.Hostname = example.com
+ log4j.appender.flume.Port = 41414
+ log4j.appender.flume.AvroReflectionEnabled = true
+ log4j.appender.flume.AvroSchemaUrl = hdfs://namenode/path/to/schema.avsc
+
+ # configure a class's logger to output to the flume appender
+ log4j.logger.org.example.MyClass = DEBUG,flume
+ #...
+
+
+Load Balancing Log4J Appender
+=============================
+
+Appends Log4j events to a list of flume agent's avro source. A client using this
+appender must have the flume-ng-sdk in the classpath (eg,
+flume-ng-sdk-1.9.0.jar). This appender supports a round-robin and random
+scheme for performing the load balancing. It also supports a configurable backoff
+timeout so that down agents are removed temporarily from the set of hosts
+Required properties are in **bold**.
+
+===================== =========== ==============================================================
+Property Name Default Description
+===================== =========== ==============================================================
+**Hosts** -- A space-separated list of host:port at which Flume (through
+ an AvroSource) is listening for events
+Selector ROUND_ROBIN Selection mechanism. Must be either ROUND_ROBIN,
+ RANDOM or custom FQDN to class that inherits from
+ LoadBalancingSelector.
+MaxBackoff -- A long value representing the maximum amount of time in
+ milliseconds the Load balancing client will backoff from a
+ node that has failed to consume an event. Defaults to no backoff
+UnsafeMode false If true, the appender will not throw exceptions on failure to
+ send the events.
+AvroReflectionEnabled false Use Avro Reflection to serialize Log4j events.
+AvroSchemaUrl -- A URL from which the Avro schema can be retrieved.
+===================== =========== ==============================================================
+
+
+Sample log4j.properties file configured using defaults:
+
+.. code-block:: properties
+
+ #...
+ log4j.appender.out2 = org.apache.flume.clients.log4jappender.LoadBalancingLog4jAppender
+ log4j.appender.out2.Hosts = localhost:25430 localhost:25431
+
+ # configure a class's logger to output to the flume appender
+ log4j.logger.org.example.MyClass = DEBUG,flume
+ #...
+
+Sample log4j.properties file configured using RANDOM load balancing:
+
+.. code-block:: properties
+
+ #...
+ log4j.appender.out2 = org.apache.flume.clients.log4jappender.LoadBalancingLog4jAppender
+ log4j.appender.out2.Hosts = localhost:25430 localhost:25431
+ log4j.appender.out2.Selector = RANDOM
+
+ # configure a class's logger to output to the flume appender
+ log4j.logger.org.example.MyClass = DEBUG,flume
+ #...
+
+Sample log4j.properties file configured using backoff:
+
+.. code-block:: properties
+
+ #...
+ log4j.appender.out2 = org.apache.flume.clients.log4jappender.LoadBalancingLog4jAppender
+ log4j.appender.out2.Hosts = localhost:25430 localhost:25431 localhost:25432
+ log4j.appender.out2.Selector = ROUND_ROBIN
+ log4j.appender.out2.MaxBackoff = 30000
+
+ # configure a class's logger to output to the flume appender
+ log4j.logger.org.example.MyClass = DEBUG,flume
+ #...
+
+Security
+========
+
+The HDFS sink, HBase sink, Thrift source, Thrift sink and Kite Dataset sink all support
+Kerberos authentication. Please refer to the corresponding sections for
+configuring the Kerberos-related options.
+
+Flume agent will authenticate to the kerberos KDC as a single principal, which will be
+used by different components that require kerberos authentication. The principal and
+keytab configured for Thrift source, Thrift sink, HDFS sink, HBase sink and DataSet sink
+should be the same, otherwise the component will fail to start.
+
+Monitoring
+==========
+
+Monitoring in Flume is still a work in progress. Changes can happen very often.
+Several Flume components report metrics to the JMX platform MBean server. These
+metrics can be queried using Jconsole.
+
+Available Component Metrics
+---------------------------
+
+The following tables show what metrics are available for components. Each component only maintains a
+set of metrics, indicated by an 'x', the unmaintained ones show default values, that is 0.
+These tables tell you where you can expect meaningful data.
+The name of the metrics should be descriptive enough, for more information you have to dig into the
+source code of the components.
+
+Sources 1
+~~~~~~~~~
+
++--------------------------+------+------+------+-----+-------+--------------------+--------+
+| | Avro | Exec | HTTP | JMS | Kafka | MultiportSyslogTCP | Scribe |
++--------------------------+------+------+------+-----+-------+--------------------+--------+
+| AppendAcceptedCount | x | | | | | | |
++--------------------------+------+------+------+-----+-------+--------------------+--------+
+| AppendBatchAcceptedCount | x | | x | x | | | |
++--------------------------+------+------+------+-----+-------+--------------------+--------+
+| AppendBatchReceivedCount | x | | x | x | | | |
++--------------------------+------+------+------+-----+-------+--------------------+--------+
+| AppendReceivedCount | x | | | | | | |
++--------------------------+------+------+------+-----+-------+--------------------+--------+
+| ChannelWriteFail | x | | x | x | x | x | x |
++--------------------------+------+------+------+-----+-------+--------------------+--------+
+| EventAcceptedCount | x | x | x | x | x | x | x |
++--------------------------+------+------+------+-----+-------+--------------------+--------+
+| EventReadFail | | | x | x | x | x | x |
++--------------------------+------+------+------+-----+-------+--------------------+--------+
+| EventReceivedCount | x | x | x | x | x | x | x |
++--------------------------+------+------+------+-----+-------+--------------------+--------+
+| GenericProcessingFail | | | x | | | x | |
++--------------------------+------+------+------+-----+-------+--------------------+--------+
+| KafkaCommitTimer | | | | | x | | |
++--------------------------+------+------+------+-----+-------+--------------------+--------+
+| KafkaEmptyCount | | | | | x | | |
++--------------------------+------+------+------+-----+-------+--------------------+--------+
+| KafkaEventGetTimer | | | | | x | | |
++--------------------------+------+------+------+-----+-------+--------------------+--------+
+| OpenConnectionCount | x | | | | | | |
++--------------------------+------+------+------+-----+-------+--------------------+--------+
+
+Sources 2
+~~~~~~~~~
+
++--------------------------+-------------------+----------------+-----------+-----------+---------+--------+
+| | SequenceGenerator | SpoolDirectory | SyslogTcp | SyslogUDP | Taildir | Thrift |
++--------------------------+-------------------+----------------+-----------+-----------+---------+--------+
+| AppendAcceptedCount | | | | | | x |
++--------------------------+-------------------+----------------+-----------+-----------+---------+--------+
+| AppendBatchAcceptedCount | x | x | | | x | x |
++--------------------------+-------------------+----------------+-----------+-----------+---------+--------+
+| AppendBatchReceivedCount | | x | | | x | x |
++--------------------------+-------------------+----------------+-----------+-----------+---------+--------+
+| AppendReceivedCount | | | | | | x |
++--------------------------+-------------------+----------------+-----------+-----------+---------+--------+
+| ChannelWriteFail | x | x | x | x | x | x |
++--------------------------+-------------------+----------------+-----------+-----------+---------+--------+
+| EventAcceptedCount | x | x | x | x | x | x |
++--------------------------+-------------------+----------------+-----------+-----------+---------+--------+
+| EventReadFail | | x | x | x | x | |
++--------------------------+-------------------+----------------+-----------+-----------+---------+--------+
+| EventReceivedCount | | x | x | x | x | x |
++--------------------------+-------------------+----------------+-----------+-----------+---------+--------+
+| GenericProcessingFail | | x | | | x | |
++--------------------------+-------------------+----------------+-----------+-----------+---------+--------+
+| KafkaCommitTimer | | | | | | |
++--------------------------+-------------------+----------------+-----------+-----------+---------+--------+
+| KafkaEmptyCount | | | | | | |
++--------------------------+-------------------+----------------+-----------+-----------+---------+--------+
+| KafkaEventGetTimer | | | | | | |
++--------------------------+-------------------+----------------+-----------+-----------+---------+--------+
+| OpenConnectionCount | | | | | | |
++--------------------------+-------------------+----------------+-----------+-----------+---------+--------+
+
+Sinks 1
+~~~~~~~
+
++------------------------+-------------+------------+---------------+-------+--------+
+| | Avro/Thrift | AsyncHBase | ElasticSearch | HBase | HBase2 |
++------------------------+-------------+------------+---------------+-------+--------+
+| BatchCompleteCount | x | x | x | x | x |
++------------------------+-------------+------------+---------------+-------+--------+
+| BatchEmptyCount | x | x | x | x | x |
++------------------------+-------------+------------+---------------+-------+--------+
+| BatchUnderflowCount | x | x | x | x | x |
++------------------------+-------------+------------+---------------+-------+--------+
+| ChannelReadFail | x | | | | x |
++------------------------+-------------+------------+---------------+-------+--------+
+| ConnectionClosedCount | x | x | x | x | x |
++------------------------+-------------+------------+---------------+-------+--------+
+| ConnectionCreatedCount | x | x | x | x | x |
++------------------------+-------------+------------+---------------+-------+--------+
+| ConnectionFailedCount | x | x | x | x | x |
++------------------------+-------------+------------+---------------+-------+--------+
+| EventDrainAttemptCount | x | x | x | x | x |
++------------------------+-------------+------------+---------------+-------+--------+
+| EventDrainSuccessCount | x | x | x | x | x |
++------------------------+-------------+------------+---------------+-------+--------+
+| EventWriteFail | x | | | | x |
++------------------------+-------------+------------+---------------+-------+--------+
+| KafkaEventSendTimer | | | | | |
++------------------------+-------------+------------+---------------+-------+--------+
+| RollbackCount | | | | | |
++------------------------+-------------+------------+---------------+-------+--------+
+
+Sinks 2
+~~~~~~~
+
++------------------------+-----------+------+------+-------+-----------+-------------+
+| | HDFSEvent | Hive | Http | Kafka | Morphline | RollingFile |
++------------------------+-----------+------+------+-------+-----------+-------------+
+| BatchCompleteCount | x | x | | | x | |
++------------------------+-----------+------+------+-------+-----------+-------------+
+| BatchEmptyCount | x | x | | x | x | |
++------------------------+-----------+------+------+-------+-----------+-------------+
+| BatchUnderflowCount | x | x | | x | x | |
++------------------------+-----------+------+------+-------+-----------+-------------+
+| ChannelReadFail | x | x | x | x | x | x |
++------------------------+-----------+------+------+-------+-----------+-------------+
+| ConnectionClosedCount | x | x | | | | x |
++------------------------+-----------+------+------+-------+-----------+-------------+
+| ConnectionCreatedCount | x | x | | | | x |
++------------------------+-----------+------+------+-------+-----------+-------------+
+| ConnectionFailedCount | x | x | | | | x |
++------------------------+-----------+------+------+-------+-----------+-------------+
+| EventDrainAttemptCount | x | x | x | | x | x |
++------------------------+-----------+------+------+-------+-----------+-------------+
+| EventDrainSuccessCount | x | x | x | x | x | x |
++------------------------+-----------+------+------+-------+-----------+-------------+
+| EventWriteFail | x | x | x | x | x | x |
++------------------------+-----------+------+------+-------+-----------+-------------+
+| KafkaEventSendTimer | | | | x | | |
++------------------------+-----------+------+------+-------+-----------+-------------+
+| RollbackCount | | | | x | | |
++------------------------+-----------+------+------+-------+-----------+-------------+
+
+Channels
+~~~~~~~~
+
++---------------------------------+------+-------+--------+-----------------+-----------------+
+| | File | Kafka | Memory | PseudoTxnMemory | SpillableMemory |
++---------------------------------+------+-------+--------+-----------------+-----------------+
+| ChannelCapacity | x | | x | | x |
++---------------------------------+------+-------+--------+-----------------+-----------------+
+| ChannelSize | x | | x | x | x |
++---------------------------------+------+-------+--------+-----------------+-----------------+
+| CheckpointBackupWriteErrorCount | x | | | | |
++---------------------------------+------+-------+--------+-----------------+-----------------+
+| CheckpointWriteErrorCount | x | | | | |
++---------------------------------+------+-------+--------+-----------------+-----------------+
+| EventPutAttemptCount | x | x | x | x | x |
++---------------------------------+------+-------+--------+-----------------+-----------------+
+| EventPutErrorCount | x | | | | |
++---------------------------------+------+-------+--------+-----------------+-----------------+
+| EventPutSuccessCount | x | x | x | x | x |
++---------------------------------+------+-------+--------+-----------------+-----------------+
+| EventTakeAttemptCount | x | x | x | x | x |
++---------------------------------+------+-------+--------+-----------------+-----------------+
+| EventTakeErrorCount | x | | | | |
++---------------------------------+------+-------+--------+-----------------+-----------------+
+| EventTakeSuccessCount | x | x | x | x | x |
++---------------------------------+------+-------+--------+-----------------+-----------------+
+| KafkaCommitTimer | | x | | | |
++---------------------------------+------+-------+--------+-----------------+-----------------+
+| KafkaEventGetTimer | | x | | | |
++---------------------------------+------+-------+--------+-----------------+-----------------+
+| KafkaEventSendTimer | | x | | | |
++---------------------------------+------+-------+--------+-----------------+-----------------+
+| Open | x | | | | |
++---------------------------------+------+-------+--------+-----------------+-----------------+
+| RollbackCounter | | x | | | |
++---------------------------------+------+-------+--------+-----------------+-----------------+
+| Unhealthy | x | | | | |
++---------------------------------+------+-------+--------+-----------------+-----------------+
+
+JMX Reporting
+-------------
+
+JMX Reporting can be enabled by specifying JMX parameters in the JAVA_OPTS environment variable using
+flume-env.sh, like
+
+ export JAVA_OPTS="-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=5445 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false"
+
+NOTE: The sample above disables the security. To enable Security, please refer http://docs.oracle.com/javase/6/docs/technotes/guides/management/agent.html
+
+Ganglia Reporting
+-----------------
+Flume can also report these metrics to
+Ganglia 3 or Ganglia 3.1 metanodes. To report metrics to Ganglia, a flume agent
+must be started with this support. The Flume agent has to be started by passing
+in the following parameters as system properties prefixed by ``flume.monitoring.``,
+and can be specified in the flume-env.sh:
+
+======================= ======= =====================================================================================
+Property Name Default Description
+======================= ======= =====================================================================================
+**type** -- The component type name, has to be ``ganglia``
+**hosts** -- Comma-separated list of ``hostname:port`` of Ganglia servers
+pollFrequency 60 Time, in seconds, between consecutive reporting to Ganglia server
+isGanglia3 false Ganglia server version is 3. By default, Flume sends in Ganglia 3.1 format
+======================= ======= =====================================================================================
+
+We can start Flume with Ganglia support as follows::
+
+ $ bin/flume-ng agent --conf-file example.conf --name a1 -Dflume.monitoring.type=ganglia -Dflume.monitoring.hosts=com.example:1234,com.example2:5455
+
+JSON Reporting
+--------------
+Flume can also report metrics in a JSON format. To enable reporting in JSON format, Flume hosts
+a Web server on a configurable port. Flume reports metrics in the following JSON format:
+
+.. code-block:: java
+
+ {
+ "typeName1.componentName1" : {"metric1" : "metricValue1", "metric2" : "metricValue2"},
+ "typeName2.componentName2" : {"metric3" : "metricValue3", "metric4" : "metricValue4"}
+ }
+
+Here is an example:
+
+.. code-block:: java
+
+ {
+ "CHANNEL.fileChannel":{"EventPutSuccessCount":"468085",
+ "Type":"CHANNEL",
+ "StopTime":"0",
+ "EventPutAttemptCount":"468086",
+ "ChannelSize":"233428",
+ "StartTime":"1344882233070",
+ "EventTakeSuccessCount":"458200",
+ "ChannelCapacity":"600000",
+ "EventTakeAttemptCount":"458288"},
+ "CHANNEL.memChannel":{"EventPutSuccessCount":"22948908",
+ "Type":"CHANNEL",
+ "StopTime":"0",
+ "EventPutAttemptCount":"22948908",
+ "ChannelSize":"5",
+ "StartTime":"1344882209413",
+ "EventTakeSuccessCount":"22948900",
+ "ChannelCapacity":"100",
+ "EventTakeAttemptCount":"22948908"}
+ }
+
+======================= ======= =====================================================================================
+Property Name Default Description
+======================= ======= =====================================================================================
+**type** -- The component type name, has to be ``http``
+port 41414 The port to start the server on.
+======================= ======= =====================================================================================
+
+We can start Flume with JSON Reporting support as follows::
+
+ $ bin/flume-ng agent --conf-file example.conf --name a1 -Dflume.monitoring.type=http -Dflume.monitoring.port=34545
+
+Metrics will then be available at **http://<hostname>:<port>/metrics** webpage.
+Custom components can report metrics as mentioned in the Ganglia section above.
+
+Custom Reporting
+----------------
+It is possible to report metrics to other systems by writing servers that do
+the reporting. Any reporting class has to implement the interface,
+``org.apache.flume.instrumentation.MonitorService``. Such a class can be used
+the same way the GangliaServer is used for reporting. They can poll the platform
+mbean server to poll the mbeans for metrics. For example, if an HTTP
+monitoring service called ``HTTPReporting`` can be used as follows::
+
+ $ bin/flume-ng agent --conf-file example.conf --name a1 -Dflume.monitoring.type=com.example.reporting.HTTPReporting -Dflume.monitoring.node=com.example:332
+
+======================= ======= ========================================
+Property Name Default Description
+======================= ======= ========================================
+**type** -- The component type name, has to be FQCN
+======================= ======= ========================================
+
+Reporting metrics from custom components
+----------------------------------------
+Any custom flume components should inherit from the
+``org.apache.flume.instrumentation.MonitoredCounterGroup`` class. The class
+should then provide getter methods for each of the metrics it exposes. See
+the code below. The MonitoredCounterGroup expects a list of attributes whose
+metrics are exposed by this class. As of now, this class only supports exposing
+metrics as long values.
+
+.. code-block:: java
+
+ public class SinkCounter extends MonitoredCounterGroup implements
+ SinkCounterMBean {
+
+ private static final String COUNTER_CONNECTION_CREATED =
+ "sink.connection.creation.count";
+
+ private static final String COUNTER_CONNECTION_CLOSED =
+ "sink.connection.closed.count";
+
+ private static final String COUNTER_CONNECTION_FAILED =
+ "sink.connection.failed.count";
+
+ private static final String COUNTER_BATCH_EMPTY =
+ "sink.batch.empty";
+
+ private static final String COUNTER_BATCH_UNDERFLOW =
+ "sink.batch.underflow";
+
+ private static final String COUNTER_BATCH_COMPLETE =
+ "sink.batch.complete";
+
+ private static final String COUNTER_EVENT_DRAIN_ATTEMPT =
+ "sink.event.drain.attempt";
+
+ private static final String COUNTER_EVENT_DRAIN_SUCCESS =
+ "sink.event.drain.sucess";
+
+ private static final String[] ATTRIBUTES = {
+ COUNTER_CONNECTION_CREATED, COUNTER_CONNECTION_CLOSED,
+ COUNTER_CONNECTION_FAILED, COUNTER_BATCH_EMPTY,
+ COUNTER_BATCH_UNDERFLOW, COUNTER_BATCH_COMPLETE,
+ COUNTER_EVENT_DRAIN_ATTEMPT, COUNTER_EVENT_DRAIN_SUCCESS
+ };
+
+
+ public SinkCounter(String name) {
+ super(MonitoredCounterGroup.Type.SINK, name, ATTRIBUTES);
+ }
+
+ @Override
+ public long getConnectionCreatedCount() {
+ return get(COUNTER_CONNECTION_CREATED);
+ }
+
+ public long incrementConnectionCreatedCount() {
+ return increment(COUNTER_CONNECTION_CREATED);
+ }
+
+ }
+
+Tools
+=====
+
+File Channel Integrity Tool
+---------------------------
+
+File Channel Integrity tool verifies the integrity of individual Events in the File channel
+and removes corrupted Events.
+
+The tools can be run as follows::
+
+ $bin/flume-ng tool --conf ./conf FCINTEGRITYTOOL -l ./datadir
+
+where datadir is the comma separated list of data directory to be verified.
+
+Following are the options available
+
+======================= ====================================================================
+Option Name Description
+======================= ====================================================================
+h/help Displays help
+**l/dataDirs** Comma-separated list of data directories which the tool must verify
+======================= ====================================================================
+
+Event Validator Tool
+--------------------
+
+Event validator tool can be used to validate the File Channel Event's in application specific way.
+The tool applies the user provider validation login on each event and drop the event which do not
+confirm to the logic.
+
+The tools can be run as follows::
+
+ $bin/flume-ng tool --conf ./conf FCINTEGRITYTOOL -l ./datadir -e org.apache.flume.MyEventValidator -DmaxSize 2000
+
+where datadir is the comma separated list of data directory to be verified.
+
+Following are the options available
+
+======================= ====================================================================
+Option Name Description
+======================= ====================================================================
+h/help Displays help
+**l/dataDirs** Comma-separated list of data directories which the tool must verify
+e/eventValidator Fully Qualified Name of Event Validator Implementation. The jar must
+ be on Flume classpath
+======================= ====================================================================
+
+The Event validator implementation must implement EventValidator interface. It's recommended
+not to throw any exception from the implementation as they are treated as invalid events.
+Additional parameters can be passed to EventValitor implementation via -D options.
+
+Let's see an example of simple size based Event Validator, which shall reject event's larger
+than maximum size specified.
+
+.. code-block:: java
+
+ public static class MyEventValidator implements EventValidator {
+
+ private int value = 0;
+
+ private MyEventValidator(int val) {
+ value = val;
+ }
+
+ @Override
+ public boolean validateEvent(Event event) {
+ return event.getBody() <= value;
+ }
+
+ public static class Builder implements EventValidator.Builder {
+
+ private int sizeValidator = 0;
+
+ @Override
+ public EventValidator build() {
+ return new DummyEventVerifier(sizeValidator);
+ }
+
+ @Override
+ public void configure(Context context) {
+ binaryValidator = context.getInteger("maxSize");
+ }
+ }
+ }
+
+
+Topology Design Considerations
+==============================
+Flume is very flexible and allows a large range of possible deployment
+scenarios. If you plan to use Flume in a large, production deployment, it is
+prudent to spend some time thinking about how to express your problem in
+terms of a Flume topology. This section covers a few considerations.
+
+Is Flume a good fit for your problem?
+-------------------------------------
+If you need to ingest textual log data into Hadoop/HDFS then Flume is the
+right fit for your problem, full stop. For other use cases, here are some
+guidelines:
+
+Flume is designed to transport and ingest regularly-generated event data over
+relatively stable, potentially complex topologies. The notion of "event data"
+is very broadly defined. To Flume, an event is just a generic blob of bytes.
+There are some limitations on how large an event can be - for instance, it
+cannot be larger than what you can store in memory or on disk on a single machine -
+but in practice, flume events can be everything from textual log entries to
+image files. The key property of an event is that they are generated in a
+continuous, streaming fashion. If your data is not regularly generated
+(i.e. you are trying to do a single bulk load of data into a Hadoop cluster)
+then Flume will still work, but it is probably overkill for your situation.
+Flume likes relatively stable topologies. Your topologies do not need to be
+immutable, because Flume can deal with changes in topology without losing data
+and can also tolerate periodic reconfiguration due to fail-over or
+provisioning. It probably won't work well if you plant to change topologies
+every day, because reconfiguration takes some thought and overhead.
+
+Flow reliability in Flume
+-------------------------
+The reliability of a Flume flow depends on several factors. By adjusting these
+factors, you can achieve a wide array of reliability options with Flume.
+
+**What type of channel you use.** Flume has both durable channels (those which
+will persist data to disk) and non durable channels (those which will lose
+data if a machine fails). Durable channels use disk-based storage, and data
+stored in such channels will persist across machine restarts or non
+disk-related failures.
+
+**Whether your channels are sufficiently provisioned for the workload.** Channels
+in Flume act as buffers at various hops. These buffers have a fixed capacity,
+and once that capacity is full you will create back pressure on earlier points
+in the flow. If this pressure propagates to the source of the flow, Flume will
+become unavailable and may lose data.
+
+**Whether you use redundant topologies.** Flume let's you replicate flows
+across redundant topologies. This can provide a very easy source of fault
+tolerance and one which is overcomes both disk or machine failures.
+
+*The best way to think about reliability in a Flume topology is to consider
+various failure scenarios and their outcomes.* What happens if a disk fails?
+What happens if a machine fails? What happens if your terminal sink
+(e.g. HDFS) goes down for some time and you have back pressure? The space of
+possible designs is huge, but the underlying questions you need to ask are
+just a handful.
+
+Flume topology design
+---------------------
+The first step in designing a Flume topology is to enumerate all sources
+and destinations (terminal sinks) for your data. These will define the edge
+points of your topology. The next consideration is whether to introduce
+intermediate aggregation tiers or event routing. If you are collecting data
+form a large number of sources, it can be helpful to aggregate the data in
+order to simplify ingestion at the terminal sink. An aggregation tier can
+also smooth out burstiness from sources or unavailability at sinks, by
+acting as a buffer. If you are routing data between different locations,
+you may also want to split flows at various points: this creates
+sub-topologies which may themselves include aggregation points.
+
+Sizing a Flume deployment
+-------------------------
+Once you have an idea of what your topology will look like, the next question
+is how much hardware and networking capacity is needed. This starts by
+quantifying how much data you generate. That is not always
+a simple task! Most data streams are bursty (for instance, due to diurnal
+patterns) and potentially unpredictable. A good starting point is to think
+about the maximum throughput you'll have in each tier of the topology, both
+in terms of *events per second* and *bytes per second*. Once you know the
+required throughput of a given tier, you can calulate a lower bound on how many
+nodes you require for that tier. To determine attainable throughput, it's
+best to experiment with Flume on your hardware, using synthetic or sampled
+event data. In general, disk-based channels
+should get 10's of MB/s and memory based channels should get 100's of MB/s or
+more. Performance will vary widely, however depending on hardware and
+operating environment.
+
+Sizing aggregate throughput gives you a lower bound on the number of nodes
+you will need to each tier. There are several reasons to have additional
+nodes, such as increased redundancy and better ability to absorb bursts in load.
+
+Troubleshooting
+===============
+
+Handling agent failures
+-----------------------
+
+If the Flume agent goes down, then the all the flows hosted on that agent are
+aborted. Once the agent is restarted, then flow will resume. The flow using
+file channel or other stable channel will resume processing events where it left
+off. If the agent can't be restarted on the same hardware, then there is an option to
+migrate the database to another hardware and setup a new Flume agent that
+can resume processing the events saved in the db. The database HA futures
+can be leveraged to move the Flume agent to another host.
+
+Compatibility
+-------------
+
+HDFS
+~~~~
+
+Currently Flume supports HDFS 0.20.2 and 0.23.
+
+AVRO
+~~~~
+
+TBD
+
+Additional version requirements
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+TBD
+
+Tracing
+-------
+
+TBD
+
+More Sample Configs
+-------------------
+
+TBD
+
+Component Summary
+=================
+
+============================================================ ====================== ====================================================================
+Component Interface Type Alias Implementation Class
+============================================================ ====================== ====================================================================
+org.apache.flume.Channel memory org.apache.flume.channel.MemoryChannel
+org.apache.flume.Channel jdbc org.apache.flume.channel.jdbc.JdbcChannel
+org.apache.flume.Channel file org.apache.flume.channel.file.FileChannel
+org.apache.flume.Channel -- org.apache.flume.channel.PseudoTxnMemoryChannel
+org.apache.flume.Channel -- org.example.MyChannel
+
+org.apache.flume.Source avro org.apache.flume.source.AvroSource
+org.apache.flume.Source netcat org.apache.flume.source.NetcatSource
+org.apache.flume.Source seq org.apache.flume.source.SequenceGeneratorSource
+org.apache.flume.Source exec org.apache.flume.source.ExecSource
+org.apache.flume.Source syslogtcp org.apache.flume.source.SyslogTcpSource
+org.apache.flume.Source multiport_syslogtcp org.apache.flume.source.MultiportSyslogTCPSource
+org.apache.flume.Source syslogudp org.apache.flume.source.SyslogUDPSource
+org.apache.flume.Source spooldir org.apache.flume.source.SpoolDirectorySource
+org.apache.flume.Source http org.apache.flume.source.http.HTTPSource
+org.apache.flume.Source thrift org.apache.flume.source.ThriftSource
+org.apache.flume.Source jms org.apache.flume.source.jms.JMSSource
+org.apache.flume.Source -- org.apache.flume.source.avroLegacy.AvroLegacySource
+org.apache.flume.Source -- org.apache.flume.source.thriftLegacy.ThriftLegacySource
+org.apache.flume.Source -- org.example.MySource
+
+org.apache.flume.Sink null org.apache.flume.sink.NullSink
+org.apache.flume.Sink logger org.apache.flume.sink.LoggerSink
+org.apache.flume.Sink avro org.apache.flume.sink.AvroSink
+org.apache.flume.Sink hdfs org.apache.flume.sink.hdfs.HDFSEventSink
+org.apache.flume.Sink hbase org.apache.flume.sink.hbase.HBaseSink
+org.apache.flume.Sink hbase2 org.apache.flume.sink.hbase2.HBase2Sink
+org.apache.flume.Sink asynchbase org.apache.flume.sink.hbase.AsyncHBaseSink
+org.apache.flume.Sink elasticsearch org.apache.flume.sink.elasticsearch.ElasticSearchSink
+org.apache.flume.Sink file_roll org.apache.flume.sink.RollingFileSink
+org.apache.flume.Sink irc org.apache.flume.sink.irc.IRCSink
+org.apache.flume.Sink thrift org.apache.flume.sink.ThriftSink
+org.apache.flume.Sink -- org.example.MySink
+
+org.apache.flume.ChannelSelector replicating org.apache.flume.channel.ReplicatingChannelSelector
+org.apache.flume.ChannelSelector multiplexing org.apache.flume.channel.MultiplexingChannelSelector
+org.apache.flume.ChannelSelector -- org.example.MyChannelSelector
+
+org.apache.flume.SinkProcessor default org.apache.flume.sink.DefaultSinkProcessor
+org.apache.flume.SinkProcessor failover org.apache.flume.sink.FailoverSinkProcessor
+org.apache.flume.SinkProcessor load_balance org.apache.flume.sink.LoadBalancingSinkProcessor
+org.apache.flume.SinkProcessor --
+
+org.apache.flume.interceptor.Interceptor timestamp org.apache.flume.interceptor.TimestampInterceptor$Builder
+org.apache.flume.interceptor.Interceptor host org.apache.flume.interceptor.HostInterceptor$Builder
+org.apache.flume.interceptor.Interceptor static org.apache.flume.interceptor.StaticInterceptor$Builder
+org.apache.flume.interceptor.Interceptor regex_filter org.apache.flume.interceptor.RegexFilteringInterceptor$Builder
+org.apache.flume.interceptor.Interceptor regex_extractor org.apache.flume.interceptor.RegexFilteringInterceptor$Builder
+
+org.apache.flume.channel.file.encryption.KeyProvider$Builder jceksfile org.apache.flume.channel.file.encryption.JCEFileKeyProvider
+org.apache.flume.channel.file.encryption.KeyProvider$Builder -- org.example.MyKeyProvider
+
+org.apache.flume.channel.file.encryption.CipherProvider aesctrnopadding org.apache.flume.channel.file.encryption.AESCTRNoPaddingProvider
+org.apache.flume.channel.file.encryption.CipherProvider -- org.example.MyCipherProvider
+
+org.apache.flume.serialization.EventSerializer$Builder text org.apache.flume.serialization.BodyTextEventSerializer$Builder
+org.apache.flume.serialization.EventSerializer$Builder avro_event org.apache.flume.serialization.FlumeEventAvroEventSerializer$Builder
+org.apache.flume.serialization.EventSerializer$Builder -- org.example.MyEventSerializer$Builder
+============================================================ ====================== ====================================================================
+
+Alias Conventions
+=================
+
+These conventions for alias names are used in the component-specific examples above, to keep the names short and consistent across all examples.
+
+========== =================
+Alias Name Alias Type
+========== =================
+a **a**\ gent
+c **c**\ hannel
+r sou\ **r**\ce
+k sin\ **k**
+g sink **g**\ roup
+i **i**\ nterceptor
+y ke\ **y**
+h **h**\ ost
+s **s**\ erializer
+========== =================
diff --git a/source/sphinx/_templates/extranav.html b/source/sphinx/_templates/extranav.html
new file mode 100644
index 00000000..46bd66c7
--- /dev/null
+++ b/source/sphinx/_templates/extranav.html
@@ -0,0 +1,23 @@
+
+<h3>Resources</h3>
+
+<ul class="this-page-menu">
+ <li><a href="https://issues.apache.org/jira/browse/FLUME">Flume Issue Tracking (Jira)</a></li>
+ <li><a href="http://cwiki.apache.org/confluence/display/FLUME">Flume Wiki</a></li>
+ <li><a href="http://cwiki.apache.org/confluence/display/FLUME/Getting+Started">Getting Started Guide</a></li>
+ <li><a href="https://builds.apache.org/job/flume-trunk/">Jenkins Continuous Integration Server</a></li>
+ <li><a href="https://builds.apache.org/analysis/">Sonar Code Quality Reports</a></li>
+</ul>
+
+<h3>Apache</h3>
+
+<ul class="this-page-menu">
+ <li><a href="http://www.apache.org">Home</a></li>
+ <li><a href="http://www.apache.org/foundation/sponsorship.html">Sponsorship</a></li>
+ <li><a href="http://www.apache.org/licenses">Licenses</a> </li>
+ <li><a href="http://www.apache.org/foundation/thanks.html">Thanks</a></li>
+ <li><a href="http://www.apachecon.com">Conferences</a></li>
+ <li><a href="http://www.apache.org/security/">Security</a></li>
+</ul>
+
+
diff --git a/source/sphinx/_themes/flume/globaltoc.html b/source/sphinx/_themes/flume/globaltoc.html
new file mode 100644
index 00000000..9fd1663c
--- /dev/null
+++ b/source/sphinx/_themes/flume/globaltoc.html
@@ -0,0 +1,2 @@
+<h3><a href="{{ pathto(master_doc) }}">{{ _('Apache Flume') }}</a></h3>
+{{ toctree() }}
\ No newline at end of file
diff --git a/source/sphinx/_themes/flume/layout.html b/source/sphinx/_themes/flume/layout.html
new file mode 100644
index 00000000..52dc35f8
--- /dev/null
+++ b/source/sphinx/_themes/flume/layout.html
@@ -0,0 +1,55 @@
+{% extends "basic/layout.html" %}
+
+{% block header %}
+{%- if logo %}
+<div class="header">
+ <table width="100%" border="0">
+ <tr>
+ <td width="10%">
+ <div class="logo">
+ <a href="{{ pathto(master_doc) }}">
+ <img class="logo" src="{{ pathto('_static/' + logo, 1) }}" alt="Logo"/>
+ </a>
+ </div>
+ </td>
+ <td width="2%">
+ <span class="trademark">™</span>
+ </td>
+ <td width="68%" align="center" class="pageTitle">{{ project }}<sup><span class="trademark">™</span></sup>
+ </td>
+ <td width="20%">
+ <a href="http://www.apache.org">
+ <img src="{{ pathto('_static/feather-small.png', 1) }}" alt="Apache Software Foundation" height="70"/>
+ </a>
+ </td>
+ </tr>
+ </table>
+</div>
+{%- endif %}
+{% endblock %}
+
+{%- block footer %}
+<div class="footer">
+ {%- if show_copyright %}
+ {%- if hasdoc('copyright') %}
+ {% trans path=pathto('copyright'), copyright=copyright|e %}© <a href="{{ path }}">Copyright</a> {{ copyright }}.{% endtrans %}
+ {%- else %}
+ {% trans copyright=copyright|e %}© Copyright {{ copyright }}.{% endtrans %}
+ {%- endif %}
+ {%- endif %}
+ {%- if last_updated %}
+ {% trans last_updated=last_updated|e %}Last updated on {{ last_updated }}.{% endtrans %}
+ {%- endif %}
+ {%- if show_sphinx %}
+ {% trans sphinx_version=sphinx_version|e %}Created using <a href="http://sphinx.pocoo.org/">Sphinx</a> {{ sphinx_version }}.{% endtrans %}
+ {%- endif %}
+</div>
+{%- endblock %}
+
+{%- block relbar1 %}{% endblock %}
+
+{%- block relbar2 %}{% endblock %}
+
+{%- block sidebarlogo %}{% endblock %}
+
+
diff --git a/source/sphinx/_themes/flume/localtoc.html b/source/sphinx/_themes/flume/localtoc.html
new file mode 100644
index 00000000..aef70af7
--- /dev/null
+++ b/source/sphinx/_themes/flume/localtoc.html
@@ -0,0 +1,4 @@
+{%- if display_toc %}
+<h3><a href="{{ pathto(master_doc) }}">{{ _('This Page') }}</a></h3>
+{{ toc }}
+{%- endif %}
\ No newline at end of file
diff --git a/source/sphinx/_themes/flume/static/flume.css_t b/source/sphinx/_themes/flume/static/flume.css_t
new file mode 100644
index 00000000..b7cb6369
--- /dev/null
+++ b/source/sphinx/_themes/flume/static/flume.css_t
@@ -0,0 +1,346 @@
+/*
+ * default.css_t
+ * ~~~~~~~~~~~~~
+ *
+ * Sphinx stylesheet -- default theme.
+ *
+ * :copyright: Copyright 2007-2011 by the Sphinx team, see AUTHORS.
+ * :license: BSD, see LICENSE for details.
+ *
+ */
+
+@import url('https://fonts.googleapis.com/css?family=Baloo+Chettan');
+@import url("basic.css");
+
+/* -- page layout ----------------------------------------------------------- */
+
+body {
+ font-family: {{ theme_bodyfont }};
+ font-size: 100%;
+ background-color: {{ theme_footerbgcolor }};
+ color: #000;
+ margin: 0;
+ padding: 0;
+}
+
+div.document {
+ background-color: {{ theme_sidebarbgcolor }};
+}
+
+div.documentwrapper {
+ float: left;
+ width: 100%;
+}
+
+div.bodywrapper {
+ margin: 0 0 0 {{ theme_sidebarwidth|toint }}px;
+}
+
+div.body {
+ background-color: {{ theme_bgcolor }};
+ color: {{ theme_textcolor }};
+ padding: 0 20px 30px 20px;
+ min-height: 30em; /* otherwise the sidebar wraps under the body */
+}
+
+{%- if theme_rightsidebar|tobool %}
+div.bodywrapper {
+ margin: 0 {{ theme_sidebarwidth|toint }}px 0 0;
+}
+{%- endif %}
+
+span.trademark {
+ font-family: sans-serif;
+ font-size: 16px;
+ position: relative;
+ top: -10px;
+}
+
+td.pageTitle {
+ font-size: 70px;
+ color: {{ theme_headertextcolor }};
+ font-family: 'Baloo Chettan', sans-serif;
+ font-weight:bold
+}
+
+div.header {
+ color: {{ theme_headertextcolor }};
+ background-color: {{ theme_headerbgcolor }};
+}
+
+div.footer {
+ color: {{ theme_footertextcolor }};
+ width: 100%;
+ padding: 9px 0 9px 0;
+ text-align: center;
+ font-size: 75%;
+ height: 30px;
+ background-color: {{ theme_footerbgcolor }};
+ bottom: 0;
+}
+
+div.footer a {
+ color: {{ theme_footertextcolor }};
+ text-decoration: underline;
+}
+
+div.related {
+ background-color: {{ theme_relbarbgcolor }};
+ line-height: 30px;
+ color: {{ theme_relbartextcolor }};
+}
+
+div.related a {
+ color: {{ theme_relbarlinkcolor }};
+}
+
+div.sphinxsidebar {
+ font-size: 85%;
+ {%- if theme_stickysidebar|tobool %}
+ top: 30px;
+ bottom: 0;
+ margin: 0;
+ position: fixed;
+ overflow: auto;
+ height: auto;
+ {%- endif %}
+ {%- if theme_rightsidebar|tobool %}
+ float: right;
+ {%- if theme_stickysidebar|tobool %}
+ right: 0;
+ {%- endif %}
+ {%- endif %}
+}
+
+{%- if theme_stickysidebar|tobool %}
+/* this is nice, but it it leads to hidden headings when jumping
+ to an anchor */
+/*
+div.related {
+ position: fixed;
+}
+
+div.documentwrapper {
+ margin-top: 30px;
+}
+*/
+{%- endif %}
+
+div.sphinxsidebar h3 {
+ font-family: {{ theme_headfont }};
+ color: {{ theme_sidebartextcolor }};
+ font-size: 1.4em;
+ font-weight: normal;
+ margin: 0;
+ padding: 0;
+}
+
+div.sphinxsidebar h3 a {
+ color: {{ theme_sidebartextcolor }};
+}
+
+div.sphinxsidebar h4 {
+ font-family: {{ theme_headfont }};
+ color: {{ theme_sidebartextcolor }};
+ font-size: 1.3em;
+ font-weight: normal;
+ margin: 5px 0 0 0;
+ padding: 0;
+}
+
+div.sphinxsidebar p {
+ color: {{ theme_sidebartextcolor }};
+}
+
+div.sphinxsidebar p.topless {
+ margin: 5px 10px 10px 10px;
+}
+
+div.sphinxsidebar ul {
+ margin: 10px;
+ padding: 0;
+ color: {{ theme_sidebartextcolor }};
+}
+
+div.sphinxsidebar a {
+ color: {{ theme_sidebarlinkcolor }};
+}
+
+div.sphinxsidebar input {
+ border: 1px solid {{ theme_sidebarlinkcolor }};
+ font-family: sans-serif;
+ font-size: 1em;
+}
+
+{% if theme_collapsiblesidebar|tobool %}
+/* for collapsible sidebar */
+div#sidebarbutton {
+ background-color: {{ theme_sidebarbtncolor }};
+}
+{% endif %}
+
+/* -- hyperlink styles ------------------------------------------------------ */
+
+a {
+ color: {{ theme_linkcolor }};
+ text-decoration: none;
+}
+
+a:visited {
+ color: {{ theme_visitedlinkcolor }};
+ text-decoration: none;
+}
+
+a:hover {
+ text-decoration: underline;
+}
+
+{% if theme_externalrefs|tobool %}
+a.external {
+ text-decoration: none;
+ border-bottom: 1px dashed {{ theme_linkcolor }};
+}
+
+a.external:hover {
+ text-decoration: none;
+ border-bottom: none;
+}
+
+a.external:visited {
+ text-decoration: none;
+ border-bottom: 1px dashed {{ theme_visitedlinkcolor }};
+}
+{% endif %}
+
+/* -- body styles ----------------------------------------------------------- */
+
+div.body h1,
+div.body h2,
+div.body h3,
+div.body h4,
+div.body h5,
+div.body h6 {
+ font-family: {{ theme_headfont }};
+ background-color: {{ theme_headbgcolor }};
+ font-weight: normal;
+ color: {{ theme_headtextcolor }};
+ border-bottom: 1px solid #ccc;
+ margin: 20px -20px 10px -20px;
+ padding: 3px 0 3px 10px;
+}
+
+div.body h1 { margin-top: 0; font-size: 200%; }
+div.body h2 { font-size: 160%; }
+div.body h3 { font-size: 140%; }
+div.body h4 { font-size: 120%; }
+div.body h5 { font-size: 110%; }
+div.body h6 { font-size: 100%; }
+
+a.headerlink {
+ color: {{ theme_headlinkcolor }};
+ font-size: 0.8em;
+ padding: 0 4px 0 4px;
+ text-decoration: none;
+}
+
+a.headerlink:hover {
+ background-color: {{ theme_headlinkcolor }};
+ color: white;
+}
+
+div.body p, div.body dd, div.body li {
+ text-align: justify;
+ line-height: 130%;
+}
+
+div.admonition p.admonition-title + p {
+ display: inline;
+}
+
+div.admonition p {
+ margin-bottom: 5px;
+}
+
+div.admonition pre {
+ margin-bottom: 5px;
+}
+
+div.admonition ul, div.admonition ol {
+ margin-bottom: 5px;
+}
+
+div.note {
+ background-color: #eee;
+ border: 1px solid #ccc;
+}
+
+div.seealso {
+ background-color: #ffc;
+ border: 1px solid #ff6;
+}
+
+div.topic {
+ background-color: #eee;
+}
+
+div.warning {
+ background-color: #ffe4e4;
+ border: 1px solid #f66;
+}
+
+p.rubric {
+ font-family: {{ theme_headfont }};
+ background-color: {{ theme_headbgcolor }};
+ font-weight: normal;
+ font-size: 160%;
+ color: {{ theme_headtextcolor }};
+ border-bottom: 1px solid #ccc;
+ margin: 20px -20px 10px -20px;
+ padding: 3px 0 3px 10px;
+}
+
+p.admonition-title {
+ display: inline;
+}
+
+p.admonition-title:after {
+ content: ":";
+}
+
+pre {
+ padding: 5px;
+ background-color: {{ theme_codebgcolor }};
+ color: {{ theme_codetextcolor }};
+ line-height: 120%;
+ border: 1px solid #ac9;
+ border-left: none;
+ border-right: none;
+}
+
+tt {
+ background-color: #ecf0f3;
+ padding: 0 1px 0 1px;
+ font-size: 0.95em;
+}
+
+th {
+ background-color: #eee;
+}
+
+.warning tt {
+ background: #efc2c2;
+}
+
+.note tt {
+ background: #d6d6d6;
+}
+
+.viewcode-back {
+ font-family: {{ theme_bodyfont }};
+}
+
+div.viewcode-block:target {
+ background-color: #f4debf;
+ border-top: 1px solid #ac9;
+ border-bottom: 1px solid #ac9;
+}
diff --git a/source/sphinx/_themes/flume/theme.conf b/source/sphinx/_themes/flume/theme.conf
new file mode 100644
index 00000000..4f0c1780
--- /dev/null
+++ b/source/sphinx/_themes/flume/theme.conf
@@ -0,0 +1,35 @@
+[theme]
+inherit = basic
+stylesheet = flume.css
+
+[options]
+rightsidebar = false
+stickysidebar = false
+collapsiblesidebar = false
+externalrefs = false
+
+bodyfont = verdana, sans-serif
+headfont = georgia, serif
+footerbgcolor = #11303d
+footertextcolor = #ffffff
+sidebarbgcolor = #1c4e63
+sidebarbtncolor = #3c6e83
+sidebartextcolor = #ffffff
+sidebarlinkcolor = #98dbdd
+relbarbgcolor = #133f52
+relbartextcolor = #ffffff
+relbarlinkcolor = #ffffff
+bgcolor = #ffffff
+textcolor = #000000
+headbgcolor = #f2f2f2
+headtextcolor = #20435c
+headlinkcolor = #c60f0f
+linkcolor = #355f7c
+visitedlinkcolor = #355f7c
+codebgcolor = #eeffcc
+codetextcolor = #333333
+headertextcolor = #ffffff
+headerbgcolor = #11303d
+
+
+
diff --git a/source/sphinx/conf.py b/source/sphinx/conf.py
new file mode 100644
index 00000000..639b1b6a
--- /dev/null
+++ b/source/sphinx/conf.py
@@ -0,0 +1,142 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# -*- coding: utf-8 -*-
+
+import sys
+import os
+from datetime import date
+
+# eventlet/gevent should not monkey patch anything.
+os.environ["GEVENT_NOPATCH"] = "yes"
+os.environ["EVENTLET_NOPATCH"] = "yes"
+#os.environ["CELERY_LOADER"] = "default"
+
+this = os.path.dirname(os.path.abspath(__file__))
+
+# If your extensions are in another directory, add it here. If the directory
+# is relative to the documentation root, use os.path.abspath to make it
+# absolute, like shown here.
+sys.path.append(os.path.join(os.pardir, "tests"))
+sys.path.append(os.path.join(this, "_ext"))
+#import celery
+
+# General configuration
+# ---------------------
+
+# Add any Sphinx extension module names here, as strings. They can be extensions
+# coming with Sphinx (named 'sphinx.ext.*') or your custom ones.
+extensions = [
+ 'sphinx.ext.autodoc',
+ #~ 'sphinx.ext.autosummary',
+ 'sphinx.ext.inheritance_diagram',
+ 'sphinx.ext.todo',
+ 'sphinx.ext.extlinks',
+ 'sphinx.ext.graphviz',
+ 'sphinx.ext.intersphinx',
+ #'rst2pdf.pdfbuilder',
+ #~ 'sphinx.ext.doctest',
+]
+
+html_show_sphinx = False
+
+# Add any paths that contain templates here, relative to this directory.
+templates_path = ['_templates']
+
+# The suffix of source filenames.
+source_suffix = '.rst'
+
+# The master toctree document.
+master_doc = 'index'
+
+# General information about the project.
+project = 'Apache Flume'
+copyright = '2009-%s The Apache Software Foundation. Apache Flume, Flume, Apache, the Apache feather logo, and the Apache Flume project logo are trademarks of The Apache Software Foundation.' % date.today().year
+
+keep_warnings = True
+
+# The version info for the project you're documenting, acts as replacement for
+# |version| and |release|, also used in various other places throughout the
+# built documents.
+#
+# The short X.Y version.
+#version = ".".join(map(str, celery.VERSION[0:2]))
+# The full version, including alpha/beta/rc tags.
+#release = celery.__version__
+
+exclude_trees = ['.build']
+
+# If true, '()' will be appended to :func: etc. cross-reference text.
+add_function_parentheses = True
+
+# If true, the current module name will be prepended to all description
+# unit titles (such as .. function::).
+#add_module_names = True
+
+#intersphinx_mapping = {
+# "http://docs.python.org/dev": None,
+# "http://kombu.readthedocs.org/en/latest/": None,
+# "http://django-celery.readthedocs.org/en/latest": None,
+#}
+
+# The name of the Pygments (syntax highlighting) style to use.
+pygments_style = 'trac'
+highlight_language = 'none'
+
+html_theme_path = ['_themes']
+
+# Add any paths that contain custom static files (such as style sheets) here,
+# relative to this directory. They are copied after the builtin static files,
+# so a file named "default.css" will overwrite the builtin "default.css".
+#html_static_path = ['../resources/images']
+html_static_path = ['images/feather-small.png']
+
+html_logo = 'images/flume-logo.png'
+
+html_use_smartypants = True
+
+# If false, no module index is generated.
+html_use_modindex = False
+
+# If false, no index is generated.
+html_use_index = False
+
+html_theme = 'flume'
+
+html_theme_options = {
+ 'sidebarbgcolor': '#3076B3',
+ 'sidebartextcolor': 'White',
+ 'headtextcolor': '#4466AA',
+ 'footerbgcolor': 'White',
+ 'footertextcolor': 'Black',
+ 'headerbgcolor': '#f9f9f9',
+ 'headertextcolor': '#4466AA',
+}
+
+html_title = 'Apache Flume'
+
+#html_sidebars = {
+# '**': ['localtoc.html', 'relations.html', 'sourcelink.html'],
+#}
+
+# Custom sidebar templates, maps document names to template names.
+html_sidebars = {
+ '**': ['globaltoc.html', 'extranav.html', 'localtoc.html'],
+}
+
+pdf_documents = [
+ ('FlumeUserGuide', 'FlumeUserGuide.tex', 'Flume User Guide', 'Apache Flume'),
+ ('FlumeDeveloperGuide', 'FlumeDeveloperGuide.tex', 'Flume Developer Guide', 'Apache Flume'),
+]
diff --git a/source/sphinx/contents.rsx b/source/sphinx/contents.rsx
new file mode 100644
index 00000000..857cfa1b
--- /dev/null
+++ b/source/sphinx/contents.rsx
@@ -0,0 +1,38 @@
+.. Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+
+========
+Contents
+========
+
+* :ref:`Project information <project-info.html>`
+* :ref:`Download Apache Flume <http://www.apache.org/dyn/closer.cgi/incubator/flume>`
+* :ref:`Flume Issue Tracking (JIRA) <https://issues.apache.org/jira/browse/FLUME>`
+* :ref:`Flume Source Code (SVN) <https://svn.apache.org/repos/asf/incubator/flume/>`
+* :ref:`License <license.html>`
+
+* :ref:`doc`
+
+* :ref:`resources`
+
+
+ .. toctree::
+ :maxdepth: 2
+
+ index
+ license
+
+ documentation
+ resources
\ No newline at end of file
diff --git a/source/sphinx/documentation.rst b/source/sphinx/documentation.rst
new file mode 100644
index 00000000..a2bdadb5
--- /dev/null
+++ b/source/sphinx/documentation.rst
@@ -0,0 +1,32 @@
+=============
+Documentation
+=============
+
+The latest released version:
+
+* `Flume User Guide <releases/content/1.9.0/FlumeUserGuide.html>`__
+* `Flume Developer Guide <releases/content/1.9.0/FlumeDeveloperGuide.html>`__
+
+The documents below are the very most recent versions of the documentation and may contain features that have not
+been released.
+
+* |user_guide_github| (unreleased version on github)
+* |developer_guide_github| (unreleased version on github)
+
+For documentation on released versions of Flume, please see the `Releases <releases/>`__ page.
+
+.. toctree::
+ :maxdepth: 1
+ :hidden:
+
+ FlumeUserGuide
+ FlumeDeveloperGuide
+
+.. |user_guide_github| raw:: html
+
+ <a href="https://github.com/apache/flume/blob/trunk/flume-ng-doc/sphinx/FlumeUserGuide.rst" target="_blank">Flume User Guide</a>
+
+.. |developer_guide_github| raw:: html
+
+ <a href="https://github.com/apache/flume/blob/trunk/flume-ng-doc/sphinx/FlumeDeveloperGuide.rst" target="_blank">Flume Developer Guide</a>
+
diff --git a/source/sphinx/download.rst b/source/sphinx/download.rst
new file mode 100644
index 00000000..2cffa8d0
--- /dev/null
+++ b/source/sphinx/download.rst
@@ -0,0 +1,35 @@
+========
+Download
+========
+
+Apache Flume is distributed under the `Apache License, version 2.0 <http://www.apache.org/licenses/LICENSE-2.0.html>`_
+
+The link in the Mirrors column should display a list of available mirrors with a default selection based on your
+inferred location. If you do not see that page, try a different browser. The checksum and signature are links to the
+originals on the main distribution server.
+
+.. csv-table::
+
+ "Apache Flume binary (tar.gz)", `apache-flume-1.9.0-bin.tar.gz <http://www.apache.org/dyn/closer.lua/flume/1.9.0/apache-flume-1.9.0-bin.tar.gz>`_, `apache-flume-1.9.0-bin.tar.gz.sha512 <http://www.apache.org/dist/flume/1.9.0/apache-flume-1.9.0-bin.tar.gz.sha512>`_, `apache-flume-1.9.0-bin.tar.gz.asc <http://www.apache.org/dist/flume/1.9.0/apache-flume-1.9.0-bin.tar.gz.asc>`_
+ "Apache Flume source (tar.gz)", `apache-flume-1.9.0-src.tar.gz <http://www.apache.org/dyn/closer.lua/flume/1.9.0/apache-flume-1.9.0-src.tar.gz>`_, `apache-flume-1.9.0-src.tar.gz.sha512 <http://www.apache.org/dist/flume/1.9.0/apache-flume-1.9.0-src.tar.gz.sha512>`_, `apache-flume-1.9.0-src.tar.gz.asc <http://www.apache.org/dist/flume/1.9.0/apache-flume-1.9.0-src.tar.gz.asc>`_
+
+It is essential that you verify the integrity of the downloaded files using the PGP or MD5 signatures. Please read
+`Verifying Apache HTTP Server Releases <http://httpd.apache.org/dev/verification.html>`_ for more information on
+why you should verify our releases.
+
+The PGP signatures can be verified using PGP or GPG. First download the `KEYS <http://www.apache.org/dist/flume/KEYS>`_
+as well as the asc signature file for the relevant distribution. Make sure you get these files from the
+`main distribution directory <http://www.apache.org/dist/flume/>`_ rather than from a mirror.
+Then verify the signatures using::
+
+ % gpg --import KEYS
+ % gpg --verify apache-flume-1.9.0-src.tar.gz.asc
+
+Apache Flume 1.9.0 is signed by Ferenc Szabo 79E8E648
+
+Alternatively, you can verify the MD5 or SHA1 signatures of the files. A program called md5, md5sum, or shasum is included in many
+Unix distributions for this purpose.
+
+.. rubric:: Previous_Releases
+
+All previous releases of Apache Flume can be found in the `archive repository <http://archive.apache.org/dist/flume>`_.
diff --git a/source/sphinx/getinvolved.rst b/source/sphinx/getinvolved.rst
new file mode 100644
index 00000000..453bd94f
--- /dev/null
+++ b/source/sphinx/getinvolved.rst
@@ -0,0 +1,15 @@
+===================
+How to Get Involved
+===================
+The Apache Flume project needs and appreciates all contributions,
+including documentation help, source code improvements, problem reports, and
+even general feedback! If you are interested
+in contributing, please visit our Wiki page on how to contribute at:
+https://cwiki.apache.org/confluence/display/FLUME/How+to+Contribute
+
+Also, consider joining the `mailing lists <mailinglists.html>`__.
+
+.. rubric:: How to Report Issues
+
+The Apache Flume project uses JIRA for issue tracking. Please report any issues
+you find at: https://issues.apache.org/jira/browse/FLUME
diff --git a/source/sphinx/images/DevGuide_image00.png b/source/sphinx/images/DevGuide_image00.png
new file mode 100755
index 00000000..dd7eff0d
Binary files /dev/null and b/source/sphinx/images/DevGuide_image00.png differ
diff --git a/source/sphinx/images/DevGuide_image01.png b/source/sphinx/images/DevGuide_image01.png
new file mode 100755
index 00000000..ed77828d
Binary files /dev/null and b/source/sphinx/images/DevGuide_image01.png differ
diff --git a/source/sphinx/images/UserGuide_image00.png b/source/sphinx/images/UserGuide_image00.png
new file mode 100755
index 00000000..e587daf9
Binary files /dev/null and b/source/sphinx/images/UserGuide_image00.png differ
diff --git a/source/sphinx/images/UserGuide_image01.png b/source/sphinx/images/UserGuide_image01.png
new file mode 100755
index 00000000..ea97b2d1
Binary files /dev/null and b/source/sphinx/images/UserGuide_image01.png differ
diff --git a/source/sphinx/images/UserGuide_image02.png b/source/sphinx/images/UserGuide_image02.png
new file mode 100755
index 00000000..85bdd952
Binary files /dev/null and b/source/sphinx/images/UserGuide_image02.png differ
diff --git a/source/sphinx/images/UserGuide_image03.png b/source/sphinx/images/UserGuide_image03.png
new file mode 100755
index 00000000..b028b8b1
Binary files /dev/null and b/source/sphinx/images/UserGuide_image03.png differ
diff --git a/source/sphinx/images/feather-small.png b/source/sphinx/images/feather-small.png
new file mode 100644
index 00000000..b944a35e
Binary files /dev/null and b/source/sphinx/images/feather-small.png differ
diff --git a/source/sphinx/images/flume-logo.png b/source/sphinx/images/flume-logo.png
new file mode 100644
index 00000000..f9ac5e2e
Binary files /dev/null and b/source/sphinx/images/flume-logo.png differ
diff --git a/source/sphinx/index.rst b/source/sphinx/index.rst
new file mode 100644
index 00000000..8cded2cd
--- /dev/null
+++ b/source/sphinx/index.rst
@@ -0,0 +1,557 @@
+.. Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+
+
+Welcome to Apache Flume
+=======================
+
+Flume is a distributed, reliable, and available service for
+efficiently collecting, aggregating, and moving large amounts of log
+data. It has a simple and flexible architecture based on streaming
+data flows. It is robust and fault tolerant with tunable reliability
+mechanisms and many failover and recovery mechanisms. It
+uses a simple extensible data model that allows for online analytic
+application.
+
+.. figure:: images/DevGuide_image00.png
+ :align: center
+ :alt: Agent component diagram
+
+.. rubric:: News
+
+.. raw:: html
+
+ <h3>January 8, 2019 - Apache Flume 1.9.0 Released</h3>
+
+The Apache Flume team is pleased to announce the release of Flume 1.9.0.
+
+Flume is a distributed, reliable, and available service for efficiently
+collecting, aggregating, and moving large amounts of streaming event data.
+
+Version 1.9.0 is the eleventh Flume release as an Apache top-level project. Flume
+1.9.0 is stable, production-ready software, and is backwards-compatible with
+previous versions of the Flume 1.x codeline.
+
+Several months of active development went into this release: about 70 patches were committed since 1.8.0, representing many features, enhancements, and bug fixes. While the full change log can be found on the 1.9.0 release page (link below), here are a few new feature highlights:
+
+ * Better SSL/TLS support
+ * Configuration Filters to provide a way to inject sensitive information like passwords into the configuration
+ * Float and Double value support in Context
+ * Kafka client upgraded to 2.0
+ * HBase 2 support
+
+Below is the list of people (from Git logs) who submitted and/or reviewed
+improvements to Flume during the 1.9.0 development cycle:
+
+ * Andras Beni
+ * Attila Simon
+ * Bessenyei Balázs Donát
+ * Denes Arvay
+ * Endre Major
+ * Ferenc Szabo
+ * Hans Uhlig
+ * hunshenshi
+ * Jehan Bruggeman
+ * Laxman Ch
+ * Mike Percy
+ * Miklos Csanady
+ * Peter Turcsanyi
+ * Ralph Goers
+ * Takafumi Saito
+ * Tristan Stevens
+ * Udai Kiran Potluri
+ * Viktor Somogyi
+ * Yan Jian
+
+The full change log and documentation are available on the
+`Flume 1.9.0 release page <releases/1.9.0.html>`__.
+
+This release can be downloaded from the Flume `Download <download.html>`__ page.
+
+Your contributions, feedback, help and support make Flume better!
+For more information on how to report problems or contribute,
+please visit our `Get Involved <getinvolved.html>`__ page.
+
+The Apache Flume Team
+
+.. raw:: html
+
+ <h3>October 4, 2017 - Apache Flume 1.8.0 Released</h3>
+
+The Apache Flume team is pleased to announce the release of Flume 1.8.0.
+
+Flume is a distributed, reliable, and available service for efficiently
+collecting, aggregating, and moving large amounts of streaming event data.
+
+Version 1.8.0 is the eleventh Flume release as an Apache top-level project. Flume
+1.8.0 is stable, production-ready software, and is backwards-compatible with
+previous versions of the Flume 1.x codeline.
+
+Several months of active development went into this release: about 80 patches were committed since 1.7.0, representing many features, enhancements, and bug fixes. While the full change log can be found on the 1.8.0 release page (link below), here are a few new feature highlights:
+
+ * Add Interceptor to remove headers from event
+ * Provide netcat UDP source as alternative to TCP
+ * Support environment variables in configuration files
+
+Below is the list of people (from Git logs) who submitted and/or reviewed
+improvements to Flume during the 1.8.0 development cycle:
+
+* Andras Beni
+* Ashish Paliwal
+* Attila Simon
+* Ben Wheeler
+* Bessenyei Balázs Donát
+* Chris Horrocks
+* Denes Arvay
+* Ferenc Szabo
+* Gabriel Commeau
+* Hari Shreedharan
+* Jeff Holoman
+* Lior Zeno
+* Marcell Hegedus
+* Mike Percy
+* Miklos Csanady
+* Peter Ableda
+* Peter Chen
+* Ping Wang
+* Pravin D'silva
+* Robin Wang
+* Roshan Naik
+* Satoshi Iijima
+* Shang Wu
+* Siddharth Ahuja
+* Takafumi Saito
+* TeddyBear1314
+* Theodore michael Malaska
+* Tristan Stevens
+* dengkai02
+* eskrm
+* filippovmn
+* loleek
+
+The full change log and documentation are available on the
+`Flume 1.8.0 release page <releases/1.8.0.html>`__.
+
+This release can be downloaded from the Flume `Download <download.html>`__ page.
+
+Your contributions, feedback, help and support make Flume better!
+For more information on how to report problems or contribute,
+please visit our `Get Involved <getinvolved.html>`__ page.
+
+The Apache Flume Team
+
+.. raw:: html
+
+ <h3>October 17, 2016 - Apache Flume 1.7.0 Released</h3>
+
+The Apache Flume team is pleased to announce the release of Flume 1.7.0.
+
+Flume is a distributed, reliable, and available service for efficiently
+collecting, aggregating, and moving large amounts of streaming event data.
+
+Version 1.7.0 is the tenth Flume release as an Apache top-level project.
+Flume 1.7.0 is stable, production-ready software, and is backwards-compatible
+with previous versions of the Flume 1.x codeline.
+
+Several months of active development went into this release: almost 100 patches were committed since 1.6.0, representing many features, enhancements, and bug fixes. While the full change log can be found on the 1.7.0 release page (link below), here are a few new feature highlights:
+
+ * Taildir source
+ * Kafka integration improvements (eg. security)
+
+Below is the list of people (from Git/SVN logs) who submitted and/or reviewed
+improvements to Flume during the 1.7.0 development cycle:
+
+* Abraham Fine
+* Alexandre Dutra
+* Andrea Rota
+* Ashish Paliwal
+* Attila Simon
+* Bessenyei Balázs Donát
+* Daniel Templeton
+* Deepesh Khandelwal
+* Denes Arvay
+* Dylan Jones
+* Fonso Li
+* Gonzalo Herreros
+* Grant Henke
+* Grigoriy Rozhkov
+* Hari Shreedharan
+* Neerja Khattar
+* Jarek Jarcec Cecho
+* Jeff Holoman
+* Johny Rufus
+* Jonathan Smith
+* Jun Seok Hong
+* Kevin Conaway
+* lfzCarlosC
+* Li Xiang
+* Liam Mousseau
+* Lionel Herbet
+* Lior Zeno
+* Michelle Casbon
+* Mike Percy
+* Miroslav Holubec
+* Muhammad Ehsan ul Haquen
+* Niccolo Becchi
+* Phil D'Amore
+* Phil Scalan
+* Philip Zeyliger
+* Ralph Goer
+* Rigo MacTaggart
+* Roshan Naik
+* Santiago M. Mola
+* Satoshi Iijima
+* Siddharth Ahuja
+* Sriharsha Chintalapani
+* Somin Mithraa
+* Sriharsha Chintalapani
+* Ted Malaska
+* tinawenqiao
+* Tristan Stevens
+
+The full change log and documentation are available on the
+`Flume 1.7.0 release page <releases/1.7.0.html>`__.
+
+This release can be downloaded from the Flume `Download <download.html>`__ page.
+
+Your contributions, feedback, help and support make Flume better!
+For more information on how to report problems or contribute,
+please visit our `Get Involved <getinvolved.html>`__ page.
+
+The Apache Flume Team
+
+
+.. raw:: html
+
+ <h3>May 20, 2015 - Apache Flume 1.6.0 Released</h3>
+
+The Apache Flume team is pleased to announce the release of Flume 1.6.0.
+
+Flume is a distributed, reliable, and available service for efficiently
+collecting, aggregating, and moving large amounts of streaming event data.
+
+Version 1.6.0 is the ninth Flume release as an Apache top-level project.
+Flume 1.6.0 is stable, production-ready software, and is backwards-compatible
+with previous versions of the Flume 1.x codeline.
+
+Several months of active development went into this release: 105 patches were committed since 1.5.2, representing many features, enhancements, and bug fixes. While the full change log can be found on the 1.6.0 release page (link below), here are a few new feature highlights:
+
+ * Flume Sink and Source for Apache Kafka
+ * A new channel that uses Kafka
+ * Hive Sink based on the new Hive Streaming support
+ * End to End authentication in Flume
+ * Simple regex search-and-replace interceptor
+
+The full change log and documentation are available on the
+`Flume 1.6.0 release page <releases/1.6.0.html>`__.
+
+This release can be downloaded from the Flume `Download <download.html>`__ page.
+
+Your contributions, feedback, help and support make Flume better!
+For more information on how to report problems or contribute,
+please visit our `Get Involved <getinvolved.html>`__ page.
+
+The Apache Flume Team
+
+.. raw:: html
+
+ <h3>November 18, 2014 - Apache Flume 1.5.2 Released</h3>
+
+The Apache Flume team is pleased to announce the release of Flume 1.5.2
+
+Flume is a distributed, reliable, and available service for efficiently
+collecting, aggregating, and moving large amounts of streaming event data.
+
+Version 1.5.2 is the eighth Flume release as an Apache top-level project.
+Flume 1.5.2 is stable, production-ready software, and is backwards-compatible
+with previous versions of the Flume 1.x codeline.
+
+Apache Flume 1.5.2 is a security and maintenance release that disables SSLv3
+on all components in Flume that support SSL/TLS. All users are encouraged to
+update to this release as soon as possible.
+
+The full change log and documentation are available on the
+`Flume 1.5.2 release page <releases/1.5.2.html>`__.
+
+This release can be downloaded from the Flume `Download <download.html>`__ page.
+
+Your contributions, feedback, help and support make Flume better!
+For more information on how to report problems or contribute,
+please visit our `Get Involved <getinvolved.html>`__ page.
+
+The Apache Flume Team
+
+.. raw:: html
+
+ <h3>July 16, 2014 - Apache Flume 1.5.0.1 Released</h3>
+
+The Apache Flume team is pleased to announce the release of Flume 1.5.0.1
+
+Flume is a distributed, reliable, and available service for efficiently
+collecting, aggregating, and moving large amounts of streaming event data.
+
+Version 1.5.0.1 is the sixth Flume release as an Apache top-level project.
+Flume 1.5.0.1 is stable, production-ready software, and is backwards-compatible
+with previous versions of the Flume 1.x codeline.
+
+Apache Flume 1.5.0.1 is a maintenance release primarily meant to add support to build
+against Apache HBase 0.98.x. This release adds a new build profile that builds Flume
+against HBase 0.98.2.
+
+Apache BigTop 0.8.0 release will ship Flume binaries built
+against HBase 0.98.x.
+
+The full change log and documentation are available on the
+`Flume 1.5.0.1 release page <releases/1.5.0.1.html>`__.
+
+This release can be downloaded from the Flume `Download <download.html>`__ page.
+
+Your contributions, feedback, help and support make Flume better!
+For more information on how to report problems or contribute,
+please visit our `Get Involved <getinvolved.html>`__ page.
+
+The Apache Flume Team
+
+.. raw:: html
+
+ <h3>May 20, 2014 - Apache Flume 1.5.0 Released</h3>
+
+The Apache Flume team is pleased to announce the release of Flume 1.5.0.
+
+Flume is a distributed, reliable, and available service for efficiently
+collecting, aggregating, and moving large amounts of streaming event data.
+
+Version 1.5.0 is the fifth Flume release as an Apache top-level project.
+Flume 1.5.0 is stable, production-ready software, and is backwards-compatible
+with previous versions of the Flume 1.x codeline.
+
+Several months of active development went into this release: 123 patches were committed since 1.4.0, representing many features, enhancements, and bug fixes. While the full change log can be found on the 1.5.0 release page (link below), here are a few new feature highlights:
+
+* New in-memory channel that can spill to disk
+* A new dataset sink that use Kite API to write data to HDFS and HBase
+* Support for Elastic Search HTTP API in Elastic Search Sink
+* Much faster replay in the File Channel.
+
+The full change log and documentation are available on the
+`Flume 1.5.0 release page <releases/1.5.0.html>`__.
+
+This release can be downloaded from the Flume `Download <download.html>`__ page.
+
+Your contributions, feedback, help and support make Flume better!
+For more information on how to report problems or contribute,
+please visit our `Get Involved <getinvolved.html>`__ page.
+
+The Apache Flume Team
+
+
+.. raw:: html
+
+ <h3>July 2, 2013 - Apache Flume 1.4.0 Released</h3>
+
+The Apache Flume team is pleased to announce the release of Flume 1.4.0.
+
+Flume is a distributed, reliable, and available service for efficiently
+collecting, aggregating, and moving large amounts of streaming event data.
+
+Version 1.4.0 is the fourth Flume release as an Apache top-level project.
+Flume 1.4.0 is stable, production-ready software, and is backwards-compatible
+with previous versions of the Flume 1.x codeline.
+
+Six months of active development went into this release: 261 patches were committed since 1.3.1, representing many features, enhancements, and bug fixes. While the full change log can be found on the 1.4.0 release page (link below), here are a few new feature highlights:
+
+* New JMS Source
+* New Solr Sink with ETL capabilities
+* Updated ElasticSearch sink to support ES version 0.90
+* Support for secure SSL transport over Avro-RPC clients, sources & sinks
+* Support for Thrift-RPC as a transport mechanism
+* Support for embedding a Flume agent within applications
+* Support for a new plugins.d directory structure for managing Flume addons
+* Support for reading Avro files via the Spooling Directory source
+* Support for writing Avro files with arbitrary schemas via the HDFS sink
+* Support for ingesting Avro-serializable objects via the log4j API
+* Improvements to the file channel to keep a backup checkpoint to avoid replays
+* Performance improvements to the file channel, including group commit
+* New file channel consistency check tool
+
+Below is the list of people (from Git/SVN logs) who submitted and/or reviewed
+improvements to Flume during the 1.4.0 development cycle:
+
+* Alexander Alten-Lorenz
+* Aline Guedes Pinto
+* Brock Noland
+* Cameron Gandevia
+* Chris Birchall
+* Christopher Nagy
+* Deepesh Khandelwal
+* Denny Ye
+* Edward Sargisson
+* Hari Shreedharan
+* Israel Ekpo
+* Ivan Bogdanov
+* Jarek Jarcec Cecho
+* Jeff Lord
+* Joey Echeverria
+* Jolly Chen
+* Juhani Connolly
+* Mark Grover
+* Mike Percy
+* Mubarak Seyed
+* Nitin Verma
+* Oliver B. Fischer
+* Patrick Wendell
+* Paul Chavez
+* Pedro Urbina Escos
+* Phil Scala
+* Rahul Ravindran
+* Ralph Goers
+* Roman Shaposhnik
+* Roshan Naik
+* Sravya Tirukkovalur
+* Steve Hoffman
+* Ted Malaska
+* Thiruvalluvan M. G.
+* Thom DeCarlo
+* Tim Bacon
+* Tom White
+* Venkat Ranganathan
+* Venkatesh Sivasubramanian
+* Will McQueen
+* Wolfgang Hoschek
+
+The full change log and documentation are available on the
+`Flume 1.4.0 release page <releases/1.4.0.html>`__.
+
+This release can be downloaded from the Flume `Download <download.html>`__ page.
+
+Your contributions, feedback, help and support make Flume better!
+For more information on how to report problems or contribute,
+please visit our `Get Involved <getinvolved.html>`__ page.
+
+The Apache Flume Team
+
+
+.. raw:: html
+
+ <h3>January 2, 2013 - Apache Flume 1.3.1 Released</h3>
+
+The Apache Flume team is pleased to announce the release of Flume version 1.3.1.
+Apache Flume 1.3.1 is the fifth release under the auspices of Apache of the
+so-called "NG" codeline, and our third release as a top-level Apache
+project! Flume 1.3.1 has been put through many stress and regression tests,
+is stable, production-ready software, and is backwards-compatible with
+Flume 1.3.0 and Flume 1.2.0.
+
+Apache Flume 1.3.1 is a maintainance release for the 1.3.0 release, and includes
+several bug fixes and performance enhancements.
+
+This release can be downloaded from the Flume download page at:
+http://flume.apache.org/download.html
+
+The change log and documentation are available on the 1.3.1 release page:
+http://flume.apache.org/releases/1.3.1.html
+
+Your help and feedback is more than welcome!
+
+.. raw:: html
+
+ <h3>December 2, 2012 - Apache Flume 1.3.0 Released</h3>
+
+The Apache Flume team is pleased to announce the release of Flume version 1.3.0.
+
+Apache Flume 1.3.0 is the fourth release under the auspices of Apache of the
+so-called "NG" codeline, and our second release as a top-level Apache
+project! Flume 1.3.0 has been put through many stress and regression tests,
+is stable, production-ready software, and is backwards-compatible with
+Flume 1.2.0.
+
+Four months of very active development went into this release: a whopping
+221 patches were committed since 1.2.0, representing many features,
+enhancements, and bug fixes. While the full change log can be found in the
+link below, here are a few new feature highlights:
+
+* New HTTP Post Source
+* New Spool Directory Source
+* New Multi-port Syslog Source
+* New Elastic Search Sink
+* New Regex Extractor Interceptor
+* File Channel Encryption
+
+This release can be downloaded from the Flume download page at:
+http://flume.apache.org/download.html
+
+The change log and documentation are available on the 1.3.0 release page:
+http://flume.apache.org/releases/1.3.0.html
+
+Your help and feedback is more than welcome!
+
+.. raw:: html
+
+ <h3>July 26, 2012 - Apache Flume 1.2.0 Released</h3>
+
+The Apache Flume team is pleased to announce the release of Flume version 1.2.0.
+
+Apache Flume 1.2.0 is the third release under the auspices of Apache of the
+so-called "NG" codeline, and our first release as a top-level Apache
+project! Flume 1.2.0 has been put through many stress and regression tests,
+is stable, production-ready software, and is backwards-compatible with
+Flume 1.1.0.
+
+Four months of very active development went into this release: a whopping
+192 patches were committed since 1.1.0, representing many features,
+enhancements, and bug fixes. While the full change log can be found in the
+link below, here are a few new feature highlights:
+
+* New durable file channel
+* New client API
+* New HBase sinks (two different implementations)
+* New Interceptor interface (a plugin processing API)
+* New JMX-based monitoring support
+
+This release can be downloaded from the Flume download page at:
+http://flume.apache.org/download.html
+
+The change log and documentation are available on the 1.2.0 release page:
+http://flume.apache.org/releases/1.2.0.html
+
+Your help and feedback is more than welcome!
+
+.. raw:: html
+
+ <h3>July 26, 2012 - Flume Meetup NYC</h3>
+
+Strata + Hadoop World (http://strataconf.com/stratany2012) will be
+held October 23-25 in NYC. A Flume Meetup is being organized
+around this event. This Meetup will be a good place for users to
+interact with each other and with Flume developers.
+
+In order to help with organization, a `form <https://docs.google.com/spreadsheet/viewform?formkey=dGRDUUxxbUI3TUJuYnQwMV9sVzJDZEE6MA#gid=0>`__
+has been set up with a few questions about what kind of Meetup the community wants, and which evening is best.
+Please fill out the `form <https://docs.google.com/spreadsheet/viewform?formkey=dGRDUUxxbUI3TUJuYnQwMV9sVzJDZEE6MA#gid=0>`__.
+Feel free to post to the User's mailing list with any questions.
+
+.. toctree::
+ :maxdepth: 1
+ :hidden:
+
+ getinvolved
+ download
+ documentation
+ releases/index
+ mailinglists
+ team
+ source
+ license
+
+..
diff --git a/source/sphinx/license.rst b/source/sphinx/license.rst
new file mode 100644
index 00000000..0eec80cc
--- /dev/null
+++ b/source/sphinx/license.rst
@@ -0,0 +1,136 @@
+.. Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+
+
+Apache License
+--------------
+
+Version 2.0, January 2004
+
+`http://www.apache.org/licenses/ <http://www.apache.org/licenses/>`_
+
+TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+**1. Definitions.**
+
+"License" shall mean the terms and conditions for use, reproduction, and distribution as defined by Sections 1 through
+9 of this document.
+
+"Licensor" shall mean the copyright owner or entity authorized by the copyright owner that is granting the License.
+
+"Legal Entity" shall mean the union of the acting entity and all other entities that control, are controlled by, or are
+under common control with that entity. For the purposes of this definition, "control" means (i) the power, direct or
+indirect, to cause the direction or management of such entity, whether by contract or otherwise, or (ii) ownership of
+fifty percent (50%) or more of the outstanding shares, or (iii) beneficial ownership of such entity.
+
+"You" (or "Your") shall mean an individual or Legal Entity exercising permissions granted by this License.
+
+"Source" form shall mean the preferred form for making modifications, including but not limited to software source code,
+documentation source, and configuration files.
+
+"Object" form shall mean any form resulting from mechanical transformation or translation of a Source form, including
+but not limited to compiled object code, generated documentation, and conversions to other media types.
+
+"Work" shall mean the work of authorship, whether in Source or Object form, made available under the License, as
+indicated by a copyright notice that is included in or attached to the work (an example is provided in the Appendix
+below).
+
+"Derivative Works" shall mean any work, whether in Source or Object form, that is based on (or derived from) the Work
+and for which the editorial revisions, annotations, elaborations, or other modifications represent, as a whole, an
+original work of authorship. For the purposes of this License, Derivative Works shall not include works that remain
+separable from, or merely link (or bind by name) to the interfaces of, the Work and Derivative Works thereof.
+
+"Contribution" shall mean any work of authorship, including the original version of the Work and any modifications or
+additions to that Work or Derivative Works thereof, that is intentionally submitted to Licensor for inclusion in the
+Work by the copyright owner or by an individual or Legal Entity authorized to submit on behalf of the copyright owner.
+For the purposes of this definition, "submitted" means any form of electronic, verbal, or written communication sent
+to the Licensor or its representatives, including but not limited to communication on electronic mailing lists, source
+code control systems, and issue tracking systems that are managed by, or on behalf of, the Licensor for the purpose of
+discussing and improving the Work, but excluding communication that is conspicuously marked or otherwise designated in
+writing by the copyright owner as "Not a Contribution."
+
+"Contributor" shall mean Licensor and any individual or Legal Entity on behalf of whom a Contribution has been received
+by Licensor and subsequently incorporated within the Work.
+
+**2. Grant of Copyright License.** Subject to the terms and conditions of this License, each Contributor hereby grants
+to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable copyright license to reproduce,
+prepare Derivative Works of, publicly display, publicly perform, sublicense, and distribute the Work and such Derivative
+Works in Source or Object form.
+
+**3. Grant of Patent License.** Subject to the terms and conditions of this License, each Contributor hereby grants
+to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable (except as stated in this section)
+patent license to make, have made, use, offer to sell, sell, import, and otherwise transfer the Work, where such license
+applies only to those patent claims licensable by such Contributor that are necessarily infringed by their
+Contribution(s) alone or by combination of their Contribution(s) with the Work to which such Contribution(s) was
+submitted. If You institute patent litigation against any entity (including a cross-claim or counterclaim in a lawsuit)
+alleging that the Work or a Contribution incorporated within the Work constitutes direct or contributory patent
+infringement, then any patent licenses granted to You under this License for that Work shall terminate as of the date
+such litigation is filed.
+
+**4. Redistribution.** You may reproduce and distribute copies of the Work or Derivative Works thereof in any medium,
+with or without modifications, and in Source or Object form, provided that You meet the following conditions:
+
+1. You must give any other recipients of the Work or Derivative Works a copy of this License; and
+
+2. You must cause any modified files to carry prominent notices stating that You changed the files; and
+
+3. You must retain, in the Source form of any Derivative Works that You distribute, all copyright, patent, trademark,
+ and attribution notices from the Source form of the Work, excluding those notices that do not pertain to any part of the
+ Derivative Works; and
+
+4. If the Work includes a "NOTICE" text file as part of its distribution, then any Derivative Works that You distribute
+ must include a readable copy of the attribution notices contained within such NOTICE file, excluding those notices that
+ do not pertain to any part of the Derivative Works, in at least one of the following places: within a NOTICE text file
+ distributed as part of the Derivative Works; within the Source form or documentation, if provided along with the
+ Derivative Works; or, within a display generated by the Derivative Works, if and wherever such third-party notices
+ normally appear. The contents of the NOTICE file are for informational purposes only and do not modify the License.
+ You may add Your own attribution notices within Derivative Works that You distribute, alongside or as an addendum to
+ the NOTICE text from the Work, provided that such additional attribution notices cannot be construed as modifying the
+ License. You may add Your own copyright statement to Your modifications and may provide additional or different license
+ terms and conditions for use, reproduction, or distribution of Your modifications, or for any such Derivative Works as
+ a whole, provided Your use, reproduction, and distribution of the Work otherwise complies with the conditions stated in
+ this License.
+
+**5. Submission of Contributions.** Unless You explicitly state otherwise, any Contribution intentionally submitted for
+inclusion in the Work by You to the Licensor shall be under the terms and conditions of this License, without any
+additional terms or conditions. Notwithstanding the above, nothing herein shall supersede or modify the terms of any
+separate license agreement you may have executed with Licensor regarding such Contributions.
+
+**6. Trademarks.** This License does not grant permission to use the trade names, trademarks, service marks, or product
+names of the Licensor, except as required for reasonable and customary use in describing the origin of the Work and
+reproducing the content of the NOTICE file.
+
+**7. Disclaimer of Warranty.** Unless required by applicable law or agreed to in writing, Licensor provides the Work
+(and each Contributor provides its Contributions) on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND,
+either express or implied, including, without limitation, any warranties or conditions of TITLE, NON-INFRINGEMENT,
+MERCHANTABILITY, or FITNESS FOR A PARTICULAR PURPOSE. You are solely responsible for determining the appropriateness of
+using or redistributing the Work and assume any risks associated with Your exercise of permissions under this License.
+
+**8. Limitation of Liability.** In no event and under no legal theory, whether in tort (including negligence), contract,
+or otherwise, unless required by applicable law (such as deliberate and grossly negligent acts) or agreed to in writing,
+shall any Contributor be liable to You for damages, including any direct, indirect, special, incidental, or
+consequential damages of any character arising as a result of this License or out of the use or inability to use the
+Work (including but not limited to damages for loss of goodwill, work stoppage, computer failure or malfunction, or any
+and all other commercial damages or losses), even if such Contributor has been advised of the possibility of such
+damages.
+
+**9. Accepting Warranty or Additional Liability.** While redistributing the Work or Derivative Works thereof, You may
+choose to offer, and charge a fee for, acceptance of support, warranty, indemnity, or other liability obligations and/or
+rights consistent with this License. However, in accepting such obligations, You may act only on Your own behalf and on
+Your sole responsibility, not on behalf of any other Contributor, and only if You agree to indemnify, defend, and hold
+each Contributor harmless for any liability incurred by, or claims asserted against, such Contributor by reason of your
+accepting any such warranty or additional liability.
+
+END OF TERMS AND CONDITIONS
\ No newline at end of file
diff --git a/source/sphinx/mailinglists.rst b/source/sphinx/mailinglists.rst
new file mode 100644
index 00000000..7eed6d72
--- /dev/null
+++ b/source/sphinx/mailinglists.rst
@@ -0,0 +1,14 @@
+=============
+Mailing lists
+=============
+
+These are the mailing lists that have been established for the Apache Flume project. For each list, there is a subscribe, unsubscribe, and an archive link.
+
+.. csv-table::
+ :header: "Name", "Subscribe", "Unsubscribe", "List Archive"
+ :widths: 25, 20, 20, 30
+
+ "Flume User List", `user subscribe <ma...@flume.apache.org>`_, `user unsubscribe <ma...@flume.apache.org>`_, `Flume User List Archive <ht...@flume.apache.org>`_
+ "Flume Developers List", `dev subscribe <ma...@flume.apache.org>`_, `dev unsubscribe <ma...@flume.apache.org>`_, `Flume Developer List Archive <ht...@flume.apache.org>`_
+ "Flume Commits List", `commit subscribe <ma...@flume.apache.org>`_, `commit unsubscribe <ma...@flume.apache.org>`_, `Flume Commits List Archive <ht...@flume.apache.org>`_
+ "Flume Issues List", `issues subscribe <ma...@flume.apache.org>`_, `issues unsubscribe <ma...@flume.apache.org>`_, `Flume Issues List Archive <ht...@flume.apache.org>`_
diff --git a/source/sphinx/releases/1.0.0.rst b/source/sphinx/releases/1.0.0.rst
new file mode 100644
index 00000000..68dd6667
--- /dev/null
+++ b/source/sphinx/releases/1.0.0.rst
@@ -0,0 +1,140 @@
+==========================
+Version 1.0.0 - Incubating
+==========================
+
+
+.. rubric:: Status of this release
+
+This release is the first release of Apache Flume within the Apache
+Incubator, and is labelled and should be considered as a beta version.
+
+.. rubric:: Release Documentation
+
+* `API Documentation <content/1.0.0-incubating/apidocs/index.html>`_
+
+.. rubric:: Changes
+
+* Bug
+ - [FLUME-830] - flume uber jar is missing files from flume-file-channel project
+ - [FLUME-831] - flume-jdbc-channel project has unnecessary direct dependency on log4j API
+ - [FLUME-833] - Audit Direct Library Deps for Flume NG
+ - [FLUME-835] - Issues during clean build of Flume NG
+ - [FLUME-850] - Upgrade the version of Hadoop we use for HDFS sink
+ - [FLUME-858] - HDFSWriterFactory is using operation == for string comparison
+ - [FLUME-863] - Use of unknown sink type leads to NullPointerException
+ - [FLUME-868] - RAT checks fail on builds.apache.org due to local maven repo
+ - [FLUME-869] - JDBC channel tests leave derby.log in module directory
+ - [FLUME-870] - LoggerSink contains two calls to Transaction#commit()
+ - [FLUME-880] - HDFSFormatterFactory is using == operator for String objects
+ - [FLUME-887] - Add maven assembly to build a source only artifact
+ - [FLUME-891] - flume-ng script doesn't build the classpath properly
+ - [FLUME-894] - Add log4j as part of the build
+ - [FLUME-898] - Create DISCLAIMER file for Flume project
+ - [FLUME-900] - RELEASENOTES file needs to be ignored by RAT
+ - [FLUME-902] - Remove thrift references in NG build
+ - [FLUME-903] - Update project metadata in main pom
+ - [FLUME-904] - Update plugin and dependency repos referenced in the main pom
+ - [FLUME-905] - ExecSource silently fails after first transaction with channel
+ - [FLUME-906] - Maven Avro plugin missing an entry in plugin dep management
+ - [FLUME-907] - Maven assembly missing CHANGELOG and other misc files
+ - [FLUME-908] - Clean out old bin and conf contents
+ - [FLUME-909] - org.apache.flume.node.TestAbstractLogicalNodeManager is failing on some machines
+ - [FLUME-910] - Typo in maven avro plugin groupId in plugin dep management
+ - [FLUME-911] - README should reference Apache Flume rather than just Flume
+ - [FLUME-912] - DEVNOTES contains outdated info
+ - [FLUME-913] - slf4j-log4j binding is excluded from the dist assembly due to test scope
+ - [FLUME-915] - Incorrect license information in various files
+ - [FLUME-916] - DISCLAIMER file has an incorrect URL
+
+* Improvement
+ - [FLUME-846] - Bump Avro version to 1.6.x
+ - [FLUME-867] - Pollable source and sink runners should reduce poll interval after several BACKOFFs
+ - [FLUME-871] - HDFS sink needs to handle blocked/hung append operation
+
+* Question
+ - [FLUME-856] - Move data across hosts
+
+* Task
+ - [FLUME-876] - Update README, NOTICE, LICENSE, and DEVNOTES files
+ - [FLUME-878] - Write release-ready Maven assembly descriptor
+ - [FLUME-879] - Document Flume's ASF release process
+ - [FLUME-885] - Set version number of project to 1.0.0-SNAPSHOT for NG branch
+ - [FLUME-899] - Add Release notes
+ - [FLUME-901] - Make Flume NG build and pass tests against Hadoop 0.23 branch
+
+Release Notes - Flume - Version NG alpha 2 - 20111107
+
+* Bug
+ - [FLUME-773] - ExecSource doesn't rollback transactions on errors
+ - [FLUME-805] - HDFS sink should mangle the names of incomplete files till they are closed
+ - [FLUME-815] - Test json config file has missing bind property
+ - [FLUME-816] - TestJdbcChannelProvider throws OOME based on RNG values
+ - [FLUME-817] - JdbcChannel can not be created by DefaultChannelFactory
+ - [FLUME-818] - PropertiesFileConfigurationProvider doesn't properly log exceptions
+ - [FLUME-822] - JDBC channel lock acquisition failure during take()
+ - [FLUME-823] - The properties configuration provider should fail if the configuration file is not found
+ - [FLUME-825] - Need to remove dependency on hadoop core from flume-ng-core project
+ - [FLUME-827] - Avro client conn failure results in 60-second wait before terminating
+ - [FLUME-848] - Typo is TestHDFSEventSink path
+ - [FLUME-861] - AvroSource is failing on ClosedChannelException
+
+* Improvement
+ - [FLUME-819] - JDBC channel plugin is not registered with Flume node
+ - [FLUME-820] - JDBC channel should support capacity specification.
+ - [FLUME-821] - Derby schema handler should create the necessary indexes for fast lookups.
+ - [FLUME-826] - Update libs
+
+* New Feature
+ - [FLUME-775] - Support text output
+ - [FLUME-814] - Add support for multiple channels to sources
+
+* Task
+ - [FLUME-812] - Enable Apache RAT checks during Maven build
+ - [FLUME-866] - Remove old plugins and log4j appender
+
+Release Notes - Flume - Version NG alpha 1 - 20111021
+
+* Sub-task
+ - [FLUME-737] - Port Flume OG sources and sinks to NG interfaces
+ - [FLUME-739] - Create NG node configuration components
+ - [FLUME-747] - Create NG command line launchers and daemon infrastructure
+ - [FLUME-760] - Implement JDBC based channel implementation
+ - [FLUME-761] - Implement HDFS Flume NG sink
+ - [FLUME-777] - Support text output for HDFS sink
+ - [FLUME-795] - Replace the non-transactional memory channel with new transactional memory channel
+
+* Bug
+ - [FLUME-769] - FLUME-728 - TestJsonFileConfigurationProvider fails due to timing issue
+ - [FLUME-784] - MemoryChannel should poll with timeout on take() rather than block indefinitely
+ - [FLUME-788] - Add more test cases to Flume-NG HDFS test
+ - [FLUME-803] - support re-entrant transaction for memory channel
+ - [FLUME-806] - Fix cast exception in MemoryChannel due to config type changes
+ - [FLUME-807] - Fix tests broken by FLUME-802 changes
+ - [FLUME-809] - Fix channel syntax javadoc bug in PropertiesFileConfigurationProvider
+ - [FLUME-811] - Remove logging of avro client that causes errors with proxy object methods
+
+* Epic
+ - [FLUME-728] - Flume NG refactoring
+
+* Improvement
+ - [FLUME-772] - MemoryChannel should push events back into channel on rollback
+ - [FLUME-774] - Move HDFS sink into a separate module
+ - [FLUME-781] - Add error checking to AvroCLICilent
+ - [FLUME-782] - Instrument AvroSource with counters
+ - [FLUME-783] - Add a batch event RPC call to AvroSource
+ - [FLUME-804] - Support help and node name arguments from the command line
+ - [FLUME-810] - Add help command line options to AvroCLIClient
+
+* New Feature
+ - [FLUME-771] - Implement NG Avro source
+ - [FLUME-778] - Implement NG Avro sink
+ - [FLUME-779] - Create an Avro CLI for sending data to the Avro source
+
+* Task
+ - [FLUME-780] - Reduce default log levels for chatty libraries
+ - [FLUME-785] - Write javadoc for builtin channels
+ - [FLUME-786] - Write javadoc for builtin sources
+ - [FLUME-787] - Write javadoc for builtin sinks
+ - [FLUME-801] - Write NG getting started guide
+ - [FLUME-802] - Complete PropertyFileConfigurationProvider implementation
+
diff --git a/source/sphinx/releases/1.1.0.rst b/source/sphinx/releases/1.1.0.rst
new file mode 100644
index 00000000..a6755c34
--- /dev/null
+++ b/source/sphinx/releases/1.1.0.rst
@@ -0,0 +1,72 @@
+==========================
+Version 1.1.0 - Incubating
+==========================
+
+
+.. rubric:: Status of this release
+
+This release is the second release of Apache Flume within the Apache
+Incubator, and is labelled and should be considered as a beta version.
+As compared to the previous release, it has substantially more features
+and functionality along with bug fixes and other enhancements. However
+it is likely to change until a stable release version.
+
+.. rubric:: Release Documentation
+
+* `API Documentation <content/1.1.0-incubating/apidocs/index.html>`_
+
+.. rubric:: Changes
+
+* Sub-task
+ - [FLUME-989] - Factor Flume Avro RPC interfaces out into separate Client SDK
+
+* Bug
+ - [FLUME-11] - Tests are setting logger level and should not be.
+ - [FLUME-889] - All events in memory channel are lost on reconfiguration
+ - [FLUME-920] - flume-ng script does not work on Ubuntu Maverick
+ - [FLUME-933] - Default[Source|Sink|Channel]Factory implementation should do reference counting for create/unregistering instances.
+ - [FLUME-936] - MemoryChannel is not thread safe
+ - [FLUME-955] - Rat failure: Legacy Avro Source missing Apache license header
+ - [FLUME-957] - Remove unused flume json config file
+ - [FLUME-960] - TestAvroSink.testFailedConnect is racy and fails often
+ - [FLUME-963] - Add additional tests to TestHDFSEventSink and demystify existing tests
+ - [FLUME-1005] - Several issues with flume-ng script
+ - [FLUME-1009] - HDFSEventSink should return BACKOFF when the channel returns null
+ - [FLUME-1018] - Context can cause NullPointerException
+ - [FLUME-1027] - Missing log4j library in Flume distribution
+ - [FLUME-1029] - Flume build issue with Avro 1.6.2 and empty local repository
+ - [FLUME-1031] - Depricate code generated by Thrift and Avro OG sources that is under com.cloudera package
+
+* Improvement
+ - [FLUME-886] - Create Log4j Appender
+ - [FLUME-919] - flume-ng script should use exec when spawning the java process
+ - [FLUME-922] - Straighten up branches for development
+ - [FLUME-925] - Update build infrastructure to follow Apache Maven guidelines
+ - [FLUME-932] - Making flume-ng components pluggage and name aware
+ - [FLUME-934] - hadoop 1.0.0 sink and corresponding profile
+ - [FLUME-935] - Create abstract implementations of basic channel/transaction semantics
+ - [FLUME-939] - Load flume-env.sh from flume_conf_dir environment variable / system property as opposed to bin directory
+ - [FLUME-945] - Add the ability to specify a default channel for multiplexing channel selector.
+ - [FLUME-956] - Configuration File Template
+ - [FLUME-958] - Miscellaneous build improvements
+ - [FLUME-964] - Remove compiler warnings where possible
+ - [FLUME-978] - Context interface is too basic requiring boilerplate user code
+ - [FLUME-984] - SinkRunner should catch unhanded exceptions and log them like PollingSourceRunner
+ - [FLUME-990] - Hive sink
+ - [FLUME-1019] - Document Sink and related interfaces, defining expected behaviors
+ - [FLUME-1021] - Document API contracts and expected behavior in additional interfaces, including Source
+ - [FLUME-1032] - Fix / clean up Flume NG build
+
+* New Feature
+ - [FLUME-865] - Implement failover sink
+ - [FLUME-892] - Support for SysLog as source
+ - [FLUME-914] - Port the IRC sink to flume ng
+ - [FLUME-930] - Support for multiplexing into different channels from single source.
+ - [FLUME-942] - Support event compatibility with Flume 0.9x
+ - [FLUME-970] - Create user guide for Flume NG
+
+* Task
+ - [FLUME-940] - Remove unused code from Flume
+ - [FLUME-949] - Collapse PollableSink into Sink interface.
+ - [FLUME-977] - Migrate trunk to 0.9.5 branch and move branch flume-728 over to trunk
+ - [FLUME-1028] - Fix jenkins build after addition of submodule
diff --git a/source/sphinx/releases/1.2.0.rst b/source/sphinx/releases/1.2.0.rst
new file mode 100644
index 00000000..9b49f27d
--- /dev/null
+++ b/source/sphinx/releases/1.2.0.rst
@@ -0,0 +1,207 @@
+=============
+Version 1.2.0
+=============
+
+.. rubric:: Status of this release
+
+This release is the first release of Apache Flume as an Apache top level project and
+is also the first release that is considered ready for production use.
+As compared to the previous releases, it has substantially more features
+and functionality along with bug fixes and other enhancements.
+
+.. rubric:: Release Documentation
+
+* `Flume User Guide <content/1.2.0/FlumeUserGuide.pdf>`_
+* `Flume Developer Guide <content/1.2.0/FlumeDeveloperGuide.pdf>`_
+* `API Documentation <content/1.2.0/apidocs/index.html>`_
+
+.. rubric:: Changes
+
+** New Feature
+ * [FLUME-896] - Implement a Durable Memory Channel
+ * [FLUME-971] - Create developer guide for Flume NG
+ * [FLUME-988] - Client SDK
+ * [FLUME-1085] - Implement a durable FileChannel
+ * [FLUME-1157] - Implement Interceptors (previously known as Decorators) for Flume 1.x
+ * [FLUME-1183] - Implement an HBase Sink which supports table level access
+ * [FLUME-1215] - Implement Timestamp Interceptor
+ * [FLUME-1252] - Asynchronous HBase Sink
+
+** Improvement
+ * [FLUME-828] - LoggerSink representation of the event's body isn't too useful
+ * [FLUME-881] - Would be nice if HDFS Sink would automatically choose best writableFormat based on fileType
+ * [FLUME-979] - ExecSource should optionally restart the command when it exits
+ * [FLUME-985] - All HDFS Operations in HDFSEventSink should have a timeout
+ * [FLUME-1001] - Support custom processors
+ * [FLUME-1011] - AvroSource should have a configurable max thread count
+ * [FLUME-1020] - Implement Kerberos security for HDFS Sink
+ * [FLUME-1030] - Retry logic for failover sink processor to handle downstream exceptions in a predictable manner.
+ * [FLUME-1032] - Fix / clean up Flume NG build
+ * [FLUME-1043] - SDK should mark slf4j deps as optional
+ * [FLUME-1048] - speed up mvn package: stop building .zip packages
+ * [FLUME-1049] - Use hadoop-1.0.0 as basis for default Flume build instead of 0.20.205
+ * [FLUME-1078] - flume-ng script has no way to add, not replace, classpath
+ * [FLUME-1090] - JDBC Channel: Minimize logging under nominal conditions
+ * [FLUME-1117] - Support output to files in Avro container format
+ * [FLUME-1122] - Flume documentation layout should be refactored
+ * [FLUME-1126] - Support RFC 3164 and 5424 syslog format timestamp parsing
+ * [FLUME-1127] - Add configuration support to AbstractAvroEventSerializer for Avro sync interval and compression support
+ * [FLUME-1132] - HDFSEventSink has spurious and verbose log message
+ * [FLUME-1140] - Adding Xms value in flume-env.sh
+ * [FLUME-1160] - ComponentConfigurationFactory catches NullPointerException
+ * [FLUME-1196] - Allow different HDFS Sinks within the same agent to write to HDFS as different users
+ * [FLUME-1198] - Implement a load-balancing sink processor
+ * [FLUME-1212] - Flume should pick HBase jars from HBASE_HOME
+ * [FLUME-1238] - Support active rolling of files created by HDFS Event Sink
+ * [FLUME-1242] - Make flume user & dev guides easily editable
+ * [FLUME-1275] - Add Regex Serializer for HBaseSink
+ * [FLUME-1287] - Add Standalone Example to Docs
+ * [FLUME-1330] - Avro Source should not use Fixed thread pool for boss threads when pool size is specified
+ * [FLUME-1338] - Produce helpful error message in case that timestamp header is missing when time based bucketing is in use
+ * [FLUME-1343] - Improve user guide
+ * [FLUME-1345] - Use apache-flume for the artifact instead of flume-ng-dist
+ * [FLUME-1351] - Add release version to Flume documentation
+
+** Bug
+ * [FLUME-862] - AvroSource breaks when config properties changes different service
+ * [FLUME-1002] - FailoverSinkProcessor replaces sinks with same priority
+ * [FLUME-1017] - syslog classes missing
+ * [FLUME-1026] - Document Thread Safety Guarantees
+ * [FLUME-1027] - Missing log4j library in Flume distribution
+ * [FLUME-1031] - Depricate code generated by Thrift and Avro OG sources that is under com.cloudera package
+ * [FLUME-1035] - slf4j error in flume sdk unit tests
+ * [FLUME-1036] - Reconfiguration of AVRO or NETCAT source causes port bind exception
+ * [FLUME-1037] - NETCAT handler theads terminate under stress test
+ * [FLUME-1040] - Release-Notes says Apache Ivy instead of Apache Flume
+ * [FLUME-1041] - maven warns of duplicate dependencies
+ * [FLUME-1046] - invoking flume-ng script from bin directory fails
+ * [FLUME-1047] - Client SDK has dependency on apache commons
+ * [FLUME-1070] - Fix javadoc for configuring hdfsSink
+ * [FLUME-1074] - AvroSink if any non-caught exception is thrown, an exception is thrown in finally clause
+ * [FLUME-1075] - HDFSEventSink begin is called when transaction opened due to other error
+ * [FLUME-1079] - Flume agent reconfiguration enters permanent bad state
+ * [FLUME-1080] - Issue with HDFSEventSink for append support
+ * [FLUME-1083] - Why does flume binary archive produces the following empty directories: bin/{ia64,amd64} ?
+ * [FLUME-1087] - Restore Client API compat with v1.1.0
+ * [FLUME-1088] - TestWAL.testThreadedAppend fails on jenkins build server
+ * [FLUME-1094] - hadoop.profile=23 build is broken by slf4j-jcl dependencies
+ * [FLUME-1096] - Add support to pass headers through AvroCLIClient
+ * [FLUME-1098] - Hadoop jars from compilation step included in assembly build
+ * [FLUME-1099] - copy-paste issue with RecoverableMemoryChannel
+ * [FLUME-1102] - HDFSEventSink rollInterval is broken
+ * [FLUME-1104] - HDFS rolls the first file incorrectly
+ * [FLUME-1108] - FILE_ROLL sink doesn't accept value 0 for unlimited wait time before rolling file
+ * [FLUME-1109] - Syslog sources need to be refactored
+ * [FLUME-1112] - HDFSCompressedDataStream append does not work
+ * [FLUME-1114] - Syslog Sources does not implement maxsize
+ * [FLUME-1116] - Extra event created for max payload size of 2500 bytes in Flume syslogtcp source
+ * [FLUME-1119] - Remove default ports for syslog sources
+ * [FLUME-1121] - Recoverable Memory Channel cannot recover data
+ * [FLUME-1124] - Lifecycle supervisor can cause thread contention, sometimes causing components to not startup.
+ * [FLUME-1125] - flume-ng script allows flume-env.sh to clobber some command-line arguments
+ * [FLUME-1128] - Conf poller should use schedule with fixed delay
+ * [FLUME-1129] - change foo to agent in sample config
+ * [FLUME-1130] - flume-ng script bad ordering on FLUME_HOME var
+ * [FLUME-1135] - flume-docs exclude is not sufficient for rat
+ * [FLUME-1136] - Remove from executor service does not always remove the runnables from the queue
+ * [FLUME-1142] - Seq source fails with multiplexing channel selector
+ * [FLUME-1148] - Refactor logging
+ * [FLUME-1149] - All sources get same channel list even if configuration is different.
+ * [FLUME-1154] - flume-ng script should first try finding java from PATH and then try using bigtop, instead of vice-versa
+ * [FLUME-1156] - If config file has empty sources, then throws NPE
+ * [FLUME-1163] - HDFSEventSink leaves .tmp files in place when Flume is stopped
+ * [FLUME-1164] - Configure should be called after stopping all events.
+ * [FLUME-1177] - Maven deps on flume-ng-configuration module are brought in transitively instead of directly
+ * [FLUME-1180] - ChannelSelectorFactory creates incorrect selector for multiplexing selector type
+ * [FLUME-1181] - Context must enforce dot-separated prefix for sub-properties.
+ * [FLUME-1182] - Syslog source cannot read format correctly from configuration
+ * [FLUME-1184] - TestFileChannel.testThreaded fails sometimes
+ * [FLUME-1188] - TestRecoverableMemoryChannel.testThreaded can fail sometimes
+ * [FLUME-1190] - DurableFileChannel requires FILE enum definition in ChannelConfigurationType
+ * [FLUME-1194] - RecoverableMemoryChannel prop misspelled -- "rentention" should be "retention"
+ * [FLUME-1200] - HDFSEventSink causes ``*.snappy`` file to be created in HDFS even when snappy isn't used (due to missing lib)
+ * [FLUME-1202] - Too many approved licenses
+ * [FLUME-1204] - Add more unit tests for hbase sink
+ * [FLUME-1205] - NPE related to checkpointing when using FileChannel
+ * [FLUME-1213] - HDFS sink should allow bucketpath rounding down.
+ * [FLUME-1216] - Need useful error message when keytab does not exist
+ * [FLUME-1217] - HDFS Event Sink generates warnings due to recent change
+ * [FLUME-1219] - Race conditions in BucketWriter / HDFSEventSink
+ * [FLUME-1220] - Load balancing channel selector needs to be in the configuration type
+ * [FLUME-1221] - ThriftLegacySource doesn't handle fields -> headers conversions for bytebuffers
+ * [FLUME-1226] - FailoverRpcClient should check for NULL batch-size property
+ * [FLUME-1229] - System.nanoTime incorrectly used in filename for HDFS file rolling
+ * [FLUME-1230] - Sink gets initialized even when not active
+ * [FLUME-1231] - Deadlock between BucketWriter and LeaseChecker on shutdown
+ * [FLUME-1232] - Cannot start agent a 3rd time when using FileChannel
+ * [FLUME-1234] - Can't use %P escape sequence for bucket path of HDFS sink
+ * [FLUME-1236] - File channel has a race condition between start and create transaction method
+ * [FLUME-1240] - Add version info to Flume NG
+ * [FLUME-1241] - Flume dist should include the flume-ng-doc directory
+ * [FLUME-1244] - Implement a load-balancing RpcClient with round/robin and random distribution capabilties.
+ * [FLUME-1245] - HDFSCompressedDataStream calls finish() on sync instead of flush()
+ * [FLUME-1246] - FileChannel hangs silently when Hadoop libs not found
+ * [FLUME-1248] - flume-ng script gets broken when it tried to load hbase classpath
+ * [FLUME-1253] - Support for running integration tests
+ * [FLUME-1254] - RpcClient can hang when communication is broken with the source.
+ * [FLUME-1270] - Incorrect default hdfs.callTimeout and hdfs.fileType of HDFSEventSink in FlumeUserGuide.rst
+ * [FLUME-1271] - Incorrect configuration causes NPE
+ * [FLUME-1280] - Make all config properties of Hbase sinks public constants
+ * [FLUME-1284] - Need host interceptor for hdfs bucket path escape sequence
+ * [FLUME-1288] - Async hbase sink should throw exception when hbase reports failure and check hbase table correctness
+ * [FLUME-1290] - HDFS Sink should accept fileType parameters of arbitrary case
+ * [FLUME-1297] - Flume tests should wait for a few seconds for agent to start.
+ * [FLUME-1301] - HDFSCompressedDataStream can lose data
+ * [FLUME-1303] - java.library.path value is being truncated at first 'n' char
+ * [FLUME-1304] - Allow for faster allocation of checkpoint file.
+ * [FLUME-1306] - LoadBalancingRpcClient should catch exception for invalid RpcClient and failover to valid one
+ * [FLUME-1309] - Integration tests not included in assembly build artifacts
+ * [FLUME-1312] - Host interceptor should support custom headers
+ * [FLUME-1314] - File channel log file can grow beyond max size which causes startup failure
+ * [FLUME-1315] - Null sink should support batching
+ * [FLUME-1316] - AvroSink should be configurable for connect-timeout and request-timeout
+ * [FLUME-1317] - Assembly build pulls in target folder from flume-ng-tests
+ * [FLUME-1319] - File Channel optimize replay of logs when a checkpoint is present
+ * [FLUME-1320] - Add safeguard for checkpoint corruption detection
+ * [FLUME-1322] - ChannelProcessor should catch Throwable to work around close() clobbering uncaught Exceptions
+ * [FLUME-1323] - Remove shutdown hook from FileChannel
+ * [FLUME-1324] - File Channel Log can contain unassigned blocks
+ * [FLUME-1325] - Components should be stopped in the reverse order that they were started
+ * [FLUME-1327] - File Channel can deadlock in when checkpoint happens in between a put/take/commit
+ * [FLUME-1329] - AvroSink can hang during Avro RPC handshake
+ * [FLUME-1331] - Start method of components throwing NoClassDefFoundError are not caught
+ * [FLUME-1333] - Disable running of saveVersion.sh on Windows
+ * [FLUME-1341] - Build fails on jenkins because a file exists in the environment
+ * [FLUME-1344] - AvroSink JMX does not report connection created count accurately
+ * [FLUME-1346] - Build warning from missing maven-sphinx version in reporting section
+ * [FLUME-1347] - Deprecate RecoverableMemoryChannel
+ * [FLUME-1348] - Update the documentation, correcting links and removing incubation.
+ * [FLUME-1349] - Document Hbase sinks.
+ * [FLUME-1352] - Add documentation for HDFS path rounddown.
+ * [FLUME-1355] - Improve user guide section about sink processors
+ * [FLUME-1356] - Document interceptors
+
+** Task
+ * [FLUME-840] - Update project committers in pom file
+ * [FLUME-991] - Make flume configuration validation component specific at time rather than at runtime
+ * [FLUME-1028] - Fix jenkins build after addition of submodule
+ * [FLUME-1050] - Update version of surefire plugin
+ * [FLUME-1073] - Default Log4j configuration file should have a rolling policy
+ * [FLUME-1082] - Add User and dev guide to Flume site
+ * [FLUME-1151] - Exclude docs directory from rat
+ * [FLUME-1189] - Test ReoverableMemoryChannel throughput versus FileChannel
+ * [FLUME-1300] - Update user guide for File Channel
+ * [FLUME-1353] - Ensure license headers are consistent
+
+** Sub-task
+ * [FLUME-748] - Create metric collection infrastructure
+ * [FLUME-962] - Failover capability for Client SDK
+ * [FLUME-992] - Create configuration stubs for sources, channels, sinks etc
+ * [FLUME-999] - Updating init scripts and variables to fit the term agent
+ * [FLUME-1052] - Core configuration component
+ * [FLUME-1053] - Basic SourceConfiguration
+ * [FLUME-1054] - Basic ChannelConfiguration
+ * [FLUME-1055] - Basic SinkConfiguration
+ * [FLUME-1105] - Allow the optional disabling of foreign keys
+ * [FLUME-1107] - Configuration keys for JDBC channel contain redundant prefix.
+ * [FLUME-1113] - JDBC Channel invokes size query on every put
diff --git a/source/sphinx/releases/1.3.0.rst b/source/sphinx/releases/1.3.0.rst
new file mode 100644
index 00000000..3db7bc6b
--- /dev/null
+++ b/source/sphinx/releases/1.3.0.rst
@@ -0,0 +1,195 @@
+=============
+Version 1.3.0
+=============
+
+.. rubric:: Status of this release
+
+This release is the second release of Apache Flume as an Apache top level project and
+is the second release that is considered ready for production use. This release
+has has substantially more features and functionality along with bug fixes and other
+enhancements.
+
+.. rubric:: Release Documentation
+
+* `Flume User Guide <content/1.3.0/FlumeUserGuide.pdf>`_
+* `Flume Developer Guide <content/1.3.0/FlumeDeveloperGuide.pdf>`_
+* `API Documentation <content/1.3.0/apidocs/index.html>`_
+
+.. rubric:: Changes
+
+
+** New Feature
+ * [FLUME-1199] - Add HTTP Post Source
+ * [FLUME-1371] - ElasticSearch Sink
+ * [FLUME-1382] - Flume adopt message from existing local Scribe
+ * [FLUME-1385] - Add a multiport syslog source
+ * [FLUME-1424] - File Channel should support encryption
+ * [FLUME-1425] - Create a SpoolDirectory Source and Client
+ * [FLUME-1488] - Load Balancing RPC client should support exponential backoff of failed nodes
+ * [FLUME-1537] - Dump RollingFileSink's counter status when agent stops
+ * [FLUME-1657] - Regex Extractor Interceptor
+
+** Improvement
+ * [FLUME-946] - Allow multiplexing channel selector to specify optional channels.
+ * [FLUME-1337] - Add IDEA files to .gitignore
+ * [FLUME-1358] - Add a regex-based filtering interceptor
+ * [FLUME-1383] - Improve various log messages in FileChannel and HDFSSink
+ * [FLUME-1408] - ScheduledExecutorService does not log uncaught Throwables, we should log them
+ * [FLUME-1418] - Improvement for Log4j configuration
+ * [FLUME-1419] - Using system time if 'timestamp' property is absent in event header
+ * [FLUME-1434] - Distinguish background worker with channel name
+ * [FLUME-1480] - Replace object descriptor with detailed component type plus name
+ * [FLUME-1487] - FileChannel format needs to be extensible
+ * [FLUME-1490] - Option to limit number of events sent in Stress source
+ * [FLUME-1496] - TestFileChannel is bloated
+ * [FLUME-1507] - Have "Topology Design Considerations" in User Guide
+ * [FLUME-1509] - Flume HDFS sink should allow for the use of different timezones when resolving sink paths
+ * [FLUME-1519] - LifecycleController prints tons of DEBUG messages
+ * [FLUME-1523] - Allow -X java opts to be passed to the agent on the command line
+ * [FLUME-1526] - LogFile log message is scary when no harm has been done
+ * [FLUME-1531] - Flume User Guide should provide more details on configuring the timestamp interceptor
+ * [FLUME-1535] - Ability to specify the capacity of MemoryChannel in bytes
+ * [FLUME-1536] - Support for batch size in StressSource
+ * [FLUME-1538] - Channels should expose channel fill ratio through JMX
+ * [FLUME-1543] - TestFileChannel should be factored into many tests
+ * [FLUME-1546] - File channel encryption: trim() passwords and warn user if he doesn't have JCE policy file
+ * [FLUME-1548] - Build dies due to older protocol buffers compiler
+ * [FLUME-1550] - Use maven-antrun-plugin to save version
+ * [FLUME-1554] - FileChannel fails to build on machines with old protocol buffer compiler
+ * [FLUME-1556] - It would be nice if NullSink logged the number of event processed every 10K or so
+ * [FLUME-1560] - TestFileChannel* tests which fill up the channel should use larger batch size than 1
+ * [FLUME-1563] - FileChannel Encryption KeyProvider configuration properties should be more consistent
+ * [FLUME-1564] - FileChannel log file creation could be clarified and tested
+ * [FLUME-1569] - MemoryChannel uses an Integer as a lock
+ * [FLUME-1575] - FIleChannel Encryption should disallow a null key
+ * [FLUME-1603] - FileChannel capacity reached message is unclear
+ * [FLUME-1607] - FileChannel We should use a regex as opposed to simple filename filter when finding logs
+ * [FLUME-1609] - FileChannel detecting when the underlying file systems are full could provide cleaner error recovery
+ * [FLUME-1631] - Retire hdfs.txnEventMax in HDFS sink
+ * [FLUME-1645] - add hdfs.fileSuffix property to HDFSEventSink
+ * [FLUME-1660] - Close "idle" hdfs handles
+ * [FLUME-1675] - Ignore netbeans config files in rat & git
+ * [FLUME-1681] - Disable empty-file unit test for Spooling File Reader
+ * [FLUME-1684] - Re-enable empty file unit test
+ * [FLUME-1689] - BodyTextSerializer should allow an option not to add a newline to each serialized event
+ * [FLUME-1692] - MultiportSyslogTCPSource user documentation and nickname
+ * [FLUME-1707] - Update FlumeDevGuide
+ * [FLUME-1706] - Website for 1.3 fails to build
+ * [FLUME-1698] - Update RELEASE-NOTES
+ * [FLUME-1711] - Update Flume User Guide
+ * [FLUME-1713] - Netcat source should allow for *not* returning "OK" upon receipt of each message.
+ * [FLUME-1740] - Remove contrib/ directory from Flume NG
+ * [FLUME-1750] - File spooling client uses -D as command line option
+ * [FLUME-1751] - User Guide Examples for File Channel encryption are broken in 1.3 rc5
+ * [FLUME-1749] - .gitignore and elipse related files should not be included in source tarball
+ * [FLUME-1752] - Update CHANGELOG for flume 1.3.0 rc6 to include latest changes
+
+
+** Bug
+ * [FLUME-1208] - Hbase sink should be built only in Hadoop-1.0 profile
+ * [FLUME-1256] - OutofMemory erros in Flume build
+ * [FLUME-1259] - Flume throws OutOfMemory error when sending data from netcat to avro source (negative test case)
+ * [FLUME-1276] - Create a static header interceptor
+ * [FLUME-1277] - Error parsing Syslog rfc 3164 messages with null values
+ * [FLUME-1310] - Make Asynch hbase sink test work with other versions of Hbase
+ * [FLUME-1354] - Update docs to show that recoverable memory channel is deprecated
+ * [FLUME-1362] - Port retrying in TestThriftLegacySource not working
+ * [FLUME-1363] - flume-ng-node - TestNetcatSource doesn't try multiple ports
+ * [FLUME-1364] - Document the necessity of the timestamp header when using time-related escapes for hdfs sink paths
+ * [FLUME-1373] - Remove hardcoded file separator in HDFSEventSink
+ * [FLUME-1374] - Support ganglia reporting
+ * [FLUME-1376] - StaticInterceptor doc update
+ * [FLUME-1377] - ChannelProcessor clobbers exception with NPE if Channel.getTransaction() throws
+ * [FLUME-1389] - Flume gives opaque error if interceptor type not specified
+ * [FLUME-1391] - Use sync() instead of syncFs() in HDFS Sink to be compatible with hadoop 0.20.2
+ * [FLUME-1392] - Inactive channels get added to source channels list causing NPE
+ * [FLUME-1398] - Improve concurrency for async hbase sink
+ * [FLUME-1412] - Commons collections is used in file channel - even though it is not in pom.xml
+ * [FLUME-1414] - VersionInfo should not create a log instance
+ * [FLUME-1416] - Version Info should have hardcoded git repo address
+ * [FLUME-1417] - File Channel checkpoint can be bad leading to the channel being unable to start.
+ * [FLUME-1420] - Exception should be thrown if we cannot instaniate an EventSerializer
+ * [FLUME-1421] - PollableSourceRunner does not name it's thread
+ * [FLUME-1422] - Fix "BarSource" Class Signature in Flume Developer Guide
+ * [FLUME-1426] - FileChannel Replay could be faster
+ * [FLUME-1428] - File Channel should not consider a file as inactive until all takes are committed.
+ * [FLUME-1432] - FileChannel should replay logs in the order they were written
+ * [FLUME-1437] - Checkpoint can miss pending takes.
+ * [FLUME-1470] - Syslog source does not parse facility correctly
+ * [FLUME-1479] - Multiple Sinks can connect to single Channel
+ * [FLUME-1482] - Flume should support exposing metrics via HTTP in JSON/some other web service format.
+ * [FLUME-1498] - File channel - Log updates and queue updates should be atomic
+ * [FLUME-1500] - Upgrade flume to use latest version of Avro - v1.7
+ * [FLUME-1504] - Test file channel times out randomly
+ * [FLUME-1506] - Child poms pull in specific versions of packages not in top level pom
+ * [FLUME-1512] - File Channel should not stop during a checkpoint.
+ * [FLUME-1513] - File Channel log close() method should not be synchronized
+ * [FLUME-1514] - Log4jAppender should also have flume-ng-configuration in the pom
+ * [FLUME-1515] - Fix flume-1.3.0 branch test failures on ASF Jenkins.
+ * [FLUME-1524] - TestMonitoredCounterGroup is flaky
+ * [FLUME-1525] - On some (slow) machines TestFileChannel can fail
+ * [FLUME-1534] - CheckpointRebuilder$ComparableFlumeEventPointer#equals does not work correctly.
+ * [FLUME-1540] - CheckpointBuilder needs to open logfiles in inactive mode
+ * [FLUME-1541] - Implement a SinkSelector for LoadBalancingSinkProcessor that includes failover mechanics
+ * [FLUME-1544] - Update dev guide to reflect the protoc requirement
+ * [FLUME-1545] - File channel missing implicit dependency on commons-lang
+ * [FLUME-1552] - TestFileChannelEncryption fails without a high encryption policy file
+ * [FLUME-1553] - TestFileChannelEncryption should be refactored to use TestFileChannelBase
+ * [FLUME-1555] - StressSource outputs bad log messages that reference (Sequence generator)
+ * [FLUME-1557] - It would be nice if SequenceGeneratorSource could do batching
+ * [FLUME-1562] - TestLoadBalancingSinkProcessor.testRoundRobinBackoffFailureRecovery is flaky, fails every once in a while...
+ * [FLUME-1565] - FileChannel Decryption in RandomReader is not thread safe
+ * [FLUME-1567] - Avro source should expose the number of active connections through JMX
+ * [FLUME-1570] - StressSource batching does not work unless maxTotalEvents is specified
+ * [FLUME-1572] - Add batching to FILE_ROLL sink
+ * [FLUME-1576] - CHECKPOINT_INCOMPLETE should be synced to disk before starting the checkpoint.
+ * [FLUME-1577] - CHECKPOINT_INCOMPLETE should be synced to disk before starting the checkpoint.
+ * [FLUME-1578] - Proposal to modify file channel encryption config
+ * [FLUME-1582] - flume-ng script should set LD_LIBRARY_PATH
+ * [FLUME-1583] - FileChannel fast full replay will always be used if enabled
+ * [FLUME-1595] - HDFS SequenceFile implementation is not durable due to not using syncFs()
+ * [FLUME-1606] - Rollbacks of Put transactions does not clear the transaction from inflight puts.
+ * [FLUME-1610] - HDFSEventSink and bucket writer have a race condition
+ * [FLUME-1611] - LogUtils regex can be precompiled
+ * [FLUME-1613] - All of the sink examples in the user guide are broken
+ * [FLUME-1616] - FileChannel will lose data in when rollback fails with IOException
+ * [FLUME-1620] - Update flume user guide for LoadBalancingSinkProcessor with the backoff changes.
+ * [FLUME-1622] - MemoryChannel throws NPE if the event has no body
+ * [FLUME-1638] - LoadBalancingRpcClient depends on com.google.common.collect.Maps
+ * [FLUME-1639] - Client SDK should not have dependency on Guava
+ * [FLUME-1650] - Fix flume-ng-hbase-sink compilation against Hadoop 2.X
+ * [FLUME-1651] - in the hadoop-0.23 profile HBase version needs to be at least 0.94.2
+ * [FLUME-1653] - Update Hadoop-23 profile to point to hadoop-2 alpha artifacts
+ * [FLUME-1655] - Doc update needed for Regex Filtering Interceptor
+ * [FLUME-1656] - flume-ng script disregards stderr from hadoop command when finding hadoop jars
+ * [FLUME-1659] - JSON Handler should return simple events, not JSONEvents.
+ * [FLUME-1662] - Convert null body in events into zero length arrays.
+ * [FLUME-1664] - Logutils.getLogs remove unneeded directory check
+ * [FLUME-1671] - Add support for custom components to MonitoredCounterGroup
+ * [FLUME-1673] - MonitoredCounterGroup publishes this reference to platform MBean server in constructor
+ * [FLUME-1683] - Fix Time Granularity Bug in SpoolingFileLineReader
+ * [FLUME-1690] - Elastic Search Sink doesn't run it's unit tests
+ * [FLUME-1712] - Regex Extractor Interceptor tests have timezone issues
+ * [FLUME-1705] - SpoolDirectory short name points at the wrong class
+ * [FLUME-1719] - Example export command in README do not properly close the string
+ * [FLUME-1723] - AsyncHBase and Avro bring in different versions of Netty
+ * [FLUME-1726] - SpoolingFileLineReader must close the reader before renaming
+ * [FLUME-1743] - Multiport syslog tcp source does not load (v1.3 rc5)
+
+
+** Test
+ * [FLUME-1492] - Create integration test for file channel
+
+** Task
+ * [FLUME-1359] - Update main pom.xml file with regards to Flume TLP promotion
+
+** Sub-task
+ * [FLUME-897] - Implement write ahead log library
+ * [FLUME-1629] - Add Audience/Stability annotations
+ * [FLUME-1694] - Fix LICENSE file for binary artifacts
+ * [FLUME-1695] - Fix tarball names and directories
+ * [FLUME-1696] - Update build instructions as Flume build requires more memory
+ * [FLUME-1697] - Update CHANGELOG after 1.3.0 RC0
+ * [FLUME-1727] - Update CHANGELOG for rc4
+
+
diff --git a/source/sphinx/releases/1.3.1.rst b/source/sphinx/releases/1.3.1.rst
new file mode 100644
index 00000000..25ce57cf
--- /dev/null
+++ b/source/sphinx/releases/1.3.1.rst
@@ -0,0 +1,41 @@
+=============
+Version 1.3.1
+=============
+
+.. rubric:: Status of this release
+
+This release is the third release of Apache Flume as an Apache top level project and
+is the third release that is considered ready for production use. This release is
+primarily a maintenance release for Flume 1.3.0, and includes
+several bug fixes and performance enhancements.
+
+.. rubric:: Release Documentation
+
+* `Flume User Guide <content/1.3.1/FlumeUserGuide.pdf>`_
+* `Flume Developer Guide <content/1.3.1/FlumeDeveloperGuide.pdf>`_
+* `API Documentation <content/1.3.1/apidocs/index.html>`_
+
+.. rubric:: Changes
+
+
+** Bug
+ * [FLUME-1761] - FileChannel can NPE when log metadata file is empty
+ * [FLUME-1763] - FileChannel Checkpoints should not be done without free space
+ * [FLUME-1773] - File Channel worker thread should not be daemon
+ * [FLUME-1774] - EventBackingStoreFactory error message asks user to delete checkpoint which is now done automatically
+ * [FLUME-1775] - FileChannel Log Background worker should catch Throwable
+ * [FLUME-1788] - Flume Thrift source can fail intermittently because of a race condition in Thrift server implementation on some Linux systems
+ * [FLUME-1789] - Unit tests TestJCEFileKeyProvider and TestFileChannelEncryption fail with IBM JDK and flume-1.3.0
+ * [FLUME-1795] - Flume thrift legacy source does not have proper logging configured
+
+
+
+** Improvement
+ * [FLUME-1571] - Channels should check for positive capacity and transaction capacity values
+ * [FLUME-1699] - Make the rename of the meta file platform neutral
+ * [FLUME-1762] - File Channel should recover automatically if the checkpoint is incomplete or bad by deleting the contents of the checkpoint directory
+ * [FLUME-1790] - Commands in EncryptionTestUtils comments require high encryption pack to be installed
+ * [FLUME-1794] - FileChannel check for full disks in the background
+
+
+
diff --git a/source/sphinx/releases/1.4.0.rst b/source/sphinx/releases/1.4.0.rst
new file mode 100644
index 00000000..57963260
--- /dev/null
+++ b/source/sphinx/releases/1.4.0.rst
@@ -0,0 +1,227 @@
+=============
+Version 1.4.0
+=============
+
+.. rubric:: Status of this release
+
+Apache Flume 1.4.0 is the fourth release of Flume as an Apache top-level project
+(TLP). Apache Flume 1.4.0 is production-ready software.
+
+.. rubric:: Release Documentation
+
+* `Flume 1.4.0 User Guide <content/1.4.0/FlumeUserGuide.html>`__ (also in `pdf <content/1.4.0/FlumeUserGuide.pdf>`__)
+* `Flume 1.4.0 Developer Guide <content/1.4.0/FlumeDeveloperGuide.html>`__ (also in `pdf <content/1.4.0/FlumeDeveloperGuide.pdf>`__)
+* `Flume 1.4.0 API Documentation <content/1.4.0/apidocs/index.html>`__
+
+.. rubric:: Changes
+
+
+Release Notes - Flume - Version v1.4.0
+
+* New Feature
+
+ * [`FLUME-924 <https://issues.apache.org/jira/browse/FLUME-924>`__] - Implement a JMS source for Flume NG
+ * [`FLUME-997 <https://issues.apache.org/jira/browse/FLUME-997>`__] - Support secure transport mechanism
+ * [`FLUME-1502 <https://issues.apache.org/jira/browse/FLUME-1502>`__] - Support for running simple configurations embedded in host process
+ * [`FLUME-1516 <https://issues.apache.org/jira/browse/FLUME-1516>`__] - FileChannel Write Dual Checkpoints to avoid replays
+ * [`FLUME-1632 <https://issues.apache.org/jira/browse/FLUME-1632>`__] - Persist progress on each file in file spooling client/source
+ * [`FLUME-1735 <https://issues.apache.org/jira/browse/FLUME-1735>`__] - Add support for a plugins.d directory
+ * [`FLUME-1894 <https://issues.apache.org/jira/browse/FLUME-1894>`__] - Implement Thrift RPC
+ * [`FLUME-1917 <https://issues.apache.org/jira/browse/FLUME-1917>`__] - FileChannel group commit (coalesce fsync)
+ * [`FLUME-2010 <https://issues.apache.org/jira/browse/FLUME-2010>`__] - Support Avro records in Log4jAppender and the HDFS Sink
+ * [`FLUME-2048 <https://issues.apache.org/jira/browse/FLUME-2048>`__] - Avro container file deserializer
+ * [`FLUME-2070 <https://issues.apache.org/jira/browse/FLUME-2070>`__] - Add a Flume Morphline Solr Sink
+
+* Improvement
+
+ * [`FLUME-1076 <https://issues.apache.org/jira/browse/FLUME-1076>`__] - Sink batch sizes vary wildy
+ * [`FLUME-1100 <https://issues.apache.org/jira/browse/FLUME-1100>`__] - HDFSWriterFactory and HDFSFormatterFactory should allow extension
+ * [`FLUME-1571 <https://issues.apache.org/jira/browse/FLUME-1571>`__] - Channels should check for positive capacity and transaction capacity values
+ * [`FLUME-1586 <https://issues.apache.org/jira/browse/FLUME-1586>`__] - File Channel should support verifying integrity of individual events.
+ * [`FLUME-1652 <https://issues.apache.org/jira/browse/FLUME-1652>`__] - Logutils.getLogs could NPE
+ * [`FLUME-1661 <https://issues.apache.org/jira/browse/FLUME-1661>`__] - ExecSource cannot execute complex Unix commands
+ * [`FLUME-1677 <https://issues.apache.org/jira/browse/FLUME-1677>`__] - Add File-channel dependency to flume-ng-node's pom.xml
+ * [`FLUME-1699 <https://issues.apache.org/jira/browse/FLUME-1699>`__] - Make the rename of the meta file platform neutral
+ * [`FLUME-1702 <https://issues.apache.org/jira/browse/FLUME-1702>`__] - HDFSEventSink should write to a hidden file as opposed to a .tmp file
+ * [`FLUME-1740 <https://issues.apache.org/jira/browse/FLUME-1740>`__] - Remove contrib/ directory from Flume NG
+ * [`FLUME-1745 <https://issues.apache.org/jira/browse/FLUME-1745>`__] - FlumeConfiguration Eats Exceptions
+ * [`FLUME-1756 <https://issues.apache.org/jira/browse/FLUME-1756>`__] - Avro client should be able to use load balancing RPC
+ * [`FLUME-1757 <https://issues.apache.org/jira/browse/FLUME-1757>`__] - Improve configuration of hbase serializers
+ * [`FLUME-1762 <https://issues.apache.org/jira/browse/FLUME-1762>`__] - File Channel should recover automatically if the checkpoint is incomplete or bad by deleting the contents of the checkpoint directory
+ * [`FLUME-1768 <https://issues.apache.org/jira/browse/FLUME-1768>`__] - Multiplexing channel selector should allow optional-only channels
+ * [`FLUME-1769 <https://issues.apache.org/jira/browse/FLUME-1769>`__] - Replicating channel selector should support optional channels
+ * [`FLUME-1770 <https://issues.apache.org/jira/browse/FLUME-1770>`__] - Flume should have serializer which supports serializer the headers to a simple string
+ * [`FLUME-1777 <https://issues.apache.org/jira/browse/FLUME-1777>`__] - AbstractSource does not provide enough implementation for sub-classes
+ * [`FLUME-1790 <https://issues.apache.org/jira/browse/FLUME-1790>`__] - Commands in EncryptionTestUtils comments require high encryption pack to be installed
+ * [`FLUME-1794 <https://issues.apache.org/jira/browse/FLUME-1794>`__] - FileChannel check for full disks in the background
+ * [`FLUME-1800 <https://issues.apache.org/jira/browse/FLUME-1800>`__] - Docs for spooling source durability changes
+ * [`FLUME-1808 <https://issues.apache.org/jira/browse/FLUME-1808>`__] - ElasticSearchSink is missing log4.properties
+ * [`FLUME-1821 <https://issues.apache.org/jira/browse/FLUME-1821>`__] - Support configuration of hbase instances to be used in AsyncHBaseSink from flume config
+ * [`FLUME-1847 <https://issues.apache.org/jira/browse/FLUME-1847>`__] - NPE in SourceConfiguration
+ * [`FLUME-1848 <https://issues.apache.org/jira/browse/FLUME-1848>`__] - HDFSDataStream logger is actually for a sequence file
+ * [`FLUME-1855 <https://issues.apache.org/jira/browse/FLUME-1855>`__] - Sequence gen source should be able to stop after a fixed number of events
+ * [`FLUME-1864 <https://issues.apache.org/jira/browse/FLUME-1864>`__] - Allow hdfs idle callback to clean up closed bucket writers
+ * [`FLUME-1874 <https://issues.apache.org/jira/browse/FLUME-1874>`__] - Ship with log4j.properties file that has a reliable time based rolling policy
+ * [`FLUME-1876 <https://issues.apache.org/jira/browse/FLUME-1876>`__] - Document hadoop dependency of FileChannel when used with EmbeddedAgent
+ * [`FLUME-1878 <https://issues.apache.org/jira/browse/FLUME-1878>`__] - FileChannel replay should print status every 10000 events
+ * [`FLUME-1886 <https://issues.apache.org/jira/browse/FLUME-1886>`__] - Add a JMS enum type to SourceType so that users don't need to enter FQCN for JMSSource
+ * [`FLUME-1889 <https://issues.apache.org/jira/browse/FLUME-1889>`__] - Add HBASE and ASYNC_HBASE enum types to SinkType so that users don't need to enter FQCNs
+ * [`FLUME-1906 <https://issues.apache.org/jira/browse/FLUME-1906>`__] - Ability to disable WAL for put operation in HBaseSink
+ * [`FLUME-1915 <https://issues.apache.org/jira/browse/FLUME-1915>`__] - Enhance NettyAvroRpcClient and the use of NettyServer to optionally use compression
+ * [`FLUME-1926 <https://issues.apache.org/jira/browse/FLUME-1926>`__] - Optionally timeout Avro Sink Rpc Clients to avoid stickiness
+ * [`FLUME-1940 <https://issues.apache.org/jira/browse/FLUME-1940>`__] - Log a snapshot of Flume metrics on shutdown
+ * [`FLUME-1945 <https://issues.apache.org/jira/browse/FLUME-1945>`__] - HBase Serializer allow key from regular expression group
+ * [`FLUME-1976 <https://issues.apache.org/jira/browse/FLUME-1976>`__] - JMS Source document should provide instruction on JMS implementation jars
+ * [`FLUME-1977 <https://issues.apache.org/jira/browse/FLUME-1977>`__] - JMS Source connectionFactory property is not documented
+ * [`FLUME-1992 <https://issues.apache.org/jira/browse/FLUME-1992>`__] - ElasticSearch dependency is marked optional
+ * [`FLUME-1994 <https://issues.apache.org/jira/browse/FLUME-1994>`__] - Add ELASTICSEARCH enum type to SinkType to eliminate need for FQCN in agent configuration files
+ * [`FLUME-2004 <https://issues.apache.org/jira/browse/FLUME-2004>`__] - Need to capture metrics on the Flume exec source such as events received, rejected, etc.
+ * [`FLUME-2005 <https://issues.apache.org/jira/browse/FLUME-2005>`__] - Minor improvements to Flume assembly config
+ * [`FLUME-2008 <https://issues.apache.org/jira/browse/FLUME-2008>`__] - it would be very convenient to have a fat jar of flume-ng-log4jappender
+ * [`FLUME-2009 <https://issues.apache.org/jira/browse/FLUME-2009>`__] - Flume project throws error when imported into Eclipse IDE (Juno)
+ * [`FLUME-2013 <https://issues.apache.org/jira/browse/FLUME-2013>`__] - Parametrize java source and target version in the main pom file
+ * [`FLUME-2015 <https://issues.apache.org/jira/browse/FLUME-2015>`__] - ElasticSearchSink: need access to IndexRequestBuilder instance during flume event processing
+ * [`FLUME-2046 <https://issues.apache.org/jira/browse/FLUME-2046>`__] - Typo in HBaseSink java doc
+ * [`FLUME-2049 <https://issues.apache.org/jira/browse/FLUME-2049>`__] - Compile ElasticSearchSink with elasticsearch 0.90
+ * [`FLUME-2062 <https://issues.apache.org/jira/browse/FLUME-2062>`__] - make it possible for HBase sink to deposit event headers into corresponding column qualifiers
+ * [`FLUME-2063 <https://issues.apache.org/jira/browse/FLUME-2063>`__] - Add Configurable charset to RegexHbaseEventSerializer
+ * [`FLUME-2076 <https://issues.apache.org/jira/browse/FLUME-2076>`__] - JMX metrics support for HTTP Source
+ * [`FLUME-2093 <https://issues.apache.org/jira/browse/FLUME-2093>`__] - binary tarball that is created by flume's assembly shouldn't contain sources
+ * [`FLUME-2100 <https://issues.apache.org/jira/browse/FLUME-2100>`__] - Increase default batchSize of Morphline Solr Sink
+ * [`FLUME-2105 <https://issues.apache.org/jira/browse/FLUME-2105>`__] - Add docs for MorphlineSolrSink
+
+* Bug
+
+ * [`FLUME-1110 <https://issues.apache.org/jira/browse/FLUME-1110>`__] - HDFS Sink throws IllegalStateException when flume-daemon shuts down
+ * [`FLUME-1153 <https://issues.apache.org/jira/browse/FLUME-1153>`__] - flume-ng script is missing some agent options in help output
+ * [`FLUME-1175 <https://issues.apache.org/jira/browse/FLUME-1175>`__] - RollingFileSink complains of Bad File Descriptor upon a reconfig event
+ * [`FLUME-1262 <https://issues.apache.org/jira/browse/FLUME-1262>`__] - Move doc generation to a different profile
+ * [`FLUME-1285 <https://issues.apache.org/jira/browse/FLUME-1285>`__] - FileChannel has a dependency on Hadoop IO classes
+ * [`FLUME-1296 <https://issues.apache.org/jira/browse/FLUME-1296>`__] - Lifecycle supervisor should check if the monitor service is still running before supervising
+ * [`FLUME-1511 <https://issues.apache.org/jira/browse/FLUME-1511>`__] - Scribe-source doesn't handle zero message request correctly.
+ * [`FLUME-1676 <https://issues.apache.org/jira/browse/FLUME-1676>`__] - ExecSource should provide a configurable charset
+ * [`FLUME-1688 <https://issues.apache.org/jira/browse/FLUME-1688>`__] - Bump AsyncHBase version to 1.4.1
+ * [`FLUME-1709 <https://issues.apache.org/jira/browse/FLUME-1709>`__] - HDFS CompressedDataStream doesn't support serializer parameter
+ * [`FLUME-1720 <https://issues.apache.org/jira/browse/FLUME-1720>`__] - LICENSE file contain entry for protobuf-<version>.jar, however proper artifact name is protobuf-java-<version>.jar
+ * [`FLUME-1731 <https://issues.apache.org/jira/browse/FLUME-1731>`__] - SpoolableDirectorySource should have configurable support for deleting files it has already completed instead of renaming
+ * [`FLUME-1741 <https://issues.apache.org/jira/browse/FLUME-1741>`__] - ElasticSearch tests leave directory data/elasticsearch/nodes/ lying around
+ * [`FLUME-1748 <https://issues.apache.org/jira/browse/FLUME-1748>`__] - HDFS Sink should check if the thread is interrupted before performing any HDFS operations
+ * [`FLUME-1755 <https://issues.apache.org/jira/browse/FLUME-1755>`__] - Load balancing RPC client has issues with downed hosts
+ * [`FLUME-1766 <https://issues.apache.org/jira/browse/FLUME-1766>`__] - AvroSource throws confusing exception when configured without a port
+ * [`FLUME-1772 <https://issues.apache.org/jira/browse/FLUME-1772>`__] - AbstractConfigurationProvider should remove component which throws exception from configure method.
+ * [`FLUME-1773 <https://issues.apache.org/jira/browse/FLUME-1773>`__] - File Channel worker thread should not be daemon
+ * [`FLUME-1774 <https://issues.apache.org/jira/browse/FLUME-1774>`__] - EventBackingStoreFactory error message asks user to delete checkpoint which is now done automatically
+ * [`FLUME-1775 <https://issues.apache.org/jira/browse/FLUME-1775>`__] - FileChannel Log Background worker should catch Throwable
+ * [`FLUME-1776 <https://issues.apache.org/jira/browse/FLUME-1776>`__] - Several modules require commons-lang but do not declare this in the pom
+ * [`FLUME-1778 <https://issues.apache.org/jira/browse/FLUME-1778>`__] - Upgrade Flume to use Avro 1.7.3
+ * [`FLUME-1784 <https://issues.apache.org/jira/browse/FLUME-1784>`__] - JMSource fix minor documentation problem and parameter name
+ * [`FLUME-1788 <https://issues.apache.org/jira/browse/FLUME-1788>`__] - Flume Thrift source can fail intermittently because of a race condition in Thrift server implementation on some Linux systems
+ * [`FLUME-1789 <https://issues.apache.org/jira/browse/FLUME-1789>`__] - Unit tests TestJCEFileKeyProvider and TestFileChannelEncryption fail with IBM JDK and flume-1.3.0
+ * [`FLUME-1795 <https://issues.apache.org/jira/browse/FLUME-1795>`__] - Flume thrift legacy source does not have proper logging configured
+ * [`FLUME-1797 <https://issues.apache.org/jira/browse/FLUME-1797>`__] - TestFlumeConfiguration is in com.apache.flume.conf namespace.
+ * [`FLUME-1799 <https://issues.apache.org/jira/browse/FLUME-1799>`__] - Generated source tarball is missing flume-ng-embedded-agent
+ * [`FLUME-1802 <https://issues.apache.org/jira/browse/FLUME-1802>`__] - Missing parameter --conf in example of the Flume User Guide
+ * [`FLUME-1803 <https://issues.apache.org/jira/browse/FLUME-1803>`__] - Generated dist tarball is missing flume-ng-embedded-agent
+ * [`FLUME-1804 <https://issues.apache.org/jira/browse/FLUME-1804>`__] - JMS source not included in binary dist
+ * [`FLUME-1805 <https://issues.apache.org/jira/browse/FLUME-1805>`__] - Embedded agent deps should be specified in dependencyManagement section of pom
+ * [`FLUME-1818 <https://issues.apache.org/jira/browse/FLUME-1818>`__] - Support various layouts in log4jappender
+ * [`FLUME-1819 <https://issues.apache.org/jira/browse/FLUME-1819>`__] - ExecSource don't flush the cache if there is no input entries
+ * [`FLUME-1820 <https://issues.apache.org/jira/browse/FLUME-1820>`__] - Should not be possible for RPC client to block indefinitely on close()
+ * [`FLUME-1822 <https://issues.apache.org/jira/browse/FLUME-1822>`__] - Update javadoc for FlumeConfiguration
+ * [`FLUME-1823 <https://issues.apache.org/jira/browse/FLUME-1823>`__] - LoadBalancingRpcClient method must throw exception if it is called after close is called.
+ * [`FLUME-1824 <https://issues.apache.org/jira/browse/FLUME-1824>`__] - Inflights can complete successfully even if checkpoint fails
+ * [`FLUME-1828 <https://issues.apache.org/jira/browse/FLUME-1828>`__] - ResettableInputStream should support seek()
+ * [`FLUME-1834 <https://issues.apache.org/jira/browse/FLUME-1834>`__] - Userguide on trunk is missing some memory channel props
+ * [`FLUME-1835 <https://issues.apache.org/jira/browse/FLUME-1835>`__] - Flume User Guide has wrong prop in Load Balancing Sink Selector
+ * [`FLUME-1844 <https://issues.apache.org/jira/browse/FLUME-1844>`__] - HDFSEventSink should have option to use RawLocalFileSystem
+ * [`FLUME-1845 <https://issues.apache.org/jira/browse/FLUME-1845>`__] - Document plugin.d directory structure
+ * [`FLUME-1849 <https://issues.apache.org/jira/browse/FLUME-1849>`__] - Embedded Agent doesn't shutdown supervisor
+ * [`FLUME-1852 <https://issues.apache.org/jira/browse/FLUME-1852>`__] - Issues with EmbeddedAgentConfiguration
+ * [`FLUME-1854 <https://issues.apache.org/jira/browse/FLUME-1854>`__] - Application class can deadlock if stopped immediately after start
+ * [`FLUME-1863 <https://issues.apache.org/jira/browse/FLUME-1863>`__] - EmbeddedAgent pom must pull in file channel
+ * [`FLUME-1865 <https://issues.apache.org/jira/browse/FLUME-1865>`__] - Rename the Sequence File formatters to Serializer to be consistent with the rest of Flume
+ * [`FLUME-1866 <https://issues.apache.org/jira/browse/FLUME-1866>`__] - ChannelProcessor is not logging ChannelExceptions.
+ * [`FLUME-1867 <https://issues.apache.org/jira/browse/FLUME-1867>`__] - There's no option to set hostname for HTTPSource
+ * [`FLUME-1868 <https://issues.apache.org/jira/browse/FLUME-1868>`__] - FlumeUserGuide mentions wrong FQCN for JSONHandler
+ * [`FLUME-1869 <https://issues.apache.org/jira/browse/FLUME-1869>`__] - Request to add "HTTP" source type to SourceType.java
+ * [`FLUME-1870 <https://issues.apache.org/jira/browse/FLUME-1870>`__] - Flume sends non-numeric values with type as float to Ganglia causing ganglia to crash
+ * [`FLUME-1872 <https://issues.apache.org/jira/browse/FLUME-1872>`__] - SpoolingDirectorySource doesn't delete tracker file when deletePolicy is "immediate"
+ * [`FLUME-1879 <https://issues.apache.org/jira/browse/FLUME-1879>`__] - Secure HBase documentation
+ * [`FLUME-1880 <https://issues.apache.org/jira/browse/FLUME-1880>`__] - Double-logging of created HDFS files
+ * [`FLUME-1882 <https://issues.apache.org/jira/browse/FLUME-1882>`__] - Allow case-insensitive deserializer value for SpoolDirectorySource
+ * [`FLUME-1890 <https://issues.apache.org/jira/browse/FLUME-1890>`__] - Flume should set the hbase keytab and principal in HBase conf object.
+ * [`FLUME-1891 <https://issues.apache.org/jira/browse/FLUME-1891>`__] - Fast replay runs even when checkpoint exists.
+ * [`FLUME-1893 <https://issues.apache.org/jira/browse/FLUME-1893>`__] - File Channel could miss possible checkpoint corruption
+ * [`FLUME-1911 <https://issues.apache.org/jira/browse/FLUME-1911>`__] - Add deprecation back to the legacy thrift code
+ * [`FLUME-1916 <https://issues.apache.org/jira/browse/FLUME-1916>`__] - HDFS sink should poll for # of active replicas. If less than required, roll the file.
+ * [`FLUME-1918 <https://issues.apache.org/jira/browse/FLUME-1918>`__] - File Channel cannot handle capacity of more than 500 Million events
+ * [`FLUME-1922 <https://issues.apache.org/jira/browse/FLUME-1922>`__] - HDFS Sink should optionally insert the timestamp at the sink
+ * [`FLUME-1924 <https://issues.apache.org/jira/browse/FLUME-1924>`__] - Bug in serializer context parsing in RollingFileSink
+ * [`FLUME-1925 <https://issues.apache.org/jira/browse/FLUME-1925>`__] - HDFS timeouts should not starve other threads
+ * [`FLUME-1929 <https://issues.apache.org/jira/browse/FLUME-1929>`__] - CheckpointRebuilder main method does not work
+ * [`FLUME-1930 <https://issues.apache.org/jira/browse/FLUME-1930>`__] - Inflights should clean up executors on close.
+ * [`FLUME-1931 <https://issues.apache.org/jira/browse/FLUME-1931>`__] - HDFS Sink has a commons-lang dependency which is missing in pom
+ * [`FLUME-1932 <https://issues.apache.org/jira/browse/FLUME-1932>`__] - no-reload-conf command line param does not work
+ * [`FLUME-1937 <https://issues.apache.org/jira/browse/FLUME-1937>`__] - Issue with maxUnderReplication in HDFS sink
+ * [`FLUME-1939 <https://issues.apache.org/jira/browse/FLUME-1939>`__] - FlumeEventQueue must check if file is open before setting the length of the file
+ * [`FLUME-1943 <https://issues.apache.org/jira/browse/FLUME-1943>`__] - ExecSource tests failing on Jenkins
+ * [`FLUME-1948 <https://issues.apache.org/jira/browse/FLUME-1948>`__] - plugins.d directory(ies) should be separately overridable, independent of FLUME_HOME
+ * [`FLUME-1949 <https://issues.apache.org/jira/browse/FLUME-1949>`__] - Documentation for sink processor lists incorrect default
+ * [`FLUME-1955 <https://issues.apache.org/jira/browse/FLUME-1955>`__] - fileSuffix does not work with compressed streams
+ * [`FLUME-1958 <https://issues.apache.org/jira/browse/FLUME-1958>`__] - Remove attlasian-ide-plugin.xml from the repo
+ * [`FLUME-1964 <https://issues.apache.org/jira/browse/FLUME-1964>`__] - hdfs sink depends on commons-io but does not specify it in the pom
+ * [`FLUME-1965 <https://issues.apache.org/jira/browse/FLUME-1965>`__] - Thrift sink alias doesn't exist
+ * [`FLUME-1969 <https://issues.apache.org/jira/browse/FLUME-1969>`__] - Update user Guide to explain the purpose of minimumRequiredSpace setting for FileChannel
+ * [`FLUME-1974 <https://issues.apache.org/jira/browse/FLUME-1974>`__] - Thrift compatibility issue with hbase-0.92
+ * [`FLUME-1975 <https://issues.apache.org/jira/browse/FLUME-1975>`__] - Use TThreadedSelectServer in ThriftSource if it is available
+ * [`FLUME-1980 <https://issues.apache.org/jira/browse/FLUME-1980>`__] - Log4jAppender should optionally drop events if append fails
+ * [`FLUME-1981 <https://issues.apache.org/jira/browse/FLUME-1981>`__] - Rpc client expiration can be done in a more thread-safe way
+ * [`FLUME-1986 <https://issues.apache.org/jira/browse/FLUME-1986>`__] - doTestInflightCorrupts should not commit transactions
+ * [`FLUME-1993 <https://issues.apache.org/jira/browse/FLUME-1993>`__] - On Windows, when using the spooling directory source, there is a file sharing violation when trying to delete tracker file
+ * [`FLUME-2002 <https://issues.apache.org/jira/browse/FLUME-2002>`__] - Flume RPC Client creates 2 threads per each log attempt if the remote flume agent goes down
+ * [`FLUME-2011 <https://issues.apache.org/jira/browse/FLUME-2011>`__] - "mvn test" fails
+ * [`FLUME-2012 <https://issues.apache.org/jira/browse/FLUME-2012>`__] - Two tests fail on Mac OS (saying they fail to load native library) with Java 7
+ * [`FLUME-2014 <https://issues.apache.org/jira/browse/FLUME-2014>`__] - Race condition when using local timestamp with BucketPath
+ * [`FLUME-2023 <https://issues.apache.org/jira/browse/FLUME-2023>`__] - Flume must login to secure HBase before creating the HTable instance
+ * [`FLUME-2025 <https://issues.apache.org/jira/browse/FLUME-2025>`__] - ThriftSource throws NPE in stop() if start() failed because socket open failed or if thrift server instance creation threw.
+ * [`FLUME-2026 <https://issues.apache.org/jira/browse/FLUME-2026>`__] - TestHTTPSource should use any available port rather than a hardcoded port number
+ * [`FLUME-2027 <https://issues.apache.org/jira/browse/FLUME-2027>`__] - Check for default replication fails on federated cluster in hdfs sink
+ * [`FLUME-2032 <https://issues.apache.org/jira/browse/FLUME-2032>`__] - HDFSEventSink doesn't work in Windows
+ * [`FLUME-2036 <https://issues.apache.org/jira/browse/FLUME-2036>`__] - Make hostname optional for HTTPSource
+ * [`FLUME-2042 <https://issues.apache.org/jira/browse/FLUME-2042>`__] - log4jappender timeout should be configurable
+ * [`FLUME-2043 <https://issues.apache.org/jira/browse/FLUME-2043>`__] - JMS Source removed on failure to create configuration
+ * [`FLUME-2044 <https://issues.apache.org/jira/browse/FLUME-2044>`__] - HDFS Sink impersonation fails after the first file
+ * [`FLUME-2051 <https://issues.apache.org/jira/browse/FLUME-2051>`__] - Surefire 2.12 cannot run a single test on Windows. Upgrade to 2.12.3
+ * [`FLUME-2054 <https://issues.apache.org/jira/browse/FLUME-2054>`__] - Support Version Info on Windows and fix failure of TestVersionInfo
+ * [`FLUME-2057 <https://issues.apache.org/jira/browse/FLUME-2057>`__] - Failures in FileChannel's TestEventQueueBackingStoreFactory on Windows
+ * [`FLUME-2060 <https://issues.apache.org/jira/browse/FLUME-2060>`__] - Failure in TestLog.testReplaySucceedsWithUnusedEmptyLogMetaDataFastReplay test on Windows
+ * [`FLUME-2072 <https://issues.apache.org/jira/browse/FLUME-2072>`__] - JMX metrics support for HBase Sink
+ * [`FLUME-2081 <https://issues.apache.org/jira/browse/FLUME-2081>`__] - JMX metrics support for SpoolDir
+ * [`FLUME-2082 <https://issues.apache.org/jira/browse/FLUME-2082>`__] - JMX support for Seq Generator Source
+ * [`FLUME-2083 <https://issues.apache.org/jira/browse/FLUME-2083>`__] - Avro Source should not start if SSL is enabled and keystore cannot be opened
+ * [`FLUME-2098 <https://issues.apache.org/jira/browse/FLUME-2098>`__] - Make Solr sink depend on the CDK version of morphlines
+
+* Documentation
+
+ * [`FLUME-1621 <https://issues.apache.org/jira/browse/FLUME-1621>`__] - Document new MemoryChannel parameters in Flume User Guide
+ * [`FLUME-1910 <https://issues.apache.org/jira/browse/FLUME-1910>`__] - Add thrift RPC documentation
+ * [`FLUME-1953 <https://issues.apache.org/jira/browse/FLUME-1953>`__] - Fix dev guide error that says sink can read from multiple channels
+ * [`FLUME-1962 <https://issues.apache.org/jira/browse/FLUME-1962>`__] - Document proper specification of lzo codec as lzop in Flume User Guide
+ * [`FLUME-1979 <https://issues.apache.org/jira/browse/FLUME-1979>`__] - Wrong propname for connection reset interval in avro sink
+ * [`FLUME-2030 <https://issues.apache.org/jira/browse/FLUME-2030>`__] - Documentation of Configuration Changes JMSSource, HBaseSink, AsyncHBaseSink and ElasticSearchSink
+
+* Task
+
+ * [`FLUME-1686 <https://issues.apache.org/jira/browse/FLUME-1686>`__] - Exclude target directories & Eclipse files from rat checks
+ * [`FLUME-2094 <https://issues.apache.org/jira/browse/FLUME-2094>`__] - Remove the deprecated - Recoverable Memory Channel
+
+* Sub-task
+
+ * [`FLUME-1626 <https://issues.apache.org/jira/browse/FLUME-1626>`__] - Support Hbase security in Hbase sink
+ * [`FLUME-1630 <https://issues.apache.org/jira/browse/FLUME-1630>`__] - Flume configuration code could be improved
+ * [`FLUME-1674 <https://issues.apache.org/jira/browse/FLUME-1674>`__] - Documentation / Wiki
+ * [`FLUME-1896 <https://issues.apache.org/jira/browse/FLUME-1896>`__] - Implement Thrift RpcClient
+ * [`FLUME-1897 <https://issues.apache.org/jira/browse/FLUME-1897>`__] - Implement Thrift Sink
+ * [`FLUME-1898 <https://issues.apache.org/jira/browse/FLUME-1898>`__] - Implement Thrift Source
+ * [`FLUME-2102 <https://issues.apache.org/jira/browse/FLUME-2102>`__] - Update LICENSE file for Flume 1.4.0
+
diff --git a/source/sphinx/releases/1.5.0.1.rst b/source/sphinx/releases/1.5.0.1.rst
new file mode 100644
index 00000000..df3bf133
--- /dev/null
+++ b/source/sphinx/releases/1.5.0.1.rst
@@ -0,0 +1,24 @@
+===============
+Version 1.5.0.1
+===============
+
+.. rubric:: Status of this release
+
+Apache Flume 1.5.0.1 is the sixth release of Flume as an Apache top-level project
+(TLP). Apache Flume 1.5.0.1 is production-ready software.
+
+.. rubric:: Release Documentation
+
+* `Flume 1.5.0.1 User Guide <content/1.5.0.1/FlumeUserGuide.html>`__ (also in `pdf <content/1.5.0.1/FlumeUserGuide.pdf>`__)
+* `Flume 1.5.0.1 Developer Guide <content/1.5.0.1/FlumeDeveloperGuide.html>`__ (also in `pdf <content/1.5.0.1/FlumeDeveloperGuide.pdf>`__)
+* `Flume 1.5.0.1 API Documentation <content/1.5.0.1/apidocs/index.html>`__
+
+.. rubric:: Changes
+
+
+Release Notes - Flume - Version v1.5.0.1
+
+** Bug
+ * [`FLUME-2389 <https://issues.apache.org/jira/browse/FLUME-2389>`__] - Spillable Memory Channel Example is incorrect
+ * [`FLUME-2397 <https://issues.apache.org/jira/browse/FLUME-2397>`__] - HBase-98 compatibility
+ * [`FLUME-2400 <https://issues.apache.org/jira/browse/FLUME-2400>`__] - Dataset Sink is not built in hbase-98 profile
diff --git a/source/sphinx/releases/1.5.0.rst b/source/sphinx/releases/1.5.0.rst
new file mode 100644
index 00000000..ca9979d1
--- /dev/null
+++ b/source/sphinx/releases/1.5.0.rst
@@ -0,0 +1,152 @@
+=============
+Version 1.5.0
+=============
+
+.. rubric:: Status of this release
+
+Apache Flume 1.5.0 is the fifth release of Flume as an Apache top-level project
+(TLP). Apache Flume 1.5.0 is production-ready software.
+
+.. rubric:: Release Documentation
+
+* `Flume 1.5.0 User Guide <content/1.5.0/FlumeUserGuide.html>`__ (also in `pdf <content/1.5.0/FlumeUserGuide.pdf>`__)
+* `Flume 1.5.0 Developer Guide <content/1.5.0/FlumeDeveloperGuide.html>`__ (also in `pdf <content/1.5.0/FlumeDeveloperGuide.pdf>`__)
+* `Flume 1.5.0 API Documentation <content/1.5.0/apidocs/index.html>`__
+
+.. rubric:: Changes
+
+
+Release Notes - Flume - Version v1.5.0
+
+** New Feature
+ * [`FLUME-1227 <https://issues.apache.org/jira/browse/FLUME-1227>`__] - Introduce some sort of SpillableChannel
+ * [`FLUME-2056 <https://issues.apache.org/jira/browse/FLUME-2056>`__] - Allow SpoolDir to pass just the filename that is the source of an event
+ * [`FLUME-2071 <https://issues.apache.org/jira/browse/FLUME-2071>`__] - Flume Context doesn't support float or double configuration values.
+ * [`FLUME-2185 <https://issues.apache.org/jira/browse/FLUME-2185>`__] - Upgrade morphlines to 0.7.0
+ * [`FLUME-2188 <https://issues.apache.org/jira/browse/FLUME-2188>`__] - flume-ng-log4jappender Support user supplied headers
+ * [`FLUME-2225 <https://issues.apache.org/jira/browse/FLUME-2225>`__] - Elasticsearch Sink for ES HTTP API
+ * [`FLUME-2294 <https://issues.apache.org/jira/browse/FLUME-2294>`__] - Add a sink for Kite Datasets
+ * [`FLUME-2309 <https://issues.apache.org/jira/browse/FLUME-2309>`__] - Spooling directory should not always consume the oldest file first.
+
+** Improvement
+ * [`FLUME-1542 <https://issues.apache.org/jira/browse/FLUME-1542>`__] - Flume User Guide sample configurations need fixing
+ * [`FLUME-1899 <https://issues.apache.org/jira/browse/FLUME-1899>`__] - Make SpoolDir work with Sub-Directories
+ * [`FLUME-2052 <https://issues.apache.org/jira/browse/FLUME-2052>`__] - Spooling directory source should be able to replace or ignore malformed characters
+ * [`FLUME-2130 <https://issues.apache.org/jira/browse/FLUME-2130>`__] - Handle larger payloads via SyslogUDPSource
+ * [`FLUME-2139 <https://issues.apache.org/jira/browse/FLUME-2139>`__] - upgrade morphline library dependency to a 0.5.0 release
+ * [`FLUME-2154 <https://issues.apache.org/jira/browse/FLUME-2154>`__] - Reducing duplicate events caused by reset-connection-interval
+ * [`FLUME-2155 <https://issues.apache.org/jira/browse/FLUME-2155>`__] - Improve replay time
+ * [`FLUME-2181 <https://issues.apache.org/jira/browse/FLUME-2181>`__] - Optionally disable File Channel fsyncs
+ * [`FLUME-2202 <https://issues.apache.org/jira/browse/FLUME-2202>`__] - AsyncHBaseSink should coalesce increments to reduce RPC roundtrips
+ * [`FLUME-2206 <https://issues.apache.org/jira/browse/FLUME-2206>`__] - ElasticSearchSink ttl field modification to mimic Elasticsearch way of specifying TTL
+ * [`FLUME-2207 <https://issues.apache.org/jira/browse/FLUME-2207>`__] - HDFS file suffix style index suffix in ElasticSearchSink
+ * [`FLUME-2212 <https://issues.apache.org/jira/browse/FLUME-2212>`__] - upgrade to Morphlines-0.8.0
+ * [`FLUME-2213 <https://issues.apache.org/jira/browse/FLUME-2213>`__] - MorphlineInterceptor should share metric registry across threads for better (aggregate) reporting
+ * [`FLUME-2217 <https://issues.apache.org/jira/browse/FLUME-2217>`__] - Preserve priority, timestamp and hostname fields in MultiportSyslogTcp and Udp sources
+ * [`FLUME-2231 <https://issues.apache.org/jira/browse/FLUME-2231>`__] - Add details in Flume Ganglia config in User Guide
+ * [`FLUME-2243 <https://issues.apache.org/jira/browse/FLUME-2243>`__] - AvroSource to use TransceiverThreadFactory for Thread naming while initializing NioServerSocketChannelFactory
+ * [`FLUME-2267 <https://issues.apache.org/jira/browse/FLUME-2267>`__] - Increase default transactionCapacity for FileChannel to 10000 from 1000
+ * [`FLUME-2275 <https://issues.apache.org/jira/browse/FLUME-2275>`__] - Improve scalability of MorphlineInterceptor under contention
+ * [`FLUME-2292 <https://issues.apache.org/jira/browse/FLUME-2292>`__] - Upgrade mapdb to 0.9.8
+ * [`FLUME-2316 <https://issues.apache.org/jira/browse/FLUME-2316>`__] - Upgrade MorphlineSolrSink to kite-0.12.0
+ * [`FLUME-2340 <https://issues.apache.org/jira/browse/FLUME-2340>`__] - Refactor to make room for Morphlines Elasticsearch Sink
+ * [`FLUME-2343 <https://issues.apache.org/jira/browse/FLUME-2343>`__] - Add user impersonation to DatasetSink
+ * [`FLUME-2351 <https://issues.apache.org/jira/browse/FLUME-2351>`__] - Ability to override any parameter from the configuration file
+ * [`FLUME-2352 <https://issues.apache.org/jira/browse/FLUME-2352>`__] - HDFSCompressedDataStream should support appendBatch
+
+** Bug
+ * [`FLUME-1666 <https://issues.apache.org/jira/browse/FLUME-1666>`__] - Syslog source strips timestamp and hostname from log message body
+ * [`FLUME-1679 <https://issues.apache.org/jira/browse/FLUME-1679>`__] - Add dependency on Guava to flume-ng-elasticsearch-sink POM
+ * [`FLUME-1892 <https://issues.apache.org/jira/browse/FLUME-1892>`__] - IRC Sink NPE
+ * [`FLUME-1951 <https://issues.apache.org/jira/browse/FLUME-1951>`__] - Remove unused future from FlumeEventQueue
+ * [`FLUME-2007 <https://issues.apache.org/jira/browse/FLUME-2007>`__] - HDFS Sink should check if file is closed and retry if it is not.
+ * [`FLUME-2088 <https://issues.apache.org/jira/browse/FLUME-2088>`__] - Minor typo in Flume User Guide JSON Reporting section
+ * [`FLUME-2109 <https://issues.apache.org/jira/browse/FLUME-2109>`__] - HTTPS support in HTTP Source
+ * [`FLUME-2110 <https://issues.apache.org/jira/browse/FLUME-2110>`__] - Scribe Source must check if category is null before inserting the headers
+ * [`FLUME-2119 <https://issues.apache.org/jira/browse/FLUME-2119>`__] - duplicate files cause flume to enter irrecoverable state
+ * [`FLUME-2121 <https://issues.apache.org/jira/browse/FLUME-2121>`__] - Upgrade Flume to log4j 1.2.17 to be in sync with HDFS/HBase
+ * [`FLUME-2122 <https://issues.apache.org/jira/browse/FLUME-2122>`__] - Minor cleanups of User guide
+ * [`FLUME-2123 <https://issues.apache.org/jira/browse/FLUME-2123>`__] - Morphline Solr sink missing short type name
+ * [`FLUME-2124 <https://issues.apache.org/jira/browse/FLUME-2124>`__] - Upgrade Morphline Solr Sink to CDK 0.4.1
+ * [`FLUME-2127 <https://issues.apache.org/jira/browse/FLUME-2127>`__] - JMX shutdown command for Flume
+ * [`FLUME-2134 <https://issues.apache.org/jira/browse/FLUME-2134>`__] - AsyncHbase Sink bugfix plus tests errors on Windows
+ * [`FLUME-2135 <https://issues.apache.org/jira/browse/FLUME-2135>`__] - Add zip to the build distribution for Windows support
+ * [`FLUME-2136 <https://issues.apache.org/jira/browse/FLUME-2136>`__] - Windows - Fix intermitent test failure in TestMonitoredCounterGroup.java
+ * [`FLUME-2137 <https://issues.apache.org/jira/browse/FLUME-2137>`__] - Fix StagedInstall.java to invoke the correct startup script on Windows
+ * [`FLUME-2142 <https://issues.apache.org/jira/browse/FLUME-2142>`__] - HTTPS tests for http source
+ * [`FLUME-2145 <https://issues.apache.org/jira/browse/FLUME-2145>`__] - TestCheckpointRebuilder.testFastReplay fails on Windows due to checkpoint file being memory mapped
+ * [`FLUME-2151 <https://issues.apache.org/jira/browse/FLUME-2151>`__] - Windows: Update TestExecSource to use native commands on Windows
+ * [`FLUME-2152 <https://issues.apache.org/jira/browse/FLUME-2152>`__] - Flume user guide says "Ganglia support" where it should say "JSON support"
+ * [`FLUME-2156 <https://issues.apache.org/jira/browse/FLUME-2156>`__] - Unregister then re-register MonitoredCounterGroup JMX MBeans on reconfigure
+ * [`FLUME-2157 <https://issues.apache.org/jira/browse/FLUME-2157>`__] - Spool directory source does not shut down correctly when Flume is reconfigured
+ * [`FLUME-2159 <https://issues.apache.org/jira/browse/FLUME-2159>`__] - Sporadic failures in TestNettyAvroRpcClient.spinThreadsCrazily()
+ * [`FLUME-2161 <https://issues.apache.org/jira/browse/FLUME-2161>`__] - Flume does not support spaces in -X java-opt command line args
+ * [`FLUME-2172 <https://issues.apache.org/jira/browse/FLUME-2172>`__] - Update protocol buffer from 2.4.1 to 2.5.0
+ * [`FLUME-2176 <https://issues.apache.org/jira/browse/FLUME-2176>`__] - SpoolDir Source, get 'File has changed' exception but actually there is no change on the file
+ * [`FLUME-2182 <https://issues.apache.org/jira/browse/FLUME-2182>`__] - Spooling Directory Source will not ingest data completely when a wide character appears at the edge of a buffer
+ * [`FLUME-2184 <https://issues.apache.org/jira/browse/FLUME-2184>`__] - flume-ng-morphline-solr-sink Build failing due to incorrect hadoop-common dependency declaration
+ * [`FLUME-2191 <https://issues.apache.org/jira/browse/FLUME-2191>`__] - HDFS Minicluster tests failing after protobuf upgrade.
+ * [`FLUME-2192 <https://issues.apache.org/jira/browse/FLUME-2192>`__] - AbstractSinkProcessor stop incorrectly calls start
+ * [`FLUME-2198 <https://issues.apache.org/jira/browse/FLUME-2198>`__] - Avro Source should disable itself if ipFilterRules contains invalid rules
+ * [`FLUME-2199 <https://issues.apache.org/jira/browse/FLUME-2199>`__] - Flume builds with new version require mvn install before site can be generated
+ * [`FLUME-2200 <https://issues.apache.org/jira/browse/FLUME-2200>`__] - HTTP Source should be able to use "port" parameter if SSL is enabled
+ * [`FLUME-2208 <https://issues.apache.org/jira/browse/FLUME-2208>`__] - Jetty's default SocketSelector leaks File descriptors
+ * [`FLUME-2209 <https://issues.apache.org/jira/browse/FLUME-2209>`__] - AsyncHBaseSink will never recover if the column family does not exists for the first start
+ * [`FLUME-2210 <https://issues.apache.org/jira/browse/FLUME-2210>`__] - UnresolvedAddressException when using multiple hostNames in Elasticsearch sink configuration
+ * [`FLUME-2220 <https://issues.apache.org/jira/browse/FLUME-2220>`__] - ElasticSearch sink - duplicate fields in indexed document
+ * [`FLUME-2229 <https://issues.apache.org/jira/browse/FLUME-2229>`__] - Backoff period gets reset too often in OrderSelector
+ * [`FLUME-2233 <https://issues.apache.org/jira/browse/FLUME-2233>`__] - MemoryChannel lock contention on every put due to bytesRemaining Semaphore
+ * [`FLUME-2235 <https://issues.apache.org/jira/browse/FLUME-2235>`__] - idleFuture should be cancelled at the start of append
+ * [`FLUME-2238 <https://issues.apache.org/jira/browse/FLUME-2238>`__] - Provide option to configure worker threads in NettyAvroRpcClient
+ * [`FLUME-2239 <https://issues.apache.org/jira/browse/FLUME-2239>`__] - Clarify File Channel's dataDirs setting in User Guide
+ * [`FLUME-2252 <https://issues.apache.org/jira/browse/FLUME-2252>`__] - Add null check before closing table in HbaseSink
+ * [`FLUME-2253 <https://issues.apache.org/jira/browse/FLUME-2253>`__] - Please delete old releases from mirroring system
+ * [`FLUME-2255 <https://issues.apache.org/jira/browse/FLUME-2255>`__] - Spooling Directory Source cannot handle channel exceptions
+ * [`FLUME-2259 <https://issues.apache.org/jira/browse/FLUME-2259>`__] - transaction closure not happening for all the scenario in hbasesink
+ * [`FLUME-2262 <https://issues.apache.org/jira/browse/FLUME-2262>`__] - Log4j Appender should use timeStamp field not getTimestamp
+ * [`FLUME-2263 <https://issues.apache.org/jira/browse/FLUME-2263>`__] - Bump Hadoop 2 version to 2.3.0
+ * [`FLUME-2264 <https://issues.apache.org/jira/browse/FLUME-2264>`__] - Log4j Appender + Avro Reflection on string results in an invalid avro schema
+ * [`FLUME-2265 <https://issues.apache.org/jira/browse/FLUME-2265>`__] - Closed bucket writers should be removed from sfwriters map
+ * [`FLUME-2266 <https://issues.apache.org/jira/browse/FLUME-2266>`__] - Update Morphline Sink to kite-0.10.0
+ * [`FLUME-2270 <https://issues.apache.org/jira/browse/FLUME-2270>`__] - Twitter Source Documentation Does not load properly
+ * [`FLUME-2272 <https://issues.apache.org/jira/browse/FLUME-2272>`__] - Getting start page returning 503 error
+ * [`FLUME-2283 <https://issues.apache.org/jira/browse/FLUME-2283>`__] - Spool Dir source must check interrupt flag before writing to channel
+ * [`FLUME-2289 <https://issues.apache.org/jira/browse/FLUME-2289>`__] - Disable maxUnderReplication test which is extremely flakey
+ * [`FLUME-2301 <https://issues.apache.org/jira/browse/FLUME-2301>`__] - Update HBaseSink tests to reflect sink returning backoff only on empty batches
+ * [`FLUME-2302 <https://issues.apache.org/jira/browse/FLUME-2302>`__] - TestHDFS Sink fails with Can't get Kerberos realm
+ * [`FLUME-2303 <https://issues.apache.org/jira/browse/FLUME-2303>`__] - HBaseSink tests can fail based on order of execution
+ * [`FLUME-2304 <https://issues.apache.org/jira/browse/FLUME-2304>`__] - DatasetSink test fails unexpectedly
+ * [`FLUME-2305 <https://issues.apache.org/jira/browse/FLUME-2305>`__] - BucketWriter#close must cancel idleFuture
+ * [`FLUME-2307 <https://issues.apache.org/jira/browse/FLUME-2307>`__] - Remove Log writetimeout
+ * [`FLUME-2311 <https://issues.apache.org/jira/browse/FLUME-2311>`__] - Use standard way of finding queue/topic
+ * [`FLUME-2312 <https://issues.apache.org/jira/browse/FLUME-2312>`__] - Add utility for adorning HTTP contexts in Jetty
+ * [`FLUME-2314 <https://issues.apache.org/jira/browse/FLUME-2314>`__] - Upgrade to Mapdb 0.9.9
+ * [`FLUME-2320 <https://issues.apache.org/jira/browse/FLUME-2320>`__] - Deadlock in DatasetSink
+ * [`FLUME-2323 <https://issues.apache.org/jira/browse/FLUME-2323>`__] - Morphline sink must increment eventDrainAttemptCount when it takes event from channel
+ * [`FLUME-2324 <https://issues.apache.org/jira/browse/FLUME-2324>`__] - Support writing to multiple HBase clusters using HBaseSink
+ * [`FLUME-2325 <https://issues.apache.org/jira/browse/FLUME-2325>`__] - BucketWriter might throw BucketClosedException incorrectly
+ * [`FLUME-2328 <https://issues.apache.org/jira/browse/FLUME-2328>`__] - FileChannel Dual Checkpoint Backup Thread not released on Application stop
+ * [`FLUME-2329 <https://issues.apache.org/jira/browse/FLUME-2329>`__] - Add an alias for the Morphline Solr Sink
+ * [`FLUME-2330 <https://issues.apache.org/jira/browse/FLUME-2330>`__] - Remove the MorphlineHandlerImpl configuration option from MorphlineSink
+ * [`FLUME-2334 <https://issues.apache.org/jira/browse/FLUME-2334>`__] - Upgrade Asynchbase to 1.5.0 as this include hbase 0.96.x support
+ * [`FLUME-2335 <https://issues.apache.org/jira/browse/FLUME-2335>`__] - TestHBaseSink#testWithoutConfigurationObject() must delete the table at the end of the test
+ * [`FLUME-2336 <https://issues.apache.org/jira/browse/FLUME-2336>`__] - HBase tests that pass in ZK configs must use a new context object
+ * [`FLUME-2338 <https://issues.apache.org/jira/browse/FLUME-2338>`__] - Support coalescing increments in HBaseSink
+ * [`FLUME-2345 <https://issues.apache.org/jira/browse/FLUME-2345>`__] - Update to Kite 0.12.0 dependency
+ * [`FLUME-2347 <https://issues.apache.org/jira/browse/FLUME-2347>`__] - Add FLUME_JAVA_OPTS which allows users to inject java properties from cmd line
+ * [`FLUME-2350 <https://issues.apache.org/jira/browse/FLUME-2350>`__] - Consume Order tests need to space out file creation
+ * [`FLUME-2357 <https://issues.apache.org/jira/browse/FLUME-2357>`__] - HDFS sink should retry closing files that previously had close errors
+ * [`FLUME-2381 <https://issues.apache.org/jira/browse/FLUME-2381>`__] - Upgrade Hadoop version in Hadoop 2 profile to 2.4.0
+ * [`FLUME-2379 <https://issues.apache.org/jira/browse/FLUME-2379>`__] - Flume's pom files are invalid and not parseable by non-maven build tools
+
+** Documentation
+ * [`FLUME-1223 <https://issues.apache.org/jira/browse/FLUME-1223>`__] - Userguide improvement rolling file sink
+ * [`FLUME-1678 <https://issues.apache.org/jira/browse/FLUME-1678>`__] - Incorrect documentation for HBase sink
+ * [`FLUME-1851 <https://issues.apache.org/jira/browse/FLUME-1851>`__] - User Guide grammar mistake
+ * [`FLUME-2064 <https://issues.apache.org/jira/browse/FLUME-2064>`__] - Typo/Grammar in flume main user doc under Scribe
+ * [`FLUME-2065 <https://issues.apache.org/jira/browse/FLUME-2065>`__] - Regex Extractor Interceptor config agent name inconsistent with rest of docs
+ * [`FLUME-2183 <https://issues.apache.org/jira/browse/FLUME-2183>`__] - Add "Other Resources" page to wiki
+ * [`FLUME-2278 <https://issues.apache.org/jira/browse/FLUME-2278>`__] - Incorrect documentation for write-timeout of File Channel
+ * [`FLUME-2319 <https://issues.apache.org/jira/browse/FLUME-2319>`__] - Incorrect property name of Ganglia Reporting documentation
+
+
diff --git a/source/sphinx/releases/1.5.2.rst b/source/sphinx/releases/1.5.2.rst
new file mode 100644
index 00000000..69b2f6f5
--- /dev/null
+++ b/source/sphinx/releases/1.5.2.rst
@@ -0,0 +1,30 @@
+===============
+Version 1.5.2
+===============
+
+.. rubric:: Status of this release
+
+Apache Flume 1.5.2 is the eighth release of Flume as an Apache top-level project
+(TLP). Apache Flume 1.5.2 is production-ready software.
+
+Apache Flume 1.5.2 is a maintenance and security release that disables SSLv3 for all security-enabled Flume
+sources and sinks.
+
+.. rubric:: Release Documentation
+
+* `Flume 1.5.2 User Guide <content/1.5.2/FlumeUserGuide.html>`__ (also in `pdf <content/1.5.2/FlumeUserGuide.pdf>`__)
+* `Flume 1.5.2 Developer Guide <content/1.5.2/FlumeDeveloperGuide.html>`__ (also in `pdf <content/1.5.2/FlumeDeveloperGuide.pdf>`__)
+* `Flume 1.5.2 API Documentation <content/1.5.2/apidocs/index.html>`__
+
+.. rubric:: Changes
+
+Release Notes - Flume - Version v1.5.2
+
+ * [`FLUME-2547 <https://issues.apache.org/jira/browse/FLUME-2547>`__] - Removing SSLv2Hello causes Java 6 clients to break
+ * [`FLUME-2548 <https://issues.apache.org/jira/browse/FLUME-2548>`__] - Enable SSLv2Hello for Avro Source and NettyAvroRpcClient
+ * [`FLUME-2549 <https://issues.apache.org/jira/browse/FLUME-2549>`__] - Enable SSLv2Hello for HttpSource
+ * [`FLUME-2551 <https://issues.apache.org/jira/browse/FLUME-2551>`__] - Add dev-support to src tarball
+ * [`FLUME-2441 <https://issues.apache.org/jira/browse/FLUME-2441>`__] - Unit test TestHTTPSource.java failed with IBM JDK 1.7
+ * [`FLUME-2520 <https://issues.apache.org/jira/browse/FLUME-2520>`__] - HTTP Source should be able to block a prefixed set of protocols.
+ * [`FLUME-2533 <https://issues.apache.org/jira/browse/FLUME-2533>`__] - HTTPS tests fail on Java 6
+ * [`FLUME-2511 <https://issues.apache.org/jira/browse/FLUME-2511>`__] - Allow configuration of enabled protocols in Avro source and Rpc client
diff --git a/source/sphinx/releases/1.6.0.rst b/source/sphinx/releases/1.6.0.rst
new file mode 100644
index 00000000..d3466667
--- /dev/null
+++ b/source/sphinx/releases/1.6.0.rst
@@ -0,0 +1,139 @@
+===============
+Version 1.6.0
+===============
+
+.. rubric:: Status of this release
+
+Apache Flume 1.6.0 is the ninth release of Flume as an Apache top-level project
+(TLP). Apache Flume 1.6.0 is production-ready software.
+
+.. rubric:: Release Documentation
+
+* `Flume 1.6.0 User Guide <content/1.6.0/FlumeUserGuide.html>`__ (also in `pdf <content/1.6.0/FlumeUserGuide.pdf>`__)
+* `Flume 1.6.0 Developer Guide <content/1.6.0/FlumeDeveloperGuide.html>`__ (also in `pdf <content/1.6.0/FlumeDeveloperGuide.pdf>`__)
+* `Flume 1.6.0 API Documentation <content/1.6.0/apidocs/index.html>`__
+
+.. rubric:: Changes
+
+Release Notes - Flume - Version v1.6.0
+
+** Sub-task
+ * [`FLUME-2250 <https://issues.apache.org/jira/browse/FLUME-2250>`__] - Add support for Kafka Source
+ * [`FLUME-2251 <https://issues.apache.org/jira/browse/FLUME-2251>`__] - Add support for Kafka Sink
+ * [`FLUME-2677 <https://issues.apache.org/jira/browse/FLUME-2677>`__] - Update versions in 1.6.0 branch
+ * [`FLUME-2686 <https://issues.apache.org/jira/browse/FLUME-2686>`__] - Update KEYS file for 1.6 release
+
+** Bug
+ * [`FLUME-1793 <https://issues.apache.org/jira/browse/FLUME-1793>`__] - Unit test TestElasticSearchLogStashEventSerializer fails with IBM JDK
+ * [`FLUME-1934 <https://issues.apache.org/jira/browse/FLUME-1934>`__] - Spoolingdir source exception when reading multiple zero size files
+ * [`FLUME-2024 <https://issues.apache.org/jira/browse/FLUME-2024>`__] - Add ExecSource flush timeout to Flume User Guide
+ * [`FLUME-2126 <https://issues.apache.org/jira/browse/FLUME-2126>`__] - Problem in elasticsearch sink when the event body is a complex field
+ * [`FLUME-2148 <https://issues.apache.org/jira/browse/FLUME-2148>`__] - Windows : Add flume-env.ps1
+ * [`FLUME-2214 <https://issues.apache.org/jira/browse/FLUME-2214>`__] - FileChannelIntegrityTool needs documentation in user guide
+ * [`FLUME-2245 <https://issues.apache.org/jira/browse/FLUME-2245>`__] - HDFS files with errors unable to close
+ * [`FLUME-2337 <https://issues.apache.org/jira/browse/FLUME-2337>`__] - export JAVA_HOME in flume-env.sh.template and increase heap size
+ * [`FLUME-2346 <https://issues.apache.org/jira/browse/FLUME-2346>`__] - idLogFileMap in Log can lose track of file ids
+ * [`FLUME-2404 <https://issues.apache.org/jira/browse/FLUME-2404>`__] - Default maxReadBufferBytes might cause OOM and cause scribe source exit
+ * [`FLUME-2408 <https://issues.apache.org/jira/browse/FLUME-2408>`__] - Remove FLUME-1899 from Flume 1.5.0 change log.
+ * [`FLUME-2416 <https://issues.apache.org/jira/browse/FLUME-2416>`__] - Use CodecPool in compressed stream to prevent leak of direct buffers
+ * [`FLUME-2420 <https://issues.apache.org/jira/browse/FLUME-2420>`__] - HDFSEventSink#stop might throw ConcurrentModificationException
+ * [`FLUME-2425 <https://issues.apache.org/jira/browse/FLUME-2425>`__] - FileChannel should trim data and checkpoint directories
+ * [`FLUME-2432 <https://issues.apache.org/jira/browse/FLUME-2432>`__] - Update Kite dependency to 0.15.0
+ * [`FLUME-2438 <https://issues.apache.org/jira/browse/FLUME-2438>`__] - Make Syslog source message body configurable
+ * [`FLUME-2450 <https://issues.apache.org/jira/browse/FLUME-2450>`__] - Improve replay index insertion speed.
+ * [`FLUME-2466 <https://issues.apache.org/jira/browse/FLUME-2466>`__] - Embedded agent name cannot contain spaces
+ * [`FLUME-2470 <https://issues.apache.org/jira/browse/FLUME-2470>`__] - Kafka Sink and Source must use camel case for all configs.
+ * [`FLUME-2472 <https://issues.apache.org/jira/browse/FLUME-2472>`__] - SyslogUtils fails when system locale is not English
+ * [`FLUME-2475 <https://issues.apache.org/jira/browse/FLUME-2475>`__] - toLowerCase/toUpperCase used without Locale parameter to manipulate Enum
+ * [`FLUME-2479 <https://issues.apache.org/jira/browse/FLUME-2479>`__] - Kafka property auto.commit.enable is incorrect for KafkaSource
+ * [`FLUME-2480 <https://issues.apache.org/jira/browse/FLUME-2480>`__] - Typo in file channel exception: "Usable space exhaused"
+ * [`FLUME-2481 <https://issues.apache.org/jira/browse/FLUME-2481>`__] - TestFileChannelEncryption fails with System locale other than en_US
+ * [`FLUME-2482 <https://issues.apache.org/jira/browse/FLUME-2482>`__] - Race condition in File Channels' Log.removeOldLogs
+ * [`FLUME-2486 <https://issues.apache.org/jira/browse/FLUME-2486>`__] - TestExecSource fails on some environments
+ * [`FLUME-2487 <https://issues.apache.org/jira/browse/FLUME-2487>`__] - SyslogParser has rounding errors in timestamp parsing
+ * [`FLUME-2488 <https://issues.apache.org/jira/browse/FLUME-2488>`__] - TestElasticSearchRestClient fails on Oracle JDK 8
+ * [`FLUME-2489 <https://issues.apache.org/jira/browse/FLUME-2489>`__] - Upgrade Apache Curator to 2.6.0
+ * [`FLUME-2492 <https://issues.apache.org/jira/browse/FLUME-2492>`__] - Flume's Kafka Source doesn't account time correctly
+ * [`FLUME-2495 <https://issues.apache.org/jira/browse/FLUME-2495>`__] - Kafka Source may miss events when channel is not available
+ * [`FLUME-2497 <https://issues.apache.org/jira/browse/FLUME-2497>`__] - TCP and UDP syslog sources parsing the timestamp incorrectly
+ * [`FLUME-2500 <https://issues.apache.org/jira/browse/FLUME-2500>`__] - Add a channel that uses Kafka
+ * [`FLUME-2505 <https://issues.apache.org/jira/browse/FLUME-2505>`__] - Test added in FLUME-2502 is flaky
+ * [`FLUME-2517 <https://issues.apache.org/jira/browse/FLUME-2517>`__] - Performance issue: SimpleDateFormat constructor takes 30% of HDFSEventSink.process()
+ * [`FLUME-2525 <https://issues.apache.org/jira/browse/FLUME-2525>`__] - flume should handle a zero byte .flumespool-main.meta file for the spooldir source
+ * [`FLUME-2538 <https://issues.apache.org/jira/browse/FLUME-2538>`__] - TestResettableFileInputStream fails on JDK 8
+ * [`FLUME-2556 <https://issues.apache.org/jira/browse/FLUME-2556>`__] - TestBucketPath.testDateFormatHours fails intermittently
+ * [`FLUME-2557 <https://issues.apache.org/jira/browse/FLUME-2557>`__] - DatasetSink doesn't work with Kerberos
+ * [`FLUME-2560 <https://issues.apache.org/jira/browse/FLUME-2560>`__] - Kafka channel entries missing for output jar generation
+ * [`FLUME-2568 <https://issues.apache.org/jira/browse/FLUME-2568>`__] - Additional fix for TestReliableSpoolingFileEventReader
+ * [`FLUME-2578 <https://issues.apache.org/jira/browse/FLUME-2578>`__] - Kafka source throws NPE if Kafka record has null key
+ * [`FLUME-2586 <https://issues.apache.org/jira/browse/FLUME-2586>`__] - HDFS Sink should have an option to try rename even if close fails
+ * [`FLUME-2592 <https://issues.apache.org/jira/browse/FLUME-2592>`__] - Specify main manifest attribute in flume tools jar
+ * [`FLUME-2594 <https://issues.apache.org/jira/browse/FLUME-2594>`__] - Close Async HBase Client if there are large number of consecutive timeouts
+ * [`FLUME-2595 <https://issues.apache.org/jira/browse/FLUME-2595>`__] - Add option to checkpoint on file channel shutdown
+ * [`FLUME-2624 <https://issues.apache.org/jira/browse/FLUME-2624>`__] - Improve Hive Sink performance
+ * [`FLUME-2626 <https://issues.apache.org/jira/browse/FLUME-2626>`__] - Remove trustmanager-type from Thrift RPC client
+ * [`FLUME-2633 <https://issues.apache.org/jira/browse/FLUME-2633>`__] - Update Kite dependency to 1.0.0
+ * [`FLUME-2639 <https://issues.apache.org/jira/browse/FLUME-2639>`__] - SecureThriftRpcClient client privileged calls throw IllegalAccessError
+ * [`FLUME-2645 <https://issues.apache.org/jira/browse/FLUME-2645>`__] - ipFilter.rules property name is wrong
+ * [`FLUME-2654 <https://issues.apache.org/jira/browse/FLUME-2654>`__] - Memory channel error message related to byteCapacity needs refining
+ * [`FLUME-2657 <https://issues.apache.org/jira/browse/FLUME-2657>`__] - Upgrade to Hive 1.0
+ * [`FLUME-2658 <https://issues.apache.org/jira/browse/FLUME-2658>`__] - Upgrade to Hbase 1.0
+ * [`FLUME-2664 <https://issues.apache.org/jira/browse/FLUME-2664>`__] - Site profile fails due to dependency issues
+ * [`FLUME-2665 <https://issues.apache.org/jira/browse/FLUME-2665>`__] - Update documentation for hdfs.closeTries based on FLUME-2586
+ * [`FLUME-2670 <https://issues.apache.org/jira/browse/FLUME-2670>`__] - Modify dependencies to not pull hadoop/hive related jars
+ * [`FLUME-2673 <https://issues.apache.org/jira/browse/FLUME-2673>`__] - Remove unused import in TestLog
+ * [`FLUME-2678 <https://issues.apache.org/jira/browse/FLUME-2678>`__] - Upgrade xalan to 2.7.2 to take care of CVE-2014-0107 vulnerability
+ * [`FLUME-2679 <https://issues.apache.org/jira/browse/FLUME-2679>`__] - Make hbase-1 the default profile
+
+** Dependency upgrade
+ * [`FLUME-2443 <https://issues.apache.org/jira/browse/FLUME-2443>`__] - org.apache.hadoop.fs.FSDataOutputStream.sync() is deprecated in hadoop 2.4
+
+** Documentation
+ * [`FLUME-1594 <https://issues.apache.org/jira/browse/FLUME-1594>`__] - Document JMX options
+ * [`FLUME-2509 <https://issues.apache.org/jira/browse/FLUME-2509>`__] - Add description for properties in Failover Sink Processor
+ * [`FLUME-2630 <https://issues.apache.org/jira/browse/FLUME-2630>`__] - Update documentation for Thrift Src/Sink SSL support and Kerberos support
+ * [`FLUME-2635 <https://issues.apache.org/jira/browse/FLUME-2635>`__] - Documentation for zookeeper dynamic configuration in flume
+ * [`FLUME-2668 <https://issues.apache.org/jira/browse/FLUME-2668>`__] - Document SecureThriftRpcClient/SecureRpcClientFactory in Flume Developer Guide
+
+** Improvement
+ * [`FLUME-1334 <https://issues.apache.org/jira/browse/FLUME-1334>`__] - Write an startscript for flume agents on Windows
+ * [`FLUME-1491 <https://issues.apache.org/jira/browse/FLUME-1491>`__] - Dynamic configuration from Zookeeper watcher
+ * [`FLUME-1521 <https://issues.apache.org/jira/browse/FLUME-1521>`__] - Document the StressSource
+ * [`FLUME-1710 <https://issues.apache.org/jira/browse/FLUME-1710>`__] - JSONEvent.getBody should not return null
+ * [`FLUME-1920 <https://issues.apache.org/jira/browse/FLUME-1920>`__] - Test case TestFileChannel fails when flume is built from paths containing the string "hadoop"
+ * [`FLUME-2237 <https://issues.apache.org/jira/browse/FLUME-2237>`__] - Example config in User Guide to use same notation for Agent name
+ * [`FLUME-2246 <https://issues.apache.org/jira/browse/FLUME-2246>`__] - event body data size can make it configurable for logger sinker
+ * [`FLUME-2273 <https://issues.apache.org/jira/browse/FLUME-2273>`__] - ElasticSearchSink: Add handling for header substitution in indexName
+ * [`FLUME-2373 <https://issues.apache.org/jira/browse/FLUME-2373>`__] - Enable configuration to switch thrift source serialization protocol
+ * [`FLUME-2385 <https://issues.apache.org/jira/browse/FLUME-2385>`__] - Flume spans log file with "Spooling Directory Source runner has shutdown" messages at INFO level
+ * [`FLUME-2401 <https://issues.apache.org/jira/browse/FLUME-2401>`__] - Optionally Compress Backup Checkpoint
+ * [`FLUME-2439 <https://issues.apache.org/jira/browse/FLUME-2439>`__] - Update DatasetSink for Kite 0.15.0
+ * [`FLUME-2462 <https://issues.apache.org/jira/browse/FLUME-2462>`__] - Remove use of deprecated methods in DatasetSink
+ * [`FLUME-2463 <https://issues.apache.org/jira/browse/FLUME-2463>`__] - Add support for Hive and HBase datasets to DatasetSink
+ * [`FLUME-2469 <https://issues.apache.org/jira/browse/FLUME-2469>`__] - DatasetSink should load dataset when needed, not at startup
+ * [`FLUME-2499 <https://issues.apache.org/jira/browse/FLUME-2499>`__] - Include Kafka Message Key in Event Header, Updated Comments
+ * [`FLUME-2502 <https://issues.apache.org/jira/browse/FLUME-2502>`__] - Spool source's directory listing is inefficient
+ * [`FLUME-2558 <https://issues.apache.org/jira/browse/FLUME-2558>`__] - Update javadoc for StressSource
+ * [`FLUME-2562 <https://issues.apache.org/jira/browse/FLUME-2562>`__] - Metrics for Flafka components
+ * [`FLUME-2591 <https://issues.apache.org/jira/browse/FLUME-2591>`__] - Kite DatasetSink 2.0
+ * [`FLUME-2613 <https://issues.apache.org/jira/browse/FLUME-2613>`__] - Tool/script for deleting individual message from queue
+ * [`FLUME-2642 <https://issues.apache.org/jira/browse/FLUME-2642>`__] - Limit kerberos relogin attempt
+
+** New Feature
+ * [`FLUME-1734 <https://issues.apache.org/jira/browse/FLUME-1734>`__] - Create a Hive Sink based on the new Hive Streaming support
+ * [`FLUME-2242 <https://issues.apache.org/jira/browse/FLUME-2242>`__] - Flume Sink and Source for Apache Kafka
+ * [`FLUME-2426 <https://issues.apache.org/jira/browse/FLUME-2426>`__] - Support interceptors in the Embedded Agent
+ * [`FLUME-2431 <https://issues.apache.org/jira/browse/FLUME-2431>`__] - Add simple regex search-and-replace interceptor
+ * [`FLUME-2570 <https://issues.apache.org/jira/browse/FLUME-2570>`__] - Add option to not pad date fields
+ * [`FLUME-2574 <https://issues.apache.org/jira/browse/FLUME-2574>`__] - SSL Support for Thrift Rpc
+ * [`FLUME-2631 <https://issues.apache.org/jira/browse/FLUME-2631>`__] - End to End authentication in Flume
+
+** Task
+ * [`FLUME-2365 <https://issues.apache.org/jira/browse/FLUME-2365>`__] - Please create a DOAP file for your TLP
+ * [`FLUME-2454 <https://issues.apache.org/jira/browse/FLUME-2454>`__] - Support batchSize to allow multiple events per transaction to the Kafka Sink
+ * [`FLUME-2455 <https://issues.apache.org/jira/browse/FLUME-2455>`__] - Documentation update for Kafka Sink
+ * [`FLUME-2523 <https://issues.apache.org/jira/browse/FLUME-2523>`__] - Document Kafka channel
+ * [`FLUME-2612 <https://issues.apache.org/jira/browse/FLUME-2612>`__] - Update kite to 0.17.1
+
+** Test
+ * [`FLUME-1501 <https://issues.apache.org/jira/browse/FLUME-1501>`__] - Flume Scribe Source needs unit tests.
diff --git a/source/sphinx/releases/1.7.0.rst b/source/sphinx/releases/1.7.0.rst
new file mode 100644
index 00000000..fc570f3d
--- /dev/null
+++ b/source/sphinx/releases/1.7.0.rst
@@ -0,0 +1,131 @@
+===============
+Version 1.7.0
+===============
+
+.. rubric:: Status of this release
+
+Apache Flume 1.7.0 is the tenth release of Flume as an Apache top-level project
+(TLP). Apache Flume 1.7.0 is production-ready software.
+
+.. rubric:: Release Documentation
+
+* `Flume 1.7.0 User Guide <content/1.7.0/FlumeUserGuide.html>`__ (also in `pdf <content/1.7.0/FlumeUserGuide.pdf>`__)
+* `Flume 1.7.0 Developer Guide <content/1.7.0/FlumeDeveloperGuide.html>`__ (also in `pdf <content/1.7.0/FlumeDeveloperGuide.pdf>`__)
+* `Flume 1.7.0 API Documentation <content/1.7.0/apidocs/index.html>`__
+
+.. rubric:: Changes
+
+Release Notes - Flume - Version v1.7.0
+
+** New Feature
+ * [`FLUME-2498 <https://issues.apache.org/jira/browse/FLUME-2498>`__] - Implement Taildir Source
+
+** Improvement
+ * [`FLUME-1899 <https://issues.apache.org/jira/browse/FLUME-1899>`__] - Make SpoolDir work with Sub-Directories
+ * [`FLUME-2526 <https://issues.apache.org/jira/browse/FLUME-2526>`__] - Build flume by jdk 7 in default
+ * [`FLUME-2628 <https://issues.apache.org/jira/browse/FLUME-2628>`__] - Add an optional parameter to specify the expected input text encoding for the netcat sourcef the netcat source
+ * [`FLUME-2704 <https://issues.apache.org/jira/browse/FLUME-2704>`__] - Configurable poll delay for spooling directory source
+ * [`FLUME-2718 <https://issues.apache.org/jira/browse/FLUME-2718>`__] - HTTP Source to support generic Stream Handler
+ * [`FLUME-2729 <https://issues.apache.org/jira/browse/FLUME-2729>`__] - Allow pollableSource backoff times to be configurable
+ * [`FLUME-2755 <https://issues.apache.org/jira/browse/FLUME-2755>`__] - Kafka Source reading multiple topics
+ * [`FLUME-2781 <https://issues.apache.org/jira/browse/FLUME-2781>`__] - A Kafka Channel defined as parseAsFlumeEvent=false cannot be correctly used by a Flume source
+ * [`FLUME-2799 <https://issues.apache.org/jira/browse/FLUME-2799>`__] - Kafka Source - Message Offset and Partition add to headers
+ * [`FLUME-2801 <https://issues.apache.org/jira/browse/FLUME-2801>`__] - Performance improvement on TailDir source
+ * [`FLUME-2810 <https://issues.apache.org/jira/browse/FLUME-2810>`__] - Add static Schema URL to AvroEventSerializer configuration
+ * [`FLUME-2820 <https://issues.apache.org/jira/browse/FLUME-2820>`__] - Support New Kafka APIs
+ * [`FLUME-2852 <https://issues.apache.org/jira/browse/FLUME-2852>`__] - Kafka Source/Sink should optionally read/write Flume records
+ * [`FLUME-2868 <https://issues.apache.org/jira/browse/FLUME-2868>`__] - Kafka Channel partition topic by key
+ * [`FLUME-2872 <https://issues.apache.org/jira/browse/FLUME-2872>`__] - Kafka Sink should be able to select which header as the key
+ * [`FLUME-2875 <https://issues.apache.org/jira/browse/FLUME-2875>`__] - Allow RollingFileSink to specify a file prefix and a file extension.
+ * [`FLUME-2909 <https://issues.apache.org/jira/browse/FLUME-2909>`__] - Bump Rat version
+ * [`FLUME-2910 <https://issues.apache.org/jira/browse/FLUME-2910>`__] - AsyncHBaseSink - Failure callbacks should log the exception that caused them
+ * [`FLUME-2911 <https://issues.apache.org/jira/browse/FLUME-2911>`__] - Add includePattern option in SpoolDirectorySource configuration
+ * [`FLUME-2918 <https://issues.apache.org/jira/browse/FLUME-2918>`__] - TaildirSource is underperforming with huge parent directories
+ * [`FLUME-2937 <https://issues.apache.org/jira/browse/FLUME-2937>`__] - Integrate checkstyle for non-test classes
+ * [`FLUME-2941 <https://issues.apache.org/jira/browse/FLUME-2941>`__] - Integrate checkstyle for test classes
+ * [`FLUME-2954 <https://issues.apache.org/jira/browse/FLUME-2954>`__] - make raw data appearing in log messages explicit
+ * [`FLUME-2955 <https://issues.apache.org/jira/browse/FLUME-2955>`__] - Add file path to the header in TaildirSource
+ * [`FLUME-2959 <https://issues.apache.org/jira/browse/FLUME-2959>`__] - Fix issues with flume-checkstyle module
+ * [`FLUME-2982 <https://issues.apache.org/jira/browse/FLUME-2982>`__] - Add localhost escape sequence to HDFS sink
+ * [`FLUME-2999 <https://issues.apache.org/jira/browse/FLUME-2999>`__] - Kafka channel and sink should enable statically assigned partition per event via header
+ * [`FLUME-2821 <https://issues.apache.org/jira/browse/FLUME-2821>`__] - Flume-Kafka Source with new Consumer
+ * [`FLUME-2822 <https://issues.apache.org/jira/browse/FLUME-2822>`__] - Flume-Kafka-Sink with new Producer
+ * [`FLUME-2823 <https://issues.apache.org/jira/browse/FLUME-2823>`__] - Flume-Kafka-Channel with new APIs
+
+** Bug
+ * [`FLUME-1668 <https://issues.apache.org/jira/browse/FLUME-1668>`__] - Hdfs Sink File Rollover
+ * [`FLUME-2132 <https://issues.apache.org/jira/browse/FLUME-2132>`__] - Exception while syncing from Flume to HDFS
+ * [`FLUME-2143 <https://issues.apache.org/jira/browse/FLUME-2143>`__] - Flume build occasionally fails with OutOfMemoryError on Windows.
+ * [`FLUME-2215 <https://issues.apache.org/jira/browse/FLUME-2215>`__] - ResettableFileInputStream can't support ucs-4 character
+ * [`FLUME-2318 <https://issues.apache.org/jira/browse/FLUME-2318>`__] - SpoolingDirectory is unable to handle empty files
+ * [`FLUME-2448 <https://issues.apache.org/jira/browse/FLUME-2448>`__] - Building flume from trunk failing with dependency error
+ * [`FLUME-2484 <https://issues.apache.org/jira/browse/FLUME-2484>`__] - NullPointerException in Kafka Sink test
+ * [`FLUME-2485 <https://issues.apache.org/jira/browse/FLUME-2485>`__] - Thrift Source tests fail on Oracle JDK 8
+ * [`FLUME-2514 <https://issues.apache.org/jira/browse/FLUME-2514>`__] - Some TestFileChannelRestart tests are extremely slow
+ * [`FLUME-2567 <https://issues.apache.org/jira/browse/FLUME-2567>`__] - Remove unneeded repository declarations in pom.xml
+ * [`FLUME-2573 <https://issues.apache.org/jira/browse/FLUME-2573>`__] - flume-ng --conf parameter is not used when starting a flume agent
+ * [`FLUME-2593 <https://issues.apache.org/jira/browse/FLUME-2593>`__] - ResettableFileInputStream returns negate values from read() method
+ * [`FLUME-2619 <https://issues.apache.org/jira/browse/FLUME-2619>`__] - Spooldir source does not log channel exceptions
+ * [`FLUME-2632 <https://issues.apache.org/jira/browse/FLUME-2632>`__] - High CPU on KafkaSink
+ * [`FLUME-2652 <https://issues.apache.org/jira/browse/FLUME-2652>`__] - Documented transaction handling semantics incorrect
+ * [`FLUME-2660 <https://issues.apache.org/jira/browse/FLUME-2660>`__] - Add documentation for EventValidator
+ * [`FLUME-2672 <https://issues.apache.org/jira/browse/FLUME-2672>`__] - NPE in KafkaSourceCounter
+ * [`FLUME-2712 <https://issues.apache.org/jira/browse/FLUME-2712>`__] - Optional channel errors slows down the Source to Main channel event rate
+ * [`FLUME-2725 <https://issues.apache.org/jira/browse/FLUME-2725>`__] - HDFS Sink does not use configured timezone for rounding
+ * [`FLUME-2732 <https://issues.apache.org/jira/browse/FLUME-2732>`__] - Make maximum tolerated failures before shutting down and recreating client in AsyncHbaseSink configurable
+ * [`FLUME-2734 <https://issues.apache.org/jira/browse/FLUME-2734>`__] - Kafka Channel timeout property is overridden by default value
+ * [`FLUME-2738 <https://issues.apache.org/jira/browse/FLUME-2738>`__] - Async HBase sink FD leak on client shutdown
+ * [`FLUME-2746 <https://issues.apache.org/jira/browse/FLUME-2746>`__] - How to include this Flume Patch in Flume 1.5.2 ?
+ * [`FLUME-2749 <https://issues.apache.org/jira/browse/FLUME-2749>`__] - Kerberos configuration error when using short names in multiple HDFS Sinks
+ * [`FLUME-2751 <https://issues.apache.org/jira/browse/FLUME-2751>`__] - Upgrade Derby version to 10.11.1.1
+ * [`FLUME-2753 <https://issues.apache.org/jira/browse/FLUME-2753>`__] - Error when specifying empty replace string in Search and Replace Interceptor
+ * [`FLUME-2754 <https://issues.apache.org/jira/browse/FLUME-2754>`__] - Hive Sink skipping first transaction in each Batch of Hive Transactions
+ * [`FLUME-2761 <https://issues.apache.org/jira/browse/FLUME-2761>`__] - Move Hive sink out of preview mode
+ * [`FLUME-2763 <https://issues.apache.org/jira/browse/FLUME-2763>`__] - flume_env script should handle jvm parameters like -javaagent -agentpath -agentlib
+ * [`FLUME-2773 <https://issues.apache.org/jira/browse/FLUME-2773>`__] - TailDirSource throws FileNotFound Exception if ~/.flume directory is not created already
+ * [`FLUME-2797 <https://issues.apache.org/jira/browse/FLUME-2797>`__] - SyslogTcpSource uses Deprecated Class + Deprecate SyslogTcpSource
+ * [`FLUME-2798 <https://issues.apache.org/jira/browse/FLUME-2798>`__] - Malformed Syslog messages can lead to OutOfMemoryException
+ * [`FLUME-2804 <https://issues.apache.org/jira/browse/FLUME-2804>`__] - Hive sink - abort remaining transactions on shutdown
+ * [`FLUME-2806 <https://issues.apache.org/jira/browse/FLUME-2806>`__] - flume-ng.ps1 Error running script to start an agent on Windows
+ * [`FLUME-2835 <https://issues.apache.org/jira/browse/FLUME-2835>`__] - Hive Sink tests need to create table with transactional property set
+ * [`FLUME-2841 <https://issues.apache.org/jira/browse/FLUME-2841>`__] - Upgrade commons-collections to 3.2.2
+ * [`FLUME-2844 <https://issues.apache.org/jira/browse/FLUME-2844>`__] - ChannelCounter of SpillableMemoryChannel doesn't register actually.
+ * [`FLUME-2881 <https://issues.apache.org/jira/browse/FLUME-2881>`__] - Windows Launch Script fails in plugins dir code
+ * [`FLUME-2886 <https://issues.apache.org/jira/browse/FLUME-2886>`__] - Optional Channels can cause OOMs
+ * [`FLUME-2889 <https://issues.apache.org/jira/browse/FLUME-2889>`__] - Fixes to DateTime computations
+ * [`FLUME-2891 <https://issues.apache.org/jira/browse/FLUME-2891>`__] - Revert FLUME-2712 and FLUME-2886
+ * [`FLUME-2897 <https://issues.apache.org/jira/browse/FLUME-2897>`__] - AsyncHBase sink NPE when Channel.getTransaction() fails
+ * [`FLUME-2901 <https://issues.apache.org/jira/browse/FLUME-2901>`__] - Document Kerberos setup for Kafka channel
+ * [`FLUME-2908 <https://issues.apache.org/jira/browse/FLUME-2908>`__] - NetcatSource - SocketChannel not closed when session is broken
+ * [`FLUME-2913 <https://issues.apache.org/jira/browse/FLUME-2913>`__] - Flume classpath too long
+ * [`FLUME-2915 <https://issues.apache.org/jira/browse/FLUME-2915>`__] - The kafka channel using new APIs will be stuck when the sink is avro sink
+ * [`FLUME-2920 <https://issues.apache.org/jira/browse/FLUME-2920>`__] - Kafka Channel Should Not Commit Offsets When Stopping
+ * [`FLUME-2922 <https://issues.apache.org/jira/browse/FLUME-2922>`__] - HDFSSequenceFile Should Sync Writer
+ * [`FLUME-2923 <https://issues.apache.org/jira/browse/FLUME-2923>`__] - Bump AsyncHBase version
+ * [`FLUME-2936 <https://issues.apache.org/jira/browse/FLUME-2936>`__] - KafkaSource tests arbitrarily fail
+ * [`FLUME-2939 <https://issues.apache.org/jira/browse/FLUME-2939>`__] - Upgrade recursive SpoolDir to use Java7 features
+ * [`FLUME-2948 <https://issues.apache.org/jira/browse/FLUME-2948>`__] - Docs: Fixed parameters on Replicating Channel Selector documentation example
+ * [`FLUME-2949 <https://issues.apache.org/jira/browse/FLUME-2949>`__] - Flume fails to build on Windows
+ * [`FLUME-2950 <https://issues.apache.org/jira/browse/FLUME-2950>`__] - ReliableSpoolingFileEventReader.rollCurrentFile is broken
+ * [`FLUME-2952 <https://issues.apache.org/jira/browse/FLUME-2952>`__] - SyslogAgent possible NPE on stop()
+ * [`FLUME-2972 <https://issues.apache.org/jira/browse/FLUME-2972>`__] - Handle offset migration in the new Kafka Channel
+ * [`FLUME-2974 <https://issues.apache.org/jira/browse/FLUME-2974>`__] - Some tests are broken in TestReliableSpoolingFileEventReader and TestSpoolingFileLineReader
+ * [`FLUME-2983 <https://issues.apache.org/jira/browse/FLUME-2983>`__] - Handle offset migration in the new Kafka Source
+
+** Documentation
+ * [`FLUME-2575 <https://issues.apache.org/jira/browse/FLUME-2575>`__] - FLUME-2548 brings SSLv2Hello back for Avro Sink, but UG says it is one of the protocols to exclude
+ * [`FLUME-2713 <https://issues.apache.org/jira/browse/FLUME-2713>`__] - Document Fault Tolerant Config parameters in FlumeUserGuide
+ * [`FLUME-2737 <https://issues.apache.org/jira/browse/FLUME-2737>`__] - Documentation for Pollable Source config parameters introduced in FLUME-2729
+ * [`FLUME-2783 <https://issues.apache.org/jira/browse/FLUME-2783>`__] - Update Website Team page with new Committer's
+ * [`FLUME-2890 <https://issues.apache.org/jira/browse/FLUME-2890>`__] - Typo in Twitter source warning
+ * [`FLUME-2934 <https://issues.apache.org/jira/browse/FLUME-2934>`__] - Document new cachePatternMatching option for TaildirSource
+ * [`FLUME-2963 <https://issues.apache.org/jira/browse/FLUME-2963>`__] - FlumeUserGuide - error in Kafka Source properties table
+ * [`FLUME-2971 <https://issues.apache.org/jira/browse/FLUME-2971>`__] - Document secure Kafka Sink/Source/Channel setup
+ * [`FLUME-2975 <https://issues.apache.org/jira/browse/FLUME-2975>`__] - Minor mistake in NetCat Source example in documentation
+ * [`FLUME-2998 <https://issues.apache.org/jira/browse/FLUME-2998>`__] - Add missing configuration parameter to SequenceSource docs
+
+** Task
+ * [`FLUME-2935 <https://issues.apache.org/jira/browse/FLUME-2935>`__] - Bump java target version to 1.7
+
+** Test
+ * [`FLUME-3003 <https://issues.apache.org/jira/browse/FLUME-3003>`__] - testSourceCounter in TestSyslogUdpSource is flaky
diff --git a/source/sphinx/releases/1.8.0.rst b/source/sphinx/releases/1.8.0.rst
new file mode 100644
index 00000000..4a942677
--- /dev/null
+++ b/source/sphinx/releases/1.8.0.rst
@@ -0,0 +1,80 @@
+===============
+Version 1.8.0
+===============
+
+.. rubric:: Status of this release
+
+Apache Flume 1.8.0 is the eleventh release of Flume as an Apache top-level project
+(TLP). Apache Flume 1.8.0 is production-ready software.
+
+.. rubric:: Release Documentation
+
+* `Flume 1.8.0 User Guide <content/1.8.0/FlumeUserGuide.html>`__ (also in `pdf <content/1.8.0/FlumeUserGuide.pdf>`__)
+* `Flume 1.8.0 Developer Guide <content/1.8.0/FlumeDeveloperGuide.html>`__ (also in `pdf <content/1.8.0/FlumeDeveloperGuide.pdf>`__)
+* `Flume 1.8.0 API Documentation <content/1.8.0/apidocs/index.html>`__
+
+.. rubric:: Changes
+
+Release Notes - Flume - Version v1.8.0
+
+** New Feature
+ * [`FLUME-2171 <https://issues.apache.org/jira/browse/FLUME-2171>`__] - Add Interceptor to remove headers from event
+ * [`FLUME-2917 <https://issues.apache.org/jira/browse/FLUME-2917>`__] - Provide netcat UDP source as alternative to TCP
+ * [`FLUME-2993 <https://issues.apache.org/jira/browse/FLUME-2993>`__] - Support environment variables in configuration files
+
+** Improvement
+ * [`FLUME-1520 <https://issues.apache.org/jira/browse/FLUME-1520>`__] - Timestamp interceptor should support custom headers
+ * [`FLUME-2945 <https://issues.apache.org/jira/browse/FLUME-2945>`__] - Bump java target version to 1.8
+ * [`FLUME-3020 <https://issues.apache.org/jira/browse/FLUME-3020>`__] - Improve HDFSEventSink Escape Ingestion by more then 10x by not getting InetAddress on every record
+ * [`FLUME-3025 <https://issues.apache.org/jira/browse/FLUME-3025>`__] - Expose FileChannel.open on JMX
+ * [`FLUME-3072 <https://issues.apache.org/jira/browse/FLUME-3072>`__] - Add IP address to headers in flume log4j appender
+ * [`FLUME-3092 <https://issues.apache.org/jira/browse/FLUME-3092>`__] - Extend the FileChannel's monitoring metrics
+ * [`FLUME-3100 <https://issues.apache.org/jira/browse/FLUME-3100>`__] - Support arbitrary header substitution for topic of Kafka
+ * [`FLUME-3144 <https://issues.apache.org/jira/browse/FLUME-3144>`__] - Improve Log4jAppender's performance by allowing logging collection of messages
+
+** Bug
+ * [`FLUME-2620 <https://issues.apache.org/jira/browse/FLUME-2620>`__] - File channel throws NullPointerException if a header value is null
+ * [`FLUME-2752 <https://issues.apache.org/jira/browse/FLUME-2752>`__] - Flume AvroSource will leak the memory and the OOM will be happened.
+ * [`FLUME-2812 <https://issues.apache.org/jira/browse/FLUME-2812>`__] - Exception in thread "SinkRunner-PollingRunner-DefaultSinkProcessor" java.lang.Error: Maximum permit count exceeded
+ * [`FLUME-2857 <https://issues.apache.org/jira/browse/FLUME-2857>`__] - Kafka Source/Channel/Sink does not restore default values when live update config
+ * [`FLUME-2905 <https://issues.apache.org/jira/browse/FLUME-2905>`__] - NetcatSource - Socket not closed when an exception is encountered during start() leading to file descriptor leaks
+ * [`FLUME-2991 <https://issues.apache.org/jira/browse/FLUME-2991>`__] - ExecSource command execution starts before starting the sourceCounter
+ * [`FLUME-3027 <https://issues.apache.org/jira/browse/FLUME-3027>`__] - Kafka Channel should clear offsets map after commit
+ * [`FLUME-3031 <https://issues.apache.org/jira/browse/FLUME-3031>`__] - sequence source should reset its counter for event body on channel exception
+ * [`FLUME-3043 <https://issues.apache.org/jira/browse/FLUME-3043>`__] - KafkaSink SinkCallback throws NullPointerException when Log4J level is debug
+ * [`FLUME-3046 <https://issues.apache.org/jira/browse/FLUME-3046>`__] - Kafka Sink and Source Configuration Improvements
+ * [`FLUME-3049 <https://issues.apache.org/jira/browse/FLUME-3049>`__] - Wrapping the exception into SecurityException in UGIExecutor.execute hides the original one
+ * [`FLUME-3057 <https://issues.apache.org/jira/browse/FLUME-3057>`__] - Build fails due to unsupported snappy-java version on ppc64le
+ * [`FLUME-3080 <https://issues.apache.org/jira/browse/FLUME-3080>`__] - Close failure in HDFS Sink might cause data loss
+ * [`FLUME-3083 <https://issues.apache.org/jira/browse/FLUME-3083>`__] - Taildir source can miss events if file updated in same second as file close
+ * [`FLUME-3085 <https://issues.apache.org/jira/browse/FLUME-3085>`__] - HDFS Sink can skip flushing some BucketWriters, might lead to data loss
+ * [`FLUME-3112 <https://issues.apache.org/jira/browse/FLUME-3112>`__] - Upgrade jackson-core library dependency
+ * [`FLUME-3127 <https://issues.apache.org/jira/browse/FLUME-3127>`__] - Upgrade libfb303 library dependency
+ * [`FLUME-3131 <https://issues.apache.org/jira/browse/FLUME-3131>`__] - Upgrade spring framework library dependencies
+ * [`FLUME-3132 <https://issues.apache.org/jira/browse/FLUME-3132>`__] - Upgrade tomcat jasper library dependencies
+ * [`FLUME-3135 <https://issues.apache.org/jira/browse/FLUME-3135>`__] - property logger in org.apache.flume.interceptor.RegexFilteringInterceptor confused
+ * [`FLUME-3141 <https://issues.apache.org/jira/browse/FLUME-3141>`__] - Small typo found in RegexHbaseEventSerializer.java
+ * [`FLUME-3152 <https://issues.apache.org/jira/browse/FLUME-3152>`__] - Add Flume Metric for Backup Checkpoint Errors
+ * [`FLUME-3155 <https://issues.apache.org/jira/browse/FLUME-3155>`__] - Use batch mode in mvn to fix Travis CI error
+ * [`FLUME-3157 <https://issues.apache.org/jira/browse/FLUME-3157>`__] - Refactor TestHDFSEventSinkOnMiniCluster to not use LeaseManager private API
+ * [`FLUME-3173 <https://issues.apache.org/jira/browse/FLUME-3173>`__] - Upgrade joda-time
+ * [`FLUME-3174 <https://issues.apache.org/jira/browse/FLUME-3174>`__] - HdfsSink AWS S3A authentication does not work on JDK 8
+ * [`FLUME-3175 <https://issues.apache.org/jira/browse/FLUME-3175>`__] - Javadoc generation fails due to Java8's strict doclint
+
+** Documentation
+ * [`FLUME-2175 <https://issues.apache.org/jira/browse/FLUME-2175>`__] - Update Developer Guide with notes on how to upgrade Protocol Buffer version
+ * [`FLUME-2817 <https://issues.apache.org/jira/browse/FLUME-2817>`__] - Sink for multi-agent flow example in user guide is set up incorrectly
+
+** Wish
+ * [`FLUME-2579 <https://issues.apache.org/jira/browse/FLUME-2579>`__] - JMS source support durable subscriptions and message listening
+
+** Question
+ * [`FLUME-2427 <https://issues.apache.org/jira/browse/FLUME-2427>`__] - java.lang.NoSuchMethodException and warning on HDFS (S3) sink
+
+** Task
+ * [`FLUME-3093 <https://issues.apache.org/jira/browse/FLUME-3093>`__] - Groundwork for version changes in root pom
+ * [`FLUME-3154 <https://issues.apache.org/jira/browse/FLUME-3154>`__] - Add HBase client version check to AsyncHBaseSink and HBaseSink
+
+** Test
+ * [`FLUME-2997 <https://issues.apache.org/jira/browse/FLUME-2997>`__] - Fix flaky junit test in SpillableMemoryChannel
+ * [`FLUME-3002 <https://issues.apache.org/jira/browse/FLUME-3002>`__] - Some tests in TestBucketWriter are flaky
diff --git a/source/sphinx/releases/1.9.0.rst b/source/sphinx/releases/1.9.0.rst
new file mode 100644
index 00000000..d4dc74ff
--- /dev/null
+++ b/source/sphinx/releases/1.9.0.rst
@@ -0,0 +1,97 @@
+===============
+Version 1.9.0
+===============
+
+.. rubric:: Status of this release
+
+Apache Flume 1.9.0 is the eleventh release of Flume as an Apache top-level project
+(TLP). Apache Flume 1.9.0 is production-ready software.
+
+.. rubric:: Release Documentation
+
+* `Flume 1.9.0 User Guide <content/1.9.0/FlumeUserGuide.html>`__ (also in `pdf <content/1.9.0/FlumeUserGuide.pdf>`__)
+* `Flume 1.9.0 Developer Guide <content/1.9.0/FlumeDeveloperGuide.html>`__ (also in `pdf <content/1.9.0/FlumeDeveloperGuide.pdf>`__)
+* `Flume 1.9.0 API Documentation <content/1.9.0/apidocs/index.html>`__
+
+.. rubric:: Changes
+
+Release Notes - Flume - Version v1.9.0
+
+** New Feature
+ * [`FLUME-2071 <https://issues.apache.org/jira/browse/FLUME-2071>`__] - Flume Context doesn't support float or double configuration values.
+ * [`FLUME-2442 <https://issues.apache.org/jira/browse/FLUME-2442>`__] - Need an alternative to providing clear text passwords in flume config
+ * [`FLUME-3142 <https://issues.apache.org/jira/browse/FLUME-3142>`__] - Adding HBase2 sink
+
+** Improvement
+ * [`FLUME-2653 <https://issues.apache.org/jira/browse/FLUME-2653>`__] - Allow inUseSuffix to be null/empty
+ * [`FLUME-2854 <https://issues.apache.org/jira/browse/FLUME-2854>`__] - Parameterize jetty version in pom
+ * [`FLUME-2977 <https://issues.apache.org/jira/browse/FLUME-2977>`__] - Upgrade RAT to 0.12
+ * [`FLUME-3050 <https://issues.apache.org/jira/browse/FLUME-3050>`__] - add counters for error conditions and expose to monitor URL
+ * [`FLUME-3182 <https://issues.apache.org/jira/browse/FLUME-3182>`__] - Please add support for SSL/TLS for syslog (tcp) & multi_port syslog (tcp) sources
+ * [`FLUME-3186 <https://issues.apache.org/jira/browse/FLUME-3186>`__] - Make asyncHbaseClient configuration parameters available from flume config
+ * [`FLUME-3223 <https://issues.apache.org/jira/browse/FLUME-3223>`__] - Flume HDFS Sink should retry close prior to performing a recoverLease
+ * [`FLUME-3227 <https://issues.apache.org/jira/browse/FLUME-3227>`__] - Add Rate Limiter to StressSource
+ * [`FLUME-3239 <https://issues.apache.org/jira/browse/FLUME-3239>`__] - Do not rename files in SpoolDirectorySource
+ * [`FLUME-3246 <https://issues.apache.org/jira/browse/FLUME-3246>`__] - Validate flume configuration to prevent larger source batch size than the channel transaction capacity
+ * [`FLUME-3269 <https://issues.apache.org/jira/browse/FLUME-3269>`__] - Support JSSE keystore/trustore -D system properties
+ * [`FLUME-3275 <https://issues.apache.org/jira/browse/FLUME-3275>`__] - Components supporting SSL/TLS should be able to specify protocol list
+ * [`FLUME-3276 <https://issues.apache.org/jira/browse/FLUME-3276>`__] - Components supporting SSL/TLS should be able to specify cipher suite list
+ * [`FLUME-3280 <https://issues.apache.org/jira/browse/FLUME-3280>`__] - Improve maven build to help code reviews by adding static code analyzer to it
+ * [`FLUME-3281 <https://issues.apache.org/jira/browse/FLUME-3281>`__] - Update to Kafka 2.0 client
+ * [`FLUME-3282 <https://issues.apache.org/jira/browse/FLUME-3282>`__] - Use slf4j in every component
+
+
+** Bug
+ * [`FLUME-1282 <https://issues.apache.org/jira/browse/FLUME-1282>`__] - Flume 1.x build fails on Maven 2
+ * [`FLUME-2232 <https://issues.apache.org/jira/browse/FLUME-2232>`__] - Add findbugs to Flume build
+ * [`FLUME-2436 <https://issues.apache.org/jira/browse/FLUME-2436>`__] - Make hadoop-2 the default build profile
+ * [`FLUME-2464 <https://issues.apache.org/jira/browse/FLUME-2464>`__] - Remove hadoop-2 profile
+ * [`FLUME-2786 <https://issues.apache.org/jira/browse/FLUME-2786>`__] - It will enter a deadlock state when modify the conf file before I stop flume-ng
+ * [`FLUME-2894 <https://issues.apache.org/jira/browse/FLUME-2894>`__] - Flume components should stop in the correct order (graceful shutdown)
+ * [`FLUME-2973 <https://issues.apache.org/jira/browse/FLUME-2973>`__] - Deadlock in hdfs sink
+ * [`FLUME-2976 <https://issues.apache.org/jira/browse/FLUME-2976>`__] - Exception when JMS source tries to connect to a Weblogic server without authentication
+ * [`FLUME-2988 <https://issues.apache.org/jira/browse/FLUME-2988>`__] - Kafka Sink metrics missing eventDrainAttemptCount
+ * [`FLUME-2989 <https://issues.apache.org/jira/browse/FLUME-2989>`__] - Kafka Channel metrics missing eventTakeAttemptCount and eventPutAttemptCount
+ * [`FLUME-3056 <https://issues.apache.org/jira/browse/FLUME-3056>`__] - TestApplication hangs indefinitely
+ * [`FLUME-3101 <https://issues.apache.org/jira/browse/FLUME-3101>`__] - taildir source may endless loop when tail a file
+ * [`FLUME-3106 <https://issues.apache.org/jira/browse/FLUME-3106>`__] - When batchSize of sink greater than transactionCapacity of Memory Channel, Flume can produce endless data
+ * [`FLUME-3107 <https://issues.apache.org/jira/browse/FLUME-3107>`__] - When batchSize of sink greater than transactionCapacity of File Channel, Flume can produce endless data
+ * [`FLUME-3114 <https://issues.apache.org/jira/browse/FLUME-3114>`__] - Upgrade commons-httpclient library dependency
+ * [`FLUME-3117 <https://issues.apache.org/jira/browse/FLUME-3117>`__] - Application can be dead loop when call System.exit() in methodconfigure
+ * [`FLUME-3133 <https://issues.apache.org/jira/browse/FLUME-3133>`__] - Add a ipHeader config in Syslog Sources
+ * [`FLUME-3201 <https://issues.apache.org/jira/browse/FLUME-3201>`__] - Fix SyslogUtil to handle RFC3164 format in december correctly
+ * [`FLUME-3218 <https://issues.apache.org/jira/browse/FLUME-3218>`__] - Fix external process config filter test
+ * [`FLUME-3222 <https://issues.apache.org/jira/browse/FLUME-3222>`__] - java.nio.file.NoSuchFileException thrown when files are being deleted from the TAILDIR source
+ * [`FLUME-3237 <https://issues.apache.org/jira/browse/FLUME-3237>`__] - Handling RuntimeExceptions coming from the JMS provider in JMSSource
+ * [`FLUME-3253 <https://issues.apache.org/jira/browse/FLUME-3253>`__] - JP Morgan Chase scan shows vulnerabilities for Splunk App using Apache Flume 1.8
+ * [`FLUME-3265 <https://issues.apache.org/jira/browse/FLUME-3265>`__] - Cannot set batch-size for LoadBalancingRpcClient
+ * [`FLUME-3270 <https://issues.apache.org/jira/browse/FLUME-3270>`__] - Close JMS resources in JMSMessageConsumer constructor in case of failure
+ * [`FLUME-3278 <https://issues.apache.org/jira/browse/FLUME-3278>`__] - Handling -D keystore parameters in Kafka components
+ * [`FLUME-3294 <https://issues.apache.org/jira/browse/FLUME-3294>`__] - Fix polling logic in TaildirSource
+ * [`FLUME-3298 <https://issues.apache.org/jira/browse/FLUME-3298>`__] - Make hadoop-common optional in flume-ng-hadoop-credential-store-config-filter
+ * [`FLUME-3299 <https://issues.apache.org/jira/browse/FLUME-3299>`__] - Fix log4j scopes in pom files
+
+** Sub-task
+ * [`FLUME-3158 <https://issues.apache.org/jira/browse/FLUME-3158>`__] - Upgrade surefire version and config
+ * [`FLUME-3243 <https://issues.apache.org/jira/browse/FLUME-3243>`__] - Increase the default of hdfs.callTimeout and document it's deprecation
+ * [`FLUME-3303 <https://issues.apache.org/jira/browse/FLUME-3303>`__] - Mention future configuration change in the release notes
+
+** Test
+ * [`FLUME-3195 <https://issues.apache.org/jira/browse/FLUME-3195>`__] - Split up the TestKafkaChannel class
+
+** Wish
+ * [`FLUME-3087 <https://issues.apache.org/jira/browse/FLUME-3087>`__] - Change log level from WARN to INFO when using default "maxIOWorkers" value.
+
+** Task
+ * [`FLUME-3183 <https://issues.apache.org/jira/browse/FLUME-3183>`__] - Maven: generate SHA-512 checksum during deploy
+
+** Dependency upgrade
+ * [`FLUME-2698 <https://issues.apache.org/jira/browse/FLUME-2698>`__] - Upgrade Jetty Version
+ * [`FLUME-3115 <https://issues.apache.org/jira/browse/FLUME-3115>`__] - Upgrade netty library dependency
+ * [`FLUME-3194 <https://issues.apache.org/jira/browse/FLUME-3194>`__] - upgrade derby to the latest (1.14.1.0) version
+
+** Documentation
+ * [`FLUME-1342 <https://issues.apache.org/jira/browse/FLUME-1342>`__] - Document JMX monitoring API
+ * [`FLUME-2723 <https://issues.apache.org/jira/browse/FLUME-2723>`__] - Document the requirement that Channel's transactionCapacity >= batchSize of the source/sink
+ * [`FLUME-3051 <https://issues.apache.org/jira/browse/FLUME-3051>`__] - Mention the incompatibility of Kafka source with 0.8.x Kafka brokers
+ * [`FLUME-3228 <https://issues.apache.org/jira/browse/FLUME-3228>`__] - Incorrect parameter name in timestamp interceptor docs
diff --git a/source/sphinx/releases/index.rst b/source/sphinx/releases/index.rst
new file mode 100644
index 00000000..e9c353ed
--- /dev/null
+++ b/source/sphinx/releases/index.rst
@@ -0,0 +1,31 @@
+Releases
+========
+
+.. rubric:: Current Release
+
+The current stable release is `Apache Flume Version 1.9.0 <1.9.0.html>`__.
+
+.. toctree::
+ :maxdepth: 1
+ :hidden:
+
+ 1.9.0
+
+.. rubric:: Previous Releases
+
+.. toctree::
+ :maxdepth: 1
+ :glob:
+
+ 1.8.0
+ 1.7.0
+ 1.6.0
+ 1.5.2
+ 1.5.0.1
+ 1.5.0
+ 1.4.0
+ 1.3.1
+ 1.3.0
+ 1.2.0
+ 1.1.0
+ 1.0.0
diff --git a/source/sphinx/source.rst b/source/sphinx/source.rst
new file mode 100644
index 00000000..48184c48
--- /dev/null
+++ b/source/sphinx/source.rst
@@ -0,0 +1,102 @@
+=================
+Source Repository
+=================
+
+.. rubric:: Overview
+
+This project uses `Git <http://git-scm.com/>`_ to manage its source code. Instructions on
+Git use can be found in the `Git documentation <http://git-scm.com/documentation/>`_.
+
+.. rubric:: Web Access
+
+The following is a link to the online source repository.
+
+.. raw:: html
+
+ <div class="highlight-none"><div class="highlight"><pre>
+ <a href="https://git-wip-us.apache.org/repos/asf?p=flume.git;a=tree;h=refs/heads/trunk;hb=trunk">https://git-wip-us.apache.org/repos/asf?p=flume.git;a=tree;h=refs/heads/trunk;hb=trunk</a>
+ </pre></div></div>
+
+.. rubric:: Anonymous Access
+
+Flume maintains an active release branch along with trunk. The release branch represents the list of commits that will go into the next release.
+All commits that go into trunk will also have to be committed using git cherry-pick and pushed to the current release branch.
+The current release branch is flume-1.4.
+
+The source can be checked out anonymously from git with this command:::
+
+ $ git clone http://git-wip-us.apache.org/repos/asf/flume.git flume
+
+.. rubric:: Developer Access
+
+Everyone can access the Git repository via HTTP, but Committers must clone the git repository via HTTPS.::
+
+ $ git clone https://git-wip-us.apache.org/repos/asf/flume.git flume
+
+Committers should first commit the patch to trunk on your local repo::
+
+ $ git pull
+ $ git commit -m "A message"
+ $ git log
+
+Copy the commit hash of the commit you just made. Then::
+
+ $ git checkout flume-1.4 (or the current release branch)
+ $ git pull
+ $ git cherry-pick <commit hash of the commit you made> (This should get committed immediately).
+
+Now push to both trunk and the release branch::
+
+ $ git push -u origin trunk
+ $ git push -u origin flume-1.4
+
+For more details, please read: `Git at Apache <https://git-wip-us.apache.org/>`_. Please read the
+`Common Errors page <https://git-wip-us.apache.org/docs/common-errors.html>`_ if you face any problems.
+
+The committer should make sure the commits are pushed to both branches. Please make sure all commits to the release branch are fast forward commits
+and there are no merge commits on the release branch.
+
+This process requires a little more work, but this guarantees that our release tags will not have accidental and local commits in its history,
+as we can force push to the release branches to remove these from history. Ideally we should try to keep the history on release branches linear,
+but if at some point we decide to start using feature branches, we might end up having merge commits on these branches too, but that is expected
+and required - since that would represent the list of commits for that feature.
+
+When a release is finalized, the current release branch will be frozen by the release manager for the release, and a new release branch will be
+branched off the current release branch. The new branch will represent the next release and all commits not meant for the current release must
+go to the new branch once the current branch is frozen. The release manager then pushes release related commits to the current branch.
+
+For example, if the next release is flume-1.4.0, all commits should go to trunk and flume-1.4.
+When the rolling release canidate, the release manager will create a new branch, say flume-1.5 from the latest commit of
+flume-1.4. All further non-release related commits should go to trunk and flume-1.5 (unless the release manager thinks otherwise - in which case it can
+go to flume-1.4 and flume-1.5).
+
+Please note that the main development branch is **trunk**, not master.
+
+.. rubric:: Access From Behind a Firewall
+
+For those users who are stuck behind a corporate firewall which is blocking HTTP access to the Subversion repository,
+you can try to access it via the developer connection::
+
+ $ git clone https://git-wip-us.apache.org/repos/asf/flume.git flume
+
+.. rubric:: Access Through a Proxy
+
+The Subversion client can go through a proxy, if you configure it to do so. To configure git to use a proxy::
+
+ $ git config http.proxy address:port
+
+Then do a git clone as usual::
+
+ $ git clone http://git-wip-us.apache.org/repos/asf/flume.git flume
+
+
+.. rubric:: SVN Access
+
+If you prefer to use svn and not git, please use the `Github read-only mirror <https://github.com/apache/flume>`_ to checkout
+the sources. Please read `this <https://github.com/blog/626-announcing-svn-support>`_ and `this <https://github.com/blog/626-announcing-svn-support>`_
+for details on how to use svn with github. Please note that the github mirror is likely to be several hours behind the Apache Flume git repo.
+
+**Note 1:** The (erstwhile) Apache Flume `SVN repository <http://svn.apache.org/repos/asf/flume/>`_ is no longer updated and should not be used to checkout
+the source code.
+
+**Note 2:** Committers must use the git-wip-us repo to commit. SVN can no longer be used to commit.
diff --git a/source/sphinx/team.rst b/source/sphinx/team.rst
new file mode 100644
index 00000000..6e5405ba
--- /dev/null
+++ b/source/sphinx/team.rst
@@ -0,0 +1,48 @@
+====
+Team
+====
+ A successful project requires many people to play many different roles. Some members write code or documentation,
+ while others are valuable as testers, submitting patches and suggestions.
+
+ The team is comprised of Members, Committers and Contributors. Members are responsible for making project management
+ decisions, voting on releases as well as acting as Committers. Committers have direct access to the source code and
+ actively evolve the codebase. Contributors improve the project through submission of patches and suggestions
+ to the Members. The number of Contributors to the project is unbounded. Get involved today. All contributions
+ to the project are greatly appreciated.
+
+ The following individuals are recognized as PMC Members or Project Committers.
+
+.. csv-table::
+ :header: "Name", "Email", "Id", "Organization", "Role"
+ :widths: 30, 25, 15, 15, 15
+
+ "Aaron Kimball", "kimballa@apache.org", "kimballa", "Zymergen", "PMC Member"
+ "Ashish Paliwal", "apaliwal@apache.org", "apaliwal", "Apple", "Committer"
+ "Andrew Bayer", "abayer@apache.org", "abayer", "CloudBees", "PMC Member"
+ "Ahmed Radwan", "ahmed@apache.org", "ahmed", "Apple", "PMC Member"
+ "Arvind Prabhakar", "arvind@apache.org", "arvind", "StreamSets", "PMC Member"
+ "Balázs Donát Bessenyei", "bessbd@apache.org", "bessbd", "Ericsson", "PMC Member"
+ "Brock Noland", "brock@apache.org", "brock", "phData", "PMC Member"
+ "Bruce Mitchener", "brucem@apache.org", "brucem", "Data Fueled", "PMC Member"
+ "Derek Deeter", "ddeeter@apache.org", "ddeeter", "Vanderbilt University", "PMC Member"
+ "Denes Arvay", "denes@apache.org", "denes", "Cloudera", "PMC Member"
+ "Eric Sammer", "esammer@apache.org", "esammer", "Splunk", "PMC Member"
+ "Hari Shreedharan", "hshreedharan@apache.org", "hshreedharan", "StreamSets", "PMC Member"
+ "Henry Robinson", "henry@apache.org", "henry", "Cloudera", "PMC Member"
+ "Jaroslav Cecho", "jarcec@apache.org", "jarcec", "StreamSets", "PMC Member"
+ "Johny Rufus", "johnyrufus@apache.org", "johnyrufus", "Microsoft", "Committer"
+ "Jonathan Hsieh", "jmhsieh@apache.org", "jmhsieh", "Cloudera", "PMC Member"
+ "Juhani Connolly", "juhanic@apache.org", "juhanic", "CyberAgent", "PMC Member"
+ "Mike Percy", "mpercy@apache.org", "mpercy", "Cloudera", "PMC Member"
+ "Mingjie Lai", "mlai@apache.org", "mlai", "Apple", "PMC Member"
+ "Mubarak Seyed", "mubarak@apache.org","mubarak", "Apple", "Committer"
+ "Nick Verbeck", "nerdynick@apache.org", "nerdynick", "", "PMC Member"
+ "Patrick Hunt", "phunt@apache.org", "phunt", "Cloudera", "PMC Member"
+ "Patrick Wendell", "pwendell@apache.org", "pwendell", "Databricks", "Committer"
+ "Prasad Mujumdar", "prasadm@apache.org", "prasadm", "BlueTalon", "PMC Member"
+ "Ralph Goers", "rgoers@apache.org", "rgoers", "Nextiva", "PMC Member"
+ "Roshan Naik", "roshannaik@apache.org", "roshannaik", "Hortonworks", "PMC Member"
+ "Attila Simon", "sati@apache.org", "sati", "RapidMiner", "Committer"
+ "Ferenc Szabo", "szaboferee@apache.org", "szaboferee", "Cloudera", "Committer"
+ "Wolfgang Hoschek", "whoschek@apache.org", "whoschek", "Cloudera", "Committer"
+ "Will McQueen", "will@apache.org", "will", "", "PMC Member"