You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@flume.apache.org by rg...@apache.org on 2022/10/04 16:18:37 UTC
[flume-site] branch asf-staging updated: Add info on Spring Boot support
This is an automated email from the ASF dual-hosted git repository.
rgoers pushed a commit to branch asf-staging
in repository https://gitbox.apache.org/repos/asf/flume-site.git
The following commit(s) were added to refs/heads/asf-staging by this push:
new 688771bb Add info on Spring Boot support
688771bb is described below
commit 688771bba61651ab3d27369e08130c4010bb6c41
Author: Ralph Goers <rg...@apache.org>
AuthorDate: Tue Oct 4 09:18:30 2022 -0700
Add info on Spring Boot support
---
source/sphinx/FlumeDeveloperGuide.rst | 13 +++-
source/sphinx/FlumeUserGuide.rst | 131 +++++++++++++++++++++++++++++++---
2 files changed, 131 insertions(+), 13 deletions(-)
diff --git a/source/sphinx/FlumeDeveloperGuide.rst b/source/sphinx/FlumeDeveloperGuide.rst
index 6a01c6a8..77cceb06 100644
--- a/source/sphinx/FlumeDeveloperGuide.rst
+++ b/source/sphinx/FlumeDeveloperGuide.rst
@@ -15,7 +15,7 @@
======================================
-Flume 1.10.1 Developer Guide
+Flume 1.11.0 Developer Guide
======================================
Introduction
@@ -90,13 +90,19 @@ Getting the source
~~~~~~~~~~~~~~~~~~
Check-out the code using Git. Click here for
-`the git repository root <https://git-wip-us.apache.org/repos/asf/flume.git>`_.
+`the git repository root <https://git-wip-us.apache.org/repos/asf/flume.git>`_
+or at `GitHub <https://github.com/apache/flume.git>`_.
+
The Flume 1.x development happens under the branch "trunk" so this command line
can be used:
git clone https://git-wip-us.apache.org/repos/asf/flume.git
+or
+
+ git clone https://github.com/apache/flume.git
+
Compile/test Flume
~~~~~~~~~~~~~~~~~~
@@ -872,7 +878,8 @@ Initializable
As of Flume 1.10.0 Sources, Sinks, and Channels may implement the Intitializable interface. Doing so
allows the component to have access the materialized configuration before any of the components have been
-started.
+started. While this ability is quite useful when using the standard configuration, it is less useful when
+configuring using Spring Boot as Spring's autowiring generally can be used to accomplish the same thing.
This example shows a Sink being configured with the name of a Source. While initializing it will
retrieve the Source from the configuration and save it. During event processing a new event will be
diff --git a/source/sphinx/FlumeUserGuide.rst b/source/sphinx/FlumeUserGuide.rst
index fb6bba80..b0dcf928 100644
--- a/source/sphinx/FlumeUserGuide.rst
+++ b/source/sphinx/FlumeUserGuide.rst
@@ -15,7 +15,7 @@
================================
-Flume 1.10.1 User Guide
+Flume 1.11.0 User Guide
================================
Introduction
@@ -509,7 +509,8 @@ and channel3, if it's "vendor" then it should go to channel2, otherwise
channel3. The mapping can be set in the agent's configuration file.
Spring Boot Setup
-==============
+=================
+As of version 1.11.0 Flume supports being packaged as a Spring Boot application.
Flume "normally" follows a paradigm where each component implements the Configurable
interface and must implement the configure method to configure itself by retrieving
@@ -545,10 +546,10 @@ Component Scanning
------------------
Spring Boot will automatically locate all the Spring components provided by Flume.
However, in order for the Flume application to be configured Spring needs the
-base Java package name for the application to locate those Spring components.
-This is accomplished by the application implementing a class that implements
-`org.apache.flume.spring.boot.config.PackageProvider` that is register as a Java
-service. For example::
+base Java package name used by the application in order for Spring to locate
+these components.This is accomplished in the application by creating a class
+that implements `org.apache.flume.spring.boot.config.PackageProvider` that is
+registered as a Java service. For example::
public class WylieCouotePackageProvider implements PackageProvider {
private static final String[] PACKAGES = {"org.acme.coyote.wylie"};
@@ -565,10 +566,11 @@ which would contain::
org.acme.coyote.wylie.config.WylieCoyotePackageProvider
-This would result in all classes in the `org.acme.coyote.wylie" package and
-it`s sub-packages being scannedfor Spring components to be included. Note
+This would result in all classes in the `org.acme.coyote.wylie` package and
+it`s sub-packages being scanned by Spring for components to be included. Note
that classes found there may also use Spring's `@Import` annotation to include
-classes in other packages.
+classes in other packages. In addition, since the getPackages method returns
+a List more than one package can be specified.
Component Wiring
----------------
@@ -644,11 +646,120 @@ construct the appropriate Flume components.
Note that all Spring Boot facilities are available to Flume applications configured this way.
+SinkGroups and Sinks may also be configured in a similar fashion as in::
+
+ flume:
+ sinkGroups:
+ rrobin:
+ backoff: true
+ selector: round_robin
+ "selector.maxTimeOut": 30000
+
+ sinks:
+ avroSinks:
+ avroSink1:
+ hostname: 192.168.10.10
+ port: 4141
+ batch-size: 100
+ compression-type: deflate
+ avroSink2:
+ hostname: 192.168.10.11
+ port: 4141
+ batch-size: 100
+ compression-type: deflate
+
+These would be configured in the Java config with::
+
+ @Bean
+ @ConfigurationProperties(prefix = "flume.sink-groups.rrobin")
+ public Map<String, String> rrobinProperties() {
+ return new HashMap<>();
+ }
+
+ @Bean
+ @ConfigurationProperties(prefix = "flume.sinks.avro-sinks")
+ public Map<String, AvroSinkConfiguration> avroSinksProperties() {
+ return new HashMap<>();
+ }
+
+ @Bean
+ public List<Sink> avroSinks(final Channel avroFileChannel,
+ final Map<String, AvroSinkConfiguration> avroSinksProperties) {
+ List<Sink> sinks = new ArrayList<>();
+ for (Map.Entry<String, AvroSinkConfiguration> entry : avroSinksProperties.entrySet()) {
+ sinks.add(configureSink(entry.getKey(), AvroSink.class, avroFileChannel,
+ entry.getValue().getProperties()));
+ }
+ return sinks;
+ }
+
+ @Bean
+ public SinkRunner avroSinkRunner(final Map<String, String> rrobinProperties, final List<Sink> avroSinks) {
+ return createSinkRunner(
+ configureSinkProcessor(rrobinProperties, LoadBalancingSinkProcessor.class, avroSinks));
+ }
+
+Note that the attribute names specified for the sources, channels, and sink groups must match the attribute names
+specified for the components in other portions of this documentation.
+
+It is important to note that a concrete class was used to capture the data for the Avro Sinks. When a simple Map
+is used Spring seems to get confused with the nested Maps. The `AvroSinkConfiguration` class would look like::
+
+ public class AvroSinkConfiguration {
+
+ private String hostName;
+ private int port;
+ private int batchSize;
+ private String compressionType;
+
+ public String getHostName() {
+ return hostName;
+ }
+
+ public void setHostName(String hostName) {
+ this.hostName = hostName;
+ }
+
+ public int getPort() {
+ return port;
+ }
+
+ public void setPort(int port) {
+ this.port = port;
+ }
+
+ public int getBatchSize() {
+ return batchSize;
+ }
+
+ public void setBatchSize(int batchSize) {
+ this.batchSize = batchSize;
+ }
+
+ public String getCompressionType() {
+ return compressionType;
+ }
+
+ public void setCompressionType(String compressionType) {
+ this.compressionType = compressionType;
+ }
+
+ public Map<String, String> getProperties() {
+ Map<String, String> map = new HashMap<>();
+ map.put("hostname", hostName);
+ map.put("port", Integer.toString(port));
+ map.put("batchSize", Integer.toString(batchSize));
+ map.put(compressionType, compressionType);
+ return map;
+ }
+ }
+
+
Configuration
=============
As mentioned in the earlier section, standard Flume agent configuration is
-read from afile that resembles a Java property file format with hierarchical property
+read from a file that resembles a Java property file format with hierarchical property
settings.
Defining the flow