[metron] branch master updated: METRON-2014 Add architectural documentation for metron-writer (merrimanr) closes apache/metron#1381

[rm...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

rmerriman pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/metron.git


The following commit(s) were added to refs/heads/master by this push:
     new 3a61933  METRON-2014 Add architectural documentation for metron-writer (merrimanr) closes apache/metron#1381
3a61933 is described below

commit 3a61933ed4fb6ec00286f4c13a22c372adf8d0dc
Author: merrimanr <me...@gmail.com>
AuthorDate: Tue Apr 16 13:16:20 2019 -0500

    METRON-2014 Add architectural documentation for metron-writer (merrimanr) closes apache/metron#1381
---
 metron-platform/metron-writer/README.md | 89 +++++++++++++++++++++++++++++++++
 1 file changed, 89 insertions(+)

diff --git a/metron-platform/metron-writer/README.md b/metron-platform/metron-writer/README.md
index ed4f053..3cc9c87 100644
--- a/metron-platform/metron-writer/README.md
+++ b/metron-platform/metron-writer/README.md
@@ -20,6 +20,95 @@ limitations under the License.
 ## Introduction
 The writer module provides some utilties for writing to outside components from within Storm.  This includes managing bulk writing.  An implemention is included for writing to HDFS in this module. Other writers can be found in their own modules.
 
+## Bulk Message Writing
+Most external components encourage messages to be written in batches for performance reasons.  The writer module includes an abstraction for doing this in an efficient manner.  This abstraction provides the following features:
+* A high-level `BulkWriterComponent` class that manages a per-sensor cache of batched messages and flushes when appropriate
+* An extension point for determining when a batch should be flushed
+* An extention point for handling a bulk message write response after a batch has been flushed
+
+### Flush Policies
+Flushing behavior is controlled by a collection of `FlushPolicy` objects.  They are responsible for 2 things:
+
+1. Determining when a batch should be flushed
+2. Handling a `BulkWriterResponse` after a batch is flushed and messages are written with a `BulkMessageWriter`
+
+The `FlushPolicy` interface defines methods for handling these responsiblities: 
+
+- `boolean shouldFlush(String sensorType, WriterConfiguration configurations, List<BulkMessage<MESSAGE_T>> messages)`
+- `void onFlush(String sensorType, BulkWriterResponse response)`
+
+There are 2 `FlushPolicy` implementations included by default:  
+
+- The `BatchSizePolicy` will flush a batch whenever the batch size reaches a configured value.  This configuration value is represented by the `batchSize` property in either the parser, enrichment or indexing configuration (whichever is appropriate in the current context).
+- The `BatchTimeoutPolicy` will flush a batch whenever the batch timeout has elapsed.  This configuration value is represented by the `batchTimeout` property in either the parser, enrichment or indexing configuration (whichever is appropriate in the current context).  A `maxBatchTimeout` is set at creation time and serves as the ceiling for a batch timeout.  In Storm topologies, this value is set to 1/2 the tuple timeout setting to ensure messages are always flushed before their tuples t [...]
+
+For example, a configuration that sets the `batchSize` and `batchTimeout` in a parser topology will look like:
+```
+{
+  "parserClassName": "org.apache.metron.parsers.bro.BasicBroParser",
+  "sensorTopic": "bro",
+  "parserConfig": {
+    "batchSize": 5
+    "batchTimeout": 2
+  }
+}
+```
+Similarly for the enrichment topology (configured in the [Global Configuration](../metron-common#global-configuration)):
+```
+{
+  "enrichment.writer.batchSize": "5",
+  "enrichment.writer.batchTimeout": "2",
+  ...
+}
+```
+And finally for the indexing topology:
+```
+{
+  "elasticsearch": {
+    "index": "bro",
+    "batchSize": 5,
+    "batchTimeout": 2,
+    "enabled": true
+  },
+  ...
+}
+```
+
+
+Additional policies can be added as needed.  For example, an `AckTuplesPolicy` is added in the Storm bolts to handle acking tuples after a batch is flushed.
+
+### Bulk Writing Workflow
+The `BulkWriterComponent` class collects messages in separate sensor-specific caches.  This class is instantiated and supplied to classes that need to write messages to external components.  A collection of default `FlushPolicy` implementations
+are created by default with the option of passing in additional `FlushPolicy` objects as needed.
+
+Batching and writing messages follows this process:
+
+1. A single message is passed to the `BulkWriterComponent.write` method and stored in the appropriate cache based on the sensor type.  A `BulkMessageWriter` is also supplied to do the actual writing when messages are flushed.
+2. The collection of `FlushPolicy` implementations are checked and a batch is flushed whenever the first `FlushPolicy.shouldFlush` returns true.
+3. If no policies signal a flush, then nothing happens.  If a policy does signal a flush, the batch of messages for that sensor are written with the supplied `BulkMessageWriter`.  Each `FlushPolicy.onFlush` method is then called with the `BulkWriterResponse`.  
+4. If a sensor type has been disabled, it's batch is flushed immediately (`FlushPolicy.shouldFlush` is not checked). 
+5. A `BulkWriterComponent.flushAll` method is available that immediately calls the `FlushPolicy.shouldFlush` methods for each sensor type in the cache.  This should be called periodically by the class containing `BulkWriterComponent` to ensure messages
+are not left sitting in the cache.  For example, the Storm bolts call this whenever a tick tuple is received.
+
+### Logging
+Logging can be enabled for the classes described in this section to provide insight into how messages are being batched and flushed.  This can be an important tool when performance tuning.
+Setting the log level to `DEBUG` on the `org.apache.metron.writer` package will produce detailed information about when batches are flushed, which sensor a flushed batch corresponds to, which policy caused the flush, and how long it took to write the batch.
+
+### Performance Tuning
+The primary purpose of the Bulk Message Writing abstraction is to enable efficient writing to external components.  This section provides recommendations and strategies for doing that.  It is assumed that most Metron installations will include a variety of sensors with different volumes and velocities.  Different sensors will likely need to be tuned differently.
+
+#### Set batches sizes higher for high volume sensors
+High volume sensors should be identified and configured with higher batch sizes (1000+ is recommended).  Use logging to verify these batches are in fact filling up.  The number of actual message written should match the batch size.  Keep in mind that large batch sizes also require more memory to hold messages.  Streaming engines like Storm limit how many messages can be processed at a time (the `topology.max.spout.pending` setting).
+
+#### Set batch timeouts lower for low volume sensors
+Low volume sensors make take longer to fill up a batch, especially if the batch size is set higher.  This can be undesirable because messages may stay cached for longer than necessary, consuming memory and increasing latency for that sensor type.
+
+#### Be careful with threads
+Each thread (executor in Storm) maintains it's own message cache.  Allocating too many threads will cause messages to be spread too thin across separate caches and batches won't fill up completely.  This should be balanced with having enough threads to take advantage of any parallel write capability offered by the endpoint that's being written to.
+
+#### Watch for high write times
+Use logging to evaluate write timing.  Unusually high write times can indicate that an endpoint is not configured correctly or undersized.
+
 ## Kafka Writer
 We have an implementation of a writer which will write batches of
 messages to Kafka.  An interesting aspect of this writer is that it can