You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@brooklyn.apache.org by he...@apache.org on 2021/08/23 07:57:07 UTC
[brooklyn-docs] 01/05: add tips for setting up ES - with thx to
@jcabrerizo
This is an automated email from the ASF dual-hosted git repository.
heneveld pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/brooklyn-docs.git
commit c30a584351b7f0989eda08bb1e7f7d087f0c6950
Author: Alex Heneveld <al...@cloudsoftcorp.com>
AuthorDate: Fri Aug 20 17:30:42 2021 +0100
add tips for setting up ES - with thx to @jcabrerizo
---
guide/ops/logging.md | 141 ++++++++++++++++++++++++++++++++++++++++++++++-----
1 file changed, 128 insertions(+), 13 deletions(-)
diff --git a/guide/ops/logging.md b/guide/ops/logging.md
index ef6d3c5..2fe535c 100644
--- a/guide/ops/logging.md
+++ b/guide/ops/logging.md
@@ -73,24 +73,43 @@ For example (on mac):
mv $LOG_FILE /path/to/archive/brooklyn.debug-$TIMESTAMP.log.zip
-## Logging aggregators
+## Logging Aggregators
Integration with systems like Logstash and Splunk is possible using standard log4j configuration.
Log4j can be configured to write to syslog using the SyslogAppender
which can then [feed its logs to Logstash](http://www.logstash.net/docs/1.4.2/inputs/syslog).
+
## Logbook
The logbook offers the possibility to query and view logs in the UI. By default, logs are stored in files as per configuration
-in `etc/org.ops4j.pax.logging.cfg`. The logbook can be configured against different log aggregation sources by adding the
-following parameters in `brooklyn.cfg`:
+in `etc/org.ops4j.pax.logging.cfg`. The logbook can be configured against different log aggregation sources by setting the
+`brooklyn.logbook.logStore` parameter in `brooklyn.cfg`, and depending which backend is used, other parameters.
-* plain log files
+For example to use the local log files written by Apache Brooklyn, you could configure:
brooklyn.logbook.logStore=org.apache.brooklyn.util.core.logbook.file.FileLogStore
brooklyn.logbook.fileLogStore.path=/var/logs/brooklyn/brooklyn.debug.log
-* or Elasticsearch released under the Apache License, version 2.0 fork created by AWS
+In production environments where log data is desired to be retained, Apache Brooklyn supports Elasticsearch backends.
+This can be a dedicated ES environment for use by Apache Brooklyn or a shared/managed ES facility that handles many logs,
+or -- for lightweight usage -- a simple local ES server running on the same instance as Apache Brooklyn.
+As with any log storage requirement, the sizing, scaling, backup and maintenance of the logging environment
+requires careful attention. Elasticsearch includes numerous options to configure these, with one suggested configuration
+outlined in more detail below.
+
+By default, only users with the `root` entitlement or an explicit `logViewer` entitlement are able to see log info through Apache Brooklyn.
+
+The Logbook UI widget can be found throughout the product:
+in the About section, where all logs can be viewed;
+on the App Inspector Entity Summary view, and
+on the App Inspector Activity view, where logs filtered by entity ID and activity ID respectively are shown.
+
+
+### Suggested Elasticsearch Setup
+
+[OpenSearch (OpenDistro for Elasticsearch)](https://opendistro.github.io/for-elasticsearch) is an Apache-licensed open-source
+backend that works well with Apache Brooklyn, with this configuration in `brooklyn.cfg`:
brooklyn.logbook.logStore=org.apache.brooklyn.util.core.logbook.opensearch.OpenSearchLogStore
brooklyn.logbook.openSearchLogStore.host=https://localhost:9200
@@ -99,16 +118,12 @@ following parameters in `brooklyn.cfg`:
brooklyn.logbook.openSearchLogStore.password=admin
brooklyn.logbook.openSearchLogStore.verifySsl=false
-Users with `root` entitlement only can query and view logs in the logbook.
-
-Logbook UI widget can be found in About section where all logs can be viewed, as well as in App Inspector Entity view and
-Activity view where logs filtered by entity ID and activity ID respectively.
-### Elasticsearch setup
+#### Routing Logs to Elastic Search
-Refer to the [official documentation](https://opendistro.github.io/for-elasticsearch/downloads.html#try) for
- installation guide. [Fluentd](https://www.fluentd.org/download) daemon can be configured to read the log files
-for Elasticsearch. See example of Fluentd `td-agent.conf` below:
+There are many solutions to routing log messages from Apache Brooklyn to Elasticsearch, either plugging in to the log4j subsystem
+or routing the log files from disk. [Fluentd](https://www.fluentd.org/download), with the following configuration in `td-agent.conf`,
+is a good simple way to forward content added to the log files:
```
<source>
@@ -135,6 +150,102 @@ for Elasticsearch. See example of Fluentd `td-agent.conf` below:
</match>
```
+#### Sizing and Rotating Logs
+
+Keeping log data obviously consumes disk storage, and serving the data requires memory.
+The log levels in Apache Brooklyn can be configured on a fine-grained log-category basis,
+and different levels and categories can be routed to different indexes with different
+retention policies.
+
+When designing your strategy for maintaining logs, some good rules of thumb are:
+
+* Allocate 4 GB RAM for a production ES instance plus 2 GB RAM for every TB of log data that is searchable
+* Consider a small number of tiers with different retention policies,
+ e.g. INFO and selected DEBUG messages (logins) to one index,
+ and all other DEBUG and lower messages sent to another index
+* Consider using rolling indices on a nightly or weekly basis, and an auto-close job and/or an auto-delete job to keep memory and disk usage at a steady state;
+ for example the INFO and selected DEBUG messages might go to an index which rotates weekly and is deleted or moved to cold storage after two years,
+ whereas DEBUG and lower messages might rotate daily and be deleted after a week
+* The amount of log data can vary depending how Apache Brooklyn is used, so monitor usage to get a feel for what is
+ maintainable, and put in place notifications if disk and/or memory usage become high
+* Review the logs and turn off unnecessary categories
+
+Instructions and links to assist with this are below.
+
+
+#### Index partitioning
+
+It’s possible to configure fluentd for sending the information to an index using an index name generated using datetime markers.
+This example will create and send the data to a new index every day:
+
+```
+<match td.apachebrokyn.*>
+ @type elasticsearch
+ hosts https://localhost:9200
+ user admin
+ password admin
+ ssl_verify false
+
+ include_timestamp true
+ index_name brooklyn-rotating-%Y.%m.%d
+ flush_interval 5s
+ <buffer tag, time>
+ timekey 60 # chunks per hours ("3600" also available)
+ flush_interval 5s
+ </buffer>
+</match>
+```
+
+Apache Brooklyn can be configured to use an index _pattern_ for querying, eg:
+
+ brooklyn.logbook.openSearchLogStore.index = brooklyn-rotating-*
+
+
+#### Index lifecycle management
+
+Policies also allow handling the lifecycle of the indexes.
+For example, to delete indexes after a period of time:
+
+```
+{
+ "policy": {
+ "description": "Delete workflow",
+ "default_state": "new",
+ "schema_version": 1,
+ "states": [
+ {
+ "name": "new",
+ "transitions": [
+ {
+ "state_name": "delete",
+ "conditions": {
+ "min_index_age": "60d"
+ }
+ }
+ ]
+ },
+ {
+ "name": "delete",
+ "actions": [
+ {
+ "delete": {}
+ }
+ ]
+ }
+ ],
+ "ism_template": {
+ "index_patterns": ["brooklyn-rotating*"],
+ "priority": 100
+ }
+ }
+}
+```
+
+With these building blocks, and others linked below, you can configure the retention policy that suits your environment,
+balancing the trade-off between data availability and resource usage.
+
+
+
## For More Information
The following resources may be useful when configuring logging:
@@ -144,3 +255,7 @@ The following resources may be useful when configuring logging:
* The [Logback Project](http://logback.qos.ch/) home page
* [Brooklyn Developer Guide]({{book.path.docs}}/dev/tips/logging.md) logging tips
* [OPS4J Pax Logging](https://ops4j1.jira.com/wiki/display/paxlogging/Configuration)
+* [Elasticsearch Best Practices](https://www.elastic.co/guide/en/elasticsearch/reference/7.x/best_practices.html)
+* [Elasticsearch Memory Usage](https://www.elastic.co/blog/significantly-decrease-your-elasticsearch-heap-memory-usage)
+* [OpenSearch Index Management](https://opensearch.org/docs/im-plugin/ism/index/) and [policies](https://opensearch.org/docs/im-plugin/ism/policies/)
+