You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@brooklyn.apache.org by he...@apache.org on 2021/08/23 07:57:06 UTC
[brooklyn-docs] branch master updated (e6f9bc7 -> fc068ea)
This is an automated email from the ASF dual-hosted git repository.
heneveld pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/brooklyn-docs.git.
from e6f9bc7 Moved policies CAMP specific content to external file to allow future overwritings of this
new c30a584 add tips for setting up ES - with thx to @jcabrerizo
new 988af71 address PR comments
new 34b425d This closes #324
new 0519dc5 new LDAP keys
new fc068ea This closes #325
The 5 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
guide/ops/configuration/brooklyn_cfg.md | 10 ++-
guide/ops/logging.md | 141 +++++++++++++++++++++++++++++---
2 files changed, 136 insertions(+), 15 deletions(-)
[brooklyn-docs] 04/05: This closes #324
Posted by he...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
heneveld pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/brooklyn-docs.git
commit 34b425d253852c7e176b27745286c317390a0e0e
Merge: e6f9bc7 988af71
Author: Alex Heneveld <al...@cloudsoftcorp.com>
AuthorDate: Mon Aug 23 08:56:47 2021 +0100
This closes #324
guide/ops/logging.md | 141 ++++++++++++++++++++++++++++++++++++++++++++++-----
1 file changed, 128 insertions(+), 13 deletions(-)
[brooklyn-docs] 02/05: new LDAP keys
Posted by he...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
heneveld pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/brooklyn-docs.git
commit 0519dc5b03067f2eb71eefa2583d5711917a97df
Author: Juan Cabrerizo <ju...@cloudsoft.io>
AuthorDate: Sat Aug 21 17:55:55 2021 +0100
new LDAP keys
---
guide/ops/configuration/brooklyn_cfg.md | 10 ++++++++--
1 file changed, 8 insertions(+), 2 deletions(-)
diff --git a/guide/ops/configuration/brooklyn_cfg.md b/guide/ops/configuration/brooklyn_cfg.md
index c372835..ecd6a3f 100644
--- a/guide/ops/configuration/brooklyn_cfg.md
+++ b/guide/ops/configuration/brooklyn_cfg.md
@@ -139,8 +139,14 @@ The other things you need to set in `brooklyn.cfg` are:
* `brooklyn.webconsole.security.ldap.allowed_realms_regex` - allows multiple realms (domains) that match regex - username must
be of form domain\user
* `brooklyn.webconsole.security.ldap.ou` *optional, by default it set to Users* - ldap ou parameter
-
-**brooklyn.cfg example configuration:**
+* `brooklyn.webconsole.security.ldap.group_config_key` *optional* to be used in combination with the next. Name of the
+ config key prefix for the valid LDAP groups to be mapped to AMP entitlements. If used only mapped groups will be added
+ to the user groups. If empty, user LDAP groups will be ignored.
+* `brooklyn.webconsole.security.ldap.fetch_user_group` *optional, by default it set to false* - whether or not the LDAP
+ groups for the user should be gathered. If true, the groups will be stored in the user session and the security context
+* `brooklyn.webconsole.security.ldap.login_info_log` *optional, by default it set to false* - whether or not the user attempts
+ to log in the system must be added to the info log
+ **brooklyn.cfg example configuration:**
~~~
brooklyn.webconsole.security.provider=org.apache.brooklyn.rest.security.provider.LdapSecurityProvider
[brooklyn-docs] 03/05: address PR comments
Posted by he...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
heneveld pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/brooklyn-docs.git
commit 988af71f6248f96af21c6dee0eb607081c6b1dda
Author: Alex Heneveld <al...@cloudsoftcorp.com>
AuthorDate: Mon Aug 23 08:56:33 2021 +0100
address PR comments
---
guide/ops/logging.md | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/guide/ops/logging.md b/guide/ops/logging.md
index 2fe535c..072280a 100644
--- a/guide/ops/logging.md
+++ b/guide/ops/logging.md
@@ -86,7 +86,7 @@ The logbook offers the possibility to query and view logs in the UI. By default,
in `etc/org.ops4j.pax.logging.cfg`. The logbook can be configured against different log aggregation sources by setting the
`brooklyn.logbook.logStore` parameter in `brooklyn.cfg`, and depending which backend is used, other parameters.
-For example to use the local log files written by Apache Brooklyn, you could configure:
+For example to use the local log files written by Apache Brooklyn, assuming `/var/logs`, you could configure (note these are the default values):
brooklyn.logbook.logStore=org.apache.brooklyn.util.core.logbook.file.FileLogStore
brooklyn.logbook.fileLogStore.path=/var/logs/brooklyn/brooklyn.debug.log
@@ -98,7 +98,7 @@ As with any log storage requirement, the sizing, scaling, backup and maintenance
requires careful attention. Elasticsearch includes numerous options to configure these, with one suggested configuration
outlined in more detail below.
-By default, only users with the `root` entitlement or an explicit `logViewer` entitlement are able to see log info through Apache Brooklyn.
+By default, only users with the `root`, `powerUser`, or an explicit `logViewer` entitlement are able to see log info through Apache Brooklyn.
The Logbook UI widget can be found throughout the product:
in the About section, where all logs can be viewed;
[brooklyn-docs] 05/05: This closes #325
Posted by he...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
heneveld pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/brooklyn-docs.git
commit fc068ea4b86077cefcd39de9049c042105907de3
Merge: 34b425d 0519dc5
Author: Alex Heneveld <al...@cloudsoftcorp.com>
AuthorDate: Mon Aug 23 08:57:02 2021 +0100
This closes #325
guide/ops/configuration/brooklyn_cfg.md | 10 ++++++++--
1 file changed, 8 insertions(+), 2 deletions(-)
[brooklyn-docs] 01/05: add tips for setting up ES - with thx to
@jcabrerizo
Posted by he...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
heneveld pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/brooklyn-docs.git
commit c30a584351b7f0989eda08bb1e7f7d087f0c6950
Author: Alex Heneveld <al...@cloudsoftcorp.com>
AuthorDate: Fri Aug 20 17:30:42 2021 +0100
add tips for setting up ES - with thx to @jcabrerizo
---
guide/ops/logging.md | 141 ++++++++++++++++++++++++++++++++++++++++++++++-----
1 file changed, 128 insertions(+), 13 deletions(-)
diff --git a/guide/ops/logging.md b/guide/ops/logging.md
index ef6d3c5..2fe535c 100644
--- a/guide/ops/logging.md
+++ b/guide/ops/logging.md
@@ -73,24 +73,43 @@ For example (on mac):
mv $LOG_FILE /path/to/archive/brooklyn.debug-$TIMESTAMP.log.zip
-## Logging aggregators
+## Logging Aggregators
Integration with systems like Logstash and Splunk is possible using standard log4j configuration.
Log4j can be configured to write to syslog using the SyslogAppender
which can then [feed its logs to Logstash](http://www.logstash.net/docs/1.4.2/inputs/syslog).
+
## Logbook
The logbook offers the possibility to query and view logs in the UI. By default, logs are stored in files as per configuration
-in `etc/org.ops4j.pax.logging.cfg`. The logbook can be configured against different log aggregation sources by adding the
-following parameters in `brooklyn.cfg`:
+in `etc/org.ops4j.pax.logging.cfg`. The logbook can be configured against different log aggregation sources by setting the
+`brooklyn.logbook.logStore` parameter in `brooklyn.cfg`, and depending which backend is used, other parameters.
-* plain log files
+For example to use the local log files written by Apache Brooklyn, you could configure:
brooklyn.logbook.logStore=org.apache.brooklyn.util.core.logbook.file.FileLogStore
brooklyn.logbook.fileLogStore.path=/var/logs/brooklyn/brooklyn.debug.log
-* or Elasticsearch released under the Apache License, version 2.0 fork created by AWS
+In production environments where log data is desired to be retained, Apache Brooklyn supports Elasticsearch backends.
+This can be a dedicated ES environment for use by Apache Brooklyn or a shared/managed ES facility that handles many logs,
+or -- for lightweight usage -- a simple local ES server running on the same instance as Apache Brooklyn.
+As with any log storage requirement, the sizing, scaling, backup and maintenance of the logging environment
+requires careful attention. Elasticsearch includes numerous options to configure these, with one suggested configuration
+outlined in more detail below.
+
+By default, only users with the `root` entitlement or an explicit `logViewer` entitlement are able to see log info through Apache Brooklyn.
+
+The Logbook UI widget can be found throughout the product:
+in the About section, where all logs can be viewed;
+on the App Inspector Entity Summary view, and
+on the App Inspector Activity view, where logs filtered by entity ID and activity ID respectively are shown.
+
+
+### Suggested Elasticsearch Setup
+
+[OpenSearch (OpenDistro for Elasticsearch)](https://opendistro.github.io/for-elasticsearch) is an Apache-licensed open-source
+backend that works well with Apache Brooklyn, with this configuration in `brooklyn.cfg`:
brooklyn.logbook.logStore=org.apache.brooklyn.util.core.logbook.opensearch.OpenSearchLogStore
brooklyn.logbook.openSearchLogStore.host=https://localhost:9200
@@ -99,16 +118,12 @@ following parameters in `brooklyn.cfg`:
brooklyn.logbook.openSearchLogStore.password=admin
brooklyn.logbook.openSearchLogStore.verifySsl=false
-Users with `root` entitlement only can query and view logs in the logbook.
-
-Logbook UI widget can be found in About section where all logs can be viewed, as well as in App Inspector Entity view and
-Activity view where logs filtered by entity ID and activity ID respectively.
-### Elasticsearch setup
+#### Routing Logs to Elastic Search
-Refer to the [official documentation](https://opendistro.github.io/for-elasticsearch/downloads.html#try) for
- installation guide. [Fluentd](https://www.fluentd.org/download) daemon can be configured to read the log files
-for Elasticsearch. See example of Fluentd `td-agent.conf` below:
+There are many solutions to routing log messages from Apache Brooklyn to Elasticsearch, either plugging in to the log4j subsystem
+or routing the log files from disk. [Fluentd](https://www.fluentd.org/download), with the following configuration in `td-agent.conf`,
+is a good simple way to forward content added to the log files:
```
<source>
@@ -135,6 +150,102 @@ for Elasticsearch. See example of Fluentd `td-agent.conf` below:
</match>
```
+#### Sizing and Rotating Logs
+
+Keeping log data obviously consumes disk storage, and serving the data requires memory.
+The log levels in Apache Brooklyn can be configured on a fine-grained log-category basis,
+and different levels and categories can be routed to different indexes with different
+retention policies.
+
+When designing your strategy for maintaining logs, some good rules of thumb are:
+
+* Allocate 4 GB RAM for a production ES instance plus 2 GB RAM for every TB of log data that is searchable
+* Consider a small number of tiers with different retention policies,
+ e.g. INFO and selected DEBUG messages (logins) to one index,
+ and all other DEBUG and lower messages sent to another index
+* Consider using rolling indices on a nightly or weekly basis, and an auto-close job and/or an auto-delete job to keep memory and disk usage at a steady state;
+ for example the INFO and selected DEBUG messages might go to an index which rotates weekly and is deleted or moved to cold storage after two years,
+ whereas DEBUG and lower messages might rotate daily and be deleted after a week
+* The amount of log data can vary depending how Apache Brooklyn is used, so monitor usage to get a feel for what is
+ maintainable, and put in place notifications if disk and/or memory usage become high
+* Review the logs and turn off unnecessary categories
+
+Instructions and links to assist with this are below.
+
+
+#### Index partitioning
+
+It’s possible to configure fluentd for sending the information to an index using an index name generated using datetime markers.
+This example will create and send the data to a new index every day:
+
+```
+<match td.apachebrokyn.*>
+ @type elasticsearch
+ hosts https://localhost:9200
+ user admin
+ password admin
+ ssl_verify false
+
+ include_timestamp true
+ index_name brooklyn-rotating-%Y.%m.%d
+ flush_interval 5s
+ <buffer tag, time>
+ timekey 60 # chunks per hours ("3600" also available)
+ flush_interval 5s
+ </buffer>
+</match>
+```
+
+Apache Brooklyn can be configured to use an index _pattern_ for querying, eg:
+
+ brooklyn.logbook.openSearchLogStore.index = brooklyn-rotating-*
+
+
+#### Index lifecycle management
+
+Policies also allow handling the lifecycle of the indexes.
+For example, to delete indexes after a period of time:
+
+```
+{
+ "policy": {
+ "description": "Delete workflow",
+ "default_state": "new",
+ "schema_version": 1,
+ "states": [
+ {
+ "name": "new",
+ "transitions": [
+ {
+ "state_name": "delete",
+ "conditions": {
+ "min_index_age": "60d"
+ }
+ }
+ ]
+ },
+ {
+ "name": "delete",
+ "actions": [
+ {
+ "delete": {}
+ }
+ ]
+ }
+ ],
+ "ism_template": {
+ "index_patterns": ["brooklyn-rotating*"],
+ "priority": 100
+ }
+ }
+}
+```
+
+With these building blocks, and others linked below, you can configure the retention policy that suits your environment,
+balancing the trade-off between data availability and resource usage.
+
+
+
## For More Information
The following resources may be useful when configuring logging:
@@ -144,3 +255,7 @@ The following resources may be useful when configuring logging:
* The [Logback Project](http://logback.qos.ch/) home page
* [Brooklyn Developer Guide]({{book.path.docs}}/dev/tips/logging.md) logging tips
* [OPS4J Pax Logging](https://ops4j1.jira.com/wiki/display/paxlogging/Configuration)
+* [Elasticsearch Best Practices](https://www.elastic.co/guide/en/elasticsearch/reference/7.x/best_practices.html)
+* [Elasticsearch Memory Usage](https://www.elastic.co/blog/significantly-decrease-your-elasticsearch-heap-memory-usage)
+* [OpenSearch Index Management](https://opensearch.org/docs/im-plugin/ism/index/) and [policies](https://opensearch.org/docs/im-plugin/ism/policies/)
+