You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@brooklyn.apache.org by tbouron <gi...@git.apache.org> on 2017/01/04 16:04:22 UTC

[GitHub] brooklyn-docs pull request #137: Update ELK docs

GitHub user tbouron opened a pull request:

    https://github.com/apache/brooklyn-docs/pull/137

    Update ELK docs

    This PR updates the advanced YAML doc with the latest version, available at https://github.com/brooklyncentral/brooklyn-elk. It also updates the links to the latest `*.bom` files

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/tbouron/brooklyn-docs fix/elk-docs

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/brooklyn-docs/pull/137.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #137
    
----
commit 3179423a15b788a512efdc19cece35e570c1ca58
Author: Thomas Bouron <th...@cloudsoftcorp.com>
Date:   2017-01-04T16:02:16Z

    Mirror docs available at https://github.com/brooklyncentral/brooklyn-elk and point links to the latest bom files

----


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] brooklyn-docs pull request #137: Update ELK docs

Posted by geomacy <gi...@git.apache.org>.

Github user geomacy commented on a diff in the pull request:

    https://github.com/apache/brooklyn-docs/pull/137#discussion_r94741580
  
    --- Diff: guide/yaml/advanced-example.md ---
    @@ -177,4 +211,41 @@ The ELK blueprint also contains a good example of a child entity.
       - type: logstash-child
     ~~~
     
    -In this example, a logstash-child is started as a child of the parent Tomcat server. The Tomcat server needs to be configured with a `children.startable.mode` to inform Brooklyn when to bring up the child. In this case we have selected background so that the child is disassociated from the parent entity, and late to specify that the parent entity should start before we start the child.
    +In this example, a logstash-child is started as a child of the parent Tomcat server. The Tomcat server needs 
    +to be configured with a `children.startable.mode` to inform Brooklyn when to bring up the child. In this case
    + we have selected background so that the child is disassociated from the parent entity, and late to specify that
    +  the parent entity should start before we start the child.
    +
    +The example also shows how to configure Logstash inputs and filters, if necessary, for a particular application, 
    +in this case Tomcat.
    +~~~yaml
    +- type: logstash-child
    +  name: Logstash
    +  brooklyn.config:
    +    logstash.elasticsearch.hosts: $brooklyn:entity("es").attributeWhenReady("urls.http.withBrackets")
    +    logstash.config.input:
    +      $brooklyn:formatString:
    +      - |
    +        input {
    +          file {
    +            path => "%s/logs/localhost_access_log.*"
    +            start_position => "beginning"
    +          }
    +        }
    +      - $brooklyn:entity("tomcat").attributeWhenReady("run.dir")
    +    logstash.config.filter: |
    +      filter {
    +        grok {
    +          match => { "message" => "%{COMBINEDAPACHELOG}" }
    +        }
    +        date {
    +          match => [ "timestamp" , "dd/MMM/yyyy:HH:mm:ss Z" ]
    +        }
    +      }
    +~~~
    +
    +
    +Configuring an appropriate visualisation on the Kibana server (access it via the URL on the summary tab for 
    +that entity) allows a dashboard to be created such as
    +
    +![kibana dashboard](logstash-snapshot.png)
    --- End diff --
    
    Need to add a copy of the logstash-snapshot.png file too.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] brooklyn-docs pull request #137: Update ELK docs

Posted by drigodwin <gi...@git.apache.org>.

Github user drigodwin commented on a diff in the pull request:

https://github.com/apache/brooklyn-docs/pull/137#discussion_r94747308

--- Diff: guide/yaml/advanced-example.md ---
@@ -28,83 +28,109 @@ For more about the ELK stack, please see the documentation [here](https://www.el

There are four blueprints that make up this application. Each of them are used to add one or more catalog items to Brooklyn. You can find them below:

-* [Elasticsearch](example_yaml/brooklyn-elasticsearch-catalog.bom)
-* [Logstash](example_yaml/brooklyn-logstash-catalog.bom)
-* [Kibana](example_yaml/brooklyn-kibana-catalog.bom)
-* [ELK](example_yaml/brooklyn-elk-catalog.bom)
+* [Elasticsearch](https://github.com/brooklyncentral/brooklyn-elk/blob/master/brooklyn-elasticsearch-catalog.bom)
+* [Logstash](https://github.com/brooklyncentral/brooklyn-elk/blob/master/brooklyn-logstash-catalog.bom)
+* [Kibana](https://github.com/brooklyncentral/brooklyn-elk/blob/master/brooklyn-kibana-catalog.bom)
+* [ELK](https://github.com/brooklyncentral/brooklyn-elk/blob/master/brooklyn-elk-catalog.bom)

#### Running the example
-First, add all four blueprints to the Brooklyn Catalog. This can be done by clicking the 'Catalog' tab, clicking the '+' symbol and pasting the YAML. Once this is done, click the 'Application' tab, then the '+' button to bring up the add application wizard. A new Catalog application will be available called 'ELK Stack'. Using the add application wizard, you should be able to deploy an ELK stack to a location of your choosing.
+First, add all four blueprints to the Brooklyn Catalog. This can be done by clicking the 'Catalog' tab, clicking the '+'
+ symbol and pasting the YAML. Once this is done, click the 'Application' tab, then the '+' button to bring up the add
+application wizard. A new Catalog application will be available called 'ELK Stack'. Using the add application wizard,
+you should be able to deploy an ELK stack to a location of your choosing. Alternatively use the `br` Brooklyn
+command line tool and add the files with `br catalog add`.

#### Exploring the example
After the application has been deployed, you can ensure it is working as expected by checking the following:

-* There is a Kibana sensor called `main.uri`, the value of which points to the Kibana front end. You can explore this front end, and observe the logs stored in Elasticsearch. Many Brooklyn applications have a `main.uri` set to point you in the right direction.
-* You can also use the Elasticsearch REST API to explore further. The Elasticsearch Cluster entity has a `urls.http.list` sensor. Using a host:port from that list you will be able to access the REST API. The following URL will give you the state of the cluster `http://<host:port>/_cluster/health?pretty=true`. As you can see the `number_of_nodes` is currently 2, indicating that the Elasticsearch nodes are communicating with each other.
+* There is a Kibana sensor called `main.uri`, the value of which points to the Kibana front end. You can explore this
+front end, and observe the logs stored in Elasticsearch. Many Brooklyn applications have a `main.uri` set to point you
+in the right direction.
+* You can also use the Elasticsearch REST API to explore further. The Elasticsearch Cluster entity has a `urls.http.list`
+sensor. Using a host:port from that list you will be able to access the REST API. The following URL will give you the
+state of the cluster `http://<host:port>/_cluster/health?pretty=true`. As you can see the `number_of_nodes` is
+currently 2, indicating that the Elasticsearch nodes are communicating with each other.

### Interesting Feature Spotlight
-We will mainly focus on the Elasticsearch blueprint, and will be clear when another blueprint is being discussed. This blueprint describes a cluster of Elasticsearch nodes. Clustering is a useful technique that is explained in more depth [here]({{site.path.guide}}/yaml/clusters.html).
+We will mainly focus on the Elasticsearch blueprint, and will be clear when another blueprint is being discussed. This blueprint describes a cluster of Elasticsearch nodes.

#### Provisioning Properties
-Our Elasticsearch blueprint has a few requirements of the location in which it is run. Firstly, it must be run on an Ubuntu machine as the example has been written specifically for this OS. Secondly, two ports must opened to ensure that the entities can be accessed from the outside world. Both of these requirements are configured via provisioning.properties as follows:
+Our Elasticsearch blueprint has a few requirements of the location in which it is run. Firstly, it must be run on an
+ Ubuntu machine as the example has been written specifically for this OS. Secondly, two ports must opened to ensure
+ that the entities can be accessed from the outside world. Both of these requirements are configured via provisioning
+ .properties as follows:
--- End diff --

Should there be a space in `provisioning.properties` here? Perhaps put backticks round it?

---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] brooklyn-docs pull request #137: Update ELK docs

Posted by geomacy <gi...@git.apache.org>.

Github user geomacy commented on a diff in the pull request:

https://github.com/apache/brooklyn-docs/pull/137#discussion_r94743700

--- Diff: guide/yaml/advanced-example.md ---
@@ -177,4 +211,41 @@ The ELK blueprint also contains a good example of a child entity.
- type: logstash-child
~~~

-In this example, a logstash-child is started as a child of the parent Tomcat server. The Tomcat server needs to be configured with a `children.startable.mode` to inform Brooklyn when to bring up the child. In this case we have selected background so that the child is disassociated from the parent entity, and late to specify that the parent entity should start before we start the child.
+In this example, a logstash-child is started as a child of the parent Tomcat server. The Tomcat server needs
+to be configured with a `children.startable.mode` to inform Brooklyn when to bring up the child. In this case
+ we have selected background so that the child is disassociated from the parent entity, and late to specify that
+ the parent entity should start before we start the child.
+
+The example also shows how to configure Logstash inputs and filters, if necessary, for a particular application,
+in this case Tomcat.
+~~~yaml
--- End diff --

it looks like you need a blank line in front of the `~~~` to have the block presented correctly

[GitHub] brooklyn-docs pull request #137: Update ELK docs

Posted by geomacy <gi...@git.apache.org>.

Github user geomacy commented on a diff in the pull request:

https://github.com/apache/brooklyn-docs/pull/137#discussion_r94742789

--- Diff: guide/yaml/advanced-example.md ---
@@ -28,83 +28,108 @@ For more about the ELK stack, please see the documentation [here](https://www.el

There are four blueprints that make up this application. Each of them are used to add one or more catalog items to Brooklyn. You can find them below:

#### Exploring the example
After the application has been deployed, you can ensure it is working as expected by checking the following:

~~~yaml
-provisioning.properties:
- osFamily: ubuntu
- inboundPorts:
+brooklyn.config:
+ elasticsearch.http.port: 9220
+ elasticsearch.tcp.port: 9330
+ provisioning.properties:
+ osFamily: ubuntu
+ inboundPorts:
- $brooklyn:config("elasticsearch.http.port")
- $brooklyn:config("elasticsearch.tcp.port")
~~~

#### VanillaSoftwareProcess
-When composing a YAML blueprint, the VanillaSoftwareProcess is a very useful entity to be aware of. A VanillaSoftwareProcess will instruct Brooklyn to provision an instance, and run a series of shell commands to setup, run, monitor and teardown your program. The commands are specified as configuration on the VanillaSoftwareProcess and there are several available. We will spotlight a few now. To simplify this blueprint, we have specified ubuntu only installs so that our commands can be tailored to this system (e.g. use apt-get rather than yum).
+When composing a YAML blueprint, the VanillaSoftwareProcess is a very useful entity to be aware of.
+A VanillaSoftwareProcess will instruct Brooklyn to provision an instance, and run a series of shell
+commands to setup, run, monitor and teardown your program. The commands are specified as configuration
+on the VanillaSoftwareProcess and there are several available. We will spotlight a few now. To simplify
+ this blueprint, we have specified ubuntu only installs so that our commands can be tailored to this
+ system (e.g. use apt-get rather than yum).

##### Customize Command
-The Customize Command is run after the application has been installed but before it is run. It is the perfect place to create and amend config files. Please refer to the following section of the Elasticsearch blueprint:
+The Customize Command is run after the application has been installed but before it is run. It is the perfect
+ place to create and amend config files. Please refer to the following section of the Elasticsearch blueprint:

~~~yaml
customize.command: |
- $brooklyn:formatString("
- sudo rm -fr sudo tee /etc/elasticsearch/elasticsearch.yml;
- echo discovery.zen.ping.multicast.enabled: false | sudo tee -a /etc/elasticsearch/elasticsearch.yml;
- echo discovery.zen.ping.unicast.enabled: true | sudo tee -a /etc/elasticsearch/elasticsearch.yml;
- echo 'discovery.zen.ping.unicast.hosts: %s' | sudo tee -a /etc/elasticsearch/elasticsearch.yml;
- echo http.port: %s | sudo tee -a /etc/elasticsearch/elasticsearch.yml;
- echo transport.tcp.port: %s | sudo tee -a /etc/elasticsearch/elasticsearch.yml;
- ",
- $brooklyn:component("parent", "").attributeWhenReady("urls.tcp.withBrackets"),
- $brooklyn:config("elasticsearch.http.port"),
- $brooklyn:config("elasticsearch.tcp.port")
- )
+ sudo rm -fr sudo tee /etc/elasticsearch/elasticsearch.yml
+ echo discovery.zen.ping.multicast.enabled: false | sudo tee -a /etc/elasticsearch/elasticsearch.yml
+ echo discovery.zen.ping.unicast.enabled: true | sudo tee -a /etc/elasticsearch/elasticsearch.yml
+ echo discovery.zen.ping.unicast.hosts: ${URLS_WITH_BRACKETS} | sudo tee -a /etc/elasticsearch/elasticsearch.yml
+ echo http.port: ${ES_HTTP_PORT} | sudo tee -a /etc/elasticsearch/elasticsearch.yml
+ echo transport.tcp.port: ${ES_TCP_PORT} | sudo tee -a /etc/elasticsearch/elasticsearch.yml
+ echo network.host: ${IP_ADDRESS} | sudo tee -a /etc/elasticsearch/elasticsearch.yml
~~~
-The purpose of this section is to create a YAML file with all of the required configuration. We use the YAML literal style `|` indicator to write a multi line command. We then use `$brooklyn:formatString` notation to build the string from configuration. We start our series of commands by using the `rm` command to remove the previous config file. We then use `echo` and `tee` to create the new config file and insert the config. Part of the configuration is a list of all hosts that is set on the parent entity- this is done by using a combination of the `component` and `attributeWhenReady` DSL commands. More on how this is generated later.
+The purpose of this section is to create a YAML file with all of the required configuration. We use the YAML
+literal style `|` indicator to write a multi line command. We then use `$brooklyn:formatString` notation to
--- End diff --

Need to delete the sentence about 'formatString', it's not used now.

[GitHub] brooklyn-docs pull request #137: Update ELK docs

Posted by tbouron <gi...@git.apache.org>.

Github user tbouron commented on a diff in the pull request:

    https://github.com/apache/brooklyn-docs/pull/137#discussion_r94744256
  
    --- Diff: guide/yaml/advanced-example.md ---
    @@ -177,4 +211,41 @@ The ELK blueprint also contains a good example of a child entity.
       - type: logstash-child
     ~~~
     
    -In this example, a logstash-child is started as a child of the parent Tomcat server. The Tomcat server needs to be configured with a `children.startable.mode` to inform Brooklyn when to bring up the child. In this case we have selected background so that the child is disassociated from the parent entity, and late to specify that the parent entity should start before we start the child.
    +In this example, a logstash-child is started as a child of the parent Tomcat server. The Tomcat server needs 
    +to be configured with a `children.startable.mode` to inform Brooklyn when to bring up the child. In this case
    + we have selected background so that the child is disassociated from the parent entity, and late to specify that
    +  the parent entity should start before we start the child.
    +
    +The example also shows how to configure Logstash inputs and filters, if necessary, for a particular application, 
    +in this case Tomcat.
    +~~~yaml
    +- type: logstash-child
    +  name: Logstash
    +  brooklyn.config:
    +    logstash.elasticsearch.hosts: $brooklyn:entity("es").attributeWhenReady("urls.http.withBrackets")
    +    logstash.config.input:
    +      $brooklyn:formatString:
    +      - |
    +        input {
    +          file {
    +            path => "%s/logs/localhost_access_log.*"
    +            start_position => "beginning"
    +          }
    +        }
    +      - $brooklyn:entity("tomcat").attributeWhenReady("run.dir")
    +    logstash.config.filter: |
    +      filter {
    +        grok {
    +          match => { "message" => "%{COMBINEDAPACHELOG}" }
    +        }
    +        date {
    +          match => [ "timestamp" , "dd/MMM/yyyy:HH:mm:ss Z" ]
    +        }
    +      }
    +~~~
    +
    +
    +Configuring an appropriate visualisation on the Kibana server (access it via the URL on the summary tab for 
    +that entity) allows a dashboard to be created such as
    +
    +![kibana dashboard](logstash-snapshot.png)
    --- End diff --
    
    Good catch, I'll add it



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] brooklyn-docs pull request #137: Update ELK docs

Posted by asfgit <gi...@git.apache.org>.

Github user asfgit closed the pull request at:

    https://github.com/apache/brooklyn-docs/pull/137


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] brooklyn-docs issue #137: Update ELK docs

Posted by tbouron <gi...@git.apache.org>.

Github user tbouron commented on the issue:

    https://github.com/apache/brooklyn-docs/pull/137
  
    @drigodwin Updated


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] brooklyn-docs issue #137: Update ELK docs

Posted by tbouron <gi...@git.apache.org>.

Github user tbouron commented on the issue:

    https://github.com/apache/brooklyn-docs/pull/137
  
    Thanks for the review @geomacy, I addressed your comments :)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---