You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@brooklyn.apache.org by he...@apache.org on 2015/03/02 17:15:06 UTC
[4/5] incubator-brooklyn git commit: Docs: adds to ops guide
Docs: adds to ops guide
- Renames cli.md to “Launching Brooklyn”
- Adds pages/sections for:
- requirements (server/OS/etc)
- cloud credentials / setup
- state backup
- log file backup
- more persistence configuration options
- more jclouds/location config options
Project: http://git-wip-us.apache.org/repos/asf/incubator-brooklyn/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-brooklyn/commit/1422f46a
Tree: http://git-wip-us.apache.org/repos/asf/incubator-brooklyn/tree/1422f46a
Diff: http://git-wip-us.apache.org/repos/asf/incubator-brooklyn/diff/1422f46a
Branch: refs/heads/master
Commit: 1422f46a005cf7b2ad0d4784446a76096f64721b
Parents: 9debe45
Author: Aled Sage <al...@gmail.com>
Authored: Tue Feb 24 12:05:48 2015 +0000
Committer: Aled Sage <al...@gmail.com>
Committed: Tue Feb 24 12:05:48 2015 +0000
----------------------------------------------------------------------
docs/guide/ops/cli.md | 143 ---------------
docs/guide/ops/index.md | 3 +-
docs/guide/ops/install-on-server.md | 12 +-
docs/guide/ops/launch.md | 198 +++++++++++++++++++++
docs/guide/ops/locations/cloud-credentials.md | 85 +++++++++
docs/guide/ops/locations/index.md | 62 +++++--
docs/guide/ops/logging.md | 28 ++-
docs/guide/ops/persistence/index.md | 105 ++++++++++-
docs/guide/ops/requirements.md | 59 ++++++
9 files changed, 530 insertions(+), 165 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-brooklyn/blob/1422f46a/docs/guide/ops/cli.md
----------------------------------------------------------------------
diff --git a/docs/guide/ops/cli.md b/docs/guide/ops/cli.md
deleted file mode 100644
index 3e44c62..0000000
--- a/docs/guide/ops/cli.md
+++ /dev/null
@@ -1,143 +0,0 @@
----
-title: CLI
-layout: website-normal
----
-
-To launch Brooklyn, from the directory where Brooklyn is unpacked, run:
-
-{% highlight bash %}
-% bin/brooklyn launch
-{% endhighlight %}
-
-With no configuration, this will launch the Brooklyn web console and REST API on [`http://localhost:8081/`](http://localhost:8081/).
-No password is set, but the server is listening only on the loopback network interface for security.
-Once [security is configured](brooklyn_properties.html), Brooklyn will listen on all network interfaces by default.
-
-You may wish to [add Brooklyn to your path](#path-setup);
-assuming you've done this, to get information the supported CLI options
-at any time, just run `brooklyn help`:
-
-{% highlight bash %}
-% bin/brooklyn help
-
-usage: brooklyn [(-q | --quiet)] [(-v | --verbose)] <command> [<args>]
-
-The most commonly used brooklyn commands are:
- help Display help information about brooklyn
- info Display information about brooklyn
- launch Starts a brooklyn application. Note that a BROOKLYN_CLASSPATH environment variable needs to be set up beforehand to point to the user application classpath.
-
-See 'brooklyn help <command>' for more information on a specific command.
-{% endhighlight %}
-
-
-
-## Configuration
-
-Brooklyn can read configuration from a variety of places:
-
-* the file `~/.brooklyn/brooklyn.properties` (unless `--noGlobalBrooklynProperties` is specified)
-* another file, if the `--localBrooklynProperties <local brooklyn.properties file>`
-* ``-D`` defines on the brooklyn (java) command-line
-* shell environment variables
-
-These properties are described in more detail [here](brooklyn_properties.html).
-
-
-
-## Path Setup
-
-In order to have easy access to the cli it is useful to configure the PATH environment variable to also point to the cli's bin directory:
-
-{% highlight bash %}
-BROOKLYN_HOME=/path/to/brooklyn/
-export PATH=$PATH:$BROOKLYN_HOME/usage/dist/target/brooklyn-dist/bin/
-{% endhighlight %}
-
-
-## Running from a Source Build
-
-Here is an example of the commands you might run to get the Brooklyn code,
-compile it and launch an application:
-
-{% highlight bash %}
-git clone https://github.com/apache/incubator-brooklyn.git
-cd brooklyn
-mvn clean install -DskipTests
-BROOKLYN_HOME=$(pwd)
-export PATH=${PATH}:${BROOKLYN_HOME}/usage/dist/target/brooklyn-dist/bin/
-export BROOKLYN_CLASSPATH=${BROOKLYN_HOME}/examples/simple-web-cluster/target/classes
-brooklyn launch --app brooklyn.demo.SingleWebServerExample --location localhost
-{% endhighlight %}
-
-
-## Extending the Classpath
-
-You can add things to the Brooklyn classpath in a number of ways:
-
-* Add ``.jar`` files to Brooklyn's ``./lib/dropins/`` directory. These are added at the end of the classpath.
-* Add ``.jar`` files to Brooklyn's ``./lib/patch/`` directory. These are added at the front of the classpath.
-* Add resources to Brooklyn's ``./conf/`` directory. This directory is at the very front of the classpath.
-* Use the ``BROOKLYN_CLASSPATH`` environment variable. If set, this is prepended to the Brooklyn classpath.
-
-
-## Cloud Explorer
-
-The `brooklyn` command line tool includes support for querying (and managing) cloud
-compute resources and blob-store resources.
-
-For example, `brooklyn cloud-compute list-instances --location aws-ec2:eu-west-1`
-will use the AWS credentials from `brooklyn.properties` and list the VM instances
-running in the given EC2 region.
-
-Use `brooklyn help` and `brooklyn help cloud-compute` to find out more information.
-
-This functionality is not intended as a generic cloud management CLI, but instead
-solves specific Brooklyn use-cases. The main use-case is discovering the valid
-configuration options on a given cloud, such as for `imageId` and `hardwareId`.
-
-
-### Cloud Compute
-
-The command `brooklyn cloud-compute` has the following options:
-
-* `list-images`: lists VM images within the given cloud, which can be chosen when
- provisioning new VMs.
- This is useful for finding the possible values for the `imageId` configuration.
-
-* `get-image <imageId1> <imageId2> ...`: retrieves metadata about the specific images.
-
-* `list-hardware-profiles`: lists the ids and the details of the hardware profiles
- available when provisioning.
- This is useful for finding the possible values for the `hardwareId` configuration.
-
-* `default-template`: retrieves metadata about the image and hardware profile that will
- be used by Brooklyn for that location, if no additional configuration options
- are supplied.
-
-* `list-instances`: lists the VM instances within the given cloud.
-
-* `terminate-instances <instanceId1> <instanceId2> ...`: Terminates the instances with
- the given ids.
-
-
-## Blob Store
-
-The command `brooklyn cloud-blobstore` is used to access a given object store, such as S3
-or Swift. It has the following options:
-
-* `list-containers`: lists the containers (i.e. buckets in S3 terminology) within the
- given object store.
-
-* `list-container <containerName>`: lists all the blobs (i.e. objects) contained within
- the given container.
-
-* `blob --container <containerName> --blob <blobName>`: retrieves the given blob
- (i.e. object), including metadata and its contents.
-
-
-## Other Topics
-
-The CLI arguments for [persistence and HA](persistence/)
-are described separately,
-as is [detailed configuration](brooklyn_properties.html).
http://git-wip-us.apache.org/repos/asf/incubator-brooklyn/blob/1422f46a/docs/guide/ops/index.md
----------------------------------------------------------------------
diff --git a/docs/guide/ops/index.md b/docs/guide/ops/index.md
index 6cfcb49..dae3071 100644
--- a/docs/guide/ops/index.md
+++ b/docs/guide/ops/index.md
@@ -2,8 +2,9 @@
title: Operations
layout: website-normal
children:
+- requirements.md
- install-on-server.md
-- cli.md
+- launch.md
- brooklyn_properties.md
- locations/
- persistence/
http://git-wip-us.apache.org/repos/asf/incubator-brooklyn/blob/1422f46a/docs/guide/ops/install-on-server.md
----------------------------------------------------------------------
diff --git a/docs/guide/ops/install-on-server.md b/docs/guide/ops/install-on-server.md
index 6904504..d4711db 100644
--- a/docs/guide/ops/install-on-server.md
+++ b/docs/guide/ops/install-on-server.md
@@ -1,6 +1,6 @@
---
layout: website-normal
-title: Installing on a Server
+title: Installing Brooklyn
---
{% include fields.md %}
@@ -10,7 +10,9 @@ Here we present two *alternatives* to install Brooklyn:
- [Running the *installation script*](#script)
- [Manual installation](#manual)
+
## <a id="script"></a> Running the installation script
+
There is a simple bash script available to help with the installation process.
#### Script prerequisites
@@ -29,6 +31,7 @@ $ chmod +x ./brooklyn-install.sh
$ ./brooklyn-install.sh -s -r <your-server-ip>
{% endhighlight %}
+
## <a id="manual"></a> Manual installation
1. [Set up the prerequisites](#prerequisites)
@@ -37,6 +40,7 @@ $ ./brooklyn-install.sh -s -r <your-server-ip>
1. [Configuring catalog.xml](#configuring-catalog)
1. [Test the installation](#confirm)
+
### <a id="prerequisites"></a>Set up the prerequisites
Before installing Apache Brooklyn, it is recommented to configure the host as follows.
@@ -79,7 +83,9 @@ $ BROOKLYN_DIR="$(pwd)"
$ export PATH=$PATH:$BROOKLYN_DIR/bin/
{% endhighlight %}
+
### <a id="configuring-properties"></a>Configuring brooklyn.properties
+
Brooklyn deploys applications to Locations. *Locations* can be clouds, machines with fixed IPs or localhost (for testing).
By default Brooklyn loads configuration parameters (including credentials for any cloud accounts) from
@@ -98,7 +104,9 @@ $ chmod 600 ~/.brooklyn/brooklyn.properties
You may need to edit `~/.brooklyn/brooklyn.properties` to ensure that brooklyn can access cloud locations for application deployment.
+
### <a id="configuring-catalog"></a>Configuring catalog.xml
+
By default Brooklyn loads the catalog of available application components and services from
`~/.brooklyn/catalog.xml`.
@@ -110,7 +118,9 @@ The `catalog.xml` is the application blueprint catalog. The above example file c
You may need to edit `~/.brooklyn/catalog.xml` to update links to any resources for download.
+
### <a id="confirm"></a>Confirm installation
+
We can do a quick test drive by launching Brooklyn:
{% highlight bash %}
http://git-wip-us.apache.org/repos/asf/incubator-brooklyn/blob/1422f46a/docs/guide/ops/launch.md
----------------------------------------------------------------------
diff --git a/docs/guide/ops/launch.md b/docs/guide/ops/launch.md
new file mode 100644
index 0000000..45d67bd
--- /dev/null
+++ b/docs/guide/ops/launch.md
@@ -0,0 +1,198 @@
+---
+title: Launching Brooklyn
+layout: website-normal
+---
+
+## Launch command
+
+To launch Brooklyn, from the directory where Brooklyn is unpacked, run:
+
+{% highlight bash %}
+% nohup bin/brooklyn launch &
+{% endhighlight %}
+
+With no configuration, this will launch the Brooklyn web console and REST API on [`http://localhost:8081/`](http://localhost:8081/).
+No password is set, but the server is listening only on the loopback network interface for security.
+Once [security is configured](brooklyn_properties.html), Brooklyn will listen on all network interfaces by default.
+
+You may wish to [add Brooklyn to your path](#path-setup);
+assuming you've done this, to get information the supported CLI options
+at any time, just run `brooklyn help`:
+
+{% highlight bash %}
+% bin/brooklyn help
+
+usage: brooklyn [(-q | --quiet)] [(-v | --verbose)] <command> [<args>]
+
+The most commonly used brooklyn commands are:
+ help Display help information about brooklyn
+ info Display information about brooklyn
+ launch Starts a brooklyn application. Note that a BROOKLYN_CLASSPATH environment variable needs to be set up beforehand to point to the user application classpath.
+
+See 'brooklyn help <command>' for more information on a specific command.
+{% endhighlight %}
+
+It is important that Brooklyn is launched with either `nohup ... &` or `... & disown`, to ensure
+it keeps running after the shell terminates.
+
+
+### Other CLI Arguments
+
+The CLI arguments for [persistence and HA](persistence/) are described separately.
+
+
+### Path Setup
+
+In order to have easy access to the cli it is useful to configure the PATH environment
+variable to also point to the cli's bin directory:
+
+{% highlight bash %}
+BROOKLYN_HOME=/path/to/brooklyn/
+export PATH=$PATH:$BROOKLYN_HOME/usage/dist/target/brooklyn-dist/bin/
+{% endhighlight %}
+
+
+### Memory Usage
+
+The amount of memory required by the Brooklyn process depends on the usage
+- for example the number of entities/VMs under management.
+
+For a standard Brooklyn deployment, the defaults are to start with 256m, and to grow to 1g of memory.
+These numbers can be overridden by setting the environment variable `JAVA_OPTS` before launching
+the `brooklyn script`:
+
+ JAVA_OPTS=-Xms1g -Xmx1g -XX:MaxPermSize=256m
+
+Brooklyn stores a task history in-memory using [soft references](http://docs.oracle.com/javase/7/docs/api/java/lang/ref/SoftReference.html).
+This means that, once the task history is large, Brooklyn will continually use the maximum allocated
+memory. It will only expunge tasks from memory when this space is required for other objects within the
+Brooklyn process.
+
+
+## Configuration
+
+### Configuration Files
+
+Brooklyn reads configuration from a variety of places. It aggregates the configuration.
+The list below shows increasing precedence (i.e. the later ones will override values
+from earlier ones, if exactly the same property is specified multiple times).
+
+1. `classpath://brooklyn/location-metadata.properties` is shipped as part of Brooklyn, containing
+ generic metadata such as jurisdiction and geographic information about Cloud providers.
+1. The file `~/.brooklyn/location-metadata.properties` (unless `--noGlobalBrooklynProperties` is specified).
+ This is intended to contain custom metadata about additional locations.
+1. The file `~/.brooklyn/brooklyn.properties` (unless `--noGlobalBrooklynProperties` is specified).
+1. Another properties file, if the `--localBrooklynProperties <local brooklyn.properties file>` is specified.
+1. Shell environment variables
+1. System properties, supplied with ``-D`` on the brooklyn (Java) command-line.
+
+These properties are described in more detail [here](brooklyn_properties.html).
+
+
+### Extending the Classpath
+
+The default Brooklyn directory structure includes:
+
+* `./conf/`: for configuration resources.
+* `./lib/patch/`: for Jar files containing patches.
+* `./lib/brooklyn/`: for the brooklyn libraries.
+* `./lib/dropins/`: for additional Jars.
+
+Resources added to `conf/` will be available on the classpath.
+
+A patch can be applied by adding a Jar to the `lib/patch/` directory, and restarting Brooklyn.
+All jars in this directory will be at the head of the classpath.
+
+Additional Jars should be added to `lib/dropins/`, prior to starting Brooklyn. These jars will
+be at the end of the classpath.
+
+The initial classpath, as set in the `brooklyn` script, is:
+
+ conf:lib/patch/*:lib/brooklyn/*:lib/dropins/*
+
+Additional entries can be added at the head of the classpath by setting the environment variable
+`BROOKLYN_CLASSPATH` before running the `brooklyn` script.
+
+
+### Replacing the web-console
+
+*Work in progress.*
+
+The Brooklyn web-console is loaded from the classpath as the resource `classpath://brooklyn.war`.
+
+To replace this, an alternative WAR with that name can be added at the head of the classpath.
+However, this approach is likely to change in a future release - consider this feature as "beta".
+
+
+## Cloud Explorer
+
+The `brooklyn` command line tool includes support for querying (and managing) cloud
+compute resources and blob-store resources.
+
+For example, `brooklyn cloud-compute list-instances --location aws-ec2:eu-west-1`
+will use the AWS credentials from `brooklyn.properties` and list the VM instances
+running in the given EC2 region.
+
+Use `brooklyn help` and `brooklyn help cloud-compute` to find out more information.
+
+This functionality is not intended as a generic cloud management CLI, but instead
+solves specific Brooklyn use-cases. The main use-case is discovering the valid
+configuration options on a given cloud, such as for `imageId` and `hardwareId`.
+
+
+### Cloud Compute
+
+The command `brooklyn cloud-compute` has the following options:
+
+* `list-images`: lists VM images within the given cloud, which can be chosen when
+ provisioning new VMs.
+ This is useful for finding the possible values for the `imageId` configuration.
+
+* `get-image <imageId1> <imageId2> ...`: retrieves metadata about the specific images.
+
+* `list-hardware-profiles`: lists the ids and the details of the hardware profiles
+ available when provisioning.
+ This is useful for finding the possible values for the `hardwareId` configuration.
+
+* `default-template`: retrieves metadata about the image and hardware profile that will
+ be used by Brooklyn for that location, if no additional configuration options
+ are supplied.
+
+* `list-instances`: lists the VM instances within the given cloud.
+
+* `terminate-instances <instanceId1> <instanceId2> ...`: Terminates the instances with
+ the given ids.
+
+
+### Blob Store
+
+The command `brooklyn cloud-blobstore` is used to access a given object store, such as S3
+or Swift. It has the following options:
+
+* `list-containers`: lists the containers (i.e. buckets in S3 terminology) within the
+ given object store.
+
+* `list-container <containerName>`: lists all the blobs (i.e. objects) contained within
+ the given container.
+
+* `blob --container <containerName> --blob <blobName>`: retrieves the given blob
+ (i.e. object), including metadata and its contents.
+
+
+## Running from a Source Build
+
+Here is an example of the commands you might run to get the Brooklyn code,
+compile it and launch an application:
+
+{% highlight bash %}
+git clone https://github.com/apache/incubator-brooklyn.git
+cd brooklyn
+mvn clean install -DskipTests
+BROOKLYN_HOME=$(pwd)
+export PATH=${PATH}:${BROOKLYN_HOME}/usage/dist/target/brooklyn-dist/bin/
+export BROOKLYN_CLASSPATH=${BROOKLYN_HOME}/examples/simple-web-cluster/target/classes
+nohup brooklyn launch --app brooklyn.demo.SingleWebServerExample --location localhost &
+{% endhighlight %}
+
+
+
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/incubator-brooklyn/blob/1422f46a/docs/guide/ops/locations/cloud-credentials.md
----------------------------------------------------------------------
diff --git a/docs/guide/ops/locations/cloud-credentials.md b/docs/guide/ops/locations/cloud-credentials.md
new file mode 100644
index 0000000..40cc108
--- /dev/null
+++ b/docs/guide/ops/locations/cloud-credentials.md
@@ -0,0 +1,85 @@
+---
+title: Cloud Setup
+layout: website-normal
+---
+
+To connect to a Cloud, Brooklyn requires appropriate credentials. These comprise the "identity" and
+"credential" in Brooklyn terminology.
+
+For private clouds (and for some clouds being targeted using a standard API), the "endpoint"
+must also be specified, which is the cloud's URL.
+
+The [jclouds guides](https://jclouds.apache.org/guides) includes documentation on configuring
+different clouds.
+
+
+## AWS
+
+### Credentials
+
+AWS has an "access key" and a "secret key", which correspond to Brooklyn's identity and credential
+respectively.
+
+These keys are the way for any programmatic mechanism to access the AWS API.
+
+To generate an access key and a secret key, see [jclouds instructions](http://jclouds.apache.org/guides/aws)
+and [AWS IAM instructions](http://docs.aws.amazon.com/IAM/latest/UserGuide/ManagingCredentials.html).
+
+An example of the expected format is shown below:
+
+ brooklyn.location.jclouds.aws-ec2.identity=ABCDEFGHIJKLMNOPQRST
+ brooklyn.location.jclouds.aws-ec2.credential=abcdefghijklmnopqrstu+vwxyzabcdefghijklm
+
+
+### Tidying up after jclouds
+
+Security groups are not always deleted by jclouds. This is due to a limitation in AWS (see
+https://issues.apache.org/jira/browse/JCLOUDS-207). In brief, AWS prevents the security group
+being deleted until there are no VMs using it. However, there is eventual consistency for
+recording which VMs still reference those security groups. After deleting the VM, it can sometimes
+take several minutes before the security group can be deleted. jclouds retries for 3 seconds, but
+does not block for longer.
+
+There is utility written by Cloudsoft for deleting these unused resources:
+http://www.cloudsoftcorp.com/blog/2013/03/tidying-up-after-jclouds.
+
+
+## Google Compute Engine
+
+### Credentials
+
+Google Compute Engine (GCE) uses a service account e-mail address for the identity, and a private key
+as the credential.
+
+To obtain these from GCE, see the [jclouds instructions](https://jclouds.apache.org/guides/google).
+
+An example of the expected format is shown below (note the credential is one long line,
+with `\n` to represent the new line characters):
+
+ brooklyn.location.jclouds.google-compute-engine.identity=123456789012@developer.gserviceaccount.com
+ brooklyn.location.jclouds.google-compute-engine.credential=-----BEGIN RSA PRIVATE KEY-----\nabcdefghijklmnopqrstuvwxyznabcdefghijk/lmnopqrstuvwxyzabcdefghij\nabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghij+lm\nnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklm\nnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxy\nzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijk\nlmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvw\nxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghi\njklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstu\nvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefg\nhijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrs\ntuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcde\nfghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvw\n-----END RSA PRIVATE KEY-----
+
+
+### Quotas
+
+GCE accounts can have low default [quotas](https://cloud.google.com/compute/docs/resource-quotas).
+
+It is easy to requesta quota increase by submitting a [quota increase form](https://support.google.com/cloud/answer/6075746?hl=en).
+
+
+### Networks
+
+GCE accounts often have a limit to the number of networks that can be created. One work around
+is to manually create a network with the required open ports, and to refer to that named network
+in Brooklyn's location configuration.
+
+To create a network, see [GCE network instructions](https://cloud.google.com/compute/docs/networking#networks_1).
+
+For example, for dev/demo purposes an "everything" network could be created that opens all ports.
+
+| Name | everything |
+| Description | opens all tcp ports |
+| Source IP Ranges | 0.0.0.0/0 |
+| Allowed protocols and ports | tcp:0-65535 and udp:0-65535 |
+
+
http://git-wip-us.apache.org/repos/asf/incubator-brooklyn/blob/1422f46a/docs/guide/ops/locations/index.md
----------------------------------------------------------------------
diff --git a/docs/guide/ops/locations/index.md b/docs/guide/ops/locations/index.md
index 235a552..18c9810 100644
--- a/docs/guide/ops/locations/index.md
+++ b/docs/guide/ops/locations/index.md
@@ -6,6 +6,7 @@ children:
- { section: Inheritance and Named Locations, title: Named Locations }
- { section: Localhost }
- { section: BYON }
+- cloud-credentials.md
- more-locations.md
- ssh-keys.md
---
@@ -72,14 +73,22 @@ The credentials for the initial OS log on are typically discovered from the clou
but in some environments this is not possible.
The keys `loginUser` and either `loginUser.password` or `loginUser.privateKeyFile` can be used to force
Brooklyn to use specific credentials for the initial login to a cloud-provisioned machine.
-(This is particularly useful when using a custom image templates where the cloud-side account management logic is not enabled.)
+
+(This custom login is particularly useful when using a custom image templates where the cloud-side account
+management logic is not enabled. For example, a vCD template can have guest customization that will change
+the root password. This setting tells AMP to only use the given password, rather than the initial
+randomly generated password that vCD returns. Without this property, there is a race for such templates:
+does Brooklyn manage to create the admin user before the guest customization changes the login and reboots,
+or is the password reset first (the latter means Brooklyn can never ssh to the VM). With this property,
+Brooklyn will always wait for guest customization to complete before it is able to ssh at all. In such
+cases, it is also recommended to use `useJcloudsSshInit=false`.)
Following a successful logon, Brooklyn performs the following steps to configure the machine:
-1. to create a new user with the same name as the user `brooklyn` is running as locally
- (this can be overridden with `user`, below)
+1. creates a new user with the same name as the user `brooklyn` is running as locally
+ (this can be overridden with `user`, below).
-1. to install the local user's `~/.ssh/id_rsa.pub` as an `authorized_keys` on the new machine,
+1. install the local user's `~/.ssh/id_rsa.pub` as an `authorized_keys` on the new machine,
to make it easy for the operator to `ssh` in
(override with `privateKeyFile`; or if there is no `id_{r,d}sa{,.pub}` an ad hoc keypair will be generated;
if there is a passphrase on the key, this must be supplied)
@@ -128,6 +137,24 @@ For more keys and more detail on the keys below, see
before making the `Location` available to entities,
optionally also using `setup.script.vars` (set as `key1:value1,key2:value2`)
+- Use `openIptables=true` to automatically configure `iptables`, to open the TCP ports required by
+ the software process. One can alternatively use `stopIptables=true` to entirely stop the
+ iptables service.
+
+- Use `installDevUrandom=true` to fall back to using `/dev/urandom` rather than `/dev/random`. This setting
+ is useful for cloud VMs where there is not enough random entropy, which can cause `/dev/random` to be
+ extremely slow (causing `ssh` to be extremely slow to respond).
+
+- Use `useJcloudsSshInit=false` to disable the use of the native jclouds support for initial commands executed
+ on the VM (e.g. for creating new users, setting root passwords, etc.). Instead, Brooklyn's ssh support will
+ be used. Timeouts and retries are more configurable within Brooklyn itself. Therefore this option is particularly
+ recommended when the VM startup is unusual (for example, if guest customizations will cause reboots and/or will
+ change login credentials).
+
+- Use `brooklyn.ssh.config.noDeleteAfterExec=true` can be used during dev/test. This prevents the scripts executed
+ on the VMs from being deleted on completion. This can help with debugging some issues. However, the contents of the
+ scripts and the stdout/stderr of their execution is also available in the AMP debug log.
+
###### VM Creation
@@ -150,12 +177,13 @@ For more keys and more detail on the keys below, see
- User metadata can be attached, using the syntax ``userMetadata=key=value,key2="value 2"``
- You can specify the number of attempts Brooklyn should make to create
- machines with ``machineCreateAttempts`` (jclouds only). This is useful for
+ machines with ``machineCreateAttempts`` (jclouds only). This is extremely useful for
working around the rare occasions in which cloud providers give machines that
are dead on arrival.
-- If you want to investigate failures, set `destroyOnFailure` to `false`
+- If you want to investigate failures, set `destroyOnFailure=false`
to keep failed VM's around. (You'll have to manually clean them up.)
+ The default is false: if a VM fails to start, or is never ssh'able, then the VM will be terminated.
### Inheritance and Named Locations
@@ -191,20 +219,21 @@ brooklyn.location.named.AWS\ Virginia\ Large\ Centos.user=root
brooklyn.location.named.AWS\ Virginia\ Large\ Centos.minRam=4096
{% endhighlight %}
-If you are confused about inheritance order, the following test may be useful:
+The precedence for configuration defined at different levels is that the most value
+defined in the most specific context will apply.
+For example, in the example below the config key is repeatedly overridden. If you deploy
+`location: named:my-aws`, Brooklyn will get `VAL5` or `KEY`:
+
{% highlight bash %}
-brooklyn.location.jclouds.KEY=VAL1
-brooklyn.location.jclouds.aws-ec2.KEY=VAL2
-brooklyn.location.named.my-aws=jclouds:aws-ec2
-brooklyn.location.named.my-aws.KEY=VAL3
+brooklyn.location.KEY=VAL1
+brooklyn.location.jclouds.KEY=VAL2
+brooklyn.location.jclouds.aws-ec2.KEY=VAL3
+brooklyn.location.jclouds.aws-ec2@us-west-1.KEY=VAL4
+brooklyn.location.named.my-aws=jclouds:aws-ec2:us-west-1
+brooklyn.location.named.my-aws.KEY=VAL5
{% endhighlight %}
-In the above example, if you deploy to `location: named:my-aws`,
-Brooklyn will get `VAL3` for `KEY`;
-this overrides `VAL2` which applies by default to all `aws-ec2` locations,
-which in turn overrides `VAL1`, which applies to all jclouds locations.
-
### Localhost
@@ -272,3 +301,4 @@ brooklyn.location.named.On-Prem\ Iron\ Example.privateKeyPassphrase=s3cr3tpassph
* [More Locations](more-locations.html)
* [SSH Keys](ssh-keys.html)
+* [Cloud Credentials](cloud-credentials.md)
http://git-wip-us.apache.org/repos/asf/incubator-brooklyn/blob/1422f46a/docs/guide/ops/logging.md
----------------------------------------------------------------------
diff --git a/docs/guide/ops/logging.md b/docs/guide/ops/logging.md
index fc1da3c..81d55e9 100644
--- a/docs/guide/ops/logging.md
+++ b/docs/guide/ops/logging.md
@@ -3,7 +3,6 @@ title: Logging
layout: website-normal
---
-
Brooklyn uses the SLF4J logging facade, which allows use of many popular frameworks including `logback`,
`java.util.logging` and `log4j`.
@@ -17,6 +16,7 @@ some other problem has occured which the user is expected to attend to
Loggers follow the ``package.ClassName`` naming standard.
+
## Standard Configuration
A `logback.xml` file is included in the `conf/` directly of the Brooklyn distro;
@@ -37,6 +37,32 @@ and create a file `logback-debug.xml` based on the
from that project.
+## Log File Backup
+
+*This sub-section is a work in progress; feedback from the community is extremely welcome.*
+
+The default rolling log files can be backed up periodically, e.g. using a CRON job.
+
+Note however that the rolling log file naming scheme will rename the historic zipped log files
+such that `brooklyn.debug-1.log.zip` is the most recent zipped log file. When the current
+`brooklyn.debug.log` is to be zipped, the previous zip file will be renamed
+`brooklyn.debug-2.log.zip`. This renaming of files can make RSYNC or backups tricky.
+
+An option is to covert/move the file to a name that includes the last-modified timestamp.
+For example (on mac):
+
+ LOG_FILE=brooklyn.debug-1.log.zip
+ TIMESTAMP=`stat -f '%Um' $LOG_FILE`
+ mv $LOG_FILE /path/to/archive/brooklyn.debug-$TIMESTAMP.log.zip
+
+
+## Logging aggregators
+
+Integration with systems like Logstash and Splunk is possible using standard logback configuration.
+Logback can be configured to [write to the syslog](http://logback.qos.ch/manual/appenders.html#SyslogAppender),
+which can then [feed its logs to Logstash](http://www.logstash.net/docs/1.4.2/inputs/syslog).
+
+
## For More Information
The following resources may be useful when configuring logging:
http://git-wip-us.apache.org/repos/asf/incubator-brooklyn/blob/1422f46a/docs/guide/ops/persistence/index.md
----------------------------------------------------------------------
diff --git a/docs/guide/ops/persistence/index.md b/docs/guide/ops/persistence/index.md
index 26696aa..1a8fe55 100644
--- a/docs/guide/ops/persistence/index.md
+++ b/docs/guide/ops/persistence/index.md
@@ -7,6 +7,7 @@ children:
- { section: Object Store Persistence }
- { section: Rebinding to State }
- { section: Writing Persistable Code }
+- { section: Persisted State Backup }
---
Brooklyn can be configured to persist its state so that the Brooklyn server can be restarted,
@@ -63,17 +64,42 @@ The state is written to the given path. The file structure under that path is:
* `./enrichers/`
In each of those directories, an XML file will be created per item - for example a file per
-entity in ./entities/. This file will capture all of the state - for example, an
+entity in `./entities/`. This file will capture all of the state - for example, an
entity's: id; display name; type; config; attributes; tags; relationships to locations, child
entities, group membership, policies and enrichers; and dynamically added effectors and sensors.
+If using the default persistence dir (i.e. no `--persistenceDir` was specified), then Brooklyn will
+write its state to `~/.brooklyn/brooklyn-persisted-state/data`. Copies of this directory
+will be automatically created in `~/.brooklyn/brooklyn-persisted-state/backups/` each time Brooklyn
+is restarted (or if a standby Brooklyn instances takes over as master).
+
+A custom directory for Brooklyn state can also be configured in `brooklyn.properties` using:
+
+ # For all Brooklyn files
+ brooklyn.base.dir=/path/to/base/dir
+
+ # Sub-directory of base.dir for writing persisted state (if relative). If directory
+ # starts with "/" (or "~/", or something like "c:\") then assumed to be absolute.
+ brooklyn.persistence.dir=data
+
+ # Sub-directory of base.dir for creating backup directories (if relative). If directory
+ # starts with "/" (or "~/", or something like "c:\") then assumed to be absolute.
+ brooklyn.persistence.backups.dir=backups
+
+This `base.dir` will also include temporary files such as the OSGi cache.
+
+If `persistence.dir` is not specified then it will use the sub-directory
+`brooklyn-persisted-state/data` of the `base.dir`. If the `backups.dir` is not specified
+the backup directories will be created in the sub-directory `backups` of the persistence dir.
+
Object Store Persistence
------------------------
Brooklyn can persist its state to any Object Store API that jclouds supports including
S3, Swift and Azure. This gives access to any compatible Object Store product or cloud provider
-including AWS-S3, SoftLayer, Rackspace, HP and Microsoft Azure.
+including AWS-S3, SoftLayer, Rackspace, HP and Microsoft Azure. For a complete list of supported
+providers, see [jclouds](http://jclouds.apache.org/guides/providers/#blobstore-providers).
To configure the Object Store, add the credentials to `~/.brooklyn/brooklyn.properties` such as:
@@ -94,10 +120,28 @@ brooklyn.location.named.softlayer-swift-ams01.credential=abcdefghijklmnopqrstuvw
Start Brooklyn pointing at this target object store, e.g.:
{% highlight bash %}
-brooklyn launch --persist auto --persistenceDir myContainerName --persistenceLocation named:softlayer-swift-ams01
+nohup brooklyn launch --persist auto --persistenceDir myContainerName --persistenceLocation named:softlayer-swift-ams01 &
{% endhighlight %}
+The following `brooklyn.properties` options can also be used:
+
+ # Location spec string for an object store (e.g. jclouds:swift:URL) where persisted state
+ # should be kept; if blank or not supplied, the file system is used.
+ brooklyn.persistence.location.spec=<location>
+
+ # Container name for writing persisted state
+ brooklyn.persistence.dir=/path/to/dataContainer
+
+ # Location spec string for an object store (e.g. jclouds:swift:URL) where backups of persisted
+ # state should be kept; defaults to the local file system.
+ brooklyn.persistence.backups.location.spec=<location>
+
+ # Container name for writing backups of persisted state;
+ # defaults to 'backups' inside the default persistence container.
+ brooklyn.persistence.backups.dir=/path/to/backupContainer
+
+
Rebinding to State
------------------
@@ -233,3 +277,58 @@ For locations, policies and enrichers they (currently) do not have attributes. H
config is persisted automatically. Normally the state of a policy or enricher is transient -
on rebind it starts afresh, for example with monitoring the performance or health metrics
rather than relying on the persisted values.
+
+
+Persisted State Backup
+----------------------
+
+### File system backup
+
+When using the file system it is important to ensure it is backed up regularly.
+
+One could use `rsync` to regularly backup the contents to another server.
+
+It is also recommended to periodically create a complete archive of the state.
+A simple mechanism is to run a CRON job periodically (e.g. every 30 minutes) that creates an
+archive of the persistence directory, and uploads that to a backup
+facility (e.g. to S3).
+
+Optionally, to avoid excessive load on the Brooklyn server, the archive-generation could be done
+on another "data" server. This could get a copy of the data via an `rsync` job.
+
+An example script to be invoked by CRON is shown below:
+
+ DATE=`date "+%Y%m%d.%H%M.%S"`
+ BACKUP_FILENAME=/path/to/archives/back-${DATE}.tar.gz
+ DATA_DIR=/path/to/base/dir/data
+
+ tar --exclude '*/backups/*' -czvf $BACKUP_FILENAME $DATA_DIR
+ # For s3cmd installation see http://s3tools.org/repositories
+ s3cmd put $BACKUP_FILENAME s3://mybackupbucket
+ rm $BACKUP_FILENAME
+
+
+### Object store backup
+
+Object Stores will normally handle replication. However, many such object stores do not handle
+versioning (i.e. to allow access to an old version, if an object has been incorrectly changed or
+deleted).
+
+The state can be downloaded periodically from the object store, archived and backed up.
+
+An example script to be invoked by CRON is shown below:
+
+ DATE=`date "+%Y%m%d.%H%M.%S"`
+ BACKUP_FILENAME=/path/to/archives/back-${DATE}.tar.gz
+ TEMP_DATA_DIR=/path/to/tempdir
+
+ amp copy-state \
+ --persistenceLocation named:my-persistence-location \
+ --persistenceDir /path/to/bucket \
+ --destinationDir $TEMP_DATA_DIR
+
+ tar --exclude '*/backups/*' -czvf $BACKUP_FILENAME $TEMP_DATA_DIR
+ # For s3cmd installation see http://s3tools.org/repositories
+ s3cmd put $BACKUP_FILENAME s3://mybackupbucket
+ rm $BACKUP_FILENAME
+ rm -r $TEMP_DATA_DIR
http://git-wip-us.apache.org/repos/asf/incubator-brooklyn/blob/1422f46a/docs/guide/ops/requirements.md
----------------------------------------------------------------------
diff --git a/docs/guide/ops/requirements.md b/docs/guide/ops/requirements.md
new file mode 100644
index 0000000..52b96e4
--- /dev/null
+++ b/docs/guide/ops/requirements.md
@@ -0,0 +1,59 @@
+---
+title: Requirements
+layout: website-normal
+---
+
+## Server Specification
+
+The size of server required by Brooklyn depends on the amount of activity. This includes:
+
+* the number of entities/VMs being managed
+* the number of VMs being deployed concurrently
+* the amount of management and monitoring required per entity
+
+For dev/test or when there are only a handful of VMs being managed, a small VM is sufficient.
+For example, an AWS m3.medium with one vCPU, 3.75GiB RAM and 4GB disk.
+
+For larger production uses, a more appropriate machine spec would be two or more cores,
+at least 8GB RAM and 20GB disk. The disk is just for logs, a small amount of persisted state, and
+any binaries for custom blueprints/integrations.
+
+
+## Supported Operating Systems
+
+The recommended operating system is CentOS 6.x or RedHat 6.x.
+
+Brooklyn has also been tested on Ubuntu 12.04 and OS X.
+
+
+## Software Requirements
+
+Brooklyn requires Java (JRE or JDK), version 6 or version 7. The most recent version 7 is recommended.
+OpenJDK is recommended. Brooklyn has also been tested on IBM J9 and Oracle's JVM.
+
+* check your `iptables` or other firewall service, making sure that incoming connections on port 8443 is not blocked
+* check that the [linux kernel entropy](increase-entropy.html) is sufficient
+
+
+## Configuration Requirements
+
+### Ports
+
+The ports used by Brooklyn are:
+
+* 8443 for https, to expose the web-console and REST api.
+* 8081 for http, to expose the web-console and REST api.
+
+Whether to use https rather than http is configurable using the CLI option `--https`;
+the port to use is configurable using the CLI option `--port <port>`.
+See [CLI](cli.html) documentation for more details.
+
+To enable remote Brooklyn access, ensure these ports are open in the firewall.
+For example, to open port 8443 in iptables, ues the command:
+
+ /sbin/iptables -I INPUT -p TCP --dport 8443 -j ACCEPT
+
+
+### Linux Kernel Entropy
+
+Check that the [linux kernel entropy](increase-entropy.html) is sufficient.