You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@brooklyn.apache.org by he...@apache.org on 2014/12/17 18:23:42 UTC
[08/50] [abbrv] incubator-brooklyn git commit: move version-specific
things to "guide"
http://git-wip-us.apache.org/repos/asf/incubator-brooklyn/blob/ba6612ec/docs/guide/use/examples/messaging/index.md
----------------------------------------------------------------------
diff --git a/docs/guide/use/examples/messaging/index.md b/docs/guide/use/examples/messaging/index.md
new file mode 100644
index 0000000..51fd0ae
--- /dev/null
+++ b/docs/guide/use/examples/messaging/index.md
@@ -0,0 +1,183 @@
+---
+layout: guide-normal
+title: Publish-Subscribe Messagiung
+toc: /toc.json
+---
+
+This example shows how a simple messaging application can be build
+in brooklyn, starting with configuring and launching a broker. For
+these examples we will use the Apache [Qpid](http://qpid.apache.org/)
+Java AMQP message broker and clients using the
+[JMS](http://docs.oracle.com/javaee/6/tutorial/doc/bnceh.html) API.
+
+{% readj ../before-begin.include.md %}
+
+Now, go to this particular example's directory:
+
+{% highlight bash %}
+% cd simple-messaging-pubsub
+{% endhighlight %}
+
+The CLI needs to know where to find your compiled examples. You can set this up by exporting
+the ``BROOKLYN_CLASSPATH`` environment variable in the following way:
+
+{% highlight bash %}
+% export BROOKLYN_CLASSPATH=$(pwd)/target/classes
+{% endhighlight %}
+
+The project ``simple-messaging-pubsub`` includes a deployment
+descriptor for our example messaging application and simple _Publish_
+and _Subscribe_ JMS test client scripts.
+
+## Single Broker
+
+The first example will include a Qpid broker, which we will customize
+to use the Oracle [BDB](http://www.oracle.com/technetwork/products/berkeleydb/overview/index.html)
+message store as an example of a typical production setup. We will
+also create a queue for use by a pair of test clients.
+
+The ``QpidBroker`` entity is created like this, which uses the
+default configuration, specifying only the AMQP port and creates
+no queues or topics:
+
+{% highlight java %}
+public class StandaloneQpidBrokerExample extends AbstractApplication {
+ @Override
+ public void init() {
+ // Configure the Qpid broker entity
+ QpidBroker broker = addChild(EntitySpec.create(QpidBroker.class)
+ .configure("amqpPort", 5672));
+ }
+}
+{% endhighlight %}
+
+To install the custom configuration files and extra libraries for
+BDB, we specify some files to copy to the broker installation, using
+the ``runtimeFiles`` property. These files should be available in
+the classpath of the application when it is running, usually by
+copying them to the ``src/main/resources`` directory. For example,
+here we copy a custom XML configuration file and a new password
+file:
+
+{% highlight java %}
+ final String CUSTOM_CONFIG_PATH = "classpath://custom-config.xml";
+ final String PASSWD_PATH = "classpath://passwd";
+
+ QpidBroker broker = addChild(EntitySpec.create(QpidBroker.class)
+ .configure("amqpPort", 5672)
+ .configure("amqpVersion", AmqpServer.AMQP_0_10)
+ .configure("runtimeFiles", ImmutableMap.builder()
+ .put(QpidBroker.CONFIG_XML, CUSTOM_CONFIG_PATH)
+ .put(QpidBroker.PASSWD, PASSWD_PATH)
+ .build()));
+{% endhighlight %}
+
+Finally, we come to the complete configuration of our ``QpidBroker``
+entity using the BDB store. The additional properties here specify
+the AMQP version and that a queue named _testQueue_ should be created
+on startup.
+
+{% highlight java %}
+ final String CUSTOM_CONFIG_PATH = "classpath://custom-config.xml";
+ final String PASSWD_PATH = "classpath://passwd";
+ final String QPID_BDBSTORE_JAR_PATH = "classpath://qpid-bdbstore-0.14.jar";
+ final String BDBSTORE_JAR_PATH = "classpath://je-5.0.34.jar";
+
+ QpidBroker broker = addChild(EntitySpec.create(QpidBroker.class)
+ .configure("amqpPort", 5672)
+ .configure("amqpVersion", AmqpServer.AMQP_0_10)
+ .configure("runtimeFiles", ImmutableMap.builder()
+ .put(QpidBroker.CONFIG_XML, CUSTOM_CONFIG_PATH)
+ .put(QpidBroker.PASSWD, PASSWD_PATH)
+ .put("lib/opt/qpid-bdbstore-0.14.jar", QPID_BDBSTORE_JAR_PATH)
+ .put("lib/opt/je-5.0.34.jar", BDBSTORE_JAR_PATH)
+ .build())
+ .configure("queue", "testQueue"));
+{% endhighlight %}
+
+
+### Running the Example
+
+You can build and run the example (on *nix or Mac) after checking
+out the Brooklyn [repository](https://www.github.com/brooklyncentral/brooklyn)
+as follows:
+
+{% highlight bash %}
+% ${BROOKLYN_HOME}/bin/brooklyn -v launch --app brooklyn.demo.StandaloneQpidBrokerExample --location localhost
+{% endhighlight %}
+
+Now, visit the Brooklyn web console on port 8081 (for pre 0.6 releases,
+use the credentials admin/password). This allows you to view the Brooklyn
+entities and their current state for debugging.
+
+Note that the installation may take some time, because the default
+deployment downloads the software from the official repos. You can
+monitor start-up activity for each entity in the ``Activity`` pane
+in the management console, and see more detail by tailing the log
+file (``tail -f brooklyn.log``).
+
+After starting up, the demo script should display a summary of all
+the Brooklyn managed entities and their attributes. This will show
+both the Qpid broker and its child entity, the queue _testQueue_
+which was created at startup. The queue entity has sensors that
+monitor the depth of unread messages, which you can check while
+running the test client scripts later.
+
+If the ``-v`` flag is passed to the startup command, all configured
+entity and sensor details will be output. This includes the broker URL,
+which is used to configure JMS clients to connect to this broker.
+This URL can also be viewed as a sensor attribute in the web console,
+named _broker.url_.
+
+This sensor is common to _all_ messaging brokers that Brooklyn
+provides, and is usually accessed by applications to allow them to
+provide it as a parameter to other entities, as shown in the code
+fragment below.
+
+{% highlight java %}
+String url = broker.getAttribute(MessageBroker.BROKER_URL)
+{% endhighlight %}
+
+Using the URL the demo script printed, you can run the test ``Subscribe``
+and then ``Publish`` classes, to send messages using the broker. Simply
+run the commands in another window, with the provided URL as the
+only argument. Note that the URLs may be different to those printed
+below, and that any unquoted ``&`` characters *must* be escaped,
+if present.
+
+{% highlight bash %}
+% URL="amqp://guest:guest@/localhost?brokerlist='tcp://localhost:5672'"
+% java -cp "./resources/lib/*:./target/classes" brooklyn.demo.Subscribe ${URL}
+% java -cp "./resources/lib/*:./target/classes" brooklyn.demo.Publish ${URL}
+{% endhighlight %}
+
+In the _Publish_ window you should see a log message every time a
+message is sent, like this:
+
+{% highlight bash %}
+2012-05-02 14:04:38,521 INFO Sent message 65
+2012-05-02 14:04:39,522 INFO Sent message 66
+{% endhighlight %}
+
+Similarly, the _Subscribe_ windows should log on reciept of these
+messages, as follows:
+
+{% highlight bash %}
+2012-05-02 14:04:32,522 INFO got message 41 test message 41
+2012-05-02 14:04:33,523 INFO got message 42 test message 42
+{% endhighlight %}
+
+### Cloud Deployment
+
+With appropriate setup (as described
+[here]({{ site.url }}/use/guide/management/index.html#startup-config))
+this can also be deployed to your favourite cloud, let's pretend
+it's Amazon Ireland, as follows:
+
+{% highlight bash %}
+% ${BROOKLYN_HOME}/bin/brooklyn launch --app brooklyn.demo.StandaloneQpidBrokerExample --location aws-ec2:eu-west-1
+{% endhighlight %}
+
+If you encounter any difficulties, please
+[tell us]({{ site.url }}/meta/contact.html) and we'll do our best
+to help.
http://git-wip-us.apache.org/repos/asf/incubator-brooklyn/blob/ba6612ec/docs/guide/use/examples/nosql-cassandra/cassandra.include.md
----------------------------------------------------------------------
diff --git a/docs/guide/use/examples/nosql-cassandra/cassandra.include.md b/docs/guide/use/examples/nosql-cassandra/cassandra.include.md
new file mode 100644
index 0000000..8fd04ae
--- /dev/null
+++ b/docs/guide/use/examples/nosql-cassandra/cassandra.include.md
@@ -0,0 +1,282 @@
+
+{% readj ../before-begin.include.md %}
+
+## Simple Cassandra Cluster
+
+Go to this particular example's directory:
+
+{% highlight bash %}
+% cd simple-nosql-cluster
+{% endhighlight %}
+
+The CLI needs to know where to find your compiled examples. You can set this up by exporting
+the ``BROOKLYN_CLASSPATH`` environment variable in the following way:
+
+{% highlight bash %}
+% export BROOKLYN_CLASSPATH=$(pwd)/target/classes
+{% endhighlight %}
+
+The project ``simple-nosql-cluster`` includes several deployment descriptors
+for deploying and managing Cassandra, under ``src/main/java``.
+
+The simplest of these, ``SimpleCassandraCluster``, will start a Cassandra cluster. The code is:
+
+{% highlight java %}
+public class SimpleCassandraCluster extends AbstractApplication {
+ public void init() {
+ addChild(EntitySpec.create(CassandraCluster.class)
+ .configure(CassandraCluster.INITIAL_SIZE, 1)
+ .configure(CassandraCluster.CLUSTER_NAME, "Brooklyn"));
+ }
+}
+{% endhighlight %}
+
+To run that example on localhost (on *nix or Mac, assuming `ssh localhost` requires no password or passphrase):
+
+{% highlight bash %}
+% ${BROOKLYN_HOME}/bin/brooklyn launch --app brooklyn.demo.SimpleCassandraCluster \
+ --location localhost
+{% endhighlight %}
+
+Then visit the Brooklyn console on ``localhost:8081``.
+Note that the installation may take some time, because the default deployment downloads the software from
+the official repos. You can monitor start-up activity for each entity in the ``Activity`` pane in the management console,
+and see more detail by tailing the log file (``tail -f brooklyn.log``).
+
+This example runs successfully on a local machine because ``INITIAL_SIZE`` is configured to just one node
+(a limitation of Cassandra is that every node must be on a different machine/VM).
+If you want to run with more than one node in the cluster, you'll need to use a location
+that either points to multiple existing machines or to a cloud provider where you can
+provision new machines.
+
+With appropriate setup of credentials (as described [here]({{ site.url }}/use/guide/management/index.html#startup-config))
+this example can also be deployed to your favourite cloud. Let's pretend it's Amazon US East, as follows:
+
+{% highlight bash %}
+% ${BROOKLYN_HOME}/bin/brooklyn launch --app brooklyn.demo.SimpleCassandraCluster \
+ --location aws-ec2:us-east-1
+{% endhighlight %}
+
+If you want more nodes in your cluster, you can either modify the deployment descriptor (i.e. change the ``INITIAL_SIZE`` value),
+or dynamically add more nodes by calling the ``resize`` effector through the web-console.
+To do the latter, select cluster entity in the tree on the left, then click on the "effectors" tab, and invoke ``resize``
+with the desired number of nodes.
+
+
+### Testing your Cluster
+
+An easy way to test your cluster is to use the ``cassandra-stress`` command line tool.
+For example, run:
+
+{% highlight bash %}
+# Substitute the id below for your VM
+NODE_IDS=ec2-54-221-69-95.compute-1.amazonaws.com
+/tmp/brooklyn-aled/installs/CassandraNode/1.2.9/apache-cassandra-1.2.9/tools/bin/cassandra-stress \
+ --nodes ${NODE_IDS} \
+ --replication-factor 1 \
+ --progress-interval 1 \
+ --num-keys 10000 \
+ --operation INSERT
+{% endhighlight %}
+
+This command will fire 10000 inserts at the cluster, via the nodes specified in the comma-separated node list.
+If you change ``INSERT`` to ``READ``, it will read each of those 10000 values.
+
+
+## High Availability Cassandra Cluster
+
+Ready for something more interesting? Try this:
+
+{% highlight bash %}
+% ${BROOKLYN_HOME}/bin/brooklyn launch --app brooklyn.demo.HighAvailabilityCassandraCluster \
+ --location aws-ec2:us-east-1
+{% endhighlight %}
+
+This launches the class ``HighAvailabilityCassandraCluster``,
+which launches a Cassandra cluster configured to replicate across availability zones.
+
+To give some background for that statement, in
+[AWS](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-regions-availability-zones.html)
+(and various other clouds), a region is a
+separate geographic area, consisting of multiple isolated locations known as availability zones.
+To ensure high availability, the Cassandra cluster and thus the data should be spread across the
+availability zones. Cassandra should be configured to ensure there is at least one replica in
+each availability zone. In
+[Cassandra terminology](http://www.datastax.com/docs/1.1/cluster_architecture/replication)
+a region is normally mapped to a "datacenter" and an availability zone to a "rack".
+
+To be properly highly available, we need some automated policies to restart failed servers
+and to replace unhealthy nodes. Brooklyn has these policies available out-of-the-box.
+To wire them up, the essential code fragment looks like this:
+
+{% highlight java %}
+public class HighAvailabilityCassandraCluster extends AbstractApplication {
+ public void init() {
+ addChild(EntitySpec.create(CassandraCluster.class)
+ .configure(CassandraCluster.CLUSTER_NAME, "Brooklyn")
+ .configure(CassandraCluster.INITIAL_SIZE, 1)
+ .configure(CassandraCluster.ENABLE_AVAILABILITY_ZONES, true)
+ .configure(CassandraCluster.NUM_AVAILABILITY_ZONES, 3)
+ .configure(CassandraCluster.ENDPOINT_SNITCH_NAME, "GossipingPropertyFileSnitch")
+ .configure(CassandraCluster.MEMBER_SPEC, EntitySpec.create(CassandraNode.class)
+ .policy(PolicySpec.create(ServiceFailureDetector.class))
+ .policy(PolicySpec.create(ServiceRestarter.class)
+ .configure(ServiceRestarter.FAILURE_SENSOR_TO_MONITOR, ServiceFailureDetector.ENTITY_FAILED)))
+ .policy(PolicySpec.create(ServiceReplacer.class)
+ .configure(ServiceReplacer.FAILURE_SENSOR_TO_MONITOR, ServiceRestarter.ENTITY_RESTART_FAILED)));
+ }
+}
+{% endhighlight %}
+
+This code is doing a lot and deserves some more detailed explanation:
+
+* The ``MEMBER_SPEC`` describes the configuration of the Cassandra nodes to be created in the cluster.
+ Assuming you're happy to use all the default thrift port etc, then the only configuration to add is
+ a couple of policies.
+* The ``ServiceFailureDetector`` policy watches the node's sensors, and generates
+ an ``ENTITY_FAILED`` event if the node goes down.
+* The ``ServiceRestarter`` policy responds to this failure-event
+ by restarting the node. Its default configuration is that: if a node does not come back up, or if it
+ fails again within three minutes, then it will emit an ``ENTITY_RESTART_FAILED`` event.
+* Finally, the ``SERVICE_REPLACER`` policy on the cluster responds to this event by replacing the
+ entire VM. It sets up a new VM in the same location, and then tears down the faulty node.
+
+> *Troubleshooting:*
+
+> *In AWS, some availability zones can be constrained for particular instance sizes (see
+ [this bug report](https://github.com/brooklyncentral/brooklyn/issues/973)
+ If you get this error, the workaround is to specify explicitly the availability zones to use.
+ This requires an additional line of code such as:*
+
+{% highlight java %}
+ .configure(AVAILABILITY_ZONE_NAMES, ImmutableList.of("us-east-1b", "us-east-1c", "us-east-1e"))
+{% endhighlight %}
+
+> *However, this prevents the blueprint from being truly portable. We're looking at fixing this issue.*
+
+
+## Wide Area Cassandra Cluster
+
+For critical enterprise use-cases, you'll want to run your Cassandra cluster across multiple regions,
+or better yet across multiple cloud providers. This gives the highest level of availability for
+the service.
+
+Try running:
+
+{% highlight bash %}
+% ${BROOKLYN_HOME}/bin/brooklyn launch --app brooklyn.demo.WideAreaCassandraCluster \
+ --location "aws-ec2:us-east-1,aws-ec2:us-west-2"
+{% endhighlight %}
+
+This launches the class ``WideAreaCassandraCluster`` across two AWS regions.
+
+Cassandra provides some great support for this with the
+[EC2MultiRegionSnitch](http://www.datastax.com/docs/1.1/cluster_architecture/replication)
+The
+[snitch](http://www.datastax.com/docs/1.1/cluster_architecture/replication#snitches)
+maps IPs to racks and data centers; it defines how the nodes are grouped together within the overall
+network topology. For wide-area deployments, it must also deal with when to use the private IPs
+(within a region) and the public IPs (between regions).
+You'll need a more generic snitch if you're going to span different cloud providers.
+Brooklyn has a custom MultiCloudSnitch that we're looking to contribute back to Cassandra.
+
+The important piece of code in ``WideAreaCassandraCluster`` is:
+
+{% highlight java %}
+public class WideAreaCassandraCluster extends AbstractApplication {
+ public void init() {
+ addChild(EntitySpec.create(CassandraFabric.class)
+ .configure(CassandraCluster.CLUSTER_NAME, "Brooklyn")
+ .configure(CassandraCluster.INITIAL_SIZE, 2) // per location
+ .configure(CassandraCluster.ENDPOINT_SNITCH_NAME, "brooklyn.entity.nosql.cassandra.customsnitch.MultiCloudSnitch")
+ .configure(CassandraNode.CUSTOM_SNITCH_JAR_URL, "classpath://brooklyn/entity/nosql/cassandra/cassandra-multicloud-snitch.jar"));
+ }
+}
+{% endhighlight %}
+
+The code below shows the wide-area example with the high-availability policies from the previous section also configured:
+
+{% highlight java %}
+public class WideAreaCassandraCluster extends AbstractApplication {
+ public void init() {
+ addChild(EntitySpec.create(CassandraFabric.class)
+ .configure(CassandraCluster.CLUSTER_NAME, "Brooklyn")
+ .configure(CassandraCluster.INITIAL_SIZE, 2) // per location
+ .configure(CassandraCluster.ENDPOINT_SNITCH_NAME, "brooklyn.entity.nosql.cassandra.customsnitch.MultiCloudSnitch")
+ .configure(CassandraNode.CUSTOM_SNITCH_JAR_URL, "classpath://brooklyn/entity/nosql/cassandra/cassandra-multicloud-snitch.jar")
+ .configure(CassandraFabric.MEMBER_SPEC, EntitySpec.create(CassandraCluster.class)
+ .configure(CassandraCluster.MEMBER_SPEC, EntitySpec.create(CassandraNode.class)
+ .policy(PolicySpec.create(ServiceFailureDetector.class))
+ .policy(PolicySpec.create(ServiceRestarter.class)
+ .configure(ServiceRestarter.FAILURE_SENSOR_TO_MONITOR, ServiceFailureDetector.ENTITY_FAILED)))
+ .policy(PolicySpec.create(ServiceReplacer.class)
+ .configure(ServiceReplacer.FAILURE_SENSOR_TO_MONITOR, ServiceRestarter.ENTITY_RESTART_FAILED))));
+ }
+}
+{% endhighlight %}
+
+To run Cassandra across multiple clouds, try running:
+
+{% highlight bash %}
+% ${BROOKLYN_HOME}/bin/brooklyn launch --app brooklyn.demo.WideAreaCassandraCluster \
+ --location "aws-ec2:us-east-1,google-compute-engine,rackspace-cloudservers-uk"
+{% endhighlight %}
+
+
+### Testing your Wide-Area Cluster
+
+You can again use the ``cassandra-stress`` command line tool to test the wide-area cluster.
+
+Note that the replication strategy (such as
+[NetworkTopologyStrategy](http://www.datastax.com/docs/1.0/cluster_architecture/replication#networktopologystrategy)
+is specified when creating a
+[keyspace](http://www.datastax.com/documentation/cassandra/1.2/webhelp/index.html#cassandra/configuration/configStorage_r.html).
+The example below specifies a minimum of 1 replica in each datacenter.
+
+To do updates against a node in a given availability zone:
+
+{% highlight bash %}
+NODE_IDS=<your node hostname>
+/tmp/brooklyn-aled/installs/CassandraNode/1.2.9/apache-cassandra-1.2.9/tools/bin/cassandra-stress \
+ --nodes ${NODE_IDS} \
+ --replication-strategy NetworkTopologyStrategy \
+ --strategy-properties=us-east-1:1,us-west-2:1 \
+ --progress-interval 1 \
+ --num-keys 10000 \
+ --operation INSERT
+{% endhighlight %}
+
+To check that the same data is available from a different region, target the reads
+against an appropriate node:
+
+{% highlight bash %}
+NODE_IDS=<your node hostname>
+/tmp/brooklyn-aled/installs/CassandraNode/1.2.9/apache-cassandra-1.2.9/tools/bin/cassandra-stress \
+ --nodes ${NODE_IDS} \
+ --replication-strategy NetworkTopologyStrategy \
+ --strategy-properties=us-east-1:1,us-west-2:1 \
+ --progress-interval 1 \
+ --num-keys 10000 \
+ --operation READ
+{% endhighlight %}
+
+To really test this, you may want to simulate the failure of a region first.
+You can kill the VMs or ``kill -9`` the processes. But remember that if Brooklyn policies are configured
+they will by default restart the processes automatically! You can disable the Brooklyn policies through
+the brooklyn web-console (select the entity, go the policies tab, select the policy, and click "disable").
+
+
+## Putting it all together: CumulusRDF
+
+If you want to try this with a real example application using the Cassandra cluster, take a look at
+[CumulusRDF](https://code.google.com/p/cumulusrdf). There is an example Brooklyn application at:
+
+{% highlight bash %}
+% ${BROOKLYN_HOME}/bin/brooklyn launch --app brooklyn.demo.CumulusRDFApplication \
+ --location "aws-ec2:us-east-1"
+{% endhighlight %}
+
+
+## Contact us!
+
+If you encounter any difficulties or have any comments, please [tell us]({{ site.url }}/meta/contact.html) and we'll do our best to help.
http://git-wip-us.apache.org/repos/asf/incubator-brooklyn/blob/ba6612ec/docs/guide/use/examples/nosql-cassandra/index.md
----------------------------------------------------------------------
diff --git a/docs/guide/use/examples/nosql-cassandra/index.md b/docs/guide/use/examples/nosql-cassandra/index.md
new file mode 100644
index 0000000..13caf53
--- /dev/null
+++ b/docs/guide/use/examples/nosql-cassandra/index.md
@@ -0,0 +1,7 @@
+---
+layout: guide-normal
+title: Cassandra Clusters
+toc: /toc.json
+---
+
+{% readj cassandra.include.md %}
http://git-wip-us.apache.org/repos/asf/incubator-brooklyn/blob/ba6612ec/docs/guide/use/examples/simple-web-cluster.md
----------------------------------------------------------------------
diff --git a/docs/guide/use/examples/simple-web-cluster.md b/docs/guide/use/examples/simple-web-cluster.md
new file mode 100644
index 0000000..e9ddac6
--- /dev/null
+++ b/docs/guide/use/examples/simple-web-cluster.md
@@ -0,0 +1,9 @@
+---
+layout: guide-normal
+title: Elastic Web Cluster
+toc: /toc.json
+---
+
+<!-- file kept to preserve old links; remove when link not used -->
+
+{% readj webcluster/webcluster.include.md %}
http://git-wip-us.apache.org/repos/asf/incubator-brooklyn/blob/ba6612ec/docs/guide/use/examples/toc.json
----------------------------------------------------------------------
diff --git a/docs/guide/use/examples/toc.json b/docs/guide/use/examples/toc.json
new file mode 100644
index 0000000..2f36c50
--- /dev/null
+++ b/docs/guide/use/examples/toc.json
@@ -0,0 +1,13 @@
+[
+{ "title": "Elastic Web Cluster",
+ "file": "{{ site.url }}/use/examples/webcluster/index.html" },
+{ "title": "Global Web Fabric",
+ "file": "{{ site.url }}/use/examples/global-web-fabric/index.html" },
+{ "title": "Whirr Hadoop Cluster",
+ "file": "{{ site.url }}/use/examples/whirrhadoop/index.html" },
+{ "title": "Publish-Subscribe Messaging",
+ "file": "{{ site.url }}/use/examples/messaging/index.html" },
+{ "title": "Cassandra Cluster",
+ "file": "{{ site.url }}/use/examples/nosql-cassandra/index.html" }
+
+]
http://git-wip-us.apache.org/repos/asf/incubator-brooklyn/blob/ba6612ec/docs/guide/use/examples/webcluster.md
----------------------------------------------------------------------
diff --git a/docs/guide/use/examples/webcluster.md b/docs/guide/use/examples/webcluster.md
new file mode 100644
index 0000000..e9ddac6
--- /dev/null
+++ b/docs/guide/use/examples/webcluster.md
@@ -0,0 +1,9 @@
+---
+layout: guide-normal
+title: Elastic Web Cluster
+toc: /toc.json
+---
+
+<!-- file kept to preserve old links; remove when link not used -->
+
+{% readj webcluster/webcluster.include.md %}
http://git-wip-us.apache.org/repos/asf/incubator-brooklyn/blob/ba6612ec/docs/guide/use/examples/webcluster/index.md
----------------------------------------------------------------------
diff --git a/docs/guide/use/examples/webcluster/index.md b/docs/guide/use/examples/webcluster/index.md
new file mode 100644
index 0000000..43ea980
--- /dev/null
+++ b/docs/guide/use/examples/webcluster/index.md
@@ -0,0 +1,7 @@
+---
+layout: guide-normal
+title: Elastic Web Cluster
+toc: /toc.json
+---
+
+{% readj webcluster.include.md %}
http://git-wip-us.apache.org/repos/asf/incubator-brooklyn/blob/ba6612ec/docs/guide/use/examples/webcluster/webcluster.include.md
----------------------------------------------------------------------
diff --git a/docs/guide/use/examples/webcluster/webcluster.include.md b/docs/guide/use/examples/webcluster/webcluster.include.md
new file mode 100644
index 0000000..11e801b
--- /dev/null
+++ b/docs/guide/use/examples/webcluster/webcluster.include.md
@@ -0,0 +1,124 @@
+
+{% readj ../before-begin.include.md %}
+
+## Simple Web Server
+
+Go to this particular example's directory:
+
+{% highlight bash %}
+% cd simple-web-cluster
+{% endhighlight %}
+
+The CLI needs to know where to find your compiled examples. You can set this up by exporting
+the ``BROOKLYN_CLASSPATH`` environment variable in the following way:
+
+{% highlight bash %}
+% export BROOKLYN_CLASSPATH=$(pwd)/target/classes
+{% endhighlight %}
+
+The project ``simple-web-cluster`` includes several deployment descriptors
+for rolling out a web application, under ``src/main/java``.
+
+
+
+The simplest of these, ``SingleWebServerExample``, starts JBoss on a single machine with a "Hello World" war deployed,
+with a single line:
+
+{% highlight java %}
+public class SingleWebServerExample extends AbstractApplication {
+ private static final String WAR_PATH = "classpath://hello-world-webapp.war";
+
+ @Override
+ public void init() {
+ addChild(EntitySpec.create(JBoss7Server.class)
+ .configure("war", WAR_PATH)
+ .configure("httpPort", 8080));
+ }
+}
+{% endhighlight %}
+
+You can run this as follows (on *nix or Mac, assuming `ssh localhost` requires no password or passphrase):
+
+{% highlight bash %}
+% ${BROOKLYN_HOME}/bin/brooklyn launch --app brooklyn.demo.SingleWebServerExample \
+ --location localhost
+{% endhighlight %}
+
+
+Then visit the webapp on port 8080, or the Brooklyn console on localhost:8081.
+Note that the installation may take some time, because the default deployment downloads the software from
+the official repos. You can monitor start-up activity for each entity in the ``Activity`` pane in the management console,
+and see more detail by tailing the log file (``tail -f brooklyn.log``).
+
+With appropriate setup (as described [here]({{ site.url }}/use/guide/management/index.html#startup-config))
+this can also be deployed to your favourite cloud, let's pretend it's Amazon Ireland, as follows:
+
+{% highlight bash %}
+% ${BROOKLYN_HOME}/bin/brooklyn launch --app brooklyn.demo.SingleWebServerExample \
+ --location aws-ec2:eu-west-1
+{% endhighlight %}
+
+
+## Elastic Three-Tier
+
+Ready for something more interesting? Try this:
+
+{% highlight bash %}
+% ${BROOKLYN_HOME}/bin/brooklyn launch --app brooklyn.demo.WebClusterDatabaseExample \
+ --location localhost
+{% endhighlight %}
+
+This launches the class ``WebClusterDatabaseExample`` (also described in the [walkthrough]({{ site.url }}/start/walkthrough/index.html))
+which launches a pool of web-servers -- of size 1 initially,
+but manually configurable (if you stop the policy first, in the GUI, then use the ``resize`` effector) --
+with an Nginx load-balancer set up in front of them, and backed by a MySQL database.
+
+The essential code fragment looks like this:
+
+{% highlight java %}
+public class WebClusterDatabaseExample extends AbstractApplication {
+ public static final String WAR_PATH = "classpath://hello-world-sql-webapp.war";
+
+ public static final String DB_SETUP_SQL_URL = "classpath://visitors-creation-script.sql";
+
+ public static final String DB_TABLE = "visitors";
+ public static final String DB_USERNAME = "brooklyn";
+ public static final String DB_PASSWORD = "br00k11n";
+
+ @Override
+ public void init() {
+ MySqlNode mysql = addChild(EntitySpec.create(MySqlNode.class)
+ .configure("creationScriptUrl", DB_SETUP_SQL_URL));
+
+ ControlledDynamicWebAppCluster web = addChild(EntitySpec.create(ControlledDynamicWebAppCluster.class)
+ .configure("memberSpec", EntitySpec.create(JBoss7Server.class)
+ .configure("httpPort", "8080+")
+ .configure("war", WAR_PATH)
+ .configure(javaSysProp("brooklyn.example.db.url"),
+ formatString("jdbc:%s%s?user=%s\\&password=%s",
+ attributeWhenReady(mysql, MySqlNode.MYSQL_URL), DB_TABLE, DB_USERNAME, DB_PASSWORD))));
+
+ web.getCluster().addPolicy(AutoScalerPolicy.builder().
+ metric(DynamicWebAppCluster.AVERAGE_REQUESTS_PER_SECOND).
+ sizeRange(1, 5).
+ metricRange(10, 100).
+ build());
+ }
+}
+{% endhighlight %}
+
+You can, of course, try this with your favourite cloud,
+tweak the database start script, or drop in your favourite WAR.
+
+
+## A Few Other Things
+
+The project includes variants of the examples shown here,
+including alternative syntax (the `*Alt*` files),
+and a web-only cluster (no database) in `WebClusterExample``.
+
+The webapp that is used is included under ``examples/hello-world-webapp``.
+
+You may wish to check out the [Global Web Fabric example]({{ site.url }}/use/examples/global-web-fabric/) next.
+
+If you encounter any difficulties, please [tell us]({{ site.url }}/meta/contact.html) and we'll do our best to help.
http://git-wip-us.apache.org/repos/asf/incubator-brooklyn/blob/ba6612ec/docs/guide/use/guide/defining-applications/Chef.png
----------------------------------------------------------------------
diff --git a/docs/guide/use/guide/defining-applications/Chef.png b/docs/guide/use/guide/defining-applications/Chef.png
new file mode 100644
index 0000000..d899de2
Binary files /dev/null and b/docs/guide/use/guide/defining-applications/Chef.png differ
http://git-wip-us.apache.org/repos/asf/incubator-brooklyn/blob/ba6612ec/docs/guide/use/guide/defining-applications/advanced-concepts.md
----------------------------------------------------------------------
diff --git a/docs/guide/use/guide/defining-applications/advanced-concepts.md b/docs/guide/use/guide/defining-applications/advanced-concepts.md
new file mode 100644
index 0000000..3b808f2
--- /dev/null
+++ b/docs/guide/use/guide/defining-applications/advanced-concepts.md
@@ -0,0 +1,137 @@
+---
+title: Advanced Concepts
+layout: guide-normal
+toc: ../guide_toc.json
+categories: [use, guide, defining-applications]
+---
+
+Lifecycle and ManagementContext
+-------------------------------
+
+Under-the-covers, at heart of the brooklyn management plane is the ``ManagementContext``.
+This is started automatically when using launching an application using the brooklyn CLI. For programmatic use, see
+``BrooklynLauncher.newLauncher().launch()``.
+
+A Brooklyn deployment consists of many entities in a hierarchical tree, with the privileged *application* entity at the top level.
+
+An application entity (``Application`` class) is responsible for starting the deployment of all its child entities (i.e. the entire entity tree under its ownership).
+
+An ``Application``'s ``start()`` method begins provisioning the child entities of the application (and their entities, recursively).
+
+Provisioning of entities typically happens in parallel automatically,
+although this can be customized. This is implemented as ***tasks*** which are tracked by the management plane and is visible in the [web-based management console]({{site.url}}/use/guide/management/index.html#console).
+
+Customized provisioning can be useful where two starting entities depend on each other. For example, it is often necessary to delay start of one entity until another entity reaches a certain state, and to supply run-time information about the latter to the former.
+
+<!-- TODO ambiguous language; need a better description of the "manage" lifecycle -->
+When new entities are created, the entity is wired up to an application by giving it a parent. The entity is then explicitly "managed", which allows other entities to discover it.
+
+Typically a Brooklyn deployment has a single management context which records:
+
+* all entities under management that are reachable by the application(s) via the parent-child relationships,
+* the state associated with each entity,
+* subscribers (listeners) to sensor events arising from the entities,
+* active tasks (jobs) associated with any the entity,
+* which Brooklyn management node is mastering (managing) each entity.
+
+<!-- TODO Distributed brooklyn not yet supported; needs clarification in docs -->
+
+In a multi-location deployment, management operates in all regions, with brooklyn entity instances being mastered in the relevant region.
+
+When management is distributed a Brooklyn deployment may consist of multiple Brooklyn management nodes each with a ``ManagementContext`` instance.
+
+<!-- TODO - Clarify the following statements.
+The management context entity forms part of the management plane.
+The management plane is responsible for the distribution of the ``Entity`` instances across multiple machines and multiple locations,
+tracking the transfer of events (subscriptions) between ``Entity`` instances, and the execution of tasks (often initiated by management policies).
+-->
+
+<a name="dependent"></a>
+Dependent Configuration
+-----------------------
+
+Under the covers Brooklyn has a sophisticated sensor event and subscription model, but conveniences around this model make it very simple to express cross-entity dependencies. Consider the example where Tomcat instances need to know the URL of a database (or a set of URLs to connect to a Monterey processing fabric, or other entities)
+
+{% highlight java %}
+setConfiguration(UsesJava.JAVA_OPTIONS, ImmutableMap.of("mysql.url",
+ attributeWhenReady(mysql, MySqlNode.MY_SQL_URL) ))
+{% endhighlight %}
+
+The ``attributeWhenReady(Entity, Sensor)`` call (a static method on the class ``DependentConfiguration``)
+causes the configuration value to be set when that given entity's attribue is ready.
+In the example, ``attributeWhenReady()`` causes the JVM system property ``mysql.url`` to be set to the value of the ``MySqlNode.MY_SQL_URL`` sensor from ``mysql`` when that value is ready. As soon as the database URL is announced by the MySql entity, the configuration value will be available to the Tomcat cluster.
+
+By default "ready" means being *set* (non-null) and, if appropriate, *non-empty* (for collections and strings) or *non-zero* (for numbers). Formally the interpretation of ready is that of "Groovy truth" defined by an ``asBoolean()`` method on the class and in the Groovy language extensions.
+
+You can customize "readiness" by supplying a ``Predicate`` (Google common) or ``Closure`` (Groovy) in a third parameter.
+This evaluates candidate values reported by the sensor until one is found to be ``true``.
+For example, passing ``{ it.size()>=3 }`` as the readiness argument would require at least three management plane URLs.
+
+More information on this can be found in the javadoc for ``DependentConfiguration``,
+along with a few other methods such as ``valueWhenAttributeReady`` which allow post-processing of an attribute value.
+
+Note that if the value of ``CONFIG_KEY`` passed to ``Entity.getConfig`` is a Closure or Task (such as returned by ``attributeWhenReady``),
+the first access of ``Entity.getConfig(CONFIG_KEY)`` will block until the task completes.
+Typically this does the right thing, blocking when necessary to generate the right start-up sequence
+without the developer having to think through the order, but it can take some getting used to.
+Be careful not to request config information until really necessary (or to use non-blocking "raw" mechanisms),
+and in complicated situations be ready to attend to circular dependencies.
+The management console gives useful information for understanding what is happening and resolving the cycle.
+
+Location
+--------
+<!-- TODO, Clarify is how geographical location works.
+-->
+
+Entities can be provisioned/started in the location of your choice. Brooklyn transparently uses [jclouds](http://www.jclouds.org) to support different cloud providers and to support BYON (Bring Your Own Nodes).
+
+The implementation of an entity (e.g. Tomcat) is agnostic about where it will be installed/started. When writing the application definition specify the location or list of possible locations (``Location`` instances) for hosting the entity.
+
+``Location`` instances represent where they run and indicate how that location (resource or service) can be accessed.
+
+For example, a ``JBoss7Server`` will usually be running in an ``SshMachineLocation``, which contains the credentials and address for sshing to the machine. A cluster of such servers may be running in a ``MachineProvisioningLocation``, capable of creating new ``SshMachineLocation`` instances as required.
+
+<!-- TODO, incorporate the following.
+
+The idea is that you could specify the location as AWS and also supply an image id. You could configure the Tomcat entity accordingly: specify the path if the image already has Tomcat installed, or specify that Tomcat must be downloaded/installed. Entities typically use _drivers_ (such as SSH-based) to install, start, and interact with their corresponding real-world instance.
+-->
+
+Policies
+--------
+Policies perform the active management enabled by Brooklyn. Entities can have zero or more ``Policy`` instances attached to them.
+
+Policies can subscribe to sensors from entities or run periodically, and
+when they run they can perform calculations, look up other values, and if deemed necessary invoke effectors or emit sensor values from the entity with which they are associated.
+
+Execution
+---------
+
+All processing, whether an effector invocation or a policy cycle, are tracked as ***tasks***. This allows several important capabilities:
+
+* active and historic processing can be observed by operators
+* the invocation context is available in the thread, to check entitlement (permissions) and maintain a
+hierarchical causal chain even when operations are run in parallel
+* processing can be managed across multiple management nodes
+
+Some executions create new entities, which can then have tasks associated with them, and the system will record, for example, that a start efector on the new entity is a task associated with that entity, with that task
+created by a task associated with a different entity.
+
+The execution of a typical overall start-up sequence is shown below:
+
+[](brooklyn-flow-websequencediagrams.com.png)
+
+
+## Integration
+
+One vital aspect of Brooklyn is its ability to communicate with the systems it starts. This is abstracted using a ***driver*** facility in Brooklyn, where a
+driver describes how a process or service can be installed and managed using a particular technology.
+
+For example, a ``TomcatServer`` may implement start and other effectors using a ``TomcatSshDriver`` which inherits from ``JavaSoftwareProcessSshDriver`` (for JVM and JMX start confguration), inheriting from ``AbstractSoftwareProcessSshDriver``
+(for SSH scripting support).
+
+Particularly for sensors, some technologies are used so frequently that they are
+packaged as ***feeds*** which can discover their configuration (including from drivers). These include JMX and HTTP (see ``JmxFeed`` and ``HttpFeed``).
+
+Brooklyn comes with entity implementations for a growing number of commonly used systems, including various web application servers, databases and NoSQL data stores, and messaging systems. See: [Extras]({{site.url}}/use/guide/extras/index.html).
+
+
http://git-wip-us.apache.org/repos/asf/incubator-brooklyn/blob/ba6612ec/docs/guide/use/guide/defining-applications/archetype.md
----------------------------------------------------------------------
diff --git a/docs/guide/use/guide/defining-applications/archetype.md b/docs/guide/use/guide/defining-applications/archetype.md
new file mode 100644
index 0000000..4894af7
--- /dev/null
+++ b/docs/guide/use/guide/defining-applications/archetype.md
@@ -0,0 +1,64 @@
+---
+title: Common Usage
+layout: guide-normal
+toc: ../guide_toc.json
+categories: [use, guide, defining-applications]
+---
+
+### Maven Archetype
+
+Brooklyn includes a maven archetype, which can be used to create the project structure for a new application.
+
+This can be done interactively using:
+{% highlight bash %}
+$ mvn archetype:generate
+{% endhighlight %}
+
+The user will be prompted for the archetype to use (i.e. group "io.brooklyn"
+and artifact "brooklyn-archetype-quickstart"), as well as options for the project
+to be created.
+
+Alternatively, all options can be supplied at the command line. For example,
+if creating a project named "autobrick" for "com.acme":
+
+{% highlight bash %}
+$ mvn archetype:generate \
+ -DarchetypeGroupId=io.brooklyn \
+ -DarchetypeArtifactId=brooklyn-archetype-quickstart \
+ -DarchetypeVersion={{ site.brooklyn-version }} \
+ -DgroupId=com.acme -DartifactId=autobrick \
+ -Dversion=0.1.0-SNAPSHOT \
+ -DpackageName=com.acme.autobrick \
+ -DinteractiveMode=false
+{% endhighlight %}
+
+This will create a directory with the artifact name (e.g. "autobrick" in the example above).
+Note that if run from a directory containing a pom, it will also modify that pom to add this as a module!
+
+The project will contain an example app. You can run this, and also replace it with your own
+application code.
+
+To build, run the commands:
+
+{% highlight bash %}
+$ cd autobrick
+$ mvn clean install assembly:assembly
+{% endhighlight %}
+
+The assembly command will build a complete standalone distribution archive in `target/autobrick-0.1.0-SNAPSHOT-dist.tar.gz`,
+suitable for redistribution and containing `./start.sh` in the root.
+
+An unpacked equivalent is placed in `target/autobrick-0.1.0-SNAPSHOT-dist`,
+thus you can run the single-node sample locally with:
+
+{% highlight bash %}
+$ cd target/autobrick-0.1.0-SNAPSHOT-dist/autobrick-0.1.0-SNAPSHOT/
+$ ./start.sh launch --single
+{% endhighlight %}
+
+This `start.sh` script has all of the same options as the default `brooklyn` script,
+including `./start.sh help` and the `--location` argument for `launch`,
+with a couple of extra `launch` options for the sample blueprints in the archetype project:
+
+- `./start.sh launch --single` will launch a single app-server instance
+- `./start.sh launch --cluster` will launch a cluster of app-servers
http://git-wip-us.apache.org/repos/asf/incubator-brooklyn/blob/ba6612ec/docs/guide/use/guide/defining-applications/basic-concepts.md
----------------------------------------------------------------------
diff --git a/docs/guide/use/guide/defining-applications/basic-concepts.md b/docs/guide/use/guide/defining-applications/basic-concepts.md
new file mode 100644
index 0000000..51d36a6
--- /dev/null
+++ b/docs/guide/use/guide/defining-applications/basic-concepts.md
@@ -0,0 +1,95 @@
+---
+title: Basic Concepts
+layout: guide-normal
+toc: ../guide_toc.json
+categories: [use, guide, defining-applications]
+---
+
+This introduces brooklyn and describes how it simplifies the deployment and management of big applications. It is
+intended for people who are using brooklyn-supported application components (such as web/app servers, data stores)
+to be able to use brooklyn to easily start their application in multiple locations with off-the-shelf management
+policies.
+
+Entities
+--------
+
+The central concept in a Brooklyn deployment is that of an ***entity***.
+An entity represents a resource under management, either *base* entities (individual machines or software processes)
+or logical collections of these entities.
+
+Fundamental to the processing model is the capability of entities to be the *parent* of other entities (the mechanism by which collections are formed),
+with every entity having a single parent entity, up to the privileged top-level ***application*** entity.
+
+Entities are code, so they can be extended, overridden, and modified. Entities can have events, operations, and processing logic associated with them, and it is through this mechanism that the active management is delivered.
+
+The main responsibilities of an entity are:
+
+- Provisioning the entity in the given location or locations
+- Holding configuration and state (attributes) for the entity
+- Reporting monitoring data (sensors) about the status of the entity
+- Exposing operations (effectors) that can be performed on the entity
+- Hosting management policies and tasks related to the entity
+
+
+Application, Parent and Membership
+-------------------------------------
+
+All entities have a ***parent*** entity, which creates and manages it, with one important exception: *applications*.
+Application entities are the top-level entities created and managed externally, manually or programmatically.
+
+Applications are typically defined in Brooklyn as an ***application descriptor***.
+This is a Java class specifying the entities which make up the application,
+by extending the class ``AbstractApplication``, and specifying how these entities should be configured and managed.
+
+All entities, including applications, can be the parent of other entities.
+This means that the "child" is typically started, configured, and managed by the parent.
+For example, an application may be the parent of a web cluster; that cluster in turn is the parent of web server processes.
+In the management console, this is represented hierarchically in a tree view.
+
+A parallel concept is that of ***membership***: in addition to one fixed parent,
+and entity may be a ***member*** of any number of special entities called ***groups***.
+Membership of a group can be used for whatever purpose is required;
+for example, it can be used to manage a collection of entities together for one purpose
+(e.g. wide-area load-balancing between locations) even though they may have been
+created by different parents (e.g. a multi-tier stack within a location).
+
+
+Configuration, Sensors and Effectors
+------------------------------------
+
+### Configuration
+
+All entities contain a map of config information. This can contain arbitrary values, typically keyed under static ``ConfigKey`` fields on the ``Entity`` sub-class. These values are inherited, so setting a configuration value at the
+application level will make it available in all entities underneath unless it is overridden.
+
+Configuration is propagated when an application "goes live" (i.e. it becomes "managed", either explicitly or when its ``start()`` method is invoked), so config values must be set before this occurs.
+
+Configuration values can be specified in a configuration file (``~/.brooklyn/brooklyn.properties``)
+to apply universally, and/or programmatically to a specific entity and its descendants
+by calling `.configure(KEY, VALUE)` in the entity spec when creating it.
+There is also an ``entity.setConfig(KEY, VALUE)`` method.
+
+Additionally, many common configuration parameters are available as "flags" which can be supplied as Strings when constructing
+then entity, in the form
+``EntitySpec.create˙(MyEntity.class).configure("config1", "value1").configure("config2", "value2")``.
+
+Documentation of the flags available for individual entities can normally be found in the javadocs.
+The ``@SetFromFlag`` annotations on ``ConfigKey`` static field definitions
+in the entity's interface is the recommended mechanism for exposing configuration options.
+
+
+### Sensors and Effectors
+
+***Sensors*** (activity information and notifications) and ***effectors*** (operations that can be invoked on the entity) are defined by entities as static fields on the ``Entity`` subclass.
+
+Sensors can be updated by the entity or associated tasks, and sensors from an entity can be subscribed to by its parent or other entities to track changes in an entity's activity.
+
+Effectors can be invoked by an entity's parent remotely, and the invoker is able to track the execution of that effector. Effectors can be invoked by other entities, but use this functionality with care to prevent too many managers!
+
+An entity consists of a Java interface (used when interacting with the entity) and a Java class. For resilience. it is recommended to store
+the entity's state in attributes (see `getAttribute(AttributeKey)``). If internal fields can be used then the data will be lost on brooklyn
+restart, and may cause problems if the entity is to be moved to a different brooklyn management node.
+
+Next: [Advanced Concepts]({{site.url}}/use/guide/defining-applications/advanced-concepts.html).
+See also: [Management > Sensors and Effectors]({{site.url}}/use/guide/management/index.html#sensors-and-effectors).
+
http://git-wip-us.apache.org/repos/asf/incubator-brooklyn/blob/ba6612ec/docs/guide/use/guide/defining-applications/brooklyn-flow-websequencediagrams.com-w400.png
----------------------------------------------------------------------
diff --git a/docs/guide/use/guide/defining-applications/brooklyn-flow-websequencediagrams.com-w400.png b/docs/guide/use/guide/defining-applications/brooklyn-flow-websequencediagrams.com-w400.png
new file mode 100644
index 0000000..c2b48ba
Binary files /dev/null and b/docs/guide/use/guide/defining-applications/brooklyn-flow-websequencediagrams.com-w400.png differ
http://git-wip-us.apache.org/repos/asf/incubator-brooklyn/blob/ba6612ec/docs/guide/use/guide/defining-applications/brooklyn-flow-websequencediagrams.com.png
----------------------------------------------------------------------
diff --git a/docs/guide/use/guide/defining-applications/brooklyn-flow-websequencediagrams.com.png b/docs/guide/use/guide/defining-applications/brooklyn-flow-websequencediagrams.com.png
new file mode 100644
index 0000000..78eb5c8
Binary files /dev/null and b/docs/guide/use/guide/defining-applications/brooklyn-flow-websequencediagrams.com.png differ
http://git-wip-us.apache.org/repos/asf/incubator-brooklyn/blob/ba6612ec/docs/guide/use/guide/defining-applications/chef-blueprints.md
----------------------------------------------------------------------
diff --git a/docs/guide/use/guide/defining-applications/chef-blueprints.md b/docs/guide/use/guide/defining-applications/chef-blueprints.md
new file mode 100644
index 0000000..4f9612a
--- /dev/null
+++ b/docs/guide/use/guide/defining-applications/chef-blueprints.md
@@ -0,0 +1,277 @@
+---
+title: Using Chef in YAML Blueprints
+layout: guide-normal
+toc: ../guide_toc.json
+categories: [use, guide, defining-applications]
+---
+
+This guide describes how Brooklyn entities can be easily created from Chef cookbooks.
+As of this writing (May 2014) some of the integration points are under active development,
+and comments are welcome.
+A plan for the full integration is online [here](https://docs.google.com/a/cloudsoftcorp.com/document/d/18ZwzmncbJgJeQjnSvMapTWg6N526cvGMz5jaqdkxMf8).
+
+This guide assumes you are familiar with the basics of [creating YAML blueprints](creating-yaml.html).
+
+
+## What you need to know about Chef
+
+Chef works in two different modes, *server* and *solo*. *Server* is where the Chef client talks to a central server
+to retrieve information about its roles, policies and cookbooks (where a cookbook defines how to install and
+configure a particular piece of software). With *solo*, the client works in isolation, therefore its configuration
+and cookbooks must be supplied by another means.
+
+Chef *client* is the Chef agent. This is a Ruby application which is installed on each and every managed host. When
+invoked in server mode, it will contact the Chef server to check for updates to cookbooks and policy; it then "runs"
+the recipes in its run lists, to converge the machine to a known state. In solo mode, it reads the locally-maintained
+cookbooks and policies. The client may be run as a daemon that checks the server regularly, or it could merely be
+run manually when required.
+
+The *policy* is a set of rules on the Chef server. A client starts with a set of *attributes*, which could be as
+simple as its name and a recipe runlist, or which may involve a more complex set of attributes about how it is to be
+configured. The client then augments this with auto-detected metadata - a tool called `ohai` is run that collects
+detailed information about the host. Next, the policy on the server modifies these attributes - overriding some,
+setting defaults for others - to produce a final set of attributes. It is these which are the input to the recipes.
+Finally, the attributes are uploaded to the server where they are stored as metadata for the node, where they can be
+inspected and modified by the system operator.
+
+Also of interest is `knife`, which is the workstation toolkit for Chef. Typically this would be installed on the
+operation engineer's workstation, where it would be used to interact with the Chef server and clients. Of particular
+interest to us is the *bootstrap* operation, which is used for setting up new Chef clients - given a virtual machine,
+it will install the Chef client on it, configure it with enough information to find the Chef server and performs its
+first run, and then kicks off the Chef client for the first time.
+
+There is often a preconception about how a Chef client is bootstrapped; mistakenly, there is the belief that the
+`knife` tool configures the Chef server with information about the client, and the client finds out about itself from
+the server. This is not the case - the bootstrap operation does not involve `knife` talking to the server. Instead,
+`knife` packages up all of the required information and sends it to the client - the client will then introduce
+itself to the server, passing on its configuration.
+
+This diagram summarises the interaction between Brooklyn, the new node, and the various Chef tools. Note that there
+is no interaction between the AMP Server and the Chef Server.
+
+[](Chef.png)
+
+### How Brooklyn interacts with Chef
+
+Brooklyn understands both the *server* and *solo* modes of operation. Server mode utilises the `knife` toolkit, and
+therefore `knife` must be installed onto the AMP server and configured appropriately. Solo mode does not have any
+special requirements; when running in solo mode, Brooklyn will install and configure the Chef client over SSH, just
+like it does most other kinds of entities.
+
+
+## Creating Blueprints from Chef
+
+In a nutshell, a new Chef-based entity can be defined as a service by specifying
+`chef:cookbook_name` as the `service_type`, along with a collection of optional configuration.
+An illustrative example is below:
+
+{% highlight yaml %}
+{% readj example_yaml/mysql-chef-1.yaml %}
+{% endhighlight %}
+
+*This works without any installation: try it now, copying-and-pasting to the Brooklyn console.
+(Don't forget to add your preferred `location: some-cloud` to the spec.)*
+
+We'll now walk through the important constituent parts,
+and then proceed to describing things which can be done to simplify the deployment.
+
+
+### Cookbook Primary Name
+
+The first thing to note is the type definition:
+
+ - type: chef:mysql
+
+This indicates that the Chef entity should be used (`brooklyn.entity.chef.ChefEntity`)
+to interpret and pass the configuration,
+and that it should be parameterised with a `brooklyn.chef.cookbook.primary.name` of `mysql`.
+This is the cookbook namespace used by default for determining what to install and run.
+
+
+### Importing Cookbooks
+
+Next we specify which cookbooks are required and where they can be pulled from:
+
+ cookbook_urls:
+ mysql: https://github.com/opscode-cookbooks/mysql/archive/v4.0.12.tar.gz
+ openssl: https://github.com/opscode-cookbooks/openssl/archive/v1.1.0.tar.gz
+ build-essential: https://github.com/opscode-cookbooks/build-essential/archive/v1.4.4.tar.gz
+
+Here, specific versions are being downloaded from the canonical github repository.
+Any URL can be used, so long as it is resolvable on either the target machine or the
+Brooklyn server; this includes `file:` and `classpath:` URLs.
+
+The archive can be ZIP or TAR or TGZ.
+
+The structure of the archive must be that a single folder is off the root,
+and in that folder contains the usual Chef recipe and auxiliary files.
+For example, the archive might contain `mysql-master/recipes/server.rb`.
+Archives such as those above from github match this format.
+The name of that folder does not matter, as often they contain version information.
+When deployed, these will be renamed to match the short name (the key in the `cookbooks_url` map,
+for instance `mysql` or `openssl`).
+
+If Chef server is configured (see below), this section can be omitted.
+
+
+### Launch Run List and Attributes
+
+The next part is to specify the Chef run list and attributes to store when launching the entity:
+
+ launch_run_list:
+ - mysql::server
+
+ launch_attributes:
+ mysql:
+ server_root_password: p4ssw0rd
+ server_repl_password: p4ssw0rd
+ server_debian_password: p4ssw0rd
+
+For the `launch_run_list`, you can use either the YAML `- recipe` syntax or the JSON `[ "recipe" ]` syntax.
+
+The `launch_attributes` key takes a map which will be stored against the `node` object in Chef.
+Thus in this example, the parameter `node['mysql']['server_root_password']` required by the mysql blueprint
+is set as specified.
+
+You can of course set many other attributes in this manner, in addition to those that are required!
+
+
+### Simple Monitoring
+
+The final section determines how Brooklyn confirms that the service is up.
+Sophisticated solutions may install monitoring agents as part of the `launch_run_list`,
+with Brooklyn configured to read monitoring information to confirm the launch was successful.
+However for convenience, two common mechanisms are available out of the box:
+
+ #service_name: mysqld
+ pid_file: /var/run/mysqld/mysqld.pid
+
+If `service_name` is supplied, Brooklyn will check the return code of the `status` command
+run against that service, ensuring it is 0. (Note that this is not universally reliable,
+although it is the same mechanism which Chef typically uses to test status when determining
+whether to start a service. Some services, e.g. postgres, will return 0 even if the service
+is not running.)
+
+If a `pid_file` is supplied, Brooklyn will check whether a process with the PID specified in that
+file is running. This has been selected for mysql because it appears to be more portable:
+the service name varies among OS's: it is `mysqld` on CentOS but `mysql` on Ubuntu!
+
+
+## Making it Simpler: Writing Chef for Blueprints
+
+The example we've just seen shows how existing Chef cookbooks can be
+used as the basis for entities. If you're *writing* the Chef recipes,
+there are a few simple techniques we've established with the Chef community
+which make blueprints literally as simple as:
+
+ - type: chef:mysql
+ brooklyn.config:
+ mysql_password: p4ssw0rd
+ pid_file: /var/run/mysqld/mysqld.pid
+
+
+### Some Basic Conventions
+
+* **A `start` recipe**:
+ The first step is to provide a `start` recipe in `recipes/start.rb`;
+ if no `launch_run_list` is supplied, this is what will be invoked to launch the entity.
+ It can be as simple as a one-line file:
+
+ include_recipe 'mysql::server'
+
+* **Using `brooklyn.config`**:
+ All the `brooklyn.config` is passed to Chef as node attributes in the `node['brooklyn']['config']` namespace.
+ Thus if the required attributes in the mysql recipe are set to take a value set in
+ `node['brooklyn']['config']['mysql_password']`, you can dispense with the `launch_attributes` section.
+
+
+## Using Chef Server
+
+The examples so far have not required Chef Server, so they will work without any external
+Chef dependencies (besides the built-in install from `https://www.opscode.com/chef/install.sh`
+and the explicitly referenced cookbooks). If you use Chef Server, however, you'll want your
+managed nodes to be integrated with it. This is easy to set up, with a few options:
+
+If you have `knife` set up in your shell environment, the Brooklyn Chef support will use it
+by default. If the recipes are installed in your Chef server, you can go ahead and remove
+the `cookbooks_url` section!
+
+Use of `solo` or `knife` can be forced by setting the `chef_mode` flag (`brooklyn.chef.mode` config key)
+to either of those values. (It defaults to `autodetect`, which will use `knife` if it is on the path and satisfies
+sanity checks).
+
+If you want to specify a different configuration, there are a number of config keys you can use:
+
+* `brooklyn.chef.knife.executableFile`: this should be point to the knife binary to use
+* `brooklyn.chef.knife.configFile`: this should point to the knife configuration to use
+* `brooklyn.chef.knife.setupCommands`: an optional set of commands to run prior to invoking knife,
+ for example to run `rvm` to get the right ruby version on the Brooklyn server
+
+If you're interested in seeing the Chef REST API be supported directly (without knife),
+please let us know. We'd like to see this too, and we'll help you along the way!
+
+
+## Tips and Tricks
+
+To help you on your way writing Chef blueprints, here are a handful of pointers
+particularly useful in this context:
+
+* Configuration keys can be inherited from the top-level and accessed using `$brooklyn:component('id').config('key_name')`.
+ An example of this is shown in the `mysql-chef.yaml` sample recipe contained in the Brooklyn code base
+ and [here](example_yaml/mysql-chef-2.yaml) for convenience.
+ Here, `p4ssw0rd` is specified only once and then used for all the attributes required by the stock mysql cookbook.
+
+* Github tarball downloads! You'll have noticed these in the example already, but they are so useful we thought
+ we'd call them out again. Except when you're developing, we recommend using specific tagged versions rather than master.
+
+* The usual machine `provisioning.properties` are supported with Chef blueprints,
+ so you can set things like `minRam` and `osFamily`
+
+* To see more configuration options, and understand the ones presented here in more detail, see the javadoc or
+ the code for the class `ChefConfig` in the Brooklyn code base.
+
+
+## Advanced Chef Integration
+
+### Adding Sensors and Effectors
+
+Custom sensors and effectors can be added using an `entity.initializer` section in the YAML blueprint.
+
+One common pattern is to have sensors which extract information from Ohai.
+Another common pattern is to install a monitoring agent as part of the run list,
+configured to talk to a monitoring store, and then to add a sensor feed which reads data from that store.
+
+On the effector side, you can add SSH-based effectors in the usual way.
+You can also describe additional chef converge targets following the pattern set down in
+`ChefLifecycleEffectorTasks`, making use of conveniences in `ChefSoloTasks` and `ChefServerTasks`,
+or provide effectors which invoke network API's of the systems under management
+(for example to supply the common `executeScript` effector as on the standard `MySqlNode`).
+
+
+### Next Steps: Simpifying sensors and effectors, transferring files, and configuring ports
+
+The Brooklyn-Chef integration is work in progress, with a few open issues we'd still like to add.
+Much of the thinking for this is set forth in the [Google document](https://docs.google.com/a/cloudsoftcorp.com/document/d/18ZwzmncbJgJeQjnSvMapTWg6N526cvGMz5jaqdkxMf8)
+indicated earlier. If you'd like to work with us to implement these, please let us know.
+
+
+## Reference
+
+A general schema for the supported YAML is below:
+
+```
+- type: chef:cookbook_name
+ cookbook_urls:
+ cookbook_name: url://for/cookbook.tgz
+ dependency1: url://for/dependency1.tgz
+ launch_run_list: [ "cookbook_name::start" ]
+ launch_attributes: # map of arguments to set in the chef node
+ service_name: cookbook_service
+ pid_file: /var/run/cookbook.pid
+```
+
+If you are interested in exploring the Java code for creating blueprints,
+start with the `TypedToyMySqlEntiyChef` class, which essentially does what this tutorial has shown;
+and then move on to the `DynamicToyMySqlEntiyChef` which starts to look at more sophisticated constructs.
+(Familiarity with BASH and basic Java blueprints may be useful at that stage.)
+
http://git-wip-us.apache.org/repos/asf/incubator-brooklyn/blob/ba6612ec/docs/guide/use/guide/defining-applications/common-usage.md
----------------------------------------------------------------------
diff --git a/docs/guide/use/guide/defining-applications/common-usage.md b/docs/guide/use/guide/defining-applications/common-usage.md
new file mode 100644
index 0000000..a3beed6
--- /dev/null
+++ b/docs/guide/use/guide/defining-applications/common-usage.md
@@ -0,0 +1,145 @@
+---
+title: Common Usage
+layout: guide-normal
+toc: ../guide_toc.json
+categories: [use, guide, defining-applications]
+---
+
+### Entity Class Hierarchy
+
+By convention in Brooklyn the following words have a particular meaning, both as types (which extend ``Group``, which extends ``Entity``) and when used as words in other entities (such as ``TomcatFabric``):
+
+- *Cluster* - a homogeneous collection of entities
+- *Fabric* - a multi-location collection of entities, with one per location; often used with a cluster per location
+- *Stack* - heterogeneous (mixed types of children)
+- *Application* - user's entry point
+
+<!---
+TODO
+-->
+
+- *entity spec* defines an entity, so that one or more such entities can be created; often used by clusters/groups to define how to instantiate new children.
+- *entity factories* are often used by clusters/groups to define how to instantiate new children.
+- *traits* (mixins) providing certain capabilities, such as Resizable and Balanceable
+- *Resizable* entities can re-sized dynamically, to increase/decrease the number of child entities.
+- *Movable* entities can be migrated between *balanceable containers*.
+- *Balanceable containers* can contain *movable* entities, where each contained entity is normally associated with
+ a piece of work within that container.
+
+### Off-the-Shelf Entities
+
+brooklyn includes a selection of entities already available for use in applications,
+including appropriate sensors and effectors, and in some cases include Cluster and Fabric variants.
+(These are also useful as templates for writing new entities.)
+
+These include:
+
+- **Web**: Tomcat, JBoss, Jetty (external), Play (external); nginx; GeoScaling
+- **Data**: MySQL, Redis, MongoDB, Infinispan, GemFire (external)
+- **Containers**: Karaf
+- **Messaging**: ActiveMQ, Qpid, Rabbit MQ
+- **PaaS**: Cloud Foundry, Stackato; OpenShift
+
+
+### <a id="locations"></a> Off-the-Shelf Locations
+
+<a id="locations"></a>
+
+Brooklyn supports deploying to any machine which admits SSH access, as well as to
+a huge variety of external and on-premise clouds. You can also connect to services,
+or use whatever technique for deployment suits you best (such as Xebia Overthere, in development!).
+
+Configuration is typically set in `~/.brooklyn/brooklyn.properties` using keys such as the following:
+
+{% highlight bash %}
+# use this key for localhost (this is the default, although if you have a passphrase you must set it)
+brooklyn.location.localhost.privateKeyFile=~/.ssh/id_rsa
+
+brooklyn.location.localhost.privateKeyPassphrase=s3cr3tPASSPHRASE
+
+# use a special key when connecting to public clouds, and a particularly special one for AWS
+brooklyn.location.jclouds.privateKeyFile=~/.ssh/public_clouds/id_rsa
+brooklyn.location.jclouds.aws-ec2.privateKeyFile=~/.ssh/public_clouds/aws_id_rsa
+
+# AWS credentials (when deploying to location jclouds:aws-ec2)
+brooklyn.location.jclouds.aws-ec2.identity=ABCDEFGHIJKLMNOPQRST
+brooklyn.location.jclouds.aws-ec2.credential=s3cr3tsq1rr3ls3cr3tsq1rr3ls3cr3tsq1rr3l
+
+# credentials for 'geoscaling' service
+brooklyn.geoscaling.username=cloudsoft
+brooklyn.geoscaling.password=xxx
+{% endhighlight %}
+
+These can also be set as environment variables (in the shell) or system properties (java command line).
+(There are also ``BROOKLYN_JCLOUDS_PRIVATE_KEY_FILE`` variants accepted.)
+
+For any jclouds provider you will typically need to set ``identity`` and ``credential``
+in the ``brooklyn.location.jclouds.provider`` namespace.
+
+To deploy to sets of machines with known IP's, assuming you have the credentials,
+use the syntax ``byon:(hosts="user@10.9.1.1,user@10.9.1.2,user@10.9.1.3")``
+(this requires your default private key to have access;
+see the ``prod1`` example below for specifying other credentials).
+
+A wide range of other fields is available, because in the real world sometimes things do get complicated.
+The following is supported from the configuration file (with whatever customization you might want available in code):
+
+- If there is a passphrase on the key file being used, you must supply it to Brooklyn for it to work, of course!
+ ``privateKeyPassphrase`` does the trick (as in ``brooklyn.location.jclouds.privateKeyPassphrase``, or other places
+ where ``privateKeyFile`` is valid). If you don't like keys, you can just use a plain old ``password``.
+
+- Hardware requirements such as ``minRam`` and ``minCores`` can be supplied, or a ``hardwareId`` (jclouds only)
+
+- Specific Secury Groups can be specified using `securityGroups`, if you want to reuse set of existing ones (jclouds only)
+
+- Specific KeyPair can be specified using `keyPair`, if you want to reuse an existing keypair (jclouds only).
+
+- Specific VM images can be specified using ``imageId`` or ``imageNameRegex`` (jclouds only)
+
+- User metadata can be attached, using the syntax ``userMetadata=key=value,key2="value 2"`` (jclouds only)
+
+- A ``user`` can be specified, with the property that -- in a jclouds world -- the user will be *created* on the machine,
+ with admin rights, authorizing the relevant public key (corresponding to the private key, or as described below).
+ Login for the root account will be disabled, as will login by password if a public key is supplied.
+ (This is skipped if ``user`` is the ``root`` or other initial login user.)
+
+- You can specify the user account to use to login to jclouds initially with the ``loginUser`` property.
+ Typically this is auto-detected by jclouds
+ (often ``root``, or ``ubuntu`` or ``ec2-user`` for known Ubuntu or Amazon Linux images),
+ but the strategy isn't foolproof, particularly in some private cloud setups. (jclouds only). In some cases, you may need to specify a `loginUser.privateKeyFile` if the image you are using doesn't allow ssh password login.
+
+- Public keys can be specified using ``publicKeyFile``,
+ although these can usually be omitted if they follow the common pattern of being
+ the private key file with the suffix ``.pub`` appended.
+ (It is useful in the case of ``loginUser.publicKeyFile``, where you shouldn't need,
+ or might not even have, the private key of the ``root`` user in order to log in.)
+
+- You can specify the number of attempts Brooklyn should make to create
+ machines with ``machineCreateAttempts`` (jclouds only). This is useful for
+ working around the rare occasions in which cloud providers give machines that
+ are dead on arrival.
+
+You can also define named locations for commonly used groups of properties,
+with the syntax ``brooklyn.location.named.your-group-name.``
+followed by the relevant properties.
+These can be accessed at runtime using the syntax ``named:your-group-name`` as the deployment location.
+
+Some more advanced examples showing the syntax and properties above are as follows:
+
+{% highlight bash %}
+# Production pool of machines for my application (deploy to named:prod1)
+brooklyn.location.named.prod1=byon:(hosts="10.9.1.1,10.9.1.2,produser2@10.9.2.{10,11,20-29}")
+brooklyn.location.named.prod1.user=produser1
+brooklyn.location.named.prod1.privateKeyFile=~/.ssh/produser_id_rsa
+brooklyn.location.named.prod1.privateKeyPassphrase=s3cr3tCOMPANYpassphrase
+
+# AWS using my company's credentials and image standard, then labelling images so others know they're mine
+brooklyn.location.named.company-jungle=jclouds:aws-ec2:us-west-1
+brooklyn.location.named.company-jungle.identity=BCDEFGHIJKLMNOPQRSTU
+brooklyn.location.named.company-jungle.privateKeyFile=~/.ssh/public_clouds/company_aws_id_rsa
+brooklyn.location.named.company-jungle.imageId=ami-12345
+brooklyn.location.named.company-jungle.minRam=2048
+brooklyn.location.named.company-jungle.userMetadata=application=my-jungle-app,owner="Bob Johnson"
+brooklyn.location.named.company-jungle.machineCreateAttempts=2
+{% endhighlight %}
+