You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@brooklyn.apache.org by nakomis <gi...@git.apache.org> on 2016/06/20 09:47:36 UTC
[GitHub] brooklyn-docs pull request #78: Adds details of setting up a HA master-slave...
GitHub user nakomis opened a pull request:
https://github.com/apache/brooklyn-docs/pull/78
Adds details of setting up a HA master-slave pair
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/nakomis/brooklyn-docs ha-supplemental
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/brooklyn-docs/pull/78.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #78
----
commit a8f837a939a28a495a26403c78a0af624f107082
Author: Martin Harris <gi...@nakomis.com>
Date: 2016-06-20T09:47:05Z
Adds details of setting up a HA master-slave pair
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] brooklyn-docs pull request #78: Adds details of setting up a HA master-slave...
Posted by drigodwin <gi...@git.apache.org>.
Github user drigodwin commented on a diff in the pull request:
https://github.com/apache/brooklyn-docs/pull/78#discussion_r67667031
--- Diff: guide/ops/high-availability-supplemental.md ---
@@ -0,0 +1,97 @@
+---
+title: High Availability (Supplemental)
+layout: website-normal
+---
+
+This document supplements the High Availability documentation available [here](http://brooklyn.apache.org/v/latest/ops/high-availability.html)
+and provides an example of how to configure a pair of Apache Brooklyn servers to run in master-standby mode with a shared NFS datastore
+
+### Prerequisites
+- Two VMs (or physical machines) have been provisioned
+- NFS or another suitable file system has been configured and is available to both VMs
+- An NFS folder has been mounted on both VMs at `/mnt/amp-persistence` and both machines can write to the folder
+
+### Launching
+To start, download the latest Apache Brooklyn release on both VMs, extract it and navigate to the root folder:
+
+{% highlight bash %}
+wget -O apache-brooklyn-0.9.0-bin.tar.gz "https://www.apache.org/dyn/closer.lua?action=download&filename=brooklyn/apache-brooklyn-0.9.0/apache-brooklyn-0.9.0-bin.tar.gz"
--- End diff --
You should use apache-brooklyn_{{site.brooklyn-version}} instead of naming a specific version here
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] brooklyn-docs pull request #78: Adds details of setting up a HA master-slave...
Posted by aledsage <gi...@git.apache.org>.
Github user aledsage commented on a diff in the pull request:
https://github.com/apache/brooklyn-docs/pull/78#discussion_r68121310
--- Diff: guide/ops/high-availability/high-availability-supplemental.md ---
@@ -0,0 +1,142 @@
+---
+title: High Availability (Supplemental)
+layout: website-normal
+---
+
+This document supplements the High Availability documentation available [here](http://brooklyn.apache.org/v/latest/ops/high-availability.html)
+and provides an example of how to configure a pair of Apache Brooklyn servers to run in master-standby mode with a shared NFS datastore
+
+### Prerequisites
+- Two VMs (or physical machines) have been provisioned
+- NFS or another suitable file system has been configured and is available to both VMs*
+- An NFS folder has been mounted on both VMs at `/mnt/brooklyn-persistence` and both machines can write to the folder
+
+\* Brooklyn can be configured to use either an object store such as S3, or a shared NFS mount. The recommended option is to use an object
+store as described in the [Object Store Persistence](./persistence/#object-store-persistence) documentation. For clarity, a shared NFS folder
+is assumed in this example
+
+### Launching
+To start, download and install the latest Apache Brooklyn release on both VMs following the 'OSX / Linux' section
+of the [Running Apache Brooklyn](../start/running.html#install-apache-brooklyn) documentation
+
+On the first VM, which will be the master node, run the following to start Brooklyn in high availability mode:
+
+{% highlight bash %}
+$ bin/brooklyn launch --highAvailability master --persist auto --persistenceDir /mnt/brooklyn-persistence
+{% endhighlight %}
+
+Once Brooklyn has launched, on the second VM, run the following command to launch Brooklyn in standby mode:
+
+{% highlight bash %}
+$ bin/brooklyn launch --highAvailability auto --persist auto --persistenceDir /mnt/brooklyn-persistence
+{% endhighlight %}
+
+### Testing
+You can now confirm that Brooklyn is running in high availibility mode on the master by logging into the web console at `http://<ip-address>:8081`.
+Similarly you can log into the web console on the standby VM where you will see a warning that the server is not the high availability master.
+To test a failover, you can simply terminate the process on the first VM and log into the web console on the second VM. Upon launch, Brooklyn will
+output its PID to the file `pid.txt`; you can terminate the process by running the following command from the same directory from which you
+launched Brooklyn:
+
+{% highlight bash %}
+$ kill -9 $(cat pid.txt)
+{% endhighlight %}
+
+It is also possiblity to check the high availability state of a running Brooklyn server using the following curl command:
+
+{% highlight bash %}
+$ curl -u myusername:mypassword http://<ip-address>:8081/v1/server/ha/state
+{% endhighlight %}
+
+This will return one of the following states:
+
+{% highlight bash %}
+
+"INITIALIZING"
+"STANDBY"
+"HOT_STANDBY"
+"HOT_BACKUP"
+"MASTER"
+"FAILED"
+"TERMINATED"
+
+{% endhighlight %}
+
+Note: The quotation characters will be included in the reply
+
+To obtain information about all of the nodes in the cluster, run the following command against any of the nodes in the cluster:
+
+{% highlight bash %}
+$ curl -u myusername:mypassword http://<ip-address>:8081/v1/server/ha/states
+{% endhighlight %}
+
+This will return a JSON document describing the Brooklyn nodes in the cluster. An example of two HA Brooklyn nodes is as follows (whitespace formatting has been
+added for clarity):
+
+{% highlight yaml %}
+
+{
+ ownId: "XkJeXUXE",
+ masterId: "yAVz0fzo",
+ nodes: {
+ yAVz0fzo: {
+ nodeId: "yAVz0fzo",
+ nodeUri: "http://<server1-ip-address>:8081/",
+ status: "MASTER",
+ localTimestamp: 1466414301065,
+ remoteTimestamp: 1466414301000
+ },
+ XkJeXUXE: {
+ nodeId: "XkJeXUXE",
+ nodeUri: "http://<server2-ip-address>:8081/",
+ status: "STANDBY",
+ localTimestamp: 1466414301066,
+ remoteTimestamp: 1466414301000
+ }
+ },
+ links: { }
+}
+
+{% endhighlight %}
+
+The examples above show how to use `curl` to manually check the status of Brooklyn via its REST API. The same REST API calls can also be used by
+automated third party monitoring tools such as Monit
+
+### Failover
+When running as a HA standby node, each standby Brooklyn server (in this case there is only one standby) will check the shared persisted state
+every 1 second to determine the state of the HA master. If no heartbeat has been recorded for thirty seconds, then an election will be performed
+and one of the standby nodes will be promoted to master. At this point all requests should be directed to the new master node
+
+In the event that tasks - such as the provisioning of a new entity - are running when a failover occurs, the new master will display the current
+state of the entity, but will not resume its provisioning or re-run any partially completed tasks. In this case it will usually be necesarry
--- End diff --
typo: necessary
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] brooklyn-docs pull request #78: Adds details of setting up a HA master-slave...
Posted by aledsage <gi...@git.apache.org>.
Github user aledsage commented on a diff in the pull request:
https://github.com/apache/brooklyn-docs/pull/78#discussion_r68121072
--- Diff: guide/ops/high-availability/high-availability-supplemental.md ---
@@ -0,0 +1,142 @@
+---
+title: High Availability (Supplemental)
+layout: website-normal
+---
+
+This document supplements the High Availability documentation available [here](http://brooklyn.apache.org/v/latest/ops/high-availability.html)
+and provides an example of how to configure a pair of Apache Brooklyn servers to run in master-standby mode with a shared NFS datastore
+
+### Prerequisites
+- Two VMs (or physical machines) have been provisioned
+- NFS or another suitable file system has been configured and is available to both VMs*
+- An NFS folder has been mounted on both VMs at `/mnt/brooklyn-persistence` and both machines can write to the folder
+
+\* Brooklyn can be configured to use either an object store such as S3, or a shared NFS mount. The recommended option is to use an object
+store as described in the [Object Store Persistence](./persistence/#object-store-persistence) documentation. For clarity, a shared NFS folder
+is assumed in this example
+
+### Launching
+To start, download and install the latest Apache Brooklyn release on both VMs following the 'OSX / Linux' section
+of the [Running Apache Brooklyn](../start/running.html#install-apache-brooklyn) documentation
+
+On the first VM, which will be the master node, run the following to start Brooklyn in high availability mode:
+
+{% highlight bash %}
+$ bin/brooklyn launch --highAvailability master --persist auto --persistenceDir /mnt/brooklyn-persistence
+{% endhighlight %}
+
+Once Brooklyn has launched, on the second VM, run the following command to launch Brooklyn in standby mode:
+
+{% highlight bash %}
+$ bin/brooklyn launch --highAvailability auto --persist auto --persistenceDir /mnt/brooklyn-persistence
+{% endhighlight %}
+
+### Testing
+You can now confirm that Brooklyn is running in high availibility mode on the master by logging into the web console at `http://<ip-address>:8081`.
+Similarly you can log into the web console on the standby VM where you will see a warning that the server is not the high availability master.
+To test a failover, you can simply terminate the process on the first VM and log into the web console on the second VM. Upon launch, Brooklyn will
+output its PID to the file `pid.txt`; you can terminate the process by running the following command from the same directory from which you
+launched Brooklyn:
+
+{% highlight bash %}
+$ kill -9 $(cat pid.txt)
+{% endhighlight %}
+
+It is also possiblity to check the high availability state of a running Brooklyn server using the following curl command:
+
+{% highlight bash %}
+$ curl -u myusername:mypassword http://<ip-address>:8081/v1/server/ha/state
+{% endhighlight %}
+
+This will return one of the following states:
+
+{% highlight bash %}
+
+"INITIALIZING"
+"STANDBY"
+"HOT_STANDBY"
+"HOT_BACKUP"
+"MASTER"
+"FAILED"
+"TERMINATED"
+
+{% endhighlight %}
+
+Note: The quotation characters will be included in the reply
+
+To obtain information about all of the nodes in the cluster, run the following command against any of the nodes in the cluster:
+
+{% highlight bash %}
+$ curl -u myusername:mypassword http://<ip-address>:8081/v1/server/ha/states
+{% endhighlight %}
+
+This will return a JSON document describing the Brooklyn nodes in the cluster. An example of two HA Brooklyn nodes is as follows (whitespace formatting has been
+added for clarity):
+
+{% highlight yaml %}
+
+{
+ ownId: "XkJeXUXE",
+ masterId: "yAVz0fzo",
+ nodes: {
+ yAVz0fzo: {
+ nodeId: "yAVz0fzo",
+ nodeUri: "http://<server1-ip-address>:8081/",
+ status: "MASTER",
+ localTimestamp: 1466414301065,
+ remoteTimestamp: 1466414301000
+ },
+ XkJeXUXE: {
+ nodeId: "XkJeXUXE",
+ nodeUri: "http://<server2-ip-address>:8081/",
+ status: "STANDBY",
+ localTimestamp: 1466414301066,
+ remoteTimestamp: 1466414301000
+ }
+ },
+ links: { }
+}
+
+{% endhighlight %}
+
+The examples above show how to use `curl` to manually check the status of Brooklyn via its REST API. The same REST API calls can also be used by
+automated third party monitoring tools such as Monit
--- End diff --
Maybe say "such as Nagios", rather than Monit?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] brooklyn-docs pull request #78: Adds details of setting up a HA master-slave...
Posted by drigodwin <gi...@git.apache.org>.
Github user drigodwin commented on a diff in the pull request:
https://github.com/apache/brooklyn-docs/pull/78#discussion_r67666709
--- Diff: guide/ops/high-availability-supplemental.md ---
@@ -0,0 +1,97 @@
+---
+title: High Availability (Supplemental)
+layout: website-normal
+---
+
+This document supplements the High Availability documentation available [here](http://brooklyn.apache.org/v/latest/ops/high-availability.html)
--- End diff --
Would this not be better as a multiple section sub-section similar to locations?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] brooklyn-docs pull request #78: Adds details of setting up a HA master-slave...
Posted by aledsage <gi...@git.apache.org>.
Github user aledsage commented on a diff in the pull request:
https://github.com/apache/brooklyn-docs/pull/78#discussion_r68120968
--- Diff: guide/ops/high-availability/high-availability-supplemental.md ---
@@ -0,0 +1,142 @@
+---
+title: High Availability (Supplemental)
+layout: website-normal
+---
+
+This document supplements the High Availability documentation available [here](http://brooklyn.apache.org/v/latest/ops/high-availability.html)
+and provides an example of how to configure a pair of Apache Brooklyn servers to run in master-standby mode with a shared NFS datastore
+
+### Prerequisites
+- Two VMs (or physical machines) have been provisioned
+- NFS or another suitable file system has been configured and is available to both VMs*
+- An NFS folder has been mounted on both VMs at `/mnt/brooklyn-persistence` and both machines can write to the folder
+
+\* Brooklyn can be configured to use either an object store such as S3, or a shared NFS mount. The recommended option is to use an object
+store as described in the [Object Store Persistence](./persistence/#object-store-persistence) documentation. For clarity, a shared NFS folder
+is assumed in this example
+
+### Launching
+To start, download and install the latest Apache Brooklyn release on both VMs following the 'OSX / Linux' section
+of the [Running Apache Brooklyn](../start/running.html#install-apache-brooklyn) documentation
+
+On the first VM, which will be the master node, run the following to start Brooklyn in high availability mode:
+
+{% highlight bash %}
+$ bin/brooklyn launch --highAvailability master --persist auto --persistenceDir /mnt/brooklyn-persistence
+{% endhighlight %}
+
+Once Brooklyn has launched, on the second VM, run the following command to launch Brooklyn in standby mode:
+
+{% highlight bash %}
+$ bin/brooklyn launch --highAvailability auto --persist auto --persistenceDir /mnt/brooklyn-persistence
+{% endhighlight %}
+
+### Testing
+You can now confirm that Brooklyn is running in high availibility mode on the master by logging into the web console at `http://<ip-address>:8081`.
+Similarly you can log into the web console on the standby VM where you will see a warning that the server is not the high availability master.
+To test a failover, you can simply terminate the process on the first VM and log into the web console on the second VM. Upon launch, Brooklyn will
--- End diff --
Ah, I see you have this already in the "Failover" section.
Maybe re-order this, so that the testing section comes after "Failover", but before "Client configuration".
You could have a "Monitoring" section here though, that says about the web-console and has the curl commands.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] brooklyn-docs pull request #78: Adds details of setting up a HA master-slave...
Posted by aledsage <gi...@git.apache.org>.
Github user aledsage commented on a diff in the pull request:
https://github.com/apache/brooklyn-docs/pull/78#discussion_r68122041
--- Diff: guide/ops/high-availability/high-availability-supplemental.md ---
@@ -0,0 +1,142 @@
+---
+title: High Availability (Supplemental)
+layout: website-normal
+---
+
+This document supplements the High Availability documentation available [here](http://brooklyn.apache.org/v/latest/ops/high-availability.html)
+and provides an example of how to configure a pair of Apache Brooklyn servers to run in master-standby mode with a shared NFS datastore
+
+### Prerequisites
+- Two VMs (or physical machines) have been provisioned
+- NFS or another suitable file system has been configured and is available to both VMs*
+- An NFS folder has been mounted on both VMs at `/mnt/brooklyn-persistence` and both machines can write to the folder
+
+\* Brooklyn can be configured to use either an object store such as S3, or a shared NFS mount. The recommended option is to use an object
+store as described in the [Object Store Persistence](./persistence/#object-store-persistence) documentation. For clarity, a shared NFS folder
+is assumed in this example
+
+### Launching
+To start, download and install the latest Apache Brooklyn release on both VMs following the 'OSX / Linux' section
+of the [Running Apache Brooklyn](../start/running.html#install-apache-brooklyn) documentation
+
+On the first VM, which will be the master node, run the following to start Brooklyn in high availability mode:
+
+{% highlight bash %}
+$ bin/brooklyn launch --highAvailability master --persist auto --persistenceDir /mnt/brooklyn-persistence
+{% endhighlight %}
+
+Once Brooklyn has launched, on the second VM, run the following command to launch Brooklyn in standby mode:
+
+{% highlight bash %}
+$ bin/brooklyn launch --highAvailability auto --persist auto --persistenceDir /mnt/brooklyn-persistence
+{% endhighlight %}
+
+### Testing
+You can now confirm that Brooklyn is running in high availibility mode on the master by logging into the web console at `http://<ip-address>:8081`.
+Similarly you can log into the web console on the standby VM where you will see a warning that the server is not the high availability master.
+To test a failover, you can simply terminate the process on the first VM and log into the web console on the second VM. Upon launch, Brooklyn will
+output its PID to the file `pid.txt`; you can terminate the process by running the following command from the same directory from which you
+launched Brooklyn:
+
+{% highlight bash %}
+$ kill -9 $(cat pid.txt)
+{% endhighlight %}
+
+It is also possiblity to check the high availability state of a running Brooklyn server using the following curl command:
+
+{% highlight bash %}
+$ curl -u myusername:mypassword http://<ip-address>:8081/v1/server/ha/state
+{% endhighlight %}
+
+This will return one of the following states:
+
+{% highlight bash %}
+
+"INITIALIZING"
+"STANDBY"
+"HOT_STANDBY"
+"HOT_BACKUP"
+"MASTER"
+"FAILED"
+"TERMINATED"
+
+{% endhighlight %}
+
+Note: The quotation characters will be included in the reply
+
+To obtain information about all of the nodes in the cluster, run the following command against any of the nodes in the cluster:
+
+{% highlight bash %}
+$ curl -u myusername:mypassword http://<ip-address>:8081/v1/server/ha/states
+{% endhighlight %}
+
+This will return a JSON document describing the Brooklyn nodes in the cluster. An example of two HA Brooklyn nodes is as follows (whitespace formatting has been
+added for clarity):
+
+{% highlight yaml %}
+
+{
+ ownId: "XkJeXUXE",
+ masterId: "yAVz0fzo",
+ nodes: {
+ yAVz0fzo: {
+ nodeId: "yAVz0fzo",
+ nodeUri: "http://<server1-ip-address>:8081/",
+ status: "MASTER",
+ localTimestamp: 1466414301065,
+ remoteTimestamp: 1466414301000
+ },
+ XkJeXUXE: {
+ nodeId: "XkJeXUXE",
+ nodeUri: "http://<server2-ip-address>:8081/",
+ status: "STANDBY",
+ localTimestamp: 1466414301066,
+ remoteTimestamp: 1466414301000
+ }
+ },
+ links: { }
+}
+
+{% endhighlight %}
+
+The examples above show how to use `curl` to manually check the status of Brooklyn via its REST API. The same REST API calls can also be used by
+automated third party monitoring tools such as Monit
+
+### Failover
+When running as a HA standby node, each standby Brooklyn server (in this case there is only one standby) will check the shared persisted state
+every 1 second to determine the state of the HA master. If no heartbeat has been recorded for thirty seconds, then an election will be performed
+and one of the standby nodes will be promoted to master. At this point all requests should be directed to the new master node
+
+In the event that tasks - such as the provisioning of a new entity - are running when a failover occurs, the new master will display the current
+state of the entity, but will not resume its provisioning or re-run any partially completed tasks. In this case it will usually be necesarry
+to remove the node and reprovision it
+
+### Client Configuration
+It is the responsibility of the client to connect to the master Brooklyn server. This can be accomplished in a variety of ways:
+
+* **Reverse Proxy**
+
+ To allow the client application to automatically fail over in the event of a master server becoming unavailable, or the promotion of a new master,
+ a reverse proxy can be configured to route traffic depending on the response returned by `http://<ip-address>:8081/v1/server/ha/state` (see above).
+ If a server returns `"MASTER"`, then traffic should be routed to that server, otherwise it should not be. The client software should be configured
+ to connect to the reverse proxy server and no action is required by the client in the event of a failover
+* **Elastic IP with manual failover**
+
+ If the cloud provider you are using supports Elastic or Floating IPs, then the IP address should be allocated to the HA master, and the client
+ application configured to connect to the floating IP address. In the event of a failure of the master node, the standby node will automatically
+ be promoted to master, and the floating IP will need to be manually re-allocated to the new master node. No action is required by the client
--- End diff --
Could add that it would be possible to automate the re-allocation of the floating IP, if the Brooklyn servers are deployed and managed by Brooklyn using the entity `org.apache.brooklyn.entity.brooklynnode.BrooklynCluster`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] brooklyn-docs pull request #78: Adds details of setting up a HA master-slave...
Posted by aledsage <gi...@git.apache.org>.
Github user aledsage commented on a diff in the pull request:
https://github.com/apache/brooklyn-docs/pull/78#discussion_r68121515
--- Diff: guide/ops/high-availability/high-availability-supplemental.md ---
@@ -0,0 +1,142 @@
+---
+title: High Availability (Supplemental)
+layout: website-normal
+---
+
+This document supplements the High Availability documentation available [here](http://brooklyn.apache.org/v/latest/ops/high-availability.html)
+and provides an example of how to configure a pair of Apache Brooklyn servers to run in master-standby mode with a shared NFS datastore
+
+### Prerequisites
+- Two VMs (or physical machines) have been provisioned
+- NFS or another suitable file system has been configured and is available to both VMs*
+- An NFS folder has been mounted on both VMs at `/mnt/brooklyn-persistence` and both machines can write to the folder
+
+\* Brooklyn can be configured to use either an object store such as S3, or a shared NFS mount. The recommended option is to use an object
+store as described in the [Object Store Persistence](./persistence/#object-store-persistence) documentation. For clarity, a shared NFS folder
+is assumed in this example
+
+### Launching
+To start, download and install the latest Apache Brooklyn release on both VMs following the 'OSX / Linux' section
+of the [Running Apache Brooklyn](../start/running.html#install-apache-brooklyn) documentation
+
+On the first VM, which will be the master node, run the following to start Brooklyn in high availability mode:
+
+{% highlight bash %}
+$ bin/brooklyn launch --highAvailability master --persist auto --persistenceDir /mnt/brooklyn-persistence
+{% endhighlight %}
+
+Once Brooklyn has launched, on the second VM, run the following command to launch Brooklyn in standby mode:
+
+{% highlight bash %}
+$ bin/brooklyn launch --highAvailability auto --persist auto --persistenceDir /mnt/brooklyn-persistence
+{% endhighlight %}
+
+### Testing
+You can now confirm that Brooklyn is running in high availibility mode on the master by logging into the web console at `http://<ip-address>:8081`.
+Similarly you can log into the web console on the standby VM where you will see a warning that the server is not the high availability master.
+To test a failover, you can simply terminate the process on the first VM and log into the web console on the second VM. Upon launch, Brooklyn will
+output its PID to the file `pid.txt`; you can terminate the process by running the following command from the same directory from which you
+launched Brooklyn:
+
+{% highlight bash %}
+$ kill -9 $(cat pid.txt)
+{% endhighlight %}
+
+It is also possiblity to check the high availability state of a running Brooklyn server using the following curl command:
+
+{% highlight bash %}
+$ curl -u myusername:mypassword http://<ip-address>:8081/v1/server/ha/state
+{% endhighlight %}
+
+This will return one of the following states:
+
+{% highlight bash %}
+
+"INITIALIZING"
+"STANDBY"
+"HOT_STANDBY"
+"HOT_BACKUP"
+"MASTER"
+"FAILED"
+"TERMINATED"
+
+{% endhighlight %}
+
+Note: The quotation characters will be included in the reply
+
+To obtain information about all of the nodes in the cluster, run the following command against any of the nodes in the cluster:
+
+{% highlight bash %}
+$ curl -u myusername:mypassword http://<ip-address>:8081/v1/server/ha/states
+{% endhighlight %}
+
+This will return a JSON document describing the Brooklyn nodes in the cluster. An example of two HA Brooklyn nodes is as follows (whitespace formatting has been
+added for clarity):
+
+{% highlight yaml %}
+
+{
+ ownId: "XkJeXUXE",
+ masterId: "yAVz0fzo",
+ nodes: {
+ yAVz0fzo: {
+ nodeId: "yAVz0fzo",
+ nodeUri: "http://<server1-ip-address>:8081/",
+ status: "MASTER",
+ localTimestamp: 1466414301065,
+ remoteTimestamp: 1466414301000
+ },
+ XkJeXUXE: {
+ nodeId: "XkJeXUXE",
+ nodeUri: "http://<server2-ip-address>:8081/",
+ status: "STANDBY",
+ localTimestamp: 1466414301066,
+ remoteTimestamp: 1466414301000
+ }
+ },
+ links: { }
+}
+
+{% endhighlight %}
+
+The examples above show how to use `curl` to manually check the status of Brooklyn via its REST API. The same REST API calls can also be used by
+automated third party monitoring tools such as Monit
+
+### Failover
+When running as a HA standby node, each standby Brooklyn server (in this case there is only one standby) will check the shared persisted state
+every 1 second to determine the state of the HA master. If no heartbeat has been recorded for thirty seconds, then an election will be performed
+and one of the standby nodes will be promoted to master. At this point all requests should be directed to the new master node
+
+In the event that tasks - such as the provisioning of a new entity - are running when a failover occurs, the new master will display the current
+state of the entity, but will not resume its provisioning or re-run any partially completed tasks. In this case it will usually be necesarry
+to remove the node and reprovision it
--- End diff --
Ambiguous: can you say "entity" rather than "node" (assuming you don't mean removing the Brooklyn server?!)
Or it could just be required to call the effector again, depending what operation was being performed when it failed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] brooklyn-docs pull request #78: Adds details of setting up a HA master-slave...
Posted by aledsage <gi...@git.apache.org>.
Github user aledsage commented on a diff in the pull request:
https://github.com/apache/brooklyn-docs/pull/78#discussion_r68119701
--- Diff: guide/ops/high-availability/high-availability-supplemental.md ---
@@ -0,0 +1,142 @@
+---
+title: High Availability (Supplemental)
+layout: website-normal
+---
+
+This document supplements the High Availability documentation available [here](http://brooklyn.apache.org/v/latest/ops/high-availability.html)
+and provides an example of how to configure a pair of Apache Brooklyn servers to run in master-standby mode with a shared NFS datastore
+
+### Prerequisites
+- Two VMs (or physical machines) have been provisioned
+- NFS or another suitable file system has been configured and is available to both VMs*
+- An NFS folder has been mounted on both VMs at `/mnt/brooklyn-persistence` and both machines can write to the folder
+
+\* Brooklyn can be configured to use either an object store such as S3, or a shared NFS mount. The recommended option is to use an object
+store as described in the [Object Store Persistence](./persistence/#object-store-persistence) documentation. For clarity, a shared NFS folder
--- End diff --
"For simplicity" rather than "for clarity"?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] brooklyn-docs issue #78: Adds details of setting up a HA master-slave pair
Posted by aledsage <gi...@git.apache.org>.
Github user aledsage commented on the issue:
https://github.com/apache/brooklyn-docs/pull/78
LGTM; merging.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] brooklyn-docs issue #78: Adds details of setting up a HA master-slave pair
Posted by aledsage <gi...@git.apache.org>.
Github user aledsage commented on the issue:
https://github.com/apache/brooklyn-docs/pull/78
@nakomis what do you think of @drigodwin 's suggestion: "would this not be better as a multiple section sub-section similar to locations?" I'm fine with either.
It's also worth adding a note (where you say `./bin/brooklyn ...`) that they should see the appropriate docs for if they are using RPMs/deb install.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] brooklyn-docs pull request #78: Adds details of setting up a HA master-slave...
Posted by aledsage <gi...@git.apache.org>.
Github user aledsage commented on a diff in the pull request:
https://github.com/apache/brooklyn-docs/pull/78#discussion_r68120602
--- Diff: guide/ops/high-availability/high-availability-supplemental.md ---
@@ -0,0 +1,142 @@
+---
+title: High Availability (Supplemental)
+layout: website-normal
+---
+
+This document supplements the High Availability documentation available [here](http://brooklyn.apache.org/v/latest/ops/high-availability.html)
+and provides an example of how to configure a pair of Apache Brooklyn servers to run in master-standby mode with a shared NFS datastore
+
+### Prerequisites
+- Two VMs (or physical machines) have been provisioned
+- NFS or another suitable file system has been configured and is available to both VMs*
+- An NFS folder has been mounted on both VMs at `/mnt/brooklyn-persistence` and both machines can write to the folder
+
+\* Brooklyn can be configured to use either an object store such as S3, or a shared NFS mount. The recommended option is to use an object
+store as described in the [Object Store Persistence](./persistence/#object-store-persistence) documentation. For clarity, a shared NFS folder
+is assumed in this example
+
+### Launching
+To start, download and install the latest Apache Brooklyn release on both VMs following the 'OSX / Linux' section
+of the [Running Apache Brooklyn](../start/running.html#install-apache-brooklyn) documentation
+
+On the first VM, which will be the master node, run the following to start Brooklyn in high availability mode:
+
+{% highlight bash %}
+$ bin/brooklyn launch --highAvailability master --persist auto --persistenceDir /mnt/brooklyn-persistence
+{% endhighlight %}
+
+Once Brooklyn has launched, on the second VM, run the following command to launch Brooklyn in standby mode:
+
+{% highlight bash %}
+$ bin/brooklyn launch --highAvailability auto --persist auto --persistenceDir /mnt/brooklyn-persistence
+{% endhighlight %}
+
+### Testing
+You can now confirm that Brooklyn is running in high availibility mode on the master by logging into the web console at `http://<ip-address>:8081`.
+Similarly you can log into the web console on the standby VM where you will see a warning that the server is not the high availability master.
+To test a failover, you can simply terminate the process on the first VM and log into the web console on the second VM. Upon launch, Brooklyn will
--- End diff --
I'd add: If the master is terminated gracefully, the secondary will be immediately promoted to mater. Otherwise, the secondary will be promoted after heartbeats are missed for a given length of time. This defaults to thirty seconds, and is configured in brooklyn.properties using `brooklyn.ha.heartbeatTimeout`.
Perhaps miss out the `kill` command here - it's described in the docs that @rdowner contributed (though without the `-9`)?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] brooklyn-docs pull request #78: Adds details of setting up a HA master-slave...
Posted by aledsage <gi...@git.apache.org>.
Github user aledsage commented on a diff in the pull request:
https://github.com/apache/brooklyn-docs/pull/78#discussion_r68119397
--- Diff: guide/ops/high-availability/high-availability-supplemental.md ---
@@ -0,0 +1,142 @@
+---
+title: High Availability (Supplemental)
+layout: website-normal
+---
+
+This document supplements the High Availability documentation available [here](http://brooklyn.apache.org/v/latest/ops/high-availability.html)
+and provides an example of how to configure a pair of Apache Brooklyn servers to run in master-standby mode with a shared NFS datastore
+
+### Prerequisites
+- Two VMs (or physical machines) have been provisioned
+- NFS or another suitable file system has been configured and is available to both VMs*
+- An NFS folder has been mounted on both VMs at `/mnt/brooklyn-persistence` and both machines can write to the folder
+
+\* Brooklyn can be configured to use either an object store such as S3, or a shared NFS mount. The recommended option is to use an object
+store as described in the [Object Store Persistence](./persistence/#object-store-persistence) documentation. For clarity, a shared NFS folder
+is assumed in this example
+
+### Launching
+To start, download and install the latest Apache Brooklyn release on both VMs following the 'OSX / Linux' section
--- End diff --
I'd word it as `...on both VMs, following the instructions in `[Running Apache Brooklyn]({{ site.path.guide }}/start/running.html#install-apache-brooklyn).`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] brooklyn-docs pull request #78: Adds details of setting up a HA master-slave...
Posted by aledsage <gi...@git.apache.org>.
Github user aledsage commented on a diff in the pull request:
https://github.com/apache/brooklyn-docs/pull/78#discussion_r68119110
--- Diff: guide/ops/high-availability/high-availability-supplemental.md ---
@@ -0,0 +1,142 @@
+---
+title: High Availability (Supplemental)
+layout: website-normal
+---
+
+This document supplements the High Availability documentation available [here](http://brooklyn.apache.org/v/latest/ops/high-availability.html)
--- End diff --
Use `This document supplements the [High Availability]]({{ site.path.guide }}/ops/high-availability.html) documentation.` Don't use absolute links to things in the docs; and I have a personal preference for the link not being "here", but being the name of what it's linking to.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] brooklyn-docs pull request #78: Adds details of setting up a HA master-slave...
Posted by aledsage <gi...@git.apache.org>.
Github user aledsage commented on a diff in the pull request:
https://github.com/apache/brooklyn-docs/pull/78#discussion_r68121198
--- Diff: guide/ops/high-availability/high-availability-supplemental.md ---
@@ -0,0 +1,142 @@
+---
+title: High Availability (Supplemental)
+layout: website-normal
+---
+
+This document supplements the High Availability documentation available [here](http://brooklyn.apache.org/v/latest/ops/high-availability.html)
+and provides an example of how to configure a pair of Apache Brooklyn servers to run in master-standby mode with a shared NFS datastore
+
+### Prerequisites
+- Two VMs (or physical machines) have been provisioned
+- NFS or another suitable file system has been configured and is available to both VMs*
+- An NFS folder has been mounted on both VMs at `/mnt/brooklyn-persistence` and both machines can write to the folder
+
+\* Brooklyn can be configured to use either an object store such as S3, or a shared NFS mount. The recommended option is to use an object
+store as described in the [Object Store Persistence](./persistence/#object-store-persistence) documentation. For clarity, a shared NFS folder
+is assumed in this example
+
+### Launching
+To start, download and install the latest Apache Brooklyn release on both VMs following the 'OSX / Linux' section
+of the [Running Apache Brooklyn](../start/running.html#install-apache-brooklyn) documentation
+
+On the first VM, which will be the master node, run the following to start Brooklyn in high availability mode:
+
+{% highlight bash %}
+$ bin/brooklyn launch --highAvailability master --persist auto --persistenceDir /mnt/brooklyn-persistence
+{% endhighlight %}
+
+Once Brooklyn has launched, on the second VM, run the following command to launch Brooklyn in standby mode:
+
+{% highlight bash %}
+$ bin/brooklyn launch --highAvailability auto --persist auto --persistenceDir /mnt/brooklyn-persistence
+{% endhighlight %}
+
+### Testing
+You can now confirm that Brooklyn is running in high availibility mode on the master by logging into the web console at `http://<ip-address>:8081`.
+Similarly you can log into the web console on the standby VM where you will see a warning that the server is not the high availability master.
+To test a failover, you can simply terminate the process on the first VM and log into the web console on the second VM. Upon launch, Brooklyn will
+output its PID to the file `pid.txt`; you can terminate the process by running the following command from the same directory from which you
+launched Brooklyn:
+
+{% highlight bash %}
+$ kill -9 $(cat pid.txt)
+{% endhighlight %}
+
+It is also possiblity to check the high availability state of a running Brooklyn server using the following curl command:
+
+{% highlight bash %}
+$ curl -u myusername:mypassword http://<ip-address>:8081/v1/server/ha/state
+{% endhighlight %}
+
+This will return one of the following states:
+
+{% highlight bash %}
+
+"INITIALIZING"
+"STANDBY"
+"HOT_STANDBY"
+"HOT_BACKUP"
+"MASTER"
+"FAILED"
+"TERMINATED"
+
+{% endhighlight %}
+
+Note: The quotation characters will be included in the reply
+
+To obtain information about all of the nodes in the cluster, run the following command against any of the nodes in the cluster:
+
+{% highlight bash %}
+$ curl -u myusername:mypassword http://<ip-address>:8081/v1/server/ha/states
+{% endhighlight %}
+
+This will return a JSON document describing the Brooklyn nodes in the cluster. An example of two HA Brooklyn nodes is as follows (whitespace formatting has been
+added for clarity):
+
+{% highlight yaml %}
+
+{
+ ownId: "XkJeXUXE",
+ masterId: "yAVz0fzo",
+ nodes: {
+ yAVz0fzo: {
+ nodeId: "yAVz0fzo",
+ nodeUri: "http://<server1-ip-address>:8081/",
+ status: "MASTER",
+ localTimestamp: 1466414301065,
+ remoteTimestamp: 1466414301000
+ },
+ XkJeXUXE: {
+ nodeId: "XkJeXUXE",
+ nodeUri: "http://<server2-ip-address>:8081/",
+ status: "STANDBY",
+ localTimestamp: 1466414301066,
+ remoteTimestamp: 1466414301000
+ }
+ },
+ links: { }
+}
+
+{% endhighlight %}
+
+The examples above show how to use `curl` to manually check the status of Brooklyn via its REST API. The same REST API calls can also be used by
+automated third party monitoring tools such as Monit
+
+### Failover
+When running as a HA standby node, each standby Brooklyn server (in this case there is only one standby) will check the shared persisted state
+every 1 second to determine the state of the HA master. If no heartbeat has been recorded for thirty seconds, then an election will be performed
+and one of the standby nodes will be promoted to master. At this point all requests should be directed to the new master node
--- End diff --
See my comment above about how to configure these timeouts. Worth including that here, assuming it's not covered anywhere else in the docs?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] brooklyn-docs pull request #78: Adds details of setting up a HA master-slave...
Posted by aledsage <gi...@git.apache.org>.
Github user aledsage commented on a diff in the pull request:
https://github.com/apache/brooklyn-docs/pull/78#discussion_r68122294
--- Diff: guide/ops/high-availability/high-availability-supplemental.md ---
@@ -0,0 +1,142 @@
+---
+title: High Availability (Supplemental)
+layout: website-normal
+---
+
+This document supplements the High Availability documentation available [here](http://brooklyn.apache.org/v/latest/ops/high-availability.html)
+and provides an example of how to configure a pair of Apache Brooklyn servers to run in master-standby mode with a shared NFS datastore
+
+### Prerequisites
+- Two VMs (or physical machines) have been provisioned
+- NFS or another suitable file system has been configured and is available to both VMs*
+- An NFS folder has been mounted on both VMs at `/mnt/brooklyn-persistence` and both machines can write to the folder
+
+\* Brooklyn can be configured to use either an object store such as S3, or a shared NFS mount. The recommended option is to use an object
+store as described in the [Object Store Persistence](./persistence/#object-store-persistence) documentation. For clarity, a shared NFS folder
+is assumed in this example
+
+### Launching
+To start, download and install the latest Apache Brooklyn release on both VMs following the 'OSX / Linux' section
+of the [Running Apache Brooklyn](../start/running.html#install-apache-brooklyn) documentation
+
+On the first VM, which will be the master node, run the following to start Brooklyn in high availability mode:
+
+{% highlight bash %}
+$ bin/brooklyn launch --highAvailability master --persist auto --persistenceDir /mnt/brooklyn-persistence
+{% endhighlight %}
+
+Once Brooklyn has launched, on the second VM, run the following command to launch Brooklyn in standby mode:
+
+{% highlight bash %}
+$ bin/brooklyn launch --highAvailability auto --persist auto --persistenceDir /mnt/brooklyn-persistence
+{% endhighlight %}
+
+### Testing
+You can now confirm that Brooklyn is running in high availibility mode on the master by logging into the web console at `http://<ip-address>:8081`.
+Similarly you can log into the web console on the standby VM where you will see a warning that the server is not the high availability master.
+To test a failover, you can simply terminate the process on the first VM and log into the web console on the second VM. Upon launch, Brooklyn will
+output its PID to the file `pid.txt`; you can terminate the process by running the following command from the same directory from which you
+launched Brooklyn:
+
+{% highlight bash %}
+$ kill -9 $(cat pid.txt)
+{% endhighlight %}
+
+It is also possiblity to check the high availability state of a running Brooklyn server using the following curl command:
+
+{% highlight bash %}
+$ curl -u myusername:mypassword http://<ip-address>:8081/v1/server/ha/state
+{% endhighlight %}
+
+This will return one of the following states:
+
+{% highlight bash %}
+
+"INITIALIZING"
+"STANDBY"
+"HOT_STANDBY"
+"HOT_BACKUP"
+"MASTER"
+"FAILED"
+"TERMINATED"
+
+{% endhighlight %}
+
+Note: The quotation characters will be included in the reply
+
+To obtain information about all of the nodes in the cluster, run the following command against any of the nodes in the cluster:
+
+{% highlight bash %}
+$ curl -u myusername:mypassword http://<ip-address>:8081/v1/server/ha/states
+{% endhighlight %}
+
+This will return a JSON document describing the Brooklyn nodes in the cluster. An example of two HA Brooklyn nodes is as follows (whitespace formatting has been
+added for clarity):
+
+{% highlight yaml %}
+
+{
+ ownId: "XkJeXUXE",
+ masterId: "yAVz0fzo",
+ nodes: {
+ yAVz0fzo: {
+ nodeId: "yAVz0fzo",
+ nodeUri: "http://<server1-ip-address>:8081/",
+ status: "MASTER",
+ localTimestamp: 1466414301065,
+ remoteTimestamp: 1466414301000
+ },
+ XkJeXUXE: {
+ nodeId: "XkJeXUXE",
+ nodeUri: "http://<server2-ip-address>:8081/",
+ status: "STANDBY",
+ localTimestamp: 1466414301066,
+ remoteTimestamp: 1466414301000
+ }
+ },
+ links: { }
+}
+
+{% endhighlight %}
+
+The examples above show how to use `curl` to manually check the status of Brooklyn via its REST API. The same REST API calls can also be used by
+automated third party monitoring tools such as Monit
+
+### Failover
+When running as a HA standby node, each standby Brooklyn server (in this case there is only one standby) will check the shared persisted state
+every 1 second to determine the state of the HA master. If no heartbeat has been recorded for thirty seconds, then an election will be performed
+and one of the standby nodes will be promoted to master. At this point all requests should be directed to the new master node
+
+In the event that tasks - such as the provisioning of a new entity - are running when a failover occurs, the new master will display the current
+state of the entity, but will not resume its provisioning or re-run any partially completed tasks. In this case it will usually be necesarry
+to remove the node and reprovision it
+
+### Client Configuration
+It is the responsibility of the client to connect to the master Brooklyn server. This can be accomplished in a variety of ways:
+
+* **Reverse Proxy**
+
+ To allow the client application to automatically fail over in the event of a master server becoming unavailable, or the promotion of a new master,
+ a reverse proxy can be configured to route traffic depending on the response returned by `http://<ip-address>:8081/v1/server/ha/state` (see above).
+ If a server returns `"MASTER"`, then traffic should be routed to that server, otherwise it should not be. The client software should be configured
+ to connect to the reverse proxy server and no action is required by the client in the event of a failover
+* **Elastic IP with manual failover**
--- End diff --
Rename to "Re-allocating an Elastic IP on Failover"
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] brooklyn-docs pull request #78: Adds details of setting up a HA master-slave...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/brooklyn-docs/pull/78
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] brooklyn-docs pull request #78: Adds details of setting up a HA master-slave...
Posted by aledsage <gi...@git.apache.org>.
Github user aledsage commented on a diff in the pull request:
https://github.com/apache/brooklyn-docs/pull/78#discussion_r68119573
--- Diff: guide/ops/high-availability/high-availability-supplemental.md ---
@@ -0,0 +1,142 @@
+---
+title: High Availability (Supplemental)
+layout: website-normal
+---
+
+This document supplements the High Availability documentation available [here](http://brooklyn.apache.org/v/latest/ops/high-availability.html)
+and provides an example of how to configure a pair of Apache Brooklyn servers to run in master-standby mode with a shared NFS datastore
+
+### Prerequisites
+- Two VMs (or physical machines) have been provisioned
+- NFS or another suitable file system has been configured and is available to both VMs*
+- An NFS folder has been mounted on both VMs at `/mnt/brooklyn-persistence` and both machines can write to the folder
+
+\* Brooklyn can be configured to use either an object store such as S3, or a shared NFS mount. The recommended option is to use an object
+store as described in the [Object Store Persistence](./persistence/#object-store-persistence) documentation. For clarity, a shared NFS folder
+is assumed in this example
+
+### Launching
+To start, download and install the latest Apache Brooklyn release on both VMs following the 'OSX / Linux' section
+of the [Running Apache Brooklyn](../start/running.html#install-apache-brooklyn) documentation
+
+On the first VM, which will be the master node, run the following to start Brooklyn in high availability mode:
+
+{% highlight bash %}
+$ bin/brooklyn launch --highAvailability master --persist auto --persistenceDir /mnt/brooklyn-persistence
--- End diff --
Given this is docs for production, let's use `--https` (and use port 8443 below).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] brooklyn-docs issue #78: Adds details of setting up a HA master-slave pair
Posted by nakomis <gi...@git.apache.org>.
Github user nakomis commented on the issue:
https://github.com/apache/brooklyn-docs/pull/78
PR comments addressed
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] brooklyn-docs pull request #78: Adds details of setting up a HA master-slave...
Posted by aledsage <gi...@git.apache.org>.
Github user aledsage commented on a diff in the pull request:
https://github.com/apache/brooklyn-docs/pull/78#discussion_r68122178
--- Diff: guide/ops/high-availability/high-availability-supplemental.md ---
@@ -0,0 +1,142 @@
+---
+title: High Availability (Supplemental)
+layout: website-normal
+---
+
+This document supplements the High Availability documentation available [here](http://brooklyn.apache.org/v/latest/ops/high-availability.html)
+and provides an example of how to configure a pair of Apache Brooklyn servers to run in master-standby mode with a shared NFS datastore
+
+### Prerequisites
+- Two VMs (or physical machines) have been provisioned
+- NFS or another suitable file system has been configured and is available to both VMs*
+- An NFS folder has been mounted on both VMs at `/mnt/brooklyn-persistence` and both machines can write to the folder
+
+\* Brooklyn can be configured to use either an object store such as S3, or a shared NFS mount. The recommended option is to use an object
+store as described in the [Object Store Persistence](./persistence/#object-store-persistence) documentation. For clarity, a shared NFS folder
+is assumed in this example
+
+### Launching
+To start, download and install the latest Apache Brooklyn release on both VMs following the 'OSX / Linux' section
+of the [Running Apache Brooklyn](../start/running.html#install-apache-brooklyn) documentation
+
+On the first VM, which will be the master node, run the following to start Brooklyn in high availability mode:
+
+{% highlight bash %}
+$ bin/brooklyn launch --highAvailability master --persist auto --persistenceDir /mnt/brooklyn-persistence
+{% endhighlight %}
+
+Once Brooklyn has launched, on the second VM, run the following command to launch Brooklyn in standby mode:
+
+{% highlight bash %}
+$ bin/brooklyn launch --highAvailability auto --persist auto --persistenceDir /mnt/brooklyn-persistence
+{% endhighlight %}
+
+### Testing
+You can now confirm that Brooklyn is running in high availibility mode on the master by logging into the web console at `http://<ip-address>:8081`.
+Similarly you can log into the web console on the standby VM where you will see a warning that the server is not the high availability master.
+To test a failover, you can simply terminate the process on the first VM and log into the web console on the second VM. Upon launch, Brooklyn will
+output its PID to the file `pid.txt`; you can terminate the process by running the following command from the same directory from which you
+launched Brooklyn:
+
+{% highlight bash %}
+$ kill -9 $(cat pid.txt)
+{% endhighlight %}
+
+It is also possiblity to check the high availability state of a running Brooklyn server using the following curl command:
+
+{% highlight bash %}
+$ curl -u myusername:mypassword http://<ip-address>:8081/v1/server/ha/state
+{% endhighlight %}
+
+This will return one of the following states:
+
+{% highlight bash %}
+
+"INITIALIZING"
+"STANDBY"
+"HOT_STANDBY"
+"HOT_BACKUP"
+"MASTER"
+"FAILED"
+"TERMINATED"
+
+{% endhighlight %}
+
+Note: The quotation characters will be included in the reply
+
+To obtain information about all of the nodes in the cluster, run the following command against any of the nodes in the cluster:
+
+{% highlight bash %}
+$ curl -u myusername:mypassword http://<ip-address>:8081/v1/server/ha/states
+{% endhighlight %}
+
+This will return a JSON document describing the Brooklyn nodes in the cluster. An example of two HA Brooklyn nodes is as follows (whitespace formatting has been
+added for clarity):
+
+{% highlight yaml %}
+
+{
+ ownId: "XkJeXUXE",
+ masterId: "yAVz0fzo",
+ nodes: {
+ yAVz0fzo: {
+ nodeId: "yAVz0fzo",
+ nodeUri: "http://<server1-ip-address>:8081/",
+ status: "MASTER",
+ localTimestamp: 1466414301065,
+ remoteTimestamp: 1466414301000
+ },
+ XkJeXUXE: {
+ nodeId: "XkJeXUXE",
+ nodeUri: "http://<server2-ip-address>:8081/",
+ status: "STANDBY",
+ localTimestamp: 1466414301066,
+ remoteTimestamp: 1466414301000
+ }
+ },
+ links: { }
+}
+
+{% endhighlight %}
+
+The examples above show how to use `curl` to manually check the status of Brooklyn via its REST API. The same REST API calls can also be used by
+automated third party monitoring tools such as Monit
+
+### Failover
+When running as a HA standby node, each standby Brooklyn server (in this case there is only one standby) will check the shared persisted state
+every 1 second to determine the state of the HA master. If no heartbeat has been recorded for thirty seconds, then an election will be performed
+and one of the standby nodes will be promoted to master. At this point all requests should be directed to the new master node
+
+In the event that tasks - such as the provisioning of a new entity - are running when a failover occurs, the new master will display the current
+state of the entity, but will not resume its provisioning or re-run any partially completed tasks. In this case it will usually be necesarry
+to remove the node and reprovision it
+
+### Client Configuration
+It is the responsibility of the client to connect to the master Brooklyn server. This can be accomplished in a variety of ways:
+
+* **Reverse Proxy**
--- End diff --
I'd use `####`, rather than just bold. And blank lines before the "Elastic IP" sub-section.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] brooklyn-docs pull request #78: Adds details of setting up a HA master-slave...
Posted by aledsage <gi...@git.apache.org>.
Github user aledsage commented on a diff in the pull request:
https://github.com/apache/brooklyn-docs/pull/78#discussion_r68120636
--- Diff: guide/ops/high-availability/high-availability-supplemental.md ---
@@ -0,0 +1,142 @@
+---
+title: High Availability (Supplemental)
+layout: website-normal
+---
+
+This document supplements the High Availability documentation available [here](http://brooklyn.apache.org/v/latest/ops/high-availability.html)
+and provides an example of how to configure a pair of Apache Brooklyn servers to run in master-standby mode with a shared NFS datastore
+
+### Prerequisites
+- Two VMs (or physical machines) have been provisioned
+- NFS or another suitable file system has been configured and is available to both VMs*
+- An NFS folder has been mounted on both VMs at `/mnt/brooklyn-persistence` and both machines can write to the folder
+
+\* Brooklyn can be configured to use either an object store such as S3, or a shared NFS mount. The recommended option is to use an object
+store as described in the [Object Store Persistence](./persistence/#object-store-persistence) documentation. For clarity, a shared NFS folder
+is assumed in this example
+
+### Launching
+To start, download and install the latest Apache Brooklyn release on both VMs following the 'OSX / Linux' section
+of the [Running Apache Brooklyn](../start/running.html#install-apache-brooklyn) documentation
+
+On the first VM, which will be the master node, run the following to start Brooklyn in high availability mode:
+
+{% highlight bash %}
+$ bin/brooklyn launch --highAvailability master --persist auto --persistenceDir /mnt/brooklyn-persistence
+{% endhighlight %}
+
+Once Brooklyn has launched, on the second VM, run the following command to launch Brooklyn in standby mode:
+
+{% highlight bash %}
+$ bin/brooklyn launch --highAvailability auto --persist auto --persistenceDir /mnt/brooklyn-persistence
+{% endhighlight %}
+
+### Testing
+You can now confirm that Brooklyn is running in high availibility mode on the master by logging into the web console at `http://<ip-address>:8081`.
+Similarly you can log into the web console on the standby VM where you will see a warning that the server is not the high availability master.
+To test a failover, you can simply terminate the process on the first VM and log into the web console on the second VM. Upon launch, Brooklyn will
+output its PID to the file `pid.txt`; you can terminate the process by running the following command from the same directory from which you
+launched Brooklyn:
+
+{% highlight bash %}
+$ kill -9 $(cat pid.txt)
+{% endhighlight %}
+
+It is also possiblity to check the high availability state of a running Brooklyn server using the following curl command:
+
+{% highlight bash %}
+$ curl -u myusername:mypassword http://<ip-address>:8081/v1/server/ha/state
--- End diff --
again https and 8443
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] brooklyn-docs pull request #78: Adds details of setting up a HA master-slave...
Posted by aledsage <gi...@git.apache.org>.
Github user aledsage commented on a diff in the pull request:
https://github.com/apache/brooklyn-docs/pull/78#discussion_r68122663
--- Diff: guide/ops/high-availability/high-availability-supplemental.md ---
@@ -0,0 +1,142 @@
+---
+title: High Availability (Supplemental)
+layout: website-normal
+---
+
+This document supplements the High Availability documentation available [here](http://brooklyn.apache.org/v/latest/ops/high-availability.html)
+and provides an example of how to configure a pair of Apache Brooklyn servers to run in master-standby mode with a shared NFS datastore
+
+### Prerequisites
+- Two VMs (or physical machines) have been provisioned
+- NFS or another suitable file system has been configured and is available to both VMs*
+- An NFS folder has been mounted on both VMs at `/mnt/brooklyn-persistence` and both machines can write to the folder
+
+\* Brooklyn can be configured to use either an object store such as S3, or a shared NFS mount. The recommended option is to use an object
+store as described in the [Object Store Persistence](./persistence/#object-store-persistence) documentation. For clarity, a shared NFS folder
+is assumed in this example
+
+### Launching
+To start, download and install the latest Apache Brooklyn release on both VMs following the 'OSX / Linux' section
+of the [Running Apache Brooklyn](../start/running.html#install-apache-brooklyn) documentation
+
+On the first VM, which will be the master node, run the following to start Brooklyn in high availability mode:
+
+{% highlight bash %}
+$ bin/brooklyn launch --highAvailability master --persist auto --persistenceDir /mnt/brooklyn-persistence
+{% endhighlight %}
+
+Once Brooklyn has launched, on the second VM, run the following command to launch Brooklyn in standby mode:
+
+{% highlight bash %}
+$ bin/brooklyn launch --highAvailability auto --persist auto --persistenceDir /mnt/brooklyn-persistence
+{% endhighlight %}
+
+### Testing
+You can now confirm that Brooklyn is running in high availibility mode on the master by logging into the web console at `http://<ip-address>:8081`.
+Similarly you can log into the web console on the standby VM where you will see a warning that the server is not the high availability master.
+To test a failover, you can simply terminate the process on the first VM and log into the web console on the second VM. Upon launch, Brooklyn will
+output its PID to the file `pid.txt`; you can terminate the process by running the following command from the same directory from which you
+launched Brooklyn:
+
+{% highlight bash %}
+$ kill -9 $(cat pid.txt)
+{% endhighlight %}
+
+It is also possiblity to check the high availability state of a running Brooklyn server using the following curl command:
+
+{% highlight bash %}
+$ curl -u myusername:mypassword http://<ip-address>:8081/v1/server/ha/state
+{% endhighlight %}
+
+This will return one of the following states:
+
+{% highlight bash %}
+
+"INITIALIZING"
+"STANDBY"
+"HOT_STANDBY"
+"HOT_BACKUP"
+"MASTER"
+"FAILED"
+"TERMINATED"
+
+{% endhighlight %}
+
+Note: The quotation characters will be included in the reply
+
+To obtain information about all of the nodes in the cluster, run the following command against any of the nodes in the cluster:
+
+{% highlight bash %}
+$ curl -u myusername:mypassword http://<ip-address>:8081/v1/server/ha/states
+{% endhighlight %}
+
+This will return a JSON document describing the Brooklyn nodes in the cluster. An example of two HA Brooklyn nodes is as follows (whitespace formatting has been
+added for clarity):
+
+{% highlight yaml %}
+
+{
+ ownId: "XkJeXUXE",
+ masterId: "yAVz0fzo",
+ nodes: {
+ yAVz0fzo: {
+ nodeId: "yAVz0fzo",
+ nodeUri: "http://<server1-ip-address>:8081/",
+ status: "MASTER",
+ localTimestamp: 1466414301065,
+ remoteTimestamp: 1466414301000
+ },
+ XkJeXUXE: {
+ nodeId: "XkJeXUXE",
+ nodeUri: "http://<server2-ip-address>:8081/",
+ status: "STANDBY",
+ localTimestamp: 1466414301066,
+ remoteTimestamp: 1466414301000
+ }
+ },
+ links: { }
+}
+
+{% endhighlight %}
+
+The examples above show how to use `curl` to manually check the status of Brooklyn via its REST API. The same REST API calls can also be used by
+automated third party monitoring tools such as Monit
+
+### Failover
+When running as a HA standby node, each standby Brooklyn server (in this case there is only one standby) will check the shared persisted state
+every 1 second to determine the state of the HA master. If no heartbeat has been recorded for thirty seconds, then an election will be performed
+and one of the standby nodes will be promoted to master. At this point all requests should be directed to the new master node
+
+In the event that tasks - such as the provisioning of a new entity - are running when a failover occurs, the new master will display the current
+state of the entity, but will not resume its provisioning or re-run any partially completed tasks. In this case it will usually be necesarry
+to remove the node and reprovision it
+
+### Client Configuration
+It is the responsibility of the client to connect to the master Brooklyn server. This can be accomplished in a variety of ways:
+
+* **Reverse Proxy**
+
+ To allow the client application to automatically fail over in the event of a master server becoming unavailable, or the promotion of a new master,
+ a reverse proxy can be configured to route traffic depending on the response returned by `http://<ip-address>:8081/v1/server/ha/state` (see above).
+ If a server returns `"MASTER"`, then traffic should be routed to that server, otherwise it should not be. The client software should be configured
--- End diff --
Add that it could take 30 seconds for the standby to be promoted, so the reverse proxy should retry for at least this period. Or that the failover time should be reconfigured to be shorter.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---