You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@whirr.apache.org by to...@apache.org on 2011/05/24 06:58:38 UTC
svn commit: r1126866 - in /incubator/whirr/trunk: ./ src/site/confluence/
src/site/resources/css/
Author: tomwhite
Date: Tue May 24 04:58:37 2011
New Revision: 1126866
URL: http://svn.apache.org/viewvc?rev=1126866&view=rev
Log:
WHIRR-310. Improve Configuration Guide. Contributed by Andrei Savu.
Modified:
incubator/whirr/trunk/CHANGES.txt
incubator/whirr/trunk/src/site/confluence/configuration-guide.confluence
incubator/whirr/trunk/src/site/confluence/faq.confluence
incubator/whirr/trunk/src/site/confluence/index.confluence
incubator/whirr/trunk/src/site/confluence/quick-start-guide.confluence
incubator/whirr/trunk/src/site/confluence/whirr-in-5-minutes.confluence
incubator/whirr/trunk/src/site/resources/css/site.css
Modified: incubator/whirr/trunk/CHANGES.txt
URL: http://svn.apache.org/viewvc/incubator/whirr/trunk/CHANGES.txt?rev=1126866&r1=1126865&r2=1126866&view=diff
==============================================================================
--- incubator/whirr/trunk/CHANGES.txt (original)
+++ incubator/whirr/trunk/CHANGES.txt Tue May 24 04:58:37 2011
@@ -90,6 +90,8 @@ Release 0.5.0 - 2011-05-16
WHIRR-141. Create a logo. (Alison Wong via tomwhite)
+ WHIRR-310. Improve Configuration Guide. (asavu via tomwhite)
+
BUG FIXES
WHIRR-253. ZooKeeper service should only authorize ingress to ZooKeeper
Modified: incubator/whirr/trunk/src/site/confluence/configuration-guide.confluence
URL: http://svn.apache.org/viewvc/incubator/whirr/trunk/src/site/confluence/configuration-guide.confluence?rev=1126866&r1=1126865&r2=1126866&view=diff
==============================================================================
--- incubator/whirr/trunk/src/site/confluence/configuration-guide.confluence (original)
+++ incubator/whirr/trunk/src/site/confluence/configuration-guide.confluence Tue May 24 04:58:37 2011
@@ -4,25 +4,63 @@ Whirr is configured using a properties f
For example working configurations, please see the recipes in the _recipes_ directory of the distribution.
+h2. General Options
+
|| Name || Command line option || Default || Description ||
| {{whirr.config}} | {{\--config}} | none | A filename of a properties file containing properties in this table. Note that Whirr properties specified in this file all have a {{whirr.}} prefix. |
| {{whirr.service-name}} | {{\--service-name}} | The default service for launching clusters | The name of the service to use. You only need to set this if you want to use a non-standard service launcher. |
| {{whirr.cluster-name}} | {{\--cluster-name}} | none | The name of the cluster to operate on. E.g. {{hadoopcluster}}. The cluster name is used to tag the instances in some cloud-specific way. For example, in Amazon it is used to form the security group name. |
+
+h2. Instance Templates Options
+
+|| Name || Command line option || Default || Description ||
| {{whirr.instance-templates}} | {{\--instance-templates}} | none | The number of instances to launch for each set of roles in a service. E.g. {{1 nn+jt,10 dn+tt}} means one instance with the roles {{nn}} (namenode) and {{jt}} (jobtracker), and ten instances each with the roles {{dn}} (datanode) and {{tt}} (tasktracker). Note that currently a role may only be specified in a single group.|
| {{whirr.instance-templates-max-percent-failures}} | {{\--instance-templates-max-percent-failures}} | none | The percentage of successfully started instances for each set of roles. E.g. {{100 nn+jt,60 dn+tt}} means all instances with the roles {{nn}} (namenode) and {{jt}} (jobtracker) has to be successfully started, and 60% of instances has to be successfully started each with the roles {{dn}} (datanode) and {{tt}} (tasktracker), otherwise a retry step is initiated with the number of nodes equal with the missing nodes per role compared to {{instance-templates}} value. If after the retry the percentage of successfully started instances is still behind the limit, then the cluster startup is considered invalid. In a valid cluster startup, with or without retry mechanism, all the failed nodes will be cleaned up immediately. Only the completely failed cluster may leave unterminated failed nodes. Default value is 100 for each roles, in that case we don't need to use this paramete
r at all. In case we would like to lower the limit from 100% to 60% for only the {{dd}} (datanode) and {{tt}} (tasktracker), then we can specify {{60 dn+tt}} for the parameter and we may left the {{100 nn+jt,}} from the beginning of the value. |
| {{whirr.instance-templates-minimum-number-of-instances}} | {{\--instance-templates-minimum-number-of-instances}} | none | The minimum number of successfully started instances for each set of roles. E.g. {{1 nn+jt,6 dn+tt}} means 1 instance with the roles {{nn}} (namenode) and {{jt}} (jobtracker) has to be successfully started, and 6 instances has to be successfully started each with the roles {{dn}} (datanode) and {{tt}} (tasktracker), otherwise a retry step is initiated with the number of nodes equal with the missing nodes per role compared to {{instance-templates}} value. If after the retry the number of successfully started instances i still behind the limit, then the cluster startup is considered invalid. In a valid cluster startup, with or without retry mechanism, all the failed nodes will be cleaned up immediately. Only the completely failed cluster may leave unterminated failed nodes. Note that we may specify only {{6 dd+tt}}, in that case the limit will be applied
only to the specified role. Default value is 100 for each roles, in that case we don't need to use this parameter at all. In case we would like to lower the limit for only the {{dd}} (datanode) and {{tt}} (tasktracker), then we can specify {{60 dn+tt}} for the parameter, skipping the {{100 nn+jt}}. |
| {{whirr.max-startup-retries}} | {{\--max-startup-retries}} | {{1}} | The number of retries in case of insufficient successfully started instances.|
+
+h2. Cloud Provider Options
+
+|| Name || Command line option || Default || Description ||
| {{whirr.provider}} | {{\--provider}} | {{aws-ec2}} | The name of the cloud provider. See the [table below|#cloud-provider-config] for possible provider names.|
-| {{whirr.login-user}} | {{\--login-user}} | none | Override the default login user used to bootstrap whirr. E.g. ubuntu or myuser:mypass |
+| {{whirr.location-id}} | {{\--location-id}} | none | The location to launch instances in. If not specified then an arbitrary location will be chosen. |
| {{whirr.identity}} | {{\--identity}} | none | The cloud identity. See the [table below|#cloud-provider-config] for how this maps to the credentials for your provider. |
| {{whirr.credential}} | {{\--credential}} | none | The cloud credential. See the [table below|#cloud-provider-config] for how this maps to the credentials for your provider. |
+| {{whirr.login-user}} | {{\--login-user}} | none | Override the default login user used to bootstrap whirr. E.g. ubuntu or myuser:mypass |
+
+h2. BlobStore Provider Options
+
+|| Name || Command line option || Default || Description ||
+| {{whirr.blobstore-provider}} | {{\--blobstore-provider}} | Computed from {{whirr.provider}} | The name of the blobstore provider. All jclouds blobstore providers are supported |
+| {{whirr.blobstore-identity}} | {{\--blobstore-identity}} | {{whirr.identity}} | The blobstore identity. See the [table below|#cloud-provider-config] for how this maps to the credentials for your provider. |
+| {{whirr.blobstore-credential}} | {{\--blobstore-credential}} | {{whirr.credential}} | The blobstore credential. See the [table below|#cloud-provider-config] for how this maps to the credentials for your provider. |
+| {{whirr.blobstore-location-id}} | {{\--blobstore-location-id}} | As close as possible to the compute nodes | The blobstore location ID |
+
+h2. Cluster State Store Options
+
+|| Name || Command line option || Default || Description ||
+| {{whirr.state-store}} | {{\--state-store}} | local | What kind of store to use for cluster state (local, blob or none). |
+| {{whirr.state-store-container}} | {{\--state-store-container}} | none | Container where to store state. Valid only for the blob state store. |
+| {{whirr.state-store-blob}} | {{\--state-store-blob}} | whirr-<{{whirr.cluster-name}}> | Blob name for state storage. Valid only for the blob state store. |
+
+h2. Instance Login Options
+
+|| Name || Command line option || Default || Description ||
+| {{whirr.cluster-user}} | {{\--cluster-user}} | Current local user | The name of the user that Whirr will create on all instances. This is the user you should use to access the cluster. |
| {{whirr.private-key-file}} | {{\--private-key-file}} | _\~/.ssh/id\_rsa_ | The filename of the private RSA SSH key used to connect to instances. Note: the public/private key must be set together, and must be passwordless. |
| {{whirr.public-key-file}} | {{\--public-key-file}} | _\~/.ssh/id\_rsa_.pub | The filename of the public RSA SSH key used to connect to instances. Note: the public/private key must be set together, and must be passwordless.|
+
+h2. Image and Hardware Selection Options
+
+|| Name || Command line option || Default || Description ||
| {{whirr.image-id}} | {{\--image-id}} | none | The ID of the image to use for instances. If not specified then a vanilla Linux image is chosen. |
| {{whirr.hardware-id}} | {{\--hardware-id}} | none | The type of hardware to use for the instance. This must be compatible with the image ID. |
-| {{whirr.location-id}} | {{\--location-id}} | none | The location to launch instances in. If not specified then an arbitrary location will be chosen. |
+| {{whirr.hardware-min-ram}} | {{\--hardware-min-ram}} | 1024 | The minimum amount of RAM each instance should have |
+
+h2. Firewall Options
+
+|| Name || Command line option || Default || Description ||
| {{whirr.client-cidrs}} | {{\--client-cidrs}} | none | A comma-separated list of [CIDR |http://en.wikipedia.org/wiki/Classless\_Inter-Domain\_Routing] blocks. E.g. {{208.128.0.0/11,108.128.0.0/11}} |
-| {{whirr.run-url-base}} | {{\--run-url-base}} | {{http://whirr.s3.amazonaws.com/VERSION/}} | Deprecated. The base URL for forming run urls from. Change this to host your own set of launch scripts, as explained in the [FAQ|faq#how-can-i-modify-the-instance-installation-and-configuration-scripts]. |
{anchor:cloud-provider-config}
h2. Cloud provider specific configuration
Modified: incubator/whirr/trunk/src/site/confluence/faq.confluence
URL: http://svn.apache.org/viewvc/incubator/whirr/trunk/src/site/confluence/faq.confluence?rev=1126866&r1=1126865&r2=1126866&view=diff
==============================================================================
--- incubator/whirr/trunk/src/site/confluence/faq.confluence (original)
+++ incubator/whirr/trunk/src/site/confluence/faq.confluence Tue May 24 04:58:37 2011
@@ -60,30 +60,29 @@ final {{a}}) would start the cluster in
h2. How can I use a custom image? How can I control the cloud hardware used?
The default image used is dependent on the Cloud provider, the hardware, and the
-service.
+service. Whirr tries to find an image with Ubuntu Server and at least 1024 MB of RAM.
Use {{image-id}} to specify the image used, and {{hardware-id}} to specify the
hardware. Both are cloud-specific.
+You can specify the amount of RAM in a cloud agnostic way by setting a value for
+{{hardware-min-ram}}.
+
In addition, on EC2 you need to set jclouds.ec2.ami-owners to include the AMI
owner if it is not Amazon, Alestic, Canonical, or RightScale.
h2. How do I log in to a node in the cluster?
-On EC2, if you know the node's address you can do
+On EC2, if you know the node's address you can do:
{code}
-ssh -i ~/.ssh/id_rsa ec2-user@host
+ssh -i ~/.ssh/id_rsa <whirr.cluster-user>@host
{code}
This assumes that you use the default private key; if this is not the case then
specify the one you used at cluster launch.
-The Amazon Linux AMI requires that you login as {{ec2-user}}. If needed, you can
-become root by doing {{sudo su -}} after logging in.
-
-Different AMIs may have different login users. Check with the documentation for
-the AMI.
+{{whirr.cluster-user}} defaults to the name of the local user running Whirr.
{anchor:how-can-i-modify-the-instance-installation-and-configuration-scripts}
h2. How can I modify the instance installation and configuration scripts?
Modified: incubator/whirr/trunk/src/site/confluence/index.confluence
URL: http://svn.apache.org/viewvc/incubator/whirr/trunk/src/site/confluence/index.confluence?rev=1126866&r1=1126865&r2=1126866&view=diff
==============================================================================
--- incubator/whirr/trunk/src/site/confluence/index.confluence (original)
+++ incubator/whirr/trunk/src/site/confluence/index.confluence Tue May 24 04:58:37 2011
@@ -2,13 +2,26 @@ h1. Apache Whirr
Apache Whirr is a set of libraries for running cloud services. Whirr is currently in the [Apache Incubator|http://incubator.apache.org/].
-Whirr provides
+Whirr provides:
* A cloud-neutral way to run services. You don't have to worry about the idiosyncrasies of each provider.
* A common service API. The details of provisioning are particular to the service.
* Smart defaults for services. You can get a properly configured system running quickly, while still being able to override settings as needed.
+You can also use Whirr as a command line tool for deploying clusters.
+
Find out more about Whirr at our [wiki|https://cwiki.apache.org/confluence/display/WHIRR].
+h2. Getting Started
+
+You can use Whirr's CLI or APIs to [get started with Whirr|quick-start-guide].
+
+There is also an [FAQ|faq] which covers how to achieve common
+tasks with Whirr, and a [configuration guide|configuration-guide] for reference.
+
+h2. Getting Involved
+
+Have you got a suggestion for improving Whirr? It's easy to [get involved|https://cwiki.apache.org/confluence/display/WHIRR/How+To+Contribute].
+
h2. Which services and cloud providers are supported?
Whirr uses [jclouds|http://code.google.com/p/jclouds/] for provisioning, so in principle it should support all the cloud providers that jclouds supports. The following table shows
@@ -18,20 +31,15 @@ the cloud provider and service combinati
|Amazon EC2|Yes|Yes|Yes|Yes|Yes|Yes|
|Rackspace Cloud Servers|Yes|Yes|Yes|Yes|Yes|Yes|
-h2. Download
-
-Download a release of Whirr from a [nearby mirror|http://www.apache.org/dyn/closer.cgi/incubator/whirr/].
+For development and local testing we are also supporting the BYON (bring your own nodes) jclouds provider. Check the _recipes_ folder for a configuration sample.
-h2. Getting Started
-
-You can use Whirr's CLI or APIs to [get started with Whirr|quick-start-guide].
+h2. What server operating systems are supported?
-There is also an [FAQ|faq] which covers how to achieve common
-tasks with Whirr, and a [configuration guide|configuration-guide] for reference.
+Each release is tested by running the integration tests on Ubuntu Server 10.04. All setup scripts should also work on Centos 5.x but we don't have a formal testing procedure in place right now.
-h2. Getting Involved
+h2. Download
-Have you got a suggestion for improving Whirr? It's easy to [get involved|https://cwiki.apache.org/confluence/display/WHIRR/How+To+Contribute].
+Download a release of Whirr from a [nearby mirror|http://www.apache.org/dyn/closer.cgi/incubator/whirr/].
h2. History
Modified: incubator/whirr/trunk/src/site/confluence/quick-start-guide.confluence
URL: http://svn.apache.org/viewvc/incubator/whirr/trunk/src/site/confluence/quick-start-guide.confluence?rev=1126866&r1=1126865&r2=1126866&view=diff
==============================================================================
--- incubator/whirr/trunk/src/site/confluence/quick-start-guide.confluence (original)
+++ incubator/whirr/trunk/src/site/confluence/quick-start-guide.confluence Tue May 24 04:58:37 2011
@@ -43,9 +43,11 @@ directory of the distribution.
{code}
whirr.cluster-name=myhadoopcluster
whirr.instance-templates=1 hadoop-jobtracker+hadoop-namenode,1 hadoop-datanode+hadoop-tasktracker
-whirr.provider=ec2
+
+whirr.provider=aws-ec2
whirr.identity=${env:AWS_ACCESS_KEY_ID}
whirr.credential=${env:AWS_SECRET_ACCESS_KEY}
+
whirr.private-key-file=${sys:user.home}/.ssh/id_rsa
whirr.public-key-file=${sys:user.home}/.ssh/id_rsa.pub
{code}
@@ -158,7 +160,7 @@ for properties is not reflected in the c
% bin/whirr launch-cluster \
--cluster-name=myhadoopcluster \
--instance-templates='1 hadoop-jobtracker+hadoop-namenode,1 hadoop-datanode+hadoop-tasktracker' \
- --provider=ec2 \
+ --provider=aws-ec2 \
--identity=$AWS_ACCESS_KEY_ID \
--credential=$AWS_SECRET_ACCESS_KEY \
--private-key-file=~/.ssh/id_rsa \
Modified: incubator/whirr/trunk/src/site/confluence/whirr-in-5-minutes.confluence
URL: http://svn.apache.org/viewvc/incubator/whirr/trunk/src/site/confluence/whirr-in-5-minutes.confluence?rev=1126866&r1=1126865&r2=1126866&view=diff
==============================================================================
--- incubator/whirr/trunk/src/site/confluence/whirr-in-5-minutes.confluence (original)
+++ incubator/whirr/trunk/src/site/confluence/whirr-in-5-minutes.confluence Tue May 24 04:58:37 2011
@@ -1,3 +1,4 @@
+
h1. Whirr in 5 minutes
The following commands install Whirr and start a 3 node ZooKeeper cluster on
@@ -8,10 +9,13 @@ already installed. Help on finding your
{code}
export AWS_ACCESS_KEY_ID=...
export AWS_SECRET_ACCESS_KEY=...
+
curl -O http://www.apache.org/dist/incubator/whirr/whirr-0.6.0-incubating/whirr-0.6.0-incubating.tar.gz
tar zxf whirr-0.6.0-incubating.tar.gz; cd whirr-0.6.0-incubating
+
ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa_whirr
bin/whirr launch-cluster --config recipes/zookeeper-ec2.properties --private-key-file ~/.ssh/id_rsa_whirr
+
echo "ruok" | nc $(awk '{print $3}' ~/.whirr/zookeeper/instances | head -1) 2181; echo
{code}
@@ -25,3 +29,4 @@ bin/whirr destroy-cluster --config recip
{code}
The various options are explained in more detail in the [Quick Start Guide|quick-start-guide].
+
Modified: incubator/whirr/trunk/src/site/resources/css/site.css
URL: http://svn.apache.org/viewvc/incubator/whirr/trunk/src/site/resources/css/site.css?rev=1126866&r1=1126865&r2=1126866&view=diff
==============================================================================
--- incubator/whirr/trunk/src/site/resources/css/site.css (original)
+++ incubator/whirr/trunk/src/site/resources/css/site.css Tue May 24 04:58:37 2011
@@ -19,7 +19,13 @@
background: none;
}
-#banner img {
- height: 123px;
- width 130px;
+#bannerLeft img {
+ height: 90px;
+ margin-left: 30px;
+ margin-top: 4px;
}
+
+#bannerRight img {
+ margin: 17px;
+}
+