You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@aurora.apache.org by wf...@apache.org on 2015/12/21 19:56:16 UTC
aurora git commit: Add a guide for installing Aurora.
Repository: aurora
Updated Branches:
refs/heads/master d3dae19d5 -> cf225b793
Add a guide for installing Aurora.
Bugs closed: AURORA-1483
Reviewed at https://reviews.apache.org/r/41564/
Project: http://git-wip-us.apache.org/repos/asf/aurora/repo
Commit: http://git-wip-us.apache.org/repos/asf/aurora/commit/cf225b79
Tree: http://git-wip-us.apache.org/repos/asf/aurora/tree/cf225b79
Diff: http://git-wip-us.apache.org/repos/asf/aurora/diff/cf225b79
Branch: refs/heads/master
Commit: cf225b793d5fc414e9a06f97e57ffc5301df7e58
Parents: d3dae19
Author: Bill Farner <wf...@apache.org>
Authored: Mon Dec 21 10:56:05 2015 -0800
Committer: Bill Farner <wf...@apache.org>
Committed: Mon Dec 21 10:56:05 2015 -0800
----------------------------------------------------------------------
README.md | 2 +-
docs/installing.md | 284 ++++++++++++++++++++++++++++++++++++++++++++++++
2 files changed, 285 insertions(+), 1 deletion(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/aurora/blob/cf225b79/README.md
----------------------------------------------------------------------
diff --git a/README.md b/README.md
index e88e1be..143ee06 100644
--- a/README.md
+++ b/README.md
@@ -72,7 +72,7 @@ Are you using Aurora too? Let us know, or submit a patch to join the list!
## Getting Started
* [Try the tutorial](docs/tutorial.md)
* [Running a Local Cluster with Vagrant](docs/vagrant.md)
-* [Deploying Aurora](docs/deploying-aurora-scheduler.md)
+* [Installing Aurora](docs/installing.md)
* [Developing Aurora](docs/developing-aurora-scheduler.md)
## Getting Help
http://git-wip-us.apache.org/repos/asf/aurora/blob/cf225b79/docs/installing.md
----------------------------------------------------------------------
diff --git a/docs/installing.md b/docs/installing.md
new file mode 100644
index 0000000..8d73725
--- /dev/null
+++ b/docs/installing.md
@@ -0,0 +1,284 @@
+# Installing Aurora
+
+- [Components](#components)
+ - [Machine profiles](#machine-profiles)
+ - [Coordinator](#coordinator)
+ - [Worker](#worker)
+ - [Client](#client)
+ - [Getting Aurora](#getting-aurora)
+ - [Building your own binary packages](#building-your-own-binary-packages)
+ - [RPMs](#rpms)
+ - [Installing the scheduler](#installing-the-scheduler)
+ - [Ubuntu 14.04](#ubuntu-1404)
+ - [CentOS 7](#centos-7)
+ - [Finalizing](#finalizing)
+ - [Configuration](#configuration)
+ - [Installing worker components](#installing-worker-components)
+ - [Ubuntu 14.04](#ubuntu-1404-1)
+ - [CentOS 7](#centos-7-1)
+ - [Configuration](#configuration-1)
+ - [Installing the client](#installing-the-client)
+ - [Ubuntu 14.04](#ubuntu-1404-2)
+ - [CentOS 7](#centos-7-2)
+ - [Configuration](#configuration-2)
+ - [See also](#see-also)
+ - [Installing Mesos](#installing-mesos)
+ - [Mesos on Ubuntu 14.04](#mesos-on-ubuntu-1404)
+ - [Mesos on CentOS 7](#mesos-on-centos-7)
+
+<!-- TODO(wfarner): Remove notes about "older RPM versions" once >=0.10.0 RPMs are released. -->
+
+## Components
+Before installing Aurora, it's important to have an understanding of the components that make up
+a functioning Aurora cluster.
+
+* **Aurora scheduler**
+
+ The scheduler will be your primary interface to the work you run in your cluster. You will
+ instruct it to run jobs, and it will manage them in Mesos for you. You will also frequently use
+ the scheduler's web interface as a heads-up display for what's running in your cluster.
+
+* **Aurora client**
+
+ The client (`aurora` command) is a command line tool that exposes primitives that you can use to
+ interact with the scheduler.
+
+ Aurora also provides an admin client (`aurora_admin` command) that contains commands built for
+ cluster administrators. You can use this tool to do things like manage user quotas and manage
+ graceful maintenance on machines in cluster.
+
+* **Aurora executor**
+
+ The executor (a.k.a. Thermos executor) is responsible for carrying out the workloads described in
+ the Aurora DSL (`.aurora` files). The executor is what actually executes user processes. It will
+ also perform health checking of tasks and register tasks in ZooKeeper for the purposes of dynamic
+ service discovery. You can find lots more detail on the executor and Thermos in the
+ [user guide](user-guide.md).
+
+* **Aurora observer**
+
+ The observer provides browser-based access to the status of individual tasks executing on worker
+ machines. It gives insight into the processes executing, and facilitates browsing of task sandbox
+ directories.
+
+* **ZooKeeper**
+
+ [ZooKeeper](http://zookeeper.apache.org) is a distributed consensus system. In an Aurora cluster
+ it is used for reliable election of the leading Aurora scheduler and Mesos master.
+
+* **Mesos master**
+
+ The master is responsible for tracking worker machines and performing accounting of their
+ resources. The scheduler interfaces with the master to control the cluster.
+
+* **Mesos agent**
+
+ The agent receives work assigned by the scheduler and executes them. It interfaces with Linux
+ isolation systems like cgroups, namespaces and Docker to manage the resource consumption of tasks.
+ When a user task is launched, the agent will launch the executor (in the context of a Linux cgroup
+ or Docker container depending upon the environment), which will in turn fork user processes.
+
+## Machine profiles
+Given that many of these components communicate over the network, there are numerous ways you could
+assemble them to create an Aurora cluster. The simplest way is to think in terms of three machine
+profiles:
+
+### Coordinator
+**Components**: ZooKeeper, Aurora scheduler, Mesos master
+
+A small number of machines (typically 3 or 5) responsible for cluster orchestration. In most cases
+it is fine to co-locate these components in anything but very large clusters (> 1000 machines).
+Beyond that point, operators will likely want to manage these services on separate machines.
+
+In practice, 5 coordinators have been shown to reliably manage clusters with tens of thousands of
+machines.
+
+
+### Worker
+**Components**: Aurora executor, Aurora observer, Mesos agent
+
+The bulk of the cluster, where services will actually run.
+
+### Client
+**Components**: Aurora client, Aurora admin client
+
+Any machines that users submit jobs from.
+
+## Getting Aurora
+Source and binary distributions can be found on our
+[downloads](https://aurora.apache.org/downloads/) page. Installing from binary packages is
+recommended for most.
+
+### Building your own binary packages
+Our package build toolchain makes it easy to build your own packages if you would like. See the
+[instructions](https://github.com/apache/aurora-packaging) to learn how.
+
+### RPMs
+We currently have work in progress to provide official RPMs. As of this writing, the suggested way
+to get RPMs is to [build them](#building-your-own-binary-packages).
+
+We do have unofficial experimental RPMs available for testing purposes.
+
+**Use these RPMs at your own risk, they are not officially released under the ASF guidelines.**
+
+ echo '[apache-aurora-wfarner]
+ name=Apache Aurora distribution maintained by wfarner
+ baseurl=http://people.apache.org/~wfarner/aurora/distributions/0.9.0/rpm/centos-7/x86_64/
+ gpgcheck = 0' | sudo tee /etc/yum.repos.d/apache-aurora-wfarner.repo > /dev/null
+
+
+## Installing the scheduler
+### Ubuntu 14.04
+
+1. Install Mesos
+
+ Skip down to [install mesos](#mesos-on-ubuntu-1404), then run:
+
+ sudo start mesos-master
+
+2. Install ZooKeeper
+
+ sudo apt-get install -y zookeeperd
+
+3. Install the Aurora scheduler
+
+ wget -c https://apache.bintray.com/aurora/aurora-scheduler_0.10.0-1_amd64.deb
+ sudo dpkg -i aurora-scheduler_0.10.0-1_amd64.deb
+
+### CentOS 7
+
+1. Install Mesos
+
+ Skip down to [install mesos](#mesos-on-centos-7), then run:
+
+ sudo systemctl start mesos-master
+
+2. Install ZooKeeper
+
+ sudo rpm -Uvh https://archive.cloudera.com/cdh4/one-click-install/redhat/6/x86_64/cloudera-cdh-4-0.x86_64.rpm
+ sudo yum install -y java-1.8.0-openjdk-headless zookeeper-server
+
+ sudo service zookeeper-server init
+ sudo systemctl start zookeeper-server
+
+3. Install the Aurora scheduler
+ If you haven't already, read the section on [how to get Aurora RPMs](#rpms).
+
+ # Note: for older Aurora RPM versions, this may be called 'aurora'.
+ sudo yum install -y aurora-scheduler
+
+Note: if you are using the unreleased 0.9.0 RPM, you will need to edit `/etc/sysconfig/aurora`:
+Change
+`-mesos_master_address='zk://127.0.0.1:2181/mesos/master'`
+To
+`-mesos_master_address='zk://127.0.0.1:2181/mesos'`
+And
+`-native_log_file_path='/var/lib/aurora/db'`
+To
+`-native_log_file_path='/var/lib/aurora/scheduler/db'`
+
+### Finalizing
+By default, the scheduler will start in an uninitialized mode. This is because external
+coordination is necessary to be certain operator error does not result in a quorum of schedulers
+starting up and believing their databases are empty when in fact they should be re-joining a
+cluster.
+
+Because of this, a fresh install of the scheduler will need intervention to start up. First,
+stop the scheduler service.
+Ubuntu: `sudo stop aurora-scheduler`
+CentOS: `sudo systemctl stop aurora`
+
+Now initialize the database:
+
+ sudo -u aurora mkdir -p /var/lib/aurora/scheduler/db
+ sudo -u aurora mesos-log initialize --path=/var/lib/aurora/scheduler/db
+
+Now you can start the scheduler back up.
+Ubuntu: `sudo start aurora-scheduler`
+CentOS: `sudo systemctl start aurora`
+
+### Configuration
+For more detail on this topic, see the dedicated page on
+[deploying the scheduler](deploying-aurora-scheduler.md)
+
+
+## Installing worker components
+### Ubuntu 14.04
+
+1. Install Mesos
+
+ Skip down to [install mesos](#mesos-on-ubuntu-1404), then run:
+
+ sudo start mesos-slave
+
+2. Install Aurora executor and observer
+
+ wget -c https://apache.bintray.com/aurora/aurora-executor_0.10.0-1_amd64.deb
+ sudo dpkg -i aurora-executor_0.10.0-1_amd64.deb
+
+### CentOS 7
+
+1. Install Mesos
+
+ Skip down to [install mesos](#mesos-on-centos-7), then run:
+
+ sudo systemctl start mesos-slave
+
+2. Install Aurora executor and observer
+ If you haven't already, read the section on [how to get Aurora RPMs](#rpms).
+
+ # Note: for older Aurora RPM versions, this may be called 'aurora-thermos'.
+ sudo yum install -y aurora-executor
+
+### Configuration
+The executor and observer typically do not require much configuration. Command line arguments can
+be passed to the executor using a command line argument on the scheduler.
+
+## Installing the client
+### Ubuntu 14.04
+
+ sudo apt-get install -y python2.7 wget
+
+ wget https://apache.bintray.com/aurora/aurora-tools_0.10.0-1_amd64.deb
+ sudo dpkg -i aurora-tools_0.10.0-1_amd64.deb
+
+### CentOS 7
+If you haven't already, read the section on [how to get Aurora RPMs](#rpms).
+
+ # Note: for older Aurora RPM versions, this may be called 'aurora-client'.
+ sudo yum install -y aurora-tools
+
+### Configuration
+Client configuration lives in a json file that describes the clusters available and how to reach
+them. By default this file is at `/etc/aurora/clusters.json`.
+
+Jobs may be submitted to the scheduler using the client, and are described with
+[job configurations](configuration-reference.md) expressed in `.aurora` files. Typically you will
+maintain a single job configuration file to describe one or more deployment environments (e.g.
+dev, test, prod) for a production job.
+
+## See also
+We have other docs that you will find useful once you have your cluster up and running:
+* [Monitor](monitoring.md) your cluster
+* Enable scheduler [security](security.md)
+* View job SLA [statistics](sla.md)
+* Understand the internals of the scheduler's [storage](storage.md)
+
+## Installing Mesos
+Mesos uses a single package for the Mesos master and slave. As a result, the package dependencies
+are identical for both.
+
+### Mesos on Ubuntu 14.04
+
+ sudo add-apt-repository ppa:openjdk-r/ppa -y
+ sudo apt-get update
+
+ sudo apt-get install -y software-properties-common wget libsvn1 libcurl3 openjdk-8-jre-headless
+
+ wget -c http://downloads.mesosphere.io/master/ubuntu/14.04/mesos_0.23.0-1.0.ubuntu1404_amd64.deb
+ sudo dpkg -i mesos_0.23.0-1.0.ubuntu1404_amd64.deb
+
+### Mesos on CentOS 7
+
+ sudo rpm -Uvh http://repos.mesosphere.io/el/7/noarch/RPMS/mesosphere-el-repo-7-1.noarch.rpm
+ sudo yum install -y mesos-0.22.0